How do you evaluate a prompt change without a labelled test set?

Question

Every guide starts with 'assemble a golden set of 200 examples'. We do not have one and building it is weeks of work. Is there a defensible middle ground for deciding whether a prompt edit made things better, or is it labels or nothing?

Dmytro Shevchenko · Accepted Answer

You do not need 200 labelled examples to start. Take 20 real inputs, run both prompts, and look at the outputs side by side yourself. Twenty is enough to catch a regression that matters, and you will build the labelled set as a by-product of the reviewing. Blind yourself to which output came from which prompt. Otherwise you will find the new one better, because you wrote it.

Riya Bansal · Answer

Format rules go in the system prompt, and it caches, which matters more as the prompt grows. The "closer to the end is followed better" effect is real but it is about recency across the whole assembled context, not about which field the text sits in. If you have a long user message, a one-line reminder at the end of it is cheap and closes most of the gap without duplicating 400 tokens.

How do you evaluate a prompt change without a labelled test set?

0 Answers

No answers yet

Add an Answer