Aditya Joshi

Answered How do you evaluate a prompt change without a labelled test set?

You do not need 200 labelled examples to start. Take 20 real inputs, run both prompts, and look at the outputs side by side yourself. Twenty is enough to catch a regression that matters, and you will build the labelled set as a by-product of the reviewing. Blind yourself to which output came from which prompt. Otherwise you will find the new one better, because you wrote it.

Answered How do you keep generated code from silently ignoring an edge case?

Ask for the edge cases first, in a separate call, and *then* give that list back as a constraint on the implementation. Asking for both in one response gets you a list that the code below it does not honour, which is worse than not asking because it looks handled. The reliable version is still tests first. Have it generate the test cases, read them, add the two it missed, then ask for an implementation that passes them.

Answered How do you stop a long prompt from drifting halfway through the output?

Splitting the request is the fix that actually holds. One call per section, with the previous section's output passed in as context. It costs more calls but the format holds all the way through because no single generation is long enough to drift. Before you do that, try the cheap version: number the sections in the prompt and ask it to emit the section number as a literal header. Drift usually starts with an invented heading, and a required literal marker makes that fail loudly instead of quiet

Asked a question What belongs in a prompt and what belongs in code?

I keep seeing prompts asked to do arithmetic, sort lists, or enforce a character limit — things a line of code does exactly and a model does approximately. Is there a rule of thumb sharper than 'deterministic things go in code'? The boundary cases are where I keep getting it wrong.