What belongs in a prompt and what belongs in code?

Question

I keep seeing prompts asked to do arithmetic, sort lists, or enforce a character limit — things a line of code does exactly and a model does approximately. Is there a rule of thumb sharper than 'deterministic things go in code'? The boundary cases are where I keep getting it wrong.

Vihaan Reddy · Accepted Answer

The rule I use: if you would be angry to find it wrong in the logs, put it in code. Arithmetic, sorting, truncation, deduplication, date maths, anything with a hard limit — code. Judgement, summarisation, classification with fuzzy boundaries, anything where two reasonable people would disagree — prompt. The boundary cases are usually a task that is secretly two tasks, and splitting them puts each half on the right side of the line.

Olivia Tremblay · Answer

One thing not mentioned yet: log the full prompt and the full response, not a summary, and keep them long enough to debug with. Almost every "the model got worse" report I have chased turned out to be an input change nobody noticed — a template that started rendering an empty section, a retrieval step returning nothing. You cannot debug what you did not record, and sampled logs always miss the interesting case.

Arjun Nair · Answer

Adding to the above — measure the failure rate before you optimise anything. I spent two weeks on prompt tuning for a step that turned out to be failing 0.4% of the time, which was well inside what the downstream process already tolerated. Instrument first. The expensive problem is rarely the one that feels most annoying.

What belongs in a prompt and what belongs in code?

0 Answers

No answers yet

Add an Answer