What is a sane way to version prompts alongside application code?

Question

Our prompts live as template literals scattered across the codebase. Changing one means a deploy, and we cannot tell which version produced a given logged output. Files in the repo, rows in a table, or a dedicated tool? Mainly interested in how you tie a stored output back to the exact prompt that made it.

Karthik Menon · Accepted Answer

Prompts in the repo, as files, versioned with the code. Hash the file contents and store the hash with every logged output. Then you can always tie an output back to the exact text that produced it, including for outputs generated six months ago by a prompt that no longer exists. We tried a database table so non-engineers could edit. It was a mistake — prompt changes are behaviour changes and they need review, staging and rollback, which is what the repo already gives you.

Aroha Ngata · Answer

At 200 pages, put the lot in context. Retrieval adds a chunking strategy, an index to keep in sync, an embedding model to maintain and a recall problem, and at that size it buys you nothing but a token bill. The crossover in our case was somewhere around 5MB, and it moved as caching got cheaper. Do the simple thing until the token cost or the latency actually hurts.

Olivia Tremblay · Answer

Benchmarked on 150 tasks: temperature 0 and 0.3 were within noise on correctness. What did differ was diversity across runs — at 0 you get the same wrong answer five times, at 0.3 one of five attempts is often right. So it depends on your loop. Single-shot into production: 0. Generating candidates you filter with tests: higher, and take the one that passes.

Budi Santoso · Answer

Ask for the edge cases first, in a separate call, and *then* give that list back as a constraint on the implementation. Asking for both in one response gets you a list that the code below it does not honour, which is worse than not asking because it looks handled. The reliable version is still tests first. Have it generate the test cases, read them, add the two it missed, then ask for an implementation that passes them.

What is a sane way to version prompts alongside application code?

0 Answers

No answers yet

Add an Answer