How do you keep generated code from silently ignoring an edge case?

Question

The generated implementation is usually right for the main path and quietly wrong at the boundary — empty array, single element, duplicate key. I have started asking for the edge cases to be listed before the code, which helps. Is there something better than that, short of writing the tests myself first?

Priya Rajan · Accepted Answer

Ask for the edge cases first, in a separate call, and *then* give that list back as a constraint on the implementation. Asking for both in one response gets you a list that the code below it does not honour, which is worse than not asking because it looks handled. The reliable version is still tests first. Have it generate the test cases, read them, add the two it missed, then ask for an implementation that passes them.

Lucas Oliveira · Answer

At 200 pages, put the lot in context. Retrieval adds a chunking strategy, an index to keep in sync, an embedding model to maintain and a recall problem, and at that size it buys you nothing but a token bill. The crossover in our case was somewhere around 5MB, and it moved as caching got cheaper. Do the simple thing until the token cost or the latency actually hurts.

Camille Dubois · Answer

Benchmarked on 150 tasks: temperature 0 and 0.3 were within noise on correctness. What did differ was diversity across runs — at 0 you get the same wrong answer five times, at 0.3 one of five attempts is often right. So it depends on your loop. Single-shot into production: 0. Generating candidates you filter with tests: higher, and take the one that passes.

How do you keep generated code from silently ignoring an edge case?

0 Answers

No answers yet

Add an Answer