How much does temperature actually matter for code generation?

Question

Common advice is temperature 0 for code. In practice I get subjectively better structure around 0.3 and I cannot tell if that is real or if I am fooling myself. Has anyone run this against a benchmark rather than by feel?

Jonas Schmidt · Accepted Answer

Benchmarked on 150 tasks: temperature 0 and 0.3 were within noise on correctness. What did differ was diversity across runs — at 0 you get the same wrong answer five times, at 0.3 one of five attempts is often right. So it depends on your loop. Single-shot into production: 0. Generating candidates you filter with tests: higher, and take the one that passes.

Minh Nguyen · Answer

Assume injection succeeds and design so that it does not matter. Do not give the model a capability whose misuse you cannot tolerate — no unbounded tool calls, no raw SQL, no sending mail on its own authority. Everything else is defence in depth: mark untrusted content clearly, never let retrieved content look like instructions, validate outputs against a schema, and require a human confirm anything that writes. Delimiters alone stop the lazy attempt and nothing more.

How much does temperature actually matter for code generation?

0 Answers

No answers yet

Add an Answer