Few-shot examples or a JSON schema — which actually holds up better?

Question

For structured extraction I can either paste three example outputs or describe a JSON schema. Examples feel more reliable in practice but cost a lot of tokens on every call. Has anyone measured this properly on a decent-sized set? Does the answer change with structured-output modes now available?

Elif Yilmaz · Answer

Prompts in the repo, as files, versioned with the code. Hash the file contents and store the hash with every logged output. Then you can always tie an output back to the exact text that produced it, including for outputs generated six months ago by a prompt that no longer exists. We tried a database table so non-engineers could edit. It was a mistake — prompt changes are behaviour changes and they need review, staging and rollback, which is what the repo already gives you.

Aditya Joshi · Answer

At 200 pages, put the lot in context. Retrieval adds a chunking strategy, an index to keep in sync, an embedding model to maintain and a recall problem, and at that size it buys you nothing but a token bill. The crossover in our case was somewhere around 5MB, and it moved as caching got cheaper. Do the simple thing until the token cost or the latency actually hurts.

Riya Bansal · Answer

We measured this on 400 extractions. Schema-only held format 91% of the time, three examples held 97%, schema plus one example held 98%. The examples cost about 600 tokens per call. Worth noting the split in the failures: schema-only failures were mostly structural (missing a nested object). Example-based failures were mostly semantic — right shape, wrong field populated. Different problems, so which one you can tolerate should drive the choice.

Karthik Menon · Answer

Benchmarked on 150 tasks: temperature 0 and 0.3 were within noise on correctness. What did differ was diversity across runs — at 0 you get the same wrong answer five times, at 0.3 one of five attempts is often right. So it depends on your loop. Single-shot into production: 0. Generating candidates you filter with tests: higher, and take the one that passes.

Mei Lin · Answer

Ask for the edge cases first, in a separate call, and *then* give that list back as a constraint on the implementation. Asking for both in one response gets you a list that the code below it does not honour, which is worse than not asking because it looks handled. The reliable version is still tests first. Have it generate the test cases, read them, add the two it missed, then ask for an implementation that passes them.

Few-shot examples or a JSON schema — which actually holds up better?

0 Answers

No answers yet

Add an Answer