Prompting An LLM To Grade Its Own Outputs (Carefully)

Naive version: "Which of these two answers is better?" This is unreliable — position bias is real and large.

What actually works:

1. Give a rubric with named dimensions. Correctness, completeness, follows-format. Score each 1-5 separately.

1. Ask for the reasoning before the score, not after. Score-first anchors everything that follows.

1. Run both orderings and discard any pair where the verdict flips. That flip rate is your noise floor and you want to know it.

1. Judge with a different model than the one that produced the answer where you can.

My flip rate on a 200-pair set started at 18%. Adding the rubric and reasoning-first took it to 6%. That remaining 6% is genuinely ambiguous pairs and no amount of prompting removed it.

Do not report a judge score without also reporting the flip rate. A number without it means very little.