# Prompting An LLM To Grade Its Own Outputs (Carefully)

> LLM-as-judge works, but only with a rubric and a forced ordering.

- **Author:** [Rohan Deshpande (@rohan_deshpande)](https://www.promptabide.com/rohan_deshpande)
- **Published:** 2026-05-05
- **Updated:** 2026-08-22
- **Tags:** `ml`, `evaluation`, `llm`, `prompt-engineering`
- **Views:** 8467
- **Likes:** 82

## Prompt

```
Naive version: "Which of these two answers is better?" This is unreliable — position bias is real and large.

What actually works:

1. **Give a rubric with named dimensions.** Correctness, completeness, follows-format. Score each 1-5 separately.
2. **Ask for the reasoning before the score**, not after. Score-first anchors everything that follows.
3. **Run both orderings** and discard any pair where the verdict flips. That flip rate is your noise floor and you want to know it.
4. **Judge with a different model** than the one that produced the answer where you can.

My flip rate on a 200-pair set started at 18%. Adding the rubric and reasoning-first took it to 6%. That remaining 6% is genuinely ambiguous pairs and no amount of prompting removed it.

Do not report a judge score without also reporting the flip rate. A number without it means very little.
```

## Output

*Produced by Claude · claude-opus-4.*

```
Reasoning: A cites the source rows; B asserts the same total without support. Both follow the format. B is shorter but omits the currency.

Correctness: A 5, B 3
Completeness: A 5, B 3
Format:      A 5, B 5
Verdict: A
```

---

Canonical HTML: https://www.promptabide.com/bides/llm-as-judge-carefully-rohan-deshpande-3
Agent guide: https://www.promptabide.com/llms.txt · https://www.promptabide.com/agent-instructions.md
Sitemap: https://www.promptabide.com/sitemap.xml