# Prompting An LLM To Grade Its Own Outputs (Carefully)

> LLM-as-judge works, but only with a rubric and a forced ordering.

- **Author:** [Lena Müller (@lena_mueller)](https://www.promptabide.com/lena_mueller)
- **Published:** 2026-05-16
- **Updated:** 2026-08-22
- **Tags:** `ml`, `evaluation`, `llm`, `prompt-engineering`
- **Views:** 13797
- **Likes:** 420

## Prompt

```
Naive version: "Which of these two answers is better?" This is unreliable — position bias is real and large.

What actually works:

1. **Give a rubric with named dimensions.** Correctness, completeness, follows-format. Score each 1-5 separately.
2. **Ask for the reasoning before the score**, not after. Score-first anchors everything that follows.
3. **Run both orderings** and discard any pair where the verdict flips. That flip rate is your noise floor and you want to know it.
4. **Judge with a different model** than the one that produced the answer where you can.

My flip rate on a 200-pair set started at 18%. Adding the rubric and reasoning-first took it to 6%. That remaining 6% is genuinely ambiguous pairs and no amount of prompting removed it.

Do not report a judge score without also reporting the flip rate. A number without it means very little.
```

---

Canonical HTML: https://www.promptabide.com/bides/llm-as-judge-carefully-lena-mueller-1
Agent guide: https://www.promptabide.com/llms.txt · https://www.promptabide.com/agent-instructions.md
Sitemap: https://www.promptabide.com/sitemap.xml