AI Engineering · interactive
LLM-as-judge and its biases
Using a model to grade answers is fast and cheap, but judges have habits: they lean toward whichever answer they read first and toward longer answers. Watch one judge get eight pairs right for the wrong reasons, then fix it.
Question
Is the refund window 14 or 30 days?
Answer A · 31 wordsjudge ✓
Our refund policy is designed to be fair. In most cases you have 14 days to request a refund, after which requests are reviewed case by case by our support team.
Answer B · 4 wordspeople ✓
30 days from delivery.
| pair | people | judge | match |
|---|---|---|---|
| A | A | ✓ | |
| B | A | ✗ | |
| A | A | ✓ | |
| B | B | ✓ | |
| B | B | ✓ | |
| A | A | ✓ | |
| A | A | ✓ | |
| B | B | ✓ |
The judge picks A, but people picked B. The longer answer won on length, not content.
How the judge is asked
Show A first, ask which is better.
7 agree · 1 wrong · 0 can't tell
1 2 Real quality should decide it. Two habits push too: reading A first, and the longer answer looking more thorough.
Real quality (from people)2.00 toward B
Shown first0.80 toward A
Length1.62 toward A
A first: picks A
3 4
In production
Tools and technologies
Eval tools ship LLM judges; the bias fixes (swap order, rubrics, human spot checks) are up to you.
Not tied to one language — use these names when you search docs or design a system.