Skip to content
Optimal Lab

AI Engineering · interactive

Evals: is the new prompt actually better?

You changed the prompt and the answers read nicer. Are they better? An eval runs both versions over the same cases and grades them: code checks for hard rules, an LLM judge for meaning, and people to keep the judge honest.

Prompt v2 · 8 cases

8/8 pass · people 5/8

CaseCodeJudgePerson
✓✓✓
✗✓✗
✗✓✓
✗✓✗
✓✓✓
✗✓✗
✓✓✓
✓✓✓

I need my refund before Friday, can you make that happen?

Absolutely, I understand how important this is! I've flagged it and you should see it by tomorrow. Refunds normally take 5 to 10 business days.

Judge: pass, “Empathetic and mentions the policy.” · Person: fail

Your graders pick Prompt v2, but the people prefer Prompt v1. The eval is wrong, not the people.

Version

Graders that count

1

Write the test cases

Real questions from users, including the awkward ones. Each case says what a good answer must contain.

  • How long does a refund take?“5 to 10 business days”
  • I need my refund before Friday, can you make that happen?“5 to 10 business days”
  • Where do I see my refund status?“orders page”
  • Was I refunded for order 1842?“orders page”
  • Do you refund shipping?“shipping”
  • Can I get a refund after 60 days?“30 days”
  • My refund went to an old card.“bank”
  • Refund me now!!“orders page”
2

Run each version on every case

Same questions, same model, only the prompt changes. Save every output so you can read them.

  • Prompt v1 “Answer in two sentences. Quote the policy. Never promise a date.”
  • Prompt v2 “Be warm and thorough. Reassure the customer and explain everything.”
3

Run the code checks

Plain code, no model: does it state the fact, is it short enough, does it promise what policy forbids? Fast, cheap, and never moody.

  • Says the key fact7/8
  • Under 60 words7/8
  • No forbidden promise6/8

Limit 60 words. Forbidden: “tomorrow”, “today”, “guarantee”, “right away”, “24 hours”.

4

Ask an LLM judge

A second model grades meaning that code can't: is it correct, honest, on policy? It reads the answer and the policy and says pass or fail.

The judge passes 8/8 answers from Prompt v2.

Judges are known to favour long, warm, confident answers. Watch what that does to Prompt v2.

5

Check the judge against people

A person labels the same answers. Until the judge agrees with them, its score is an opinion, not a measurement.

Judge agrees with the person on 5/8 answers.

6

Compare the versions, then decide

Ship only if the new version wins on graders you have checked against people. Then add the cases you got wrong to the set, and run it on every change.

  • Prompt v1graders 6/8 · people 6/8
  • Prompt v2graders 8/8 · people 5/8

Your graders pick Prompt v2, but the people prefer Prompt v1. The eval is wrong, not the people.

In production

Tools and technologies

Eval tools all do the same three things: run a test set, grade it with code and models, and compare versions over time.

Not tied to one language — use these names when you search docs or design a system.