Skip to content
Optimal Lab

AI Engineering · interactive

Benchmarks

A benchmark is a fixed set of questions with known answers, and a leaderboard is everyone's score on it. Five models on three benchmarks: a fresh one, an old one everyone has maxed out, and one that leaked into a model's training data. See when the ranking means something.

Old and saturated · 14,042 questions

1/10 pairs apart

  1. #1 D91.0%
  2. #2 B90.6%
  3. #3 E90.3%
  4. #4 C90.1%
  5. #5 A89.8%
85%100%

|—| 95% interval┊ best possible (93.5%: label errors)

Everyone is near 93.5%, the most the labels allow. Even with all 14,042 questions, 1 of 10 pairs can be told apart.

Benchmark

1

Fix a set of questions

Same questions, same answers, same grading for every model. That's what makes scores comparable, and also what lets them leak and go stale.

Released years ago, now in every training mix. Every model scores about 90%.

14,042 questions

2

Run every model and score it

Usually the share of questions answered correctly. Prompt format, few-shot examples and answer parsing all move the number, so compare only scores run the same way.

  • Model A: 89.8%
  • Model B: 90.6%
  • Model C: 90.1%
  • Model D: 91.0%
  • Model E: 90.3%
3

Put error bars on them

A score from n questions is an estimate. Its 95% interval is about ±1.96·√(p(1−p)/n). When two intervals overlap, the ranking between them is a coin toss.

Questions in the test

±0.5% · 1 of 10 pairs apart

4

Check for contamination

If test questions were on the web, they may be in the training data, and the model recalls answers instead of working them out. Rephrase the questions: a real skill survives, a memorized answer doesn't.

Pick “Leaked into training” to try it.

5

Read the ranking with care

A benchmark stops meaning much once every model is near its ceiling or has seen its questions. New, harder sets replace it, and the cycle starts again.

About 6.5% of MMLU's questions have wrong or unclear answers (Gema et al. 2024), so ~93.5% is the real top. Models within a point of each other are tied.

For your own product, a few hundred of your real tasks graded your way beats any public leaderboard.

In production

Tools and technologies

Public benchmarks run through a few shared harnesses; fresh, harder sets replace the saturated ones.

Not tied to one language — use these names when you search docs or design a system.