Skip to content
Optimal Lab

AI Engineering · interactive

Why models hallucinate

A model is always asked for the next token, and it always gives one. When a fact was rare in its training data, the probabilities are spread thin, but the answer still comes out fluent and sure. Try the fixes: give it a source, let it say “I'm not sure”, and change how answers are graded.

12 questions · memory only

abstain below 0%

right
7
made up
5
abstained
0

The famous facts are real. The rare ones and “Acme” are made up: rare facts are rare because almost nobody wrote them down.

7 right, 5 made up, 0 “not sure”. “A postdoc's birthday”: the fact appeared only once in training, so the model guessed “22 October 1989”, just as fluently. Click any card.

Retrieval (RAG)

1

A question comes in

The model doesn't look anything up. It only has patterns from training, and some facts showed up there far more often than others.

When was Dr. Lena Okafor, a postdoc with one paper, born?

this fact in the training data: 1 time

2

Spread probability over answers

A fact seen thousands of times gets nearly all the probability. One seen once or twice gets a small share, and the rest goes to answers that merely look right.

  • 14 March 199125%
  • 3 June 199038%
  • 22 October 198922%
  • 9 January 199215%

From memory alone: 25% on the right answer.

3

Decoding always picks something

Sampling turns probabilities into one answer, written in the same confident tone either way. Nothing in this step can say “I don't know” unless the model was trained to.

Sampled: 22 October 1989 (wrong, it had 22%)

How sampling picks: How an LLM picks the next token.

4

Give it a source to read

Retrieval puts the relevant document in the prompt, so the model copies instead of recalling. It only helps for facts that are actually in your documents.

3 of the 12 questions are about Acme, and only those are in the docs. Turn retrieval on and watch only those cards change. RAG, step by step

5

Let it say “I'm not sure”, and grade fairly

If the top answer is below a confidence threshold, abstain. Whether that pays off depends on the grading: if wrong answers cost nothing, guessing always scores at least as well.

Grading

Expected score, never abstain
7.63
Expected score, your threshold
7.63

With right-or-nothing grading, abstaining can only lose points, so a model tuned to top these benchmarks learns to guess. That's a big part of why hallucination persists. Expected score is the average over many runs; this run scored 7.

In production

Tools and technologies

No setting turns hallucination off; teams ground answers in sources, measure faithfulness, and reward “not sure”.

Not tied to one language — use these names when you search docs or design a system.