Skip to content
Optimal Lab

AI Engineering · interactive

Distillation

A big model is slow and expensive to run. Distillation uses it as a teacher: it labels a pile of inputs, and a small student trains on its answers. Keep the teacher's whole probability distribution rather than its top pick and the student learns far more from each example.

Unlabelled inputs

8,000

Teacher, 70B

94% · $15/M

Soft labels

T = 4

Student, 3B

80% · $0.3/M

Your app

180 tok/s

one photo of a cat · what the student is told · T = 4

  • cat46.2%
  • fox25.4%
  • dog21.3%
  • car7.1%

1.76 bits of information per example · fox is far more likely than car: that is the knowledge

Student, 3B, 8,000 soft labels at T = 4: 80% (teacher 94%, ceiling for this size 86%).

Labels

Step 1 of 5

  1. 1

    Pick a student

    Small enough to be cheap and fast where it will run. Its size caps how good it can get, however well it is taught.

    ceiling 86% · $0.3/M tokens · 180 tok/s

  2. 2

    Let the teacher label the data

    The inputs need no human labels; the teacher's answers are the labels. For an LLM, that means the teacher writes the answers the student will imitate.

    teacher’s scores for the cat photo: cat 6.0 · fox 3.6 · dog 2.9 · car -1.5

  3. 3

    Keep the whole distribution

    Softmax turns the scores into probabilities. Dividing the scores by a temperature first spreads probability onto the similar classes, so the student also learns what a cat is like. Too hot and everything flattens.

    Temperature

    softmax(scores ÷ 4) → 46% / 25% / 21% / 7% · teaches 3.0× more per example than a hard label

  4. 4

    Train the student on them

    The student learns to match the teacher's distribution (or its top pick). Soft labels carry more information, so the student needs fewer examples to get close to its ceiling.

    Labelled examples

    top pickdistribution5005%13%2,00017%42%8,00051%80%32,00084%86%

    labelling 8,000 examples with the teacher: about $24

  5. 5

    Serve the student

    Nearly the teacher's quality at a fraction of the price and latency. Hosted LLMs are usually distilled from their outputs only (hard labels), because the logits aren't exposed; open-weight teachers allow the full distribution.

    accuracy$/M tokenstok/sTeacher, 70B94%$1525Student, 3B80%$0.3180

    The softmax and its temperature are real (Hinton et al. 2015). The accuracy curves are a teaching model.

In production

Tools and technologies

Distillation is how most small models are made, from logit matching to training on a teacher's outputs.

Not tied to one language — use these names when you search docs or design a system.