Skip to content
Optimal Lab

AI Engineering · interactive

Pretraining

Pretraining is one task repeated trillions of times: guess the next token, see how wrong the guess was, nudge the weights. Nobody labels anything; the text is its own answer key. Watch a tiny model learn which word follows which, one batch at a time.

Web text

360 lines

Clean + dedupe

240 kept

Tokenizer

30 tokens

Model

weights = 0

Loss

3.40

Held-out text

24 lines

random guessing 3.40step 0step 13
— held-out loss— batch loss
  • the →any 3%
  • sat →any 3%
  • jumped →any 3%
  • and →any 3%

Before training: every weight is 0, so all 30 words are equally likely next. Loss is ln(30) = 3.40. Press Play.

Step 0 of 13
  1. 1

    Crawl the web, then throw away spam, boilerplate and duplicates. Repeated text gets memorized instead of learned from, so deduplication matters.

    360 lines → 240 kept (90 duplicates, 30 junk)

    • the dog slept on the rug
    • the child ran past the mat in the sun
    • the cat slept on the wall and then slept
    • the dog slept on the fence in the sun
    • the dog looked at the wall and then slept
    • click here to subscribe !!!(boilerplate or spam)
    • the cat jumped over the wall

    24 clean lines are held out: the model never trains on them, so their loss shows what it really learned.

  2. 2

  3. 3

  4. 4

  5. 5

In production

Tools and technologies

Open pretraining runs share the same parts: a web crawl, a cleaning pipeline, a tokenizer and a distributed trainer.

Not tied to one language — use these names when you search docs or design a system.