Skip to content
Optimal Lab

AI Engineering · interactive

How text becomes tokens

A model never reads letters or words. It reads tokens: chunks a tokenizer learned by merging the most common pairs of characters, over and over. Train one, then type your own text.

Characters only

vocabulary 7

    1. h
    2. u
    3. g
    ×10
    1. p
    2. u
    3. g
    ×5
    1. p
    2. u
    3. n
    ×12
    1. b
    2. u
    3. n
    ×4
    1. h
    2. u
    3. g
    4. s
    ×5

Next merge: u + g (20 times)

Start: every word is split into single characters. 7 tokens in the vocabulary.

Merge 0 of 6
  1. 1

    Training starts from the smallest pieces. Each distinct character is one token in the vocabulary.

    b g h n p s u

  2. 2

  3. 3

  4. 4

  5. 5

In production

Tools and technologies

Every model ships with its own tokenizer; you count tokens to budget prompts and estimate cost, whatever your stack.

Not tied to one language — use these names when you search docs or design a system.