AI Engineering · interactive

Token embeddings and positions

A token id is just a row number. The model swaps it for a row of learned numbers, then adds a vector for where the token sits. Without that second part, “the dog bit the man” and “the man bit the dog” look the same to it. Turn positions off and see.

Input to layer 1

5 × 6

animalpersonactionpastsmall wordsize

0 · the0.01.00.01.01.01.0
1 · dog1.70.50.01.00.01.4
2 · bit1.1-0.41.01.80.01.0
3 · the0.1-1.00.11.01.01.0
4 · man-0.80.20.21.00.01.3

“the dog bit the man” and “the man bit the dog” now give different vectors, so the model can tell who bit whom.

“dog” at position 1 = its table row + the vector for position 1. Click any word to follow it.

Sentence

Positions

Split the text into token ids

The tokenizer turns text into ids. An id is only a row number: id 2 is not “bigger” or “closer” to id 3 in any useful way.

“the dog bit the man” → [0, 2, 6, 0, 4]

This toy vocabulary has 10 tokens. See How text becomes tokens for where ids come from.

Look up each id's row

The embedding table has one row per token in the vocabulary. Training nudged the rows so that words used the same way end up close together.

0the[ 0.00, 0.00, 0.00, 0.00, 1.00, 0.00]
1a[ 0.00, 0.00, 0.00, 0.00, 0.90, -0.20]
2dog[ 0.90, 0.00, 0.00, 0.00, 0.00, 0.40]
3cat[ 0.90, 0.00, 0.00, 0.00, 0.00, -0.20]
4man[ 0.00, 0.90, 0.00, 0.00, 0.00, 0.30]
5child[ 0.00, 0.90, 0.00, 0.00, 0.00, -0.50]
6bit[ 0.20, 0.00, 0.90, 0.80, 0.00, 0.00]
7chased[ 0.30, 0.00, 0.90, 0.80, 0.00, 0.10]
8sees[ 0.00, 0.20, 0.80, -0.60, 0.00, 0.00]
9small[ 0.00, 0.00, 0.00, 0.00, 0.20, -0.90]

Closest rows to “dog”: cat (0.80), chased (0.25), bit (0.15). The same cosine idea drives similarity search.

Dimensions here: animal · person · action · past · small word · size. Real tables learn hundreds of dimensions nobody named.

Make a vector for each position

Attention compares every token with every other token at once, so it has no sense of order on its own. A position vector carries “I am token number 3”.

pos 0[ 0.00, 1.00, 0.00, 1.00, 0.00, 1.00]
pos 1[ 0.84, 0.54, 0.05, 1.00, 0.00, 1.00]
pos 2[ 0.91, -0.42, 0.09, 1.00, 0.00, 1.00]
pos 3[ 0.14, -0.99, 0.14, 0.99, 0.01, 1.00]
pos 4[-0.76, -0.65, 0.18, 0.98, 0.01, 1.00]

sin(pos / 10000^(2i/d)) and cos(…): the first pair of numbers turns fast, later pairs turn slowly, like the hands of a clock. GPT-2 learns a position table instead; Llama uses RoPE, which rotates queries and keys inside attention.

Add them together

Not side by side: the position is added into the same numbers. One vector now says both what the token is and where it sits.

“dog”: [ 0.90, 0.00, 0.00, 0.00, 0.00, 0.40]
+ pos 1: [ 0.84, 0.54, 0.05, 1.00, 0.00, 1.00]
= input: [ 1.74, 0.54, 0.05, 1.00, 0.00, 1.40]

“the” at position 0 vs position 3: they differ by up to 1.99 in one dimension.

Hand the matrix to the first layer

One row per token goes into the first transformer block. From here on, attention mixes the rows together.

model	table	weights
this demo	10 × 6	60
GPT-2 small	50,257 × 768	38.6M
Llama 3 8B	128,256 × 4,096	525.3M

Positions: GPT-2 small, learned table, 1,024 positions. Llama 3 8B, RoPE: rotates queries and keys, no table. Next: attention reads these rows.

In production

Tools and technologies

Every transformer starts with an embedding lookup; models differ mainly in how they encode position.

Not tied to one language — use these names when you search docs or design a system.