AI Engineering · interactive

A transformer block

A large language model is one block repeated dozens of times. Attention lets the tokens share information, a feed-forward layer works on each token alone, and residual connections add each result onto a running vector instead of replacing it. Remove those, and the tokens blur together.

Layer 1 of 12 · “down”

with residuals

“down”, out of layer 1

d11.31
d20.77
d3-2.04
d42.59

How alike the 4 tokens are, layer by layer

— with residuals— without

After 1 layer the tokens are 8% alike: each one still carries its own meaning. Turn off the residual connections and stack more layers.

Follow the token

Residual connections

Layers stacked1

Normalize each token

LayerNorm shifts and scales each token's vector to mean 0 and spread 1. It keeps the numbers in a steady range as layers stack up.

in: [ 0.10, 0.60, -0.80, 0.90]
normed: [-0.16, 0.62, -1.55, 1.09]

Attention: tokens look at each other

This is the only place tokens share information. Each token blends in the tokens before it, weighted by how well they match.

The2%
cat22%
sat6%
down71%

How these weights are computed: Attention, one head. Real blocks run many heads side by side.

Add the change back onto the token

The residual connection: x + attention(x). The sub-layer only adds a change, so the original token survives every layer, and gradients have a short path back during training.

x: [ 0.10, 0.60, -0.80, 0.90]
change: [ 0.17, 0.46, -1.33, 0.69]
result: [ 0.27, 1.06, -2.13, 1.59]

Normalize again

A second LayerNorm, so the feed-forward layer also gets steady numbers.

normed: [ 0.05, 0.60, -1.64, 0.98]

Feed-forward: each token on its own

Expand to 16 numbers (4× wider), drop the negatives (ReLU), and project back to 4. Most of a model's weights, and much of what it knows, live in these layers.

16 hidden units for “down”, 10 switched on

out: [ 1.03, -0.29, 0.09, 1.00]

Add back, then repeat the block

The second residual add gives the block's output. It goes straight into the next block, which has its own weights. Only after the last block does the model predict the next token.

output: [ 1.31, 0.77, -2.04, 2.59]

Blocks: 32
Attention / block: 41.9M
Feed-forward / block: 176.2M
Whole model: 8.03B

81% of each block is the feed-forward layer. Llama 3 8B is 32 copies of this block (4,096 numbers per token) plus 1.05B in its embedding tables; its published size is 8.0B. Next: how the last block’s output picks a token.

In production

Tools and technologies

GPT-2, Llama and nearly every LLM since stack this same block; the configs only change its size and count.

Not tied to one language — use these names when you search docs or design a system.