Skip to content
Optimal Lab

AI Engineering · interactive

Attention: which words does a word look at?

“The animal didn't cross the street because it was too tired.” What was tired? Attention is how each token pulls meaning from the tokens around it. Change the last word and watch where it looks.

One attention head · query “tired”

  1. The2%
  2. animal47%
  3. didn't2%
  4. cross4%
  5. the2%
  6. street2%
  7. because2%
  8. it19%
  9. was2%
  10. too2%
  11. tired13%

“tired” pays most attention to “animal” (47%). Click any word on the left to make it the query.

The sentence ends with…

1

Give every token a query, key and value

Three learned projections of the same token: the query asks “what am I looking for?”, the key says “what do I contain?”, the value is what gets passed on.

query
[6.0, 0.0, 1.2, 0.0]
key
[0.5, 0.0, 0.3, 0.0]
value
[0.6, 0.0, 0.0, 0.4]

Dimensions here: animate · place · refers back · filler. Real heads use 64 to 128 learned dimensions nobody named.

2

Score the query against every key

A dot product: multiply matching dimensions and add. Big when the query asks for what the key holds.

  • The0.00
  • animal6.00
  • didn't0.00
  • cross1.20
  • the0.00
  • street0.00
  • because0.00
  • it4.20
  • was0.00
  • too0.00
  • tired3.36
3

Divide by √d (√4 = 2)

Without this, scores grow with the vector size and softmax would put almost all weight on one word.

  • The0.00
  • animal3.00
  • didn't0.00
  • cross0.60
  • the0.00
  • street0.00
  • because0.00
  • it2.10
  • was0.00
  • too0.00
  • tired1.68
4

Hide the future

A model that writes left to right (GPT, Claude, Llama) may only attend to earlier tokens. The mask sets later scores to −∞.

0 of 11 tokens hidden from “tired”.

5

Softmax the scores into weights

eˣ of each score, divided by the total. Every weight is between 0 and 1, and they add up to 1.

  • The2%
  • animal47%
  • didn't2%
  • cross4%
  • the2%
  • street2%
  • because2%
  • it19%
  • was2%
  • too2%
  • tired13%
6

Blend the values

Multiply each value by its weight and add them up. That sum is the new vector for this token, now carrying meaning from the words it attended to.

  • animate0.65
  • place0.13
  • refers back0.19
  • filler0.21

“tired” now leans animate: it took most of its meaning from “animal”. A real model runs dozens of heads like this in parallel, in every layer, then passes the result on to predict the next token.

In production

Tools and technologies

Every transformer runs this same score, mask, softmax and blend, in every head and every layer.

Not tied to one language — use these names when you search docs or design a system.