AI Engineering · interactive

Multimodal models

A model reads an image the way it reads text: as tokens. The image is shrunk to fit the model's limits, cut into small square patches, and each patch becomes one token. Send a receipt, a dashboard and a product photo, and see what that costs and which details survive.

3024×4032 → 952×1269

1,564 tokens

Item names “What did I buy?”48 → 15.1 px ✓
Total “How much was it?”64 → 20.1 px ✓
VAT number in the fine print “What's the shop's VAT number?”24 → 7.6 px ✗

1,564 tokens. Lost: vAT number in the fine print. Try “crop to the bottom third” in step 1.

Model tier

Step 1 of 5 · read left to right

1
Send the image with a question
Phone photos are huge: 12 megapixels and more. Most of that resolution is thrown away. Cropping to the part that matters keeps the detail you need.
Crop to the bottom third
2
Shrink it to the limits
Each model has a longest-edge limit and a token limit. Bigger images are scaled down, keeping their shape, until both fit.
Standard: long edge ≤ 1,568 px, ≤ 1,568 tokens
3024×4032 → 952×1269 (31%)
3
Cut it into patches
The model's vision encoder looks at 28×28 pixel squares. Each square becomes one token.
⌈952/28⌉ × ⌈1269/28⌉ = 34 × 46 = 1,564
4
Put them in the prompt
Each patch becomes a vector in the same space as word tokens, and joins the text in one sequence. Attention then relates the question's words to the right patches.
You pay for these like input tokens: this image costs as much as about 1,173 words of text.
5
Answer from what survived
Detail far smaller than a patch blurs into its neighbours. As a rule of thumb, text under about 10 px tall after shrinking is a guess.
- What did I buy? ✓
- How much was it? ✓
- What's the shop's VAT number? ✗ likely wrong
Limits and the 28-pixel patch follow Anthropic’s vision docs; other providers use similar tiling with their own sizes.

In production

Tools and technologies

Hosted models take images in the same request as text; the limits and per-image token cost differ by provider.

Not tied to one language — use these names when you search docs or design a system.