AI · interactive
Scaling laws
With a fixed training budget, you can train a bigger model on less data or a smaller one on more. Loss follows a predictable curve in both, so there's a best split. Each curve below is one budget; the dot is its sweet spot.
x: parameters (log scale). y: loss, lower is better. Each curve is a valley; more compute moves it down.
- Best size
- 5.6B params
- Training tokens
- 296.4B
- Loss
- 2.14
With 10^22 FLOPs, the best model has about 5.6B parameters trained on 296.4B tokens: ~53 tokens per parameter. Go bigger and it's undertrained; smaller and it can't absorb the data.
1 2 3 DeepMind's Chinchilla showed earlier models were too big for their data. Scale both together.
From the fitted constants in that paper.
4
In production
Tools and technologies
Labs plan training runs with fitted scaling curves.
Not tied to one language — use these names when you search docs or design a system.