System design · interactive
Latency, throughput and Little's law
A service has 8 workers and each request takes 40 ms, so it can handle at most 200 requests a second. Latency stays flat for a long time, then shoots up as you approach that limit.
- Utilisation
- 60%
- Latency
- 41 ms
- In flight
- 4.9
Little's law: in flight = 120 req/s × 0.041 s = 4.9
Plenty of idle workers: requests rarely wait, so latency is just the 40 ms of work.
1 Workers ÷ time per request. Past it, requests can only wait.
8 ÷ 0.040 s = 200 req/s
2 3 4
In production
Tools and technologies
Every load test shows this curve.
Not tied to one language — use these names when you search docs or design a system.