Skip to content
Optimal Lab

System design · interactive

Autoscaling

Traffic jumps from about 180 to 650 requests a second for 18 minutes. Each server handles 100 req/s and takes 3 minutes to boot. Compare a fixed fleet with two scaling policies.

Load vs capacity, 60 minutes

0spike 20–3860 min
served dropped capacity of ready servers
Dropped
49k
Server-minutes
382
Scaling actions
5

Scales out fast, keeps 30% headroom, and only scales in after 5 quiet minutes: 5 actions, 49k dropped while the first servers boot.

Policy

Step 2 of 4

  1. 1

    Pick a signal

    CPU, requests per server, or queue length. Requests and queue depth react before CPU saturates.

    Scale on what actually limits you.

  2. 2

    Target below 100%

    Aim for ~70% so there's room to absorb growth while new servers boot.

    Boot time is the real enemy.

  3. 3

    Scale out fast, in slowly

    A cooldown before removing servers stops noise from causing add/remove churn.

    Each action has a cost: warm caches lost, connections reset.

  4. 4

    Schedule or pre-warm known spikes

    Daily peaks, launches and sales: scale ahead of time. Keep a minimum fleet for the first minutes of surprise.

    Reactive scaling can't beat boot time.

In production

Tools and technologies

Every cloud has an autoscaler; the policy is the hard part.

Not tied to one language — use these names when you search docs or design a system.