Skip to content
Optimal Lab

AI Engineering · interactive

Running models locally

Downloading an open model is easy; the question is whether it fits in your machine's memory and how fast it writes. Pick a model, a precision, a context length and a machine, and see the memory it needs and the tokens per second you'll get.

Llama 3.1 8B · 4-bit (Q4_K_M) · 32K context

7.9 tok/s

  • weights 4.9 GB
  • KV cache 4.3 GB
  • runtime 1.0 GB
  • ┊ GPU memory 11 GB

The same setup on every machine

  • Gaming PC, 8 GB GPU9.7 tok/s (offloaded)
  • Laptop, 16 GB unified7.9 tok/s
  • Workstation, 24 GB GPU66 tok/s
  • Desktop, 128 GB unified36 tok/s

Fits in 11 GB of fast memory: about 7.9 tokens a second.

10 GB needed · 11 GB fast memory

Step 1 of 5

  1. 1

    Pick a model

    The parameter count sets almost everything: more weights, better answers, more memory, slower tokens.

  2. 2

    Choose the precision

    Quantization stores each weight in fewer bits. Weights take params × bits ÷ 8 bytes. 4-bit is the usual local choice.

    8.0B × 4.85 bits ÷ 8 = 4.9 GB

    A small quality loss; the usual choice for local use.

  3. 3

    Add room for the context

    The KV cache keeps keys and values for every token in the conversation, for every layer. It grows with the context you allow.

    128 KB per token × 32,768 = 4.3 GB

  4. 4

    Check it fits

    It has to fit in memory the GPU can read fast. A discrete GPU can push extra layers to system RAM, which works but is slow. A laptop's unified memory is shared with everything else.

    Apple-silicon base chip. The OS and apps keep about 5 GB.

  5. 5

    Estimate the speed

    Each new token reads every weight once. So speed is about memory bandwidth ÷ model size, not raw compute.

    120 GB/s × 60% ÷ 9.2 GB ≈ 7.9 tok/s

    People read about 250 words a minute, roughly 5 tokens a second; much slower than that and a chat feels sluggish. Reading the prompt in (prefill) is much faster than writing.

In production

Tools and technologies

A handful of runtimes load quantized open models on laptops and gaming PCs; the arithmetic is the same in all of them.

Not tied to one language — use these names when you search docs or design a system.