Skip to content
Optimal Lab

System design · interactive

Streaming responses

A model answer takes several seconds to generate. Streaming only helps if every hop between the model and the browser passes chunks on immediately. One buffering proxy undoes it all.

t = 0.0 s

  1. Model

    0/29 tokens

  2. API server

    0/29 tokens

  3. Proxy / CDN

    0/29 tokens

  4. Browser

    0/29 tokens

User's screen

…

The API streams, but the proxy collects the whole response before passing it on. The user still waits 8 s.

Setup

0.0 s

Step 3 of 4

  1. 1

    The model emits tokens one by one

    The first token arrives in well under a second; the full answer takes many seconds.

    Ask the provider API for stream: true.

  2. 2

    Your server forwards, not collects

    Send chunks to the client as they arrive, over Server-Sent Events or a chunked HTTP response.

    Content-Type: text/event-stream

  3. 3

    Turn off buffering in proxies

    Nginx, some load balancers, CDNs and compression middleware wait for the whole body by default. Disable it for streaming routes.

    X-Accel-Buffering: no

  4. 4

    Handle disconnects and timeouts

    If the user leaves, cancel the model call so you stop paying for it. Raise idle timeouts or send heartbeats on long streams.

    Long-lived connections need different limits.

In production

Tools and technologies

Streaming has to be enabled at every hop.

Not tied to one language — use these names when you search docs or design a system.