Skip to content
Optimal Lab

System design · interactive

Background jobs

Signing up triggers a welcome email, an avatar resize and a CRM sync. Doing them inside the request makes sign-up slow and fragile. Putting them on a queue lets the request finish at once and lets failures retry on their own.

POST /signup

waiting

  • Send welcome email—
  • Resize avatar—
  • Sync to CRM—

The sign-up request creates the user…

Where the work runs

step 1 of 3

Step 1 of 4

  1. 1

    Keep requests short

    Anything slow or unreliable that the user doesn't need to wait for goes to a queue.

    Emails, images, webhooks, reports.

  2. 2

    Workers pull and process

    Scale workers separately from web servers. Jobs survive web server restarts.

    Enqueue in the same transaction (outbox) to avoid lost jobs.

  3. 3

    Retry with backoff, then dead-letter

    Transient errors retry; permanent ones land in a dead-letter queue for a person to inspect and replay.

    Jobs must be idempotent.

  4. 4

    Delayed and recurring jobs

    Send a reminder in 3 days; run a report every night. Use one scheduler so a job doesn't run once per server.

    Sidekiq, BullMQ, Celery, SQS.

In production

Tools and technologies

Every web framework has a job queue library.

Not tied to one language — use these names when you search docs or design a system.