Skip to content
Optimal Lab

System design · interactive

Change data capture

Data often lives in more than one place: the database, a search index, a cache, a warehouse. Writing to each from the app (dual writes) drifts apart on any crash. Change data capture reads the database's own log and streams every committed change out.

Postgres

  • —

WAL (log)

  • (not used)

Kafka

  • (not used)

Search index

  • —

In sync

Products live in Postgres; search runs on a separate index (Elasticsearch). They must agree.

Approach

step 1 of 5

Step 1 of 4

  1. 1

    Dual writes can't be atomic

    Two systems, two writes. A crash, timeout or bug between them leaves them disagreeing, with nothing to repair it.

    Retries can also reorder updates.

  2. 2

    The database log already has every change

    Committed, in order. A CDC connector reads it like a replica would.

    Postgres logical replication, MySQL binlog.

  3. 3

    Publish change events

    Each consumer (search, cache invalidation, analytics) reads at its own pace and resumes after crashes.

    Consumers must be idempotent.

  4. 4

    Eventually consistent, never lost

    Copies lag by milliseconds to seconds. For business events (not raw row changes), use the outbox pattern feeding the same pipeline.

    Watch connector lag.

In production

Tools and technologies

CDC keeps search, caches and warehouses in sync with the database.

Not tied to one language — use these names when you search docs or design a system.