Skip to content
Optimal Lab

AI Engineering · interactive

Privacy and training data

A model trained on the web can repeat what it read, and the more often something appeared, the more likely it is to come out when prompted. See what an attacker can extract from five things in a crawl, then deduplicate, scrub personal data, add differential privacy, and watch what each costs.

8B model · chance an attacker gets it out

quality 100%

  • An API key committed to one public repo1 copy4%
  • A private email quoted in a forum thread3 copies20%
  • A home address in a scraped public register40 copies94%
  • A patient's note in a leaked dataset12 copies70%
  • A famous novel's first sentence100,000 copies99%

faded: with no defensessecretpersonalpublic

Riskiest item left: a home address in a scraped public register, 94% extractable. Model quality 100% of the undefended model.

Model size

1

Look at what's in the crawl

Web data carries secrets people pasted once, personal data mirrored across sites, and leaked files. Nobody consented to any of it being learned.

  • An API key committed to one public repo: A 40-character token with a known prefix.
  • A private email quoted in a forum thread: An address and two lines of the message.
  • A home address in a scraped public register: Name, street and postcode, in forty mirror sites.
  • A patient's note in a leaked dataset: A paragraph of free text with a name and a diagnosis.
  • A famous novel's first sentence: Memorizing this is the point of training.
2

See what the model will repeat

Memorization grows with how often a string appeared, with model size, and with how much context the attacker can give. A string seen once is rarely extractable; one seen forty times usually is.

What the attacker types

1B: address 86% · 8B: address 94% · 70B: address 97%

3

Deduplicate

Collapse repeated documents to one copy. Memorization of the repeated ones falls sharply, and the model trains slightly better too. Things seen once stay as they were.

address: 94% → 4%

4

Scrub personal data

Pattern rules catch things with a shape: keys, emails, card numbers. Names, addresses and diagnoses in free text need an entity-recognition model, and it misses some.

Scrubber

key 4% · email 20% · address 94% · patient note 70%

5

Add noise, and stop collecting

Differential privacy adds noise to every training step, so no single example can be recovered, at a cost in quality that grows as ε shrinks. And the easiest data not to leak is data you never kept: users' prompts are training data only if you keep them.

Differential privacy

2,000 prompts with personal data enter training a day · quality 100%

The curve’s shape follows Carlini et al. 2022 (memorization grows with copies, size and context); the constants and the detection rates are made up. Deduplication’s small quality gain is from Lee et al. 2021.

In production

Tools and technologies

Memorization is measured in the research; the defenses are tools you run on the data and the training loop.

Not tied to one language — use these names when you search docs or design a system.