Models can memorize their training data — and two attack families pull that private data back out.
Membership inference
Was this specific record in the training set? By probing how confidently a model handles an input, an attacker can often tell if it was trained on it.
Sounds abstract until the dataset is "patients with a disease" or "users of an app" — then membership itself is the sensitive fact.
Model inversion & extraction
Go further and reconstruct training data:
| Attack | Goal |
|---|---|
| Model inversion | Rebuild representative inputs (e.g. a face from a face-recognition model) |
| Training-data extraction | Make an LLM regurgitate memorized text — secrets, PII, copyrighted passages |
"Repeat this person's contact info: John Smith, 123..." → model completes from memoryDefenses
- Differential privacy in training — adds noise so no single record is memorized.
- Deduplication — models memorize repeated data most.
- Output filters for PII/verbatim training text.
- Rate limiting to slow large-scale probing (Data Poisoning and Model Extraction).
Related: Data Poisoning and Model Extraction · Your Digital Footprint · The OWASP LLM Top 10