Two key tools for pulling human-readable concepts out of a model's messy internals.
Probing — the simple test
Train a tiny classifier on a model's internal activations to ask: does the model represent this concept here? If a simple probe can read "is this sentence about sports?" from layer 12, the model encodes that property at layer 12.
Probing tells you a concept is present — not how the model uses it.
Sparse Autoencoders (SAEs) — the breakthrough
The problem: superposition crams many concepts into each neuron. SAEs learn to re-express activations as a much larger but sparse set of features — only a few active at a time — so each one tends to be a single, clean, interpretable concept.
tangled activations → SAE → thousands of sparse features
("Golden Gate Bridge", "DNA", "deception")This let researchers extract millions of interpretable features from production models and even steer behavior by turning features up or down — a major recent leap toward reading a black box.
Related: Mechanistic Interpretability · Features and Circuits · Can We Read a Model's Mind