We can build a neural network, train it, and watch it work — yet no one can fully explain why it produces a given output. Its "knowledge" is spread across billions of numbers with no human-readable logic. That's the black box problem.
Why it's opaque
Unlike hand-written code, a model's behavior emerges from weights tuned by gradient descent, not rules a person wrote. There's no commented function for "detect sarcasm" — just patterns of activation no one designed or annotated.
Why it matters
| Stakes | Problem |
|---|---|
| Trust | Hard to rely on a decision you can't explain |
| Safety | Can't verify it's not secretly misaligned |
| Bias | Can't see why it discriminates (Algorithmic Bias) |
| Law | "Right to explanation" rules need explanations |
It connects straight to philosophy: if we can't see inside, how would we ever know whether anyone is home — or just simulating it (The Chinese Room)?
Cracking it open is the goal of Mechanistic Interpretability.
Related: Mechanistic Interpretability · Algorithmic Bias · The Chinese Room