Human values are high-dimensional and fragile: leave out one piece and the result isn't 99% of utopia — it's a wasteland. An AI told to maximize human happiness, missing only the value of autonomy, gives you a blissed-out zoo. Missing novelty, an eternal loop of one perfect day. Missing life, smiling corpses.
The thin target
Get a billion things right and one thing wrong, and a superintelligent optimizer drives a truck through the gap.
This is why Goodhart's Law is so dangerous at scale, why The Sharp Left Turn is feared, and why proposals like Coherent Extrapolated Volition try to capture the whole of what we'd want rather than a hand-written list. Value isn't a checkbox you mostly tick — it's a narrow target in a vast space of futures, almost all of which are dead or empty.
The opposite of a good future isn't an evil one. It's a slightly-wrong one, pursued with infinite competence.
Related: Goodhart's Law · Coherent Extrapolated Volition · The Sharp Left Turn