Bostrom's Vulnerable World Hypothesis asks us to imagine an urn of inventions. Most balls are white (beneficial) or grey (mixed). The hypothesis: somewhere in the urn may be a black ball — a technology that, once discovered, by default destroys the civilization that finds it.
The black-ball problem
We've been lucky so far. But suppose making a world-ending weapon required only household items — the moment that recipe is known, someone uses it, and there's no taking it back.
Civilization keeps drawing from the urn. We've pulled grey balls (nuclear weapons) and survived through luck and tight control. A black ball offers no second chance.
The unsettling fix
Bostrom argues that surviving a black ball might require extreme preventive measures — pervasive surveillance and strong global governance — trading freedom for survival. That tension is the heart of the Orwellian outcome, and AI is a leading candidate for the next ball drawn.
Related: 1984 and the Orwellian Outcome · The Singleton · Longtermism and Existential Risk