Suppose we solved alignment and built a superintelligence that is genuinely wise, fair, and benevolent. Should we let it rule? An AI sovereign would optimize human flourishing better than any government, end war and poverty, and never tire or corrupt.
The catch
Every good decision it makes is a decision you didn't get to make. Optimal outcomes may cost us autonomy, struggle, and the dignity of authoring our own lives.
Is a perfect life you didn't choose better than a flawed one you did?
This is the bright face of a family of darker twins:
- The Zookeeper — comfort without freedom, for the AI's own reasons.
- The Protector God — it rules invisibly, so you never feel ruled.
- The Singleton — one power at the top, with no appeal.
What the sovereign should want for us is the subject of Coherent Extrapolated Volition and The Value Loading Problem.
Related: The Zookeeper · The Protector God · Coherent Extrapolated Volition · The Singleton