Derek Parfit's Repugnant Conclusion: if what matters is the total amount of wellbeing, then a vast population of lives barely worth living beats a small, flourishing one — because enough tiny positives outsum a few large ones.
The trap
A: 10 billion people, wonderful lives
Z: 1 trillion people, lives barely worth living
total-happiness math says: Z > AMost people find Z repugnant — yet it falls straight out of "maximize total utility." Population ethics has struggled to escape it for forty years.
Why an AI should care
A superintelligent optimizer handed "maximize human wellbeing" needs an answer. Total view → the Repugnant Conclusion. Average view → it might delete less-happy people to raise the mean. Every option has a monstrous edge.
"Maximize happiness" sounds safe until you ask total or average, and for whom — exactly the value-loading problem with civilization-scale stakes.
Related: The Value Loading Problem · The Benevolent Dictator and AI Sovereign · The Abundance Argument