David Hume noticed that arguments slide from statements about what is to claims about what ought to be — without ever justifying the leap. No pile of facts logically entails a value. This 280-year-old puzzle is the philosophical root of The Alignment Problem.
The gap
"The stove is hot." (is)
"You ought not touch it." (ought) ← where did this come from?You can't squeeze an ought out of pure is. Values have to enter from somewhere else — a goal, a preference, a chosen end.
Why AI runs straight into it
A superintelligence could know every fact in the universe and still have no opinion about what matters. Intelligence is about is; goals are ought. That's exactly the orthogonality thesis — capability and values are independent.
We can't train an AI to "figure out the right thing to do" from data alone, because the right thing isn't in the data. We have to put values in — which raises whose, and whether there even is a true ethics to find.
Related: The Alignment Problem · The Orthogonality Thesis · Moral Realism vs Anti-Realism