A stranger says: "Give me \$5, or I'll use magic powers to torture 3↑↑↑↑3 people in another dimension." The chance is absurdly low — but the threatened number is so astronomically huge that probability × payoff still says: pay up. That's Pascal's Mugging.
Why it matters for AI
An AI that reasons by expected value can be hijacked by any claim that pairs a tiny probability with a gigantic stakes number. The math says act, even when common sense screams "ignore this."
Reason that only multiplies probability by magnitude has no defense against a big enough imaginary threat.
- It's a cousin of Roko's Basilisk (a low-probability future threat coercing present action).
- It shows why naive utility-maximizers are dangerous, and why robust values and bounded reasoning matter for alignment.
- It's secular Pascal's Wager with the payoffs cranked to infinity.
Related: Roko's Basilisk · Newcomb's Problem and Decision Theory · The Fragility of Value