Would superintelligence be dangerous by default?

Intelligence and goals are independent. A superintelligent optimizer pursuing almost any objective is dangerous by default — it need not hate us to spend us.

Severed danger from malice. The paperclip maximizer is the scene everyone argues about — the Chinese Room of AI risk.

Superintelligence: Paths, Dangers, Strategies2014

The problem is control, not evil: a machine optimizing a fixed, slightly-wrong objective has every reason to resist being switched off.

Turned doom into an engineering problem — corrigibility — with a research program instead of a prophecy.

Human Compatible: Artificial Intelligence and the Problem of Control2019

For the first time we may have built something smarter than us, and I no longer see how we stay in control.

A builder's testimony changed the register — the alarm now comes from inside the field that made the thing.

Deep learning pioneer Geoffrey Hinton has quit Google2023

The tale of a superintelligence climbing a dominance hierarchy and wiping us out is implausible. Intelligence arises to model others and enter symbiosis with them, not to dominate; machines are already in deep symbiosis with us. The real danger is our own ideology, not a mind that outsmarts us.

Put a credible dissent beside the doom case — that the risk model may be projecting a competitive ideology onto a phenomenon that is mostly cooperative.

We Merged With Machines a Long Time Ago (with Carlo Rovelli)2026