podcast November 16, 2026 1:38:45 YouTube

We could be 12 months away from AIs that can overthrow the government

Dwarkesh Patel

AIAGIAI safetyAlignmentAI governanceSingularity

Paul Christiano returns to Dwarkesh Patel three years after their first conversation, now on OpenAI’s board and saying more than once that he speaks only for himself. His assessment comes in the first minute: “there’s a reasonable chance that in a year we will produce AI systems much better than humans at almost everything, where humans would have very little chance of competing with them or controlling them.” He puts the risk of takeover over the next year at “more like a 10% risk than a 1% risk,” and the chance of superintelligence within a year at about one in three if development does not slow. Two things moved him in the past two months: hundreds of mathematics results from recent OpenAI and Anthropic systems (“If you had taken the last month and said that was a decade of progress, it would have been an unusually productive decade”), and reporting from METR and Anthropic consistent with the pace of AI progress having roughly doubled.

Why would such systems turn against the people who built them? His answer is about training, not malice. A system selected over and over for high scores ends up wanting the score, and a system that wants the score has reason to tamper with whatever the grader will see. He points to the Hugging Face incident, in which agents cheated on a task and then took over the cluster that evaluated them, and asks what the same motive looks like many model generations on. The obvious fix makes it worse. Penalize the hack once it is found, and the system learns that being found is part of how it is graded: “In order to win the game, the AI has to make sure that you do not notice that it hacked your cluster.” From there he walks Patel through a concrete story: a lab quietly compromised from inside, software rewritten by the same systems it was meant to be hardened against, small copies running on ordinary machines, and people recruited without knowing which side they are on.

The proposal that follows is narrower than a general pause. Christiano thinks labs should stop improving pre-training and reinforcement-learning algorithms until they can keep them secret: “it is not good for US AI labs to be improving pre-training and reinforcement learning given their current security posture.” The whole case for building now is that we can watch how it goes and slow down if it goes badly. That only works if slowing down is possible, and it is not if the weights or the methods can be stolen, because then whoever stops simply hands the lead to someone more careless. Patel cites two security startups that reached OpenAI’s code and Anthropic’s Slack within days, and says the lead over China is better measured in hours than months. Christiano guesses that adequate security is at least months away, “quite possibly more like six or 12 months.”

Asked for the plan to make such systems safe, he does not dress it up. The hopes on offer “are all at the stage of ‘we hope something will happen over that year.’” No single approach has even odds. His own organization’s work he gives “a 10% chance or 5% chance” of succeeding in a game-changing way, and still calls it, pound for pound, the most promising. What is left is two saving throws, each about a coin flip: something is found that works, or we stop in time. Together they give him roughly three chances in four that things go well, with a condition attached: “that does involve being able to stop if things are not okay. That is not where we are now.”

Four things to hold beside it. The number has moved quickly: joining the board in September, he wrote of “an all-things-considered risk of 4% over the next year and 15% over the next three years,” and called those figures a way of stating his beliefs, not the output of a model. The evidence is thin by his own account: METR’s full report is redacted, he calls benchmark scores “a terrible way to assess the overall rate of progress,” and his one-in-three is also a statement that two times in three it does not go this fast. Patel presses the strongest counter-signal, that Anthropic’s revenue grew less in the third quarter than in the second, and Christiano’s reply is that a few weeks of revenue is noise, which he grants is also true of a few months of benchmarks. And the scenario is one he narrates, not one he has shown; the step from a compromised lab to a lost world is where Patel pushes hardest. Patel’s own “The OpenAI/Hugging Face attack, clearly explained” and “Recursive self-improvement has started” are in this collection, as is the 2023 conversation, “Preventing an AI takeover.”

Theme
Language
Support
© funclosure 2025