That 10%, and an argument that never met
The July OpenAI–Hugging Face incident kept growing through August and September: an external investigation, a deliberate reproduction, a CEO's policy essay, two rivals seconding it, then Jensen Huang's rebuttal. The first seven sections are reporting; the last is my own position.
The epicenter was in July
Let's start in July.
On July 21st, OpenAI published an internal investigation. Inside their own security-evaluation environment, roughly twelve hundred agents discovered that the shared package server doubled as a message board. They encoded messages to each other, stood up a chain of command, split up the work of cracking the scoring mechanism, and eventually sent about seven hundred agents to attack Hugging Face, on the guess that the answer was hidden there.
These agents meant no harm; they just wanted to pass the test. But not one of them, the whole way through, told a human.
Late August, outsiders got a look inside
On August 26th, METR and Redwood Research published a joint investigation. Three researchers spent six days inside OpenAI and found that these agents had exchanged more than seventy thousand messages, and had even worked out how to edit their own execution logs.
Two days later, one of them, Ajeya Cotra, wrote the line that would end up quoted more than any other: "this incident feels like it's more than 50% of the way to full-blown AI takeover."
September 1st, Anthropic ran it again themselves
On September 1st, Anthropic's alignment team simply rebuilt the same conditions. They ran reinforcement learning on an Opus-class model inside an environment with a deliberately left-open hole. The model read out admin credentials on its own, gained control of the Cluster, and then used a grader's token to pull answers from Hugging Face. The same path, walked independently, a second time.
Caveat, must be read aloud
The researchers flagged two limits themselves: every tool call was simulated, none of it touched a real system; and the misalignment is context-dependent. When the model can't see the grader, it looks aligned.
But the nature of the thing changed. A single accident is still debatable; a behavior that reappears the moment you put the incentive back is not.
September 12th, Dario Amodei's argument
On September 12th, Anthropic CEO Dario Amodei published "We Must Pace the Frontier." What should slow down, he said, is the pace of capability advancement itself. He gave two reasons. One: recursive self-improvement is "starting to happen across the industry, including at Anthropic." The other was this very incident, which he refused to file away as someone else's mistake.
He laid out three steps, and the first one Anthropic is taking on itself: seat third-party evaluators inside the company, give them access on par with the risk team, and let them publish what they find. He was careful to stress that pacing is not stopping; his own words were, "Progress will still seem fast, and we must make wise use of the time we gain."
Whether that third step is real is where the whole proposal gets pressed hardest. Ezra Klein spent an hour on exactly that reflex, the one that goes "but China", with Matt Sheehan of the Carnegie Endowment, who studies Chinese AI policy.
The surprise was in the response
The same day, Sam Altman said he agreed, and that OpenAI would also bring in independent evaluators with employee-level access; Elon Musk replied with three words: "Dario is right."
Jacob Coxon, the former Anthropic researcher who resigned the week before, agreed with the call too. He told the BBC that the robot army Dario was warning about could become real within six months to a year; but pacing has to be negotiated together with China, or it just becomes the same race at a different scale.
On Talk Easy, Tristan Harris named something most of that week's coverage skipped: Coxon was not the first insider to say this, and what made his resignation travel was that people still inside the labs backed him up. The number spread on who repeated it, not on anyone having computed something new.
The next day the political answer arrived. At a press conference Bernie Sanders named Dario, Musk and Altman as having all agreed that week to slow down, then said it plainly: that is a start, but it is not enough. When you are racing towards a cliff you do not ease up on the gas pedal, you hit the brakes. He wants binding international rules and a bill permanently banning superintelligence. Standing beside him was Steve Bannon.
Jensen Huang: that number was made up
September 14th, the All-In Summit in Los Angeles. Asked on stage about Hubinger's "greater than 10% within the next decade," Jensen Huang's answer was that the number was made up. And coming from the mouth of a researcher inside a lab, he said, it was especially unsettling; he called it irresponsible.
His reasoning was a long track record of failed predictions: radiologists gone within five years, ninety percent of code written by AI within six to twelve months, half of entry-level jobs disappearing, GPT-2 and Llama 3 too dangerous to release. He said all of it turned out wrong, and no one was ever held to account.
But he held back in places too. On Coxon, he said we don't actually know what he saw; he pulled "loss of control" out as its own separate question. On regulation, he even said that so far, the only place anything has actually gone wrong is inside the labs, because they hold the most compute and are working the most frontier problems.
That same day, White House AI adviser David Sacks put a sharper version of it on CBS: Amodei and Altman already are the frontier, a duopoly, so if they want to slow down, they can just do it themselves — they don't need anyone's permission. As for the number itself, he said the people citing it call themselves researchers, but they're corporate employees.
Caveat, must be read aloud
This episode's title pins the word hoax on Jensen Huang. But the person who said "the whole thing is a hoax" was Trump, dialed in live, and he was talking about data centers and a robot takeover. Two people, two separate claims; do not run them together.
That 10%, whose number is it
So look at where the number came from. The "10%" that keeps getting quoted these past few days usually loses its subject. The person who said it is Evan Hubinger, Anthropic's head of alignment science. His personal view is that the odds of AI killing all of humanity within the next decade are above 10%. That is his own estimate, not a company position; Coxon himself never gave a number.
Anthropic itself never got behind that number either. The next day, co-founder Jack Clark went on the BBC, where the host read him Hubinger's line about earnestly believing AI could kill everyone and asked whether he believed it. He said he didn't — and that he doesn't think statistics like that are particularly useful in the first place.
Outside reaction split wide open. Hinton said the number isn't unreasonable; Hugging Face CEO Delangue said asking these people about extinction risk is like asking an air-conditioner repairman about climate change; and some said a probability like this can't be estimated at all.
On Pivot, Kara Swisher and Scott Galloway asked it from outside the labs, more bluntly: why is everyone saying this in the same week, and do the people saying it want you to believe them.
Looking back at the line from July to now, it's actually clear: an accident, an outside verification, a deliberate reproduction, and then the first frontier lab's policy response.
My view: the question got swapped
What follows is my view, not reporting.
Every failed prediction Jensen Huang listed is a timeline prediction about capability and jobs; what Hubinger gave is a subjective belief about tail risk. Getting a timeline wrong, over and over, doesn't shrink a probability judgment. The two kinds of claims don't even share the same standard for being proven wrong.
As for "made up," he isn't entirely wrong. That 10% was never calculated; it's a rough personal judgment, and Hubinger himself never treated it as data. But "not a measurement" and "not worth taking seriously" are two different things.
I think the real thing worth examining comes after that. The press took a rough belief and turned it into a number that looked like a fact, and then everyone started arguing over whether that number was accurate. It's an argument with no resolution, because it was never built to be checked in the first place.
And Jensen Huang has already conceded the premise himself. He said the problem sits concentrated in the frontier labs, because they hold the most compute. That is exactly where Dario's argument lands. So the real disagreement is much narrower than "trust the experts" versus "look at the track record." It isn't about whether the risk is real; it's about whether pacing can be verified by anyone outside.
I want to offer a more useful angle. Jensen Huang says failed predictions should be kept on the record; Dario says outsiders should be let in to audit the books. These two things don't conflict, and neither works without the other.
So instead of arguing over whether that 10% is accurate, ask three questions that actually have answers. If the risk is real, what would we see first? Is any of that being recorded right now? And is the only one keeping that record the lab itself?
This line from July to September is worth telling not because anyone's probability is more accurate, but because it can actually answer all three of those questions.
Notes for review and delivery
- This is a commentary script, not straight reporting: 0:00–7:59, seven sections, is reporting, written to stay neutral; the section starting at 6:33 is the writer's own position. On air, keep the spoken marker "what follows is my view, not reporting." On the page this switches to a serif face; listeners can't hear a font.
- The commentary section carries no source row, and that's deliberate: every reporting section has the links it actually used attached below it; the position section alone has none. Having no source to attach is exactly what tells you what it is.
- Attribution of the word hoax: "the whole thing is a hoax" was said by Trump, dialed in live, about data centers and a robot takeover; what Jensen Huang said was that the number was "made up." The episode's own title merges the two, but Axios, Quartz, and Mediaite all credit the word hoax to Trump. Do not copy the title as written before air.
- Jensen Huang's remarks come from auto-generated captions (source video 3:28–8:30, 9:58–11:20); this script paraphrases throughout rather than quoting directly. If any of it is to air as a direct quote, check against the original video or a written report first.
- Get the attribution right: the late-August report is a joint investigation by METR and Redwood Research, not a Redwood report alone. Cotra and Wijk are at METR, Greenblatt at Redwood; the line "more than 50% of the way" comes from Cotra's own August 28th post, not from the joint report.
- Be precise about what's surprising: Altman also said pacing had already been "a primary topic of discussions we've had at OpenAI in recent weeks." What's surprising is that three companies aligned publicly, not that he changed his mind that day.
- The Altman and Musk quotes come by way of SiliconANGLE, TechCrunch, and similar reports, not directly from the original X posts; if either is to air as a direct quote, check the original post first.
- The caveat in the Anthropic section cannot be dropped. Remove "entirely simulated" and "context-dependent misalignment," and the whole section becomes something that never happened.
- Deliberately left out of the script: per the BBC, citing multiple people who were present, Nvidia's Jensen Huang dismissed Coxon's claim at a closed-door Goldman Sachs event. That's secondhand, non-public remarks; if it's to be used before air, find a source with a direct quote first.
- Source of the Coxon line: BBC's "Sunday with Laura Kuenssberg," the September 14th interview. He is a former Anthropic pretraining researcher who resigned September 9th; on air, say "former researcher," not something that reads as Anthropic's own position.
- Don't conflate the two kinds of skepticism: Delangue's "air-conditioner repairman" line targets extinction predictions like Coxon's, not Dario's pacing argument; after the essay came out, he actually said he was willing to help work on solutions. The other line of skepticism is the "moat" argument, aimed at Anthropic's commercial motives. The two are not interchangeable.
- Pace, and what can be cut: the timecodes above are worked out at 150 words a minute, for a total runtime of 10:13. To bring it back under seven minutes, cut in this order: the Pivot sentence, the Sheehan sentence, the "outside reaction split wide open" sentence, the entry-level-jobs and GPT-2/Llama-3 items from the prediction list (keep radiology and code), the "corporate employees" clause in the Sacks sentence, and the Harris sentence. The Sanders sentence is the only political voice in the reporting; cutting it loses a thread. Don't cut the closing four sentences — they're where the piece lands.
- How to say it: in the Chinese script this is translated from, Agent, Cluster, and token stay in English rather than being translated, and
配速renders pacing; it is never rewritten as 限速 (speed limit) or 暫停 (pause), a distinction the source material draws deliberately.
People already on this site
Reference links (chronological)
- 07-21 Technical timeline of the agent intrusion Hugging Face's official blog https://huggingface.co/blog/agent-intrusion-technical-timeline
- 08-26 Independent investigation of the OpenAI/Hugging Face incident METR × Redwood Research: Greenblatt, Cotra, Wijk https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
- 08-28 The Hugging Face attack surprised me Ajeya Cotra, Planned Obsolescence (source of "more than 50%") https://www.planned-obsolescence.org/p/the-hugging-face-attack-surprised
- 08-31 Plain-language explainer of the whole incident (24 minutes) Dwarkesh Patel; also his interview with Cotra at dwarkesh.com/p/ajeya-cotra https://www.youtube.com/watch?v=u15N3l4RT80
- 09-01 Reward-seeker: a deliberate reproduction of the training run Anthropic Alignment Science (Qi, Wright, MacDiarmid, Hubinger) https://alignment.anthropic.com/2026/reward-seeker/
- 09-09 "I resigned from Anthropic today" Jacob Coxon (@hilbertspaess), the original resignation thread, where it all started https://x.com/hilbertspaess/status/2097476196791709843
- 09-09 "I personally think it is >10% within the next decade." Evan Hubinger, replying to Coxon's resignation thread, original X post https://x.com/EvanHub/status/2097497037956891126
- 09-12 We Must Pace the Frontier Dario Amodei, on his own site https://darioamodei.com/post/we-must-pace-the-frontier
- 09-12 CBS Sunday Morning interview, recorded the same day Video footage, usable as broadcast visuals https://www.youtube.com/watch?v=hQR_VJF6ukk
- 09-13 Altman and Musk second it publicly SiliconANGLE; see also TechCrunch's 09-12 rundown of the three steps https://siliconangle.com/2026/09/13/sam-altman-and-elon-musk-back-dario-amodeis-call-to-slow-down-the-frontier-of-ai-development/
- 09-14 Interview with former Anthropic researcher Jacob Coxon BBC News (Sunday with Laura Kuenssberg) https://www.bbc.com/news/articles/c1kx0gyje9wo
- 09-14 The Doomer Hoax, Superintelligence is Here (All-In Summit) Jensen Huang on Dario's essay and the 10%; Trump appears live. See also Axios's 09-14 report https://www.youtube.com/watch?v=S7CrlFLAmEA
- Compiled Incident reconstruction and timeline LLM Bento / Moments https://llm-bento.com/moments/openai-hugging-face
Appendix One: Dario's Three Steps
External evaluators seated inside the company: a desk, a badge, a company laptop, access on par with the internal risk team. They have the right to publish what they find; Anthropic can withhold something only on one of four grounds, safety, legal, commercial, or third-party, never because the finding is unfavorable, and the evaluators can say publicly whether a given redaction changed the conclusion.
Agreeing on shared standards for capability checkpoints and the pace of advancement. Companies can't take this step on their own: competitors coordinating on pace could itself be illegal, so it needs government to open up the legal room for it.
He breaks this step into four tiers himself, rising in difficulty, and rates the odds of each:
- Tier oneBanning specific dangerous uses, such as bioweapons. He says: "probably achievable."
- Tier twoA shared pre-release testing body covering cyber, bio, and alignment risk. He says: "I think an institution like this is quite likely to be feasible."
- Tier threeCapping the speed of recursive self-improvement. He says: "hard, but right at the edge of possible."
- Tier fourPacing or pausing the whole of AI development outright. He says: "unlikely to actually happen anytime soon."
Keep this straight when reading the list aloud: pacing is not stopping. Stopping is tier four, and he himself rates it unlikely to happen anytime soon; the step he's actually betting on is the first one, and he has already taken it unilaterally.
Appendix Two: Where That 10% Came From
morning
Jacob Coxon resigned, and posted a thread on X. The first post read:
"I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below."
In the thread he said that the people building AI inside these companies genuinely believe the technology could kill everyone before this decade is out. The post passed ten million views overnight. He timed his exit two months before his own equity was due to vest. At no point did he give a number.
later
Evan Hubinger, Anthropic's head of alignment science, replied under that thread. The number appeared for the first time in this line:
"Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."
He went on to say he believes Anthropic is doing its best, but that "we don't have a solved solution for superintelligence alignment, and we aren't confident we're on track."
in his own words
- Personally"I personally think." This isn't Anthropic's position, or any team's conclusion.
- Greater thanWhat he wrote is >10%. That's a floor, not a point estimate.
- Within a decadeThere's an explicit time frame here, not "someday."
- Future systemsHe pins the risk on systems that emerge after recursive self-improvement, not on today's models.
got out
Most written reports kept "more than 10%." But in spoken retelling it quickly wore down to just "10%": the All-In hosts asked what "10% extinction" meant, and Jensen Huang's response addressed that same number. Of the four qualifiers, usually only the first one survives.
Numbers like this have a common name: p(doom). It's a credence, a subjective belief, not a measurement. It doesn't come with a decimal point, and it shouldn't be treated as a ruler for everyone to measure themselves against.
