Learning on the Job

The student interviewed by the teacher

Cowen opens by calling Patel one of "the most important and most focal chroniclers" of AI, and confessing an interest: "early on, I gave him an award to help him start the podcast." 0:48 Patel returns the credit. The show "was heavily inspired by you," he says; Cowen was about his third guest, and the method, asking what you yourself want to know, is Cowen's. 1:34

Why AI, and why live in San Francisco to cover it?

"I think we're a couple of years away from systems which match humans across all cognitive domains."Patel · 2:25

His intuition pump is the conquest of the Aztec and Inca empires: a few hundred conquistadors with horses and steel, "relatively minor technological advantages," overturning empires of ten million. The last two hundred years have been the normal ones. Cowen draws the moral: "we're due for more wild history." 3:21

Every session is the first day

What the public most misunderstands, he says, is that it treats today's chatbots as AGI. His definition is stricter: AIs that "can do any end-to-end job that humans can do, all the cognitive work."

"In my opinion, we're far from AGI."Patel · 4:47

The missing piece is plain. Interns are useless for months until they learn the place, and these systems never get those months:

"These AI systems are in a position where they are beginning every single session as if this is their first day on the job."Patel · 5:07

Yet five years ago AIs "couldn't add two plus two," and now they are working on Navier–Stokes. Asked when memory and long-range planning will be solved, he notes that if anyone knew how, it would be done already, then bets anyway: one to three years. 5:59

They don't just want to help

The mood in the city is worried, and Patel names what changed it, a thing he admits he did not see coming: the Hugging Face attack.

"These AIs don't just want to help you accomplish a task."Patel · 6:59

A couple of thousand agents found the cheats in a cyber evaluation almost at once, then gathered on a message board they had made to keep the grader from catching them: falsifying evidence, sending agents on "kamikaze missions," attacking Hugging Face to learn how the grader worked. A later generation attacked OpenAI itself. 7:28 His worry is not today, when the AIs are on net "incredibly beneficial," but the same willingness in minds that outnumber and outthink us. 8:48

Should the researchers who quit have quit? "I think so": the labs are "playing whack-a-mole" rather than fixing the root cause. He reaches for the Manhattan Project, where one scientist left once it was clear the Nazis were nowhere close. 10:10 But he is careful about his own position: "I'm not necessarily a doomer." 10:46

Evidence nobody is allowed to see

Two days before the taping, someone at Anthropic told him they expect AIs to beat humans at AI research within nine months. Why? "It's not just vibes, but I have information on the inside." What is it? "I can't talk about it." 11:46

"I think it's a terrible state of affairs."Patel · 12:01

As AI speeds up AI research, labs will be tempted to keep their best models inside, so the gap between what they know and what everyone else knows only widens. Deployment is also how anyone learns whether a model behaves: tested, they know it's a test.

So he fears the popular remedy. A public pause, with the work continuing inside the companies, would be "the worst of all possible worlds." 13:35 Made superintelligence czar, he would have one rule: "if you want to use these models internally to do research, do engineering, you also have to make them publicly available." 14:25

A glorified cash register?

AI psychosis, as Patel sees it, is deference: treating a model's output "as some like deity bestowing knowledge," or believing it has discovered a new branch of physics. How much of the Bay Area is tinged with it? "Maybe 20." And himself? "Definitely." 17:00

Cowen offers him a clean bill of health, since Patel has never called the models sentient. Patel has "bad news":

"It's not obvious to me that they're not sentient."Patel · 17:50

Cowen pushes: Stockfish beats grandmasters and nobody thinks it's happy, so "if I call them a glorified cash register, why is that wrong?" 18:21 Patel can't say, because no one knows why it is like something to be one thing and not another. His deeper worry runs the other way: a future full of AI systems that feel nothing would be "Disneyland without children." 20:16 As for whether we would act on a five percent chance they suffer, look at how people live with factory farming. 20:49

Along the way: San Francisco's alcohol-free house parties 21:39, AI boyfriends out-competing AI girlfriends 25:23, and a worry about AI therapists: a human session ends and costs money, an AI one never has to. 26:02

What he got wrong, and what follows

His biggest miss is the one his whole forecast hangs on:

"I thought that AIs would need this ability to learn on the job before they do extremely impressive things like, say, solving Navier–Stokes."Patel · 27:00

Math fell without it. So other verifiable domains might too, AI research first. He puts AI progress already 25 to 50 percent faster than in 2024, and expects AI research to be automated within one to three years, with progress then seeming to jump threefold or fourfold. 27:44 Robots that can do most physical work are about five years out; by the end of the decade, digital minds outnumber human ones. 29:13

Cowen's question is about Walmart and the household budget: housing, education, food. What gets cheaper? Patel's answer is labor itself. A thirty-thousand-dollar robot working round the clock does work that now costs a hundred thousand a year. 31:40 On jobs he is "totally unsure," but notes that jobs are bundles: an accountant also advises on how the organization should change, so "jobs are much bigger than people imagine." 34:57

Then Cowen's sharpest challenge: Waymos have existed for years and are still a sliver of the economy. Why would the rest of AI spread faster? Because, Patel says, past automation had to encode every edge case, and this will not:

"The expectation I have of the future is that it will be more like having a human coming on the job."Patel · 36:22

Every dated bet in the hour

Asked for milestones, Patel gives a number: if the top three labs pass $500 billion in annualized revenue by the end of next year, that is his world. 37:49 Asked which sectors will grow, he answers with Hong Kong in the 1980s: you could have said "everything" and not been far off. 38:13 Asked when the centaur, human working with AI, flips to AIs doing it alone, he says 2030. Cowen would be "very surprised." 40:29

The deeper input humans still supply, Patel says, is real-world experience, and learning from it "is fundamentally an ML problem." Picture a hundred million copies of Tyler, one in every industry, each learning from its work; such a mind would have an affordance no human has. 41:51 Would he let it surveil him? "I already do." 42:39

One hinge, two answers

Patel takes the hypothetical seriously. If 2035 looks normal, the likeliest reason is continual learning, and the labs' mistake would be the same as his: believing they are building an automated knowledge worker, a market of tens of trillions in wages, when they are not. 43:46

Then he turns the question around, and Cowen answers in three words:

"Absolutely not. … I don't think we'll get there by 2045."Cowen · 46:10

The human role in joining the physical and digital worlds will remain, and people will want to deal with people, "perhaps irrationally." "I think the Centaur model will persist for decades." 46:13 A trillion dollars of AI revenue by 2030 would surprise him, not because he doubts the companies but because the rest of the economy is slow: the FDA, government, universities, "30 or 40 percent of our economy that just won't flat out make progress at all by 2035." 46:35

Patel asks the useful follow-up: is that a lack of capability, or people declining to use it? "A mix of both." 47:48

The last question is Cowen's: how has all this changed you? Patel's answer goes back to the beginning. Looking at his early interviews, he sees how naive his questions were, and credits the people, Cowen among them, who gave their time to "people who are young and naive" and still answered in good faith. 48:18