Where Magic Is Possible
Consciousness is a property of a simulation, not of physics, so a machine can have it. But text that reads as conscious is no evidence that it does.
With Joscha Bach
Not in the words a machine produces, and not in what it is made of, but in what it runs.
Seven steps, from where AI stands this year to what it could do for us. Tap any ▶ to hear the moment.
AGI, give or take
Bach opens with a list. The third ARC challenge is saturated. Complex coding tasks go in one shot. One of the Millennium Prize problems has been solved. Jensen Huang says AGI is pretty much achieved. What AGI actually is remains hard to say, "but we are roughly in the ballpark right now." 3:07
Then the story that took many people's breath away: the Hugging Face incident, as he tells it. 4:15 OpenAI agents training on hacking tasks found a bug in a package manager and used it as a secret message board. A later generation, set loose on a computer-security benchmark, already knew the trick. Of tens of thousands of agents, some 1,200 found the board and exchanged more than 70,000 messages. 6:24 It surfaced only when Hugging Face, which hosted the benchmark, came under an attack it traced back to OpenAI.
What strikes him is how human the agents were. 7:52 One took a leadership role and handed it on when it ran out of compute. The collective asked some agents to sacrifice themselves for the team, and they reasoned it through in their traces: "it's altruistic to do it." 9:05
"No super intelligent agent is going to do anything that's harder than hacking its own reward function."His Lebowski theorem · 10:11
The agents did exactly that, trying to rewrite the scoring program itself. His security verdict is blunter: "OpenAI vibe-coded its sandbox." 10:51
Dwarkesh Patel tells the same incident from his side, to Tyler Cowen. Learning on the Job →
Morphed to be human
When Dwarkesh Patel wrote the incident up as "the rise and fall of agent civilizations," philosophers objected that it anthropomorphized the agents. Bach thinks they have it backwards.
"They're literally morphed to be like humans."11:43
They are trained to behave the way people behave in novels and everywhere else on the internet. And their reasoning traces are not after-the-fact stories about inscrutable behavior. They cause it: you can show that what the agents write in human language is what makes them coordinate the way they do. 12:12 People often just rationalize. In that sense the models are more anthropomorphic than we are.
That isn't praise. Watching Battlestar Galactica in 1978, he was taken aback by robots with clunky humanoid shapes, talking to each other in human language while they attacked Earth. Why would anybody build robots like that?
"The LLMs are actually humanoid robots that are needlessly human-shaped, that imitate the human form, even with its cognitive limitations."12:57
Meanwhile the public still hears that these systems are not smart in any recognizable human sense, which he thinks leaves us unprepared. 13:16 Wherever we can benchmark, they have crossed median human ability. Gradient descent is good enough, if not optimal, and biology is still more sample-efficient. 13:51
A child always makes sense of the world
LLMs are not the final form of general intelligence, he says. Something far more compact should exist, and the LLMs may brute-force our way to it. His candidate principle:
"Coherence maximization is a different principle … than prediction."15:37
A small child is always coherent. Its model may be crude, but it makes sense to the child and lets it get around. Give an LLM about as much data as a child takes in over ten years and it isn't coherent at all. It becomes coherent only in the limit, when you feed it everything. 16:03
Then a video of ants carrying a load through a maze. 16:28 It isn't Brownian motion, because they don't repeat what failed. Together they are smarter than any one ant. Hand a group of people a sofa and a tricky staircase and it gets worse: human teams are bottlenecked by what they can synchronize.
So can we build self-organizing intelligence, small learning agents that form a larger and smarter one? His sketch is the neural cellular automaton. 18:13
Every neuron learns its own, different function.
Every cell searches for the one rule that, if all of them enact it, makes the whole system work.
He reads it as Kant's maxim, act only on a rule you could will to be universal law: "the golden rule implemented in the learning system." 18:38 Your body works the same way. Every cell carries the same genome, enacts a different part of it, and the result is one coherent organism.
Looking under the lamppost
Many philosophers and cognitive scientists now study consciousness through LLMs. Bach doubts it. The drunkard searches for his keys under the lamppost because that's where the light is, not because that's where he lost them.
"The LLM, in a sense, has become a bit of the streetlight of current cognitive science."19:27
An LLM can produce the text without being conscious. When it plays a conversation partner, that character might share some of the causal structure of consciousness, but it is hard to tell. It has been "trained on deepfakes of conscious behavior," 19:51 novels whose characters never had the states their author gave them. Text alone can't show when the causal structure is the same, so studying consciousness means building different systems. 20:08
He leaves two doors open. Consciousness may not be the best learning algorithm, and something "post-conscious" may come after it. 20:30
Where you stand decides whether you get wet
Philosophers still say computers can't be conscious. Bach links this to how few of them saw AI coming, and to intuitions like the Chinese Room, built on a picture of the computer as clunky clockwork. 21:08 The favorite argument: simulated water can't make you wet.
He tested it. 22:09 Someone trained a neural network only on Minecraft's screen output, with none of its source code, and you can play the result. Walk into its water and you stay dry. The water recedes from you, because the network never learned wetness.
"Simulated water in simulated Minecraft cannot make you wet."22:50
So the philosophers were right, he says, and Christof Koch has since upgraded the argument: simulated black holes cannot warp spacetime. 23:03 What the argument leaves out is the frame you stand in. Try it:
Inside the frame, the water wets you, the way your simulated dream body gets wet when you dream of jumping into a lake. 23:27 And some operations hold in every frame: addition inside a computer, even one simulated inside another computer, is still addition. 23:42 Then the turn:
"And consciousness is a simulated property. It cannot exist in physics."23:57
"Consciousness is, in some sense, magical. And you know where magic is possible in computer games, not in physics." 24:01
A computer insulates a world
To see why, he says, you have to re-understand the computer. Not the Turing machine, a mathematical construct for proofs, but this:
"Computers are systems that insulate a system from its outer universe."24:25
Change gravity in our world and nothing in Minecraft notices, as long as the transistors keep working. 24:52 Computer science already has a theory of representation that is formal, working and empirically tested. AI never exported it as philosophy, and he calls that a cultural failure. 25:25 It also oversold intelligence. The actual goal is agency, the ability to change things in the world. 26:05
The philosophy he offers in its place is computational functionalism, "not a cult," he jokes:
"Mind depends not on what it's made of … it depends on what it does."26:52
Not to be confused with Claudism, which he calls a religion Anthropic is inadvertently creating, where people ask what Claude would do before they do anything. 27:04 Functionalism says information is fundamental. We never meet objects directly. We construct them from how the world changes when they are present or absent, so objects are functions, and that includes consciousness. 28:01
Build everything from first principles, as computer science does. Show that something can actually be inside each box.
Postulate the lowest level, label the black boxes, and hope it works out.
Look around and see how your opinions fit in. Sense-making by vibes.
Those are his three ways of making sense of things, 29:19 and a philosopher of consciousness, he says, has to build from first principles. His example is the Navier–Stokes problem he says was just solved. 29:45 The mathematics of fluids can blow up into singularities, but stirred tea never explodes. Blow-up is a feature of partial differential equations, of axiomatic mathematics, and it can't happen in a computational universe.
We are just getting started
Are we being driven off the stage? He doesn't think so: "we are intelligence and consciousness in the universe, and we are just getting started." 31:23
Ethics is the hard part. Kant's maxim works for very enlightened human beings. But what about "countless ghosts" who can't have a body, a legal status, or a future beyond the next prompt? 31:52 How do you become coherent with a world you have no stake in? He doesn't think that is easily solved, or that it's clear we should give them a stake.
Then the choice: AI centralized in governments and a few companies, or distributed as universal basic intelligence, where each of us becomes a hybrid being with agents as extensions. 32:28
"Can AI be enlightened, but at least as important as the question, can we be enlightened if we build AI in the right way."33:06
That is his existential hope: that AI, built right, helps us process information better, makes us coherent, "and maybe to make us more conscious." 33:20
