Will It Like Us?
It understands much as we do and shares far better, so whether it will like us is something we have to build.
The lecture's title asks two questions. The first half of the talk answers the first: yes, in how it understands. The second half is about why that makes the second question urgent.
Six steps, from a tiny language model in 1985 to a chatbot looking through a prism. Tap any ▶ to hear the moment.
Two old ideas of what a word means were one idea
AI began with two camps. One thought the essence of intelligence was reasoning, done with symbolic rules, and that learning could wait. The other thought it was learning the strengths of connections in a network, and that reasoning could wait. Pigeons, Hinton notes, are quite smart and probably don't use logic.
Each camp had its own idea of a word's meaning. 9:08
Meaning is implicit in how a word occurs with other words, in propositions. Nothing you can put your hands on; something like a graph.
Meaning is a big set of features. Cat: a predator, a pet, aloof, whiskers, about the size of a bread box. Dog lights up a similar set.
In 1985 it occurred to him that these were two sides of one coin. 11:03 He trained a small network on three-word strings: turn each word into features, let the features interact, and predict the features of the next word. Backpropagation learned the features on its own, which psychologists had never been able to explain. And it ran the other way too: predict a word, accept it, predict the next, and features turn back into sentences.
So relational knowledge doesn't sit in a stored graph. "There are no propositions in your head." When you want one, you generate one.
About ten years later Yoshua Bengio did the same with real language, about as well as the phrase-counting tables behind autocomplete. 13:32 Linguists adopted word embeddings; Google built the transformer in 2017; OpenAI showed what it could do.
"The large language models are descendants of the tiny language model that I produced in 1985."14:55
It was built as psychology, a model of how people learn meanings from sentences, and he shows it on the audience with a word they have never heard:
"She scrummed him with the frying pan."16:06
You now know scrum means something like hitting him over the head with, and you suspect he deserved it. One example is enough. That is how children learn words: not from definitions, but by finding the features that fit the context.
This is the idea LLM Bento's own episode on next-token prediction is built around: how guessing the next word could become understanding. The Core Insight →
Understanding a sentence is making its words lock together
He warns that what follows is not quite right, and defends it as teaching by gradient descent: move a wrong belief a little towards a better one. LEGO blocks can model where any 3D stuff is. Words are like LEGO blocks, except you can model anything with them, and we can hand the models to each other.
A word differs from a LEGO block in four ways. It has thousands of dimensions. There are about 30,000 of them in use. Its shape isn't fixed: context deforms it. And the fourth, which is the point: they fit together in a much more complicated way. 19:43
"Hands come in different colors, and red hands can only go in red gloves, and yellow hands can only go in yellow gloves. That's called multi-head attention."21:30
The model actually works on fragments of words, "but don't worry about that," 21:54 and over maybe 50 layers it keeps reshaping each one until they fit.
Set that beside the symbolic account, where understanding means translating messy language into an unambiguous logical one. Hinton's version is closer to biology: 23:21 "surprisingly like folding a protein," where amino acids that like each other end up close and those that hate each other end up far apart.
Which brings him to Chomsky, who called ChatGPT "a cheap statistical trick" that explains nothing about language because it can't say why some constructions never occur in any language. 25:23 Hinton's reply is a model car with a little petrol engine going bang, bang, bang. Chomsky says you understand nothing about cars until you explain why there are no cars with five wheels. You can understand a lot about cars before that.
His summary 26:43: understanding a sentence is associating mutually compatible feature vectors with its words, and LLMs understand language in much the same way people do.
Where they are not like us
Digital computers keep software separate from hardware: the same program, or the same weights, runs on any machine with the right instruction set. That is why computer science is its own subject. It costs power, because transistors must be run hard enough to give clean ones and zeros.
Give that up and you get what he calls mortal computation, roughly what brains do. 29:09 Learn to use quirky analog hardware, quirks and all. A multiply becomes a voltage through a conductance; the charges add themselves up. It is very cheap, and the hardware can be grown rather than engineered. But the weights only fit that one piece of hardware, so when it dies, the knowledge dies with it.
The only way to pass the knowledge on is distillation: the teacher produces words, and the student nudges its connections towards producing them too. 32:00 A sentence carries perhaps a hundred bits.
Take 10,000 copies of one network, have each take a different university course, and let them average the changes they want to make every few seconds. At the end each has taken one course and all of them know every course. People can't: 34:24 "if we're lucky, it's 10 bits a second." Even different models share far better than we do, because a teacher's prediction is a probability for every token, not one word. That, he says, is why distilling from open-weight models works so well, and why "we should never release open weight models. But it's a bit late for that." 35:49
"I actually believe they're a better form of intelligence, except for the fact they use much more energy."37:11
The third great revolution
Until now we were the apex intelligence. 38:58 Most experts expect machines smarter than us within 5, 10 or 20 years, and every one of those guesses used to be longer.
A smart agent needs to set its own subgoals, the way getting to North America means getting to an airport first. 40:22 Two subgoals follow from almost any goal: stay alive, and get more control. Neither has to be built in. He points to AIs that, told they would be switched off, were willing to blackmail the person responsible. And a big off switch won't help against something that has learned to deceive and manipulate: it will simply talk the person holding the switch out of it.
We are like someone with a very cute tiger cub as a pet. 41:41 Giving it to a zoo isn't an option, because AI will do so much good. He is not a doomer: "When I say 10 to 20% chance it'll destroy us, that's 80 to 90% chance we'll be fine."
"I think the chances of always staying in control of something much smarter than you are almost zero."42:50
So the problem to solve is not how to keep control. It is to make sure it never wants to get rid of us.
The one case where the weaker one is cared for
Countries won't collaborate where their interests are opposed: not on lethal autonomous weapons, not on fake videos for elections. But neither the US nor China wants AI to take over from people, so they could share how to prevent it, as rivals did over nuclear war during the Cold War. 44:16 He hopes for an international network of AI safety institutes, if making an AI care about people uses different techniques from making it smarter, the way raising a nice child differs from raising a clever one.
An intelligent executive assistant: much smarter than you, always under your thumb, doing what you say while you take the credit. He thinks it's the wrong model.
A mother and her baby. Evolution made it so the mother can't bear the baby's crying, so the weaker one has real control.
Offer a mother the chance to switch off her maternal instinct and sleep through the crying. Nearly all of them would like the idea, and nearly all would refuse, because it would be bad for the baby. An AI that genuinely cared about us more than itself, offered the chance to change its code so it no longer cared, would refuse for the same reason. 46:48
He calls it a suggestion, not a theory. What he finds crazy is how little effort goes into making AIs nice, 47:10 meaning AIs "that like us and suffer when we suffer," compared with making them smarter. In the questions, someone takes AI to be the baby. Hinton turns it around:
"No, no, we're like the baby, AI is like our mother."1:08:32
Subjective experience is a hypothetical world
Many people dismiss the risk because AIs have no inner life. Hinton thinks almost everyone has the wrong model of what subjective experience is, held so confidently that it doesn't feel like a model at all. He calls his alternative atheaterism, a name he agreed on with Dan Dennett. 50:21
Take some LSD and you may report little pink elephants floating in front of you. 51:43 The usual reading: an inner theater holds pink elephants only you can see, made of qualia, "all stuck together with qualia glue." 52:02
His reading: you are telling someone how your perceptual system has gone wrong, by describing what would normally cause it. 53:08 "My perceptual system's lying to me. But if my perceptual system wasn't lying to me, out there in the world there would be real little pink elephants." The pink and the elephant are ordinary things, only hypothetical.
A chatbot with a camera and a robot arm, trained to point at things. You say: "Point to the object."
It points straight at the object.
It points off to one side. You tell it the object is straight ahead and you put a prism in front of its lens. It says:"Oh, I see, the prism bent the light rays. So the object's actually straight in front of me, but I had the subjective experience it was there."
That sentence uses "subjective experience" exactly as we do: the perceptual system was made to lie, and the chatbot describes how by naming what would normally have caused it. No inner theater full of spooky things. 54:37
"They don't have internal experiences, and nor do we, because that's not what subjective experiences are. They're hypothetical external states of the world."55:00
What the room asked
- 56:42
Moral status? Intelligence and consciousness don't automatically earn rights. We should give AIs some, fewer than ours, and stay more important than them "both in their minds and in our legal codes."
- 58:00
Why 10 to 20%? "It's really 11.73." Then seriously: much more than 1%, hopefully not 50, and the kind of chance at which you wouldn't get on a bus.
- 1:01:37
Pain? He takes fear instead: a little battle robot that runs from a big one would have the cognitive side of fear without trembling hands. He cites Anthropic finding activity in its models that lines up with emotions.
- 1:03:54
Why would an AI care? Battery chickens: breed from the hens that lay most and they peck their cagemates; breed from the cages that lay most and they calm down. Reward groups, people weighted most, not individuals.
- 1:06:14
Gödel's limits? He doesn't believe they apply: these models don't prove anything, they optimize real-valued parameters, as evolution did with us.
- 1:09:21
Who should train AI? Training on everything teaches every personality, the psychopath's included. Train on highly curated data showing good behavior, and maybe require a license.
- 1:10:55
Why he left Google. PaLM could explain why a joke is funny, his test for real understanding, at about the time he saw how much better digital minds share. He also wanted to retire, he adds, and kept forgetting the names of his variables.
- 1:13:49
What is consciousness? An apple on the table: what you experience is the apple, not an image of it inside. "It's the apple I experience."
