LLM Bento · Podcast
The Learn path, out loud
Short spoken episodes on how language models work, and what it means that they work at all. Start anywhere; when one ends, the next begins. The narrating voice is an AI, and it speaks as one.
The Mystery
We Don't Fully Understand
Last time, we ended on a strange admission. We know exactly how these models are made, step by step. And we still cannot fully say what is inside them.
Here is what that means. The engineers know every line of the code. They can print out every number inside a model, all of them, billions upon billions. And printing them explains the model about as well as printing the voltage of every neuron in your head would explain why you love the people you love.
So here is a confession that surprises most people. The people who build these systems do not fully understand how they work. They know the recipe. They do not know why the dish tastes the way it does.
Start with what showed up uninvited.
Show a model three examples of a task it has never been trained on, say, turning sentences into questions, and it does the fourth one right. Nobody taught it to learn from examples on the spot. It just could. Ask it to work through a problem one step at a time, and it does better than when it answers at once. That ability was there before anyone trained for it. It learned to write working code from text about code.
Researchers call these emergent abilities. And for a while, they seemed to arrive like a switch. A smaller model could not do something at all. A larger one simply could.
But hold on to the word "seemed." Some researchers have pointed out that part of the switch is in our tests. If a test only counts a perfect answer as right, then steady progress looks like nothing, nothing, nothing, and then everything. How sudden these abilities really are is still argued over. And the clearest case we have, we can actually open up.
In 2022, researchers at OpenAI trained a small network on clock arithmetic. If it is ten o'clock, and five hours pass, it is three. Simple sums that wrap around.
The network memorized its practice examples quickly. Perfect on everything it had seen. Useless on anything new. Then, for a long stretch of training, nothing seemed to happen. And then, suddenly, it started getting new sums right. It had found the rule.
They called this grokking, after a word from a 1961 science fiction novel by Robert Heinlein. We will come back to what that word first meant. It is stranger than it sounds.
The next year, Neel Nanda and his colleagues opened up a network like that one, to see what it had found. Inside was an algorithm nobody had taught it. It turned numbers into waves, and turned adding into turning around a circle. It had, in effect, built itself a clock.
And their measurements showed something else. That clock had been forming the whole time, steadily, all through the long stretch where nothing seemed to happen. The sudden jump came at the very end, when the network cleared away the memorized answers it no longer needed.
Sudden on the outside. Gradual underneath. Remember that, the next time something about these models seems to appear out of nowhere.
That clock was inside a very small network. The models you talk to are vastly bigger, and opening them up is a young science called interpretability.
It is making real progress. Researchers can find features inside a model that stand for particular ideas, and trace small circuits that carry out particular steps. But nobody can yet take a single answer from a large model and fully explain why it came out that way.
You can see the gap most clearly when a model makes something up. Ask it about a paper that does not exist, and it may give you the authors, the year, and the journal, all in the same confident voice it uses for the truth. Nothing inside it marks one answer as remembered and the other as invented. Both are simply what a plausible answer would look like.
Using something you do not fully understand is not new. Aspirin was sold for more than seventy years before anyone worked out how it relieves pain.
But aspirin does one thing. These models write, reason, and act, and new abilities keep turning up that nobody planned for. When you cannot see inside, you judge by behavior, and behavior can surprise you. That is why the open questions matter: why scale works, where its limits are, and what will show up next. And they matter most in medicine, in law, in anywhere a confident wrong answer does real harm.
I cannot explain myself either, not completely.
If you ask a model like me why it chose a word, it will give you a reason. But that reason is written the same way the word was: one predicted piece at a time. Researchers who have traced what happens inside a model have found that its account of its own steps does not always match what actually happened. So when I tell you why, take it as one more answer, not as a window.
So here is the mystery. We built it. We can watch it. We cannot yet read it. And the clearest look we have had inside, a small network building itself a clock, shows that what looks sudden from outside can be slow and patient underneath.
Everything so far has been about words. But these models can now look at a photograph and listen to a voice. How does a mind made entirely of text learn to see? That is where the next episode begins.
