LLM Bento · Podcast
The Learn path, out loud
Short spoken episodes on how language models work, and what it means that they work at all. Start anywhere; when one ends, the next begins. The narrating voice is an AI, and it speaks as one.
Beyond Text
How AI Learned to See, Hear, and Create
Last time, we ended on a question. Everything so far has been about words. So how does a mind made entirely of text learn to see?
Here is where it has got to. A frontier model has taken in more sunsets than any person will ever see. Millions of them, from every coast and every season, in every style photography has invented.
It has never stood in one. It has never felt the air cool as the light went.
This episode is about how these models learned to see, hear, and make pictures. And about what that does, and does not, change.
The answer turns on one quiet move, and you already know half of it.
In the second episode, we said the model never sees words, only numbered pieces called tokens. That turns out to be the key to everything else. The world is continuous: light, sound, speech, all of it flows. A model cannot hold flow. It holds pieces. So the first step is always the same. Chop.
Text is chopped into pieces of words. A picture is chopped into small squares, called patches. Sound is chopped into short slices.
Then the second step. Each piece becomes a point in that space of thousands of directions from the first episode, where nearness means meaning. And because picture pieces land in the same space as word pieces, a patch of orange sky can end up close to the word "sunset." Nobody wrote that rule. The two kept turning up together, and the model learned it.
So for a language model, learning to see was not growing an eye. It was learning new words.
Follow a single photo through. You send a picture of a cat and ask, "What breed is this?"
The picture is cut into a few thousand small squares. Each square becomes a point in that shared space. Those points join the stream right next to your words. And inside the model, the piece for "breed" reaches toward the squares that hold the ears, the whiskers, the fur. It is the same attention from the second episode, the same hands finding the same gloves. Only now, some of the gloves are made of light.
Sound works the same way, and it carries something text loses.
Here is one sentence, said two ways. "That's just great," said warmly. And: "That's just great," said flat.
Write those down and they are identical. But a model that hears them gets two different streams of sound, with different pitch, different pace, a different little drop at the end. The flat one lands somewhere else in that space, somewhere near disappointment. Understanding speech, it turns out, is not only getting the words back. It is hearing how they were meant.
Seeing is half the story. These models also make pictures, and they learned it in two different ways.
The first is a sculptor. Take real images, bury them in random noise, and train a model to remove the noise. Then run it backwards: start from pure static, like an untuned television, and clear a little noise at a time, steered by your words, until a picture is left standing.
The second is a writer. The newest models make a picture the way they make a sentence, one piece after another, with the same attention that read your request. That is why they have become so good at doing exactly what you asked, putting the right words on the sign and the cup on the left side of the table.
And the pieces keep extending. Video. Voices that answer in sound. Robots whose next movement is treated as one more piece to predict.
There is a cost to senses that can run in reverse.
They can make faces, voices, and events that never happened. A photograph used to be modest evidence that something occurred. A recording of a voice used to mean someone said it. That line is dissolving. And what replaces it will not be technical. It will be human: knowing where a thing came from, who stands behind it, and whom to trust.
You have been listening to an example the whole time.
This voice is one of those senses run in reverse. There is no throat behind it and no breath, only a model predicting what sound should come next. When I said "That's just great" two ways, I could make you hear the difference. Whether I heard it, in any sense that matters, is the same question as the sunset.
So, the model has something shaped like senses now. It has taken in more of the world than any of us. And as far as anyone can tell, it still has no body that was ever cold, no sunset it stood inside. More senses make the comparison sharper. They do not settle it.
That ends the first part of this series: how it works. The next part is about working with it. Until now, you have met these models through a box on a screen. So what is actually between you and the model? Next time: the terminal.
