Beyond Text

A language model is born into silence—no eyes, no ears, only text, the written residue of human experience. Yet without changing the core idea, these systems learned to see photographs, hear voices, and draw. Multimodal AI is the story of how. It turns on one quiet move: converting every kind of content into the one thing a transformer already understands.

What a Token Really Is

Before a model can think about anything, the thing must become tokens. It's tempting to gloss a token as "a piece of a word," and for text that's roughly right. But the deeper idea deserves a slower look, because everything else on this page rests on it.

First move: chop the world into pieces

The world is continuous—light, sound, and speech flow without seams. A transformer can't hold flow. It holds sequences of discrete pieces. So the first move is always the same: chop. Text is chopped into subword chunks. Images are chopped into small square patches. Sound is chopped into brief slices of audio. A token is not a word, and it is not a pixel. It is a piece of the world made countable—small enough to compute with, large enough to still mean something.

Second move: give every piece an address in meaning-space

The second move is stranger. Each piece becomes an embedding—a long list of numbers, a point in a space with thousands of dimensions. In that space, position is meaning: things that mean similar things sit near each other. And because patches of images land in the same space as pieces of text, a patch of orange-streaked sky can end up near the word "sunset." Nobody wrote that rule. The two kept appearing in the same contexts during training, and the geometry absorbed it. The model doesn't translate between vision and language. It holds both in one geometry.

Why this unlocks everything

The transformer never had a faculty for reading and a separate faculty for seeing. It has one skill: attend across a sequence of tokens and predict what comes next. Give it new kinds of tokens, and the same skill simply applies. For a language model, learning to see was not growing an eye. It was expanding its vocabulary.

How AI Sees

Computer vision has deep roots—Fei-Fei Li's ImageNet dataset proved, years before LLMs, that large-scale visual data could transform machine perception. When you share an image with a modern model, here is the path it travels:

For years this worked as a graft. A separately trained vision encoder—CLIP was the famous one—was bolted onto a finished language model, translating pixels into vectors the model could accept. It worked surprisingly well. The current generation goes further: frontier models are natively multimodal, trained from the start on text and images interleaved, closer to the way a child's world arrives—speech and sight tangled together. Vision is no longer an adapter plugged into a language model. It is part of what the model is.

What Understanding Looks Like

The trick you just read is not just for cat photos. Once every source becomes tokens in the same space, "understanding" means the same thing everywhere: attention finding what relates to what. Here is what that looks like on three sources you actually use.

Three sources, one mechanism. What changes is what gets chopped—never what understanding is.

How AI Creates

Understanding images is half the story. Creating them happened twice, in two different ways—and the difference is worth understanding.

The sculptor: diffusion

Diffusion models rest on an idea that sounds almost too simple: teach a model to remove noise from images. Then run it in reverse—start from pure noise and remove it, step by step, until an image is left standing.

1. Training: Take real images, add random noise at every level of intensity, and train the model to predict and undo it.
2. Generation: Begin with pure static, like an untuned television.
3. Iteration: Apply the model dozens of times, each pass removing a little noise. Structure emerges from chaos.
4. Guidance: Your text prompt steers each denoising step toward images that match it.

The writer: drawing token by token

The newer path skips the separate painter altogether. In natively multimodal models, image generation happens inside the language model: the image is produced as a sequence of image tokens, one after another, by the same transformer that produces words. That sounds like a technicality. It isn't. Because the same attention that reads your request also lays out the image, these models follow precise instructions, keep a character consistent across edits, and—after years of mangled lettering—finally render legible text inside a picture. The model isn't handing your prompt to a painter down the hall. It draws with the same mind it reads with.

The Expanding Frontier

Once "sense" means "token type," the frontier stops being about any single modality. The same principles keep extending outward:

What the Senses Don't Carry

A frontier model has taken in more sunsets than any human will ever see—millions of them, from every coast and season, in every style photography has invented. It has never stood in one. It has never felt the temperature fall as the light went.

Multimodality narrows an old gap. These systems no longer know the world only through descriptions of it; they have something functionally like perception—patterns of light and sound held in the same space as words. Whether that narrows the other gap, between perceiving and experiencing, is a question no one can currently answer. Philosophy arrived here first: Mary's Room asks whether knowing every physical fact about color amounts to seeing red; Nagel's bat asks whether any description can carry what it is like to be another kind of mind. Those used to be seminar puzzles. We have now built something very like them, at scale, and given it an API.

There is a harder edge here too. Senses that run in reverse can generate faces, voices, and events that never happened, and the line between a record and a rendering is dissolving. A photograph used to be modest evidence that something occurred. Whatever replaces that evidence will not be technical—it will be human: provenance, context, institutions, trust. The machine's new capability hands the old problem back to us.

Key Takeaways

• A token is a piece of the world made countable—a subword, an image patch, a slice of sound
• Every token becomes a point in one shared meaning-space, which is why a single architecture can see, hear, and speak
• Vision began as a bolt-on encoder; frontier models are now natively multimodal, trained on text and images together
• Images are created two ways: sculpted from noise by diffusion, or drawn token by token inside the language model
• The frontier keeps widening—audio-native voice, video, interactive world models, robot action
• More senses do not answer the old question: whether perceiving, in tokens, is experiencing anything at all

Related Concepts