Beyond Text
How AI Sees and Creates Images
First came text. Now AI can see, hear, and create images. Multimodal AI represents the next frontier—systems that understand and generate multiple types of content. The same core ideas that power language models are expanding into entirely new domains.
Everything Becomes Tokens
The key insight: the same architecture that processes language can process other things—if you can convert them into tokens.
From Content to Tokens
- Text: "Hello world" → tokens based on subwords
- Images: Image → patches → vision encoder → tokens
- Audio: Sound waves → spectrograms → audio tokens
- Video: Frames → image tokens + temporal tokens
How AI Sees: Vision Models
When you share an image with Claude or GPT-4V, here's what happens:
1. Divide into Patches
The image is split into small squares (typically 14x14 or 16x16 pixels each). A 1024x1024 image might become a grid of 64x64 patches.
2. Encode to Vectors
A vision encoder (usually a pre-trained model like CLIP) converts each patch into a numerical vector that captures its visual meaning.
3. Combine with Text
These visual tokens are fed to the language model alongside text tokens. The model processes them together using the same attention mechanism.
4. Generate Response
The model produces text that references what it "saw" in the image, answering questions or describing content.
What Vision Models Can Do
- • Describe image contents in natural language
- • Answer questions about what's in an image
- • Read and extract text (OCR)
- • Analyze charts, diagrams, and documents
- • Understand spatial relationships
- • Identify objects, people, scenes
Creating Images: Diffusion Models
Image generation uses a different but equally elegant approach: diffusion models.
The core idea is surprisingly simple: train a model to remove noise from images. Then use it in reverse—starting from pure noise and gradually removing it until an image appears.
How Diffusion Works
- Training: Take real images, add random noise at various levels, train the model to predict and remove that noise.
- Generation: Begin with pure random static—like TV snow.
- Iteration: Apply the model repeatedly, each step removing a bit of noise. Structure emerges from chaos.
- Guidance: Text descriptions steer the denoising process toward images matching the prompt.
The Expanding Frontier
The same principles are extending to other modalities:
🎤 Speech
Models like Whisper transcribe speech to text with near-human accuracy. Voice synthesis creates natural-sounding speech from text.
🎵 Music
AI can now generate music from text descriptions—complete songs with structure, melody, and lyrics.
🎬 Video
Early video generation models can create short clips from text. Longer, coherent video remains a frontier challenge.
🤖 Robotics
Multimodal models are being used to give robots the ability to understand instructions and their environment.
Toward Unified Intelligence
The trend is clear: AI is becoming increasingly multimodal. Future systems will likely process text, images, audio, and video seamlessly—much as humans do.
Different modalities carry different information:
- Text is precise but can miss visual details
- Images capture appearance but not temporal change
- Audio conveys tone and emotion text often misses
- Video shows how things change over time
A truly intelligent system needs all of these—and needs to understand how they relate to each other.
What This Means
Multimodal AI has profound implications:
New Creative Tools
Artists, designers, and creators have powerful new tools. The boundary between imagination and creation is thinner than ever.
New Challenges
Deepfakes, synthetic media, and the difficulty of distinguishing real from generated content create serious societal challenges.
New Questions
If AI can create any image, what does "photography" mean? If it can write and illustrate, what is human creativity? These questions are becoming urgent.
Key Takeaways
- Multimodal AI converts all content types into tokens for unified processing
- Vision models use encoders to convert images into tokens LLMs can understand
- Diffusion models generate images by learning to remove noise
- Audio, music, and video are following similar paths
- The future points toward unified, multimodal AI systems
Related Concepts
Journey Complete
You've explored the fundamentals of how LLMs work—from the wonder of machine language to multimodal AI. This is just the beginning. The field evolves rapidly, and there's always more to learn.