Phase 5 · Transformers · Scale & inference
Multimodal transformers
How models see images and hear audio.
Open this lesson Create a free account
Why it matters
Today's assistants read screenshots, charts, PDFs, product photos and handwriting, and some listen and speak. Underneath, almost all of it is the transformer you just built: images and audio are cut into pieces, turned into vectors, and processed as tokens. Knowing how explains what these models are good at, what images cost, and why they still misread small text or miscount objects.
What you'll be able to do
- Turn an image into a sequence of tokens with patches, as a vision transformer does
- Explain contrastive training (CLIP) and use it for zero-shot classification
- Describe how vision-language models feed image tokens into a language model
- Estimate what images cost in tokens, and where multimodal models still fail