Glossary · Phase 5 · Transformers
Vision encoder
The part of a multimodal model that turns an image into a list of vectors, often trained CLIP-style.
Learn it properly
Multimodal transformers · How models see images and hear audio. About 35 minutes.
Other terms from the same lesson
- CLIP: OpenAI's 2021 model, trained on image–caption pairs to place matching pictures and text close together in one shared embedding space.
- embedding space: The map that embeddings live on, where nearby points mean similar things. CLIP puts images and text on the same map.
- patch embedding: The layer that turns each small square of an image into a list of numbers, like a token embedding for pictures.
- projector: A small layer that maps a vision encoder's vectors into a language model's embedding space, so the model can read them like tokens.
- spectrogram: A picture of sound: how much energy each frequency has over time.
- zero-shot classification: Sorting things into labels the model was never trained on, by comparing each item with a text description of each label.