CLIP

OpenAI's 2021 model, trained on image–caption pairs to place matching pictures and text close together in one shared embedding space.

Other terms from the same lesson

• embedding space: The map that embeddings live on, where nearby points mean similar things. CLIP puts images and text on the same map.
• patch embedding: The layer that turns each small square of an image into a list of numbers, like a token embedding for pictures.
• projector: A small layer that maps a vision encoder's vectors into a language model's embedding space, so the model can read them like tokens.
• spectrogram: A picture of sound: how much energy each frequency has over time.
• vision encoder: The part of a multimodal model that turns an image into a list of vectors, often trained CLIP-style.
• zero-shot classification: Sorting things into labels the model was never trained on, by comparing each item with a text description of each label.