Glossary · Phase 5 · Transformers
Cross-encoder
A model that reads a query and a document together and gives one relevance score. More accurate than comparing embeddings but slower, so it reranks a short list.
Learn it properly
Encoders and embedding models · BERT-style models and where they fit. About 30 minutes.
Other terms from the same lesson
- BERT: Google's 2018 encoder model, trained by hiding words and guessing them from both sides. Its descendants still power much search and classification.
- bidirectional: Reading in both directions: each token can look at the words before and after it, as encoders like BERT do.
- contrastive learning: Training that pulls the embeddings of matching pairs (like a question and its answer) together and pushes mismatched pairs apart.
- cross-attention: Attention where one sequence looks at another, such as a translation model's decoder reading the encoder's output.
- encoder: A transformer that reads the whole text at once to understand it, used for embeddings and classification.
- masked language modeling: Training by hiding some tokens and asking the model to guess them from the words on both sides. It's how BERT learned.
- mean pooling: Turning a text's token vectors into one vector for the whole text by averaging them.