Chunking

Cutting long documents into smaller pieces so the most relevant pieces can be found and sent to a model.

Other terms from the same lesson

• metadata: Facts stored alongside a piece of content, such as which document it came from, its date or its section, used to filter and cite.
• overlap: In chunking, the words repeated at the end of one chunk and the start of the next, so an idea split across a boundary still appears whole somewhere.