Neural language models

Why it matters

A bigram table can't remember more than one character, and a table for longer contexts explodes. In 2003, Bengio and colleagues replaced the table with a neural network over learned embeddings, and every modern language model descends from that idea. This lesson builds one and shows exactly what it fixes, and what it still can't do, which is the opening for attention.

What you'll be able to do

• Explain why counting tables can't scale to longer contexts
• Build a neural language model: embed the previous characters, then predict the next
• Show how more context lowers the loss and improves generated text
• See what the embeddings learn, and the limits that point toward attention

Before this lesson

• A bigram model
• Embedding layers

Next in this phase: Attention: the intuition