Train a real n-gram language model on nursery rhymes, news, or your own text — then watch it predict the next word, just like autocomplete on your phone!
A corpus is a collection of text the model learns from. Nursery rhymes give childlike language; news gives formal language.
An n-gram is a sequence of N words. Bigrams = 2 words, trigrams = 3. "the cat sat" has the bigram "the cat" and "cat sat".
After seeing "the cat", what comes next? Count how often each word follows "the cat" in the corpus to get probabilities.
Given the last N-1 words, the model picks the next word by its probability. High prob = common continuation.
| Context (prefix) | Next Word | Count | Probability |
|---|
You trained an n-gram model and explored how corpus choice shapes language style!
The text a language model learns from. The corpus completely determines what the model knows and how it "talks".
A sequence of N consecutive words. Bigram = 2 words, trigram = 3. Used to capture context for prediction.
Given context words, what's the probability of each next word? Counted from how often it appears in the corpus.
A measure of how "surprised" a language model is by new text. Lower perplexity = better model!
How many previous words the model looks at. N=2 uses 1 word context; GPT-4 uses thousands of tokens!
What if a word combination never appeared in training? Smoothing gives a small probability to unseen n-grams.