Test the model on text it has not seen Lec 2 · Aug 27 · recapped Lec 3 · Sep 1
A model can do well on sentences it has memorized. To find out whether it has learned useful patterns, we need to give it new text and see how well it predicts the words that actually appear.
The goal is to put more probability on the kinds of sentences people use. Since the total probability is fixed, giving more to those sentences leaves less for other possibilities. This is a prediction test, not a complete test of grammar: an unusual but perfectly grammatical sentence can still have low probability.
- Training set: the text used to learn the model's parameters.
- Test set: separate text kept out of training and model selection.
- Evaluation metric: a number that summarizes the model's performance on that test set.
Start with a next-word guessing game Lec 2 · Aug 27
The Shannon game asks how well we can predict what comes next. Try completing “I always order pizza with cheese and ___”. The words already present make some answers much more plausible than others.
The slides illustrate the prediction with possible continuations: mushrooms 0.1, pepperoni 0.1, anchovies 0.01, …, fried rice 0.0001, …, and 1e-100.
If the next word really is pepperoni, a model that gave it more probability made a better prediction at this step. We can repeat this check across all the words in an unseen test set.
A unigram model struggles with this game because it ignores the sentence so far. A model that uses context has a chance to make more focused predictions.
Perplexity turns those predictions into one score Lec 2 · Aug 27 · recapped Lec 3 · Sep 1
For a fixed test set, we want the observed sentences to receive a high \(P(\text{sentence})\). Multiplying the probabilities of all their words gives a score for the whole set, but that product gets smaller as the text gets longer.
Perplexity accounts for length. Take the inverse of the test set's probability, then take the root corresponding to the number of words. This gives a score on a per-word scale, where lower is better:
The factor \(1/N\) prevents the score from getting worse merely because the test text is longer. Each \(-\log P\) measures how surprising an observed word was to the model; their sum is the negative log-likelihood. The final formula averages these values over the words, then takes the exponential.
On the same test set, minimizing perplexity is equivalent to maximizing the probability assigned to that text. It does not require the model to give every possible acceptable sentence the highest score.
Imagine a sequence of 50 random digits. Our model gives each digit the same probability, \(\tfrac{1}{10}\). The probability of the full sequence is that value multiplied by itself 50 times:
\[ \mathrm{PP}(W) = \Big(\big(\tfrac{1}{10}\big)^{50}\Big)^{-1/50} = \big(\tfrac{1}{10}\big)^{-1} = 10 \]The length cancels out. A perplexity of 10 means the model is, in this sense, as uncertain as choosing uniformly among 10 options at each step. This is the intuition behind the slides' term weighted average branching factor: an effective number of choices, rather than a literal count of possible next words.
Compare models under the same conditions
The slides report an experiment using 38 million words of the Wall Street Journal for training and 1.5 million words for testing:
| Model | Unigram | Bigram | Trigram |
|---|---|---|---|
| Perplexity | 962 | 170 | 109 |
In this experiment, using more context improves prediction: the trigram model has the lowest perplexity. This result illustrates the value of context, but increasing \(n\) does not guarantee improvement in every dataset.
The slides ask what affects perplexity. Consider two broad groups of factors: the amount of training text and how closely it matches the test text; and the model's vocabulary and context size. For a fair comparison, use the same test set and vocabulary, with the same tokenization and treatment of unknown words. Changing what counts as a word changes what the score measures.
Also check whether the model helps the actual task Lec 3 · Sep 1
Predicting held-out text is quick to measure. A practical system may care about something else, such as whether a summary is useful. These lead to two ways of evaluating a language model:
| Extrinsic evaluation: test the task | Intrinsic evaluation: test the model itself |
|---|---|
| Put the language model inside the application, such as a summarization system, and measure that application's results. This gives more direct evidence of usefulness, but takes time. We also have to decide which tasks to test and how many are enough. | Measure the language model on its own, for example with perplexity. This is fast and useful for early experiments, but only gives an indirect indication of performance in an application. The evaluation text needs to represent the intended use; a mismatch can make the score misleading. |
Even when training and test text are similar, lower perplexity does not by itself establish that a system will produce better summaries or answers. Use a task evaluation to check that claim.
- Evaluate on separate, unseen text. Perplexity is inverse probability on a per-word scale, or \(\exp\) of the average negative log-likelihood.
- Lower perplexity means better prediction under the same evaluation conditions. A uniform choice among \(k\) options has perplexity \(k\).
- Perplexity is quick to calculate; evaluating the real task gives more direct evidence of practical value.
- If the model assigns zero probability to any test n-gram, the test sequence has probability zero and perplexity has no finite value: it diverges to infinity. See why zero counts cause trouble and how smoothing addresses them.