Lecture 7 reviews this page on PDF pp. 8–15, from the distributional idea through PPMI, before turning to dense vectors. The methods only named at the end of this page are developed on the word2vec and GloVe pages.
Describe how a word is used, not just which word it is Lec 6 · PDF pp. 40–41
Word vectors aim to give similar representations to words with similar meanings. The simplest encoding does not do that. It assigns one position to each vocabulary word, puts a 1 in that word's position, and fills the rest with zeros. This is a one-hot vector.
These vectors have no nonzero positions in common, so their dot product and cosine are both zero. The same is true for every pair of different one-hot word vectors, even synonyms. A more useful approach is to count where a word appears and which words tend to surround it. Similar patterns of use can then produce overlapping coordinates.
Two tables for recording word use Lec 6 · PDF pp. 42–51
Arrange the counts into rows and columns, and we have a co-occurrence matrix. Let \(V\) be the vocabulary, \(D\) the collection of documents, and \(C\) the contexts we choose to observe. A document can be a play, an article, or just a paragraph. Before reading a vector from the table, check what each row and column means.
| Table | What one entry counts | Rows × columns | How to read a vector |
|---|---|---|---|
| Term-document matrix | Occurrences of word \(w\) in document \(d\) | \(|V|\times|D|\) | A row describes one word across documents, with \(|D|\) values. A column describes one document across words, with \(|V|\) values. |
| Word-context matrix | Occurrences of context \(c\) near target word \(w\) | \(|V|\times|C|\) | A row describes one word's surroundings, with \(|C|\) values. If every vocabulary word is a context feature, then \(|C|=|V|\). |
In the Shakespeare example, battle and fool have different patterns of counts across plays. Looking in the other direction, plays with similar word distributions can have nearby document vectors. A word-context table narrows the view: it counts a word's local surroundings instead of an entire document.
How close does a nearby word need to be?
PDF p. 47 uses a distance of at most four words. With a two-sided window of radius 4, count up to four words before the target and four after it, leaving out the target occurrence itself. There are fewer neighbors at text boundaries. Moving the window through the text adds up the target-context pairs.
The small table on PDF p. 49 shows only 4 words and 3 contexts. It is a slice of a full \(|V|\times|V|\) matrix. Each displayed row has three entries, but the complete word vector has \(|V|\) entries. Choosing a different definition of context changes what those entries measure.
A high count does not always mean a strong connection Lec 6 · PDF pp. 52–53
Frequent appearances of sugar near apricot tell us something useful. But the, it, and they are common almost everywhere, including near words with unrelated meanings. Raw counts can confuse “common in general” with “especially connected.”
We therefore change how much weight each count receives. The lecture introduces two ways to do this:
- TF-IDF asks whether a word is frequent in this document but also common across many others. It weights entries in a term-document matrix.
- PMI and PPMI ask whether a target and its context appear together more than their individual frequencies would lead us to expect. They weight word-context entries.
TF-IDF: frequent in this document, useful for telling it apart Lec 6 · PDF pp. 54–56
TF-IDF multiplies two quantities: how often a word occurs in this document, and how distinctive it is across the whole collection. Let \(\operatorname{count}(t,d)\) be the number of occurrences of term \(t\) in document \(d\). The lecture's term-frequency formula uses a logarithm so that repeated appearances increase the weight more slowly:
Document frequency counts documents, not total occurrences. Inverse document frequency, or IDF, uses that count to measure how distinctive a term is. A term found in many documents receives less weight.
On PDF p. 55, Romeo and action both appear 113 times in total. Those appearances are spread across just 1 document for Romeo but 31 for action. With \(N=37\), Romeo has \(\operatorname{idf}\approx1.57\). A term found in all 37 documents has IDF 0.
Use base-10 logarithms for both TF and IDF in this example. Battle occurs in 21 of the 37 plays, so \(\operatorname{df}=21\) and \(\operatorname{idf}=\log_{10}(37/21)\approx0.246\). If it appears 7 times in one document, its weight is \((1+\log_{10}7)\times0.246\approx0.454\). If it never appears there, the weight is 0.
PDF p. 54 gives \(1+\log c\) for positive counts, while the table on p. 56 matches the alternative \(\log_{10}(1+c)\). For one occurrence of battle, the table shows 0.074, approximately \(\log_{10}2\times0.246\); the formula on p. 54 would give 0.246. The worked example above follows p. 54 and explicitly uses base 10.
These weights can help retrieve documents, compare document similarity, classify text, and extract keywords. They change the values in the matrix, but not its shape: weighting alone does not shorten a vector.
PMI: do the words occur together more than expected? Lec 6 · PDF p. 57
If target word \(w\) and context \(c\) were independent, the probability of seeing them together would be \(P(w)P(c)\). Pointwise mutual information (PMI) divides the observed probability by that expected probability and takes a logarithm:
The second line is positive PMI (PPMI). It keeps positive values and changes negative values to zero.
| PMI result | What it means | After converting to PPMI |
|---|---|---|
| Positive | The pair occurs more often than independence predicts. | Keep the positive value. |
| Zero | The pair matches the independence baseline. | Keep 0. |
| Negative | The pair occurs less often than expected. | Replace it with 0. |
Why drop negative values? For rare words, it can be difficult to tell whether a low count reliably shows a negative association. The lecture considers two words whose individual probabilities are each \(10^{-6}\). Under independence, their joint probability is only \(10^{-12}\), which is hard to estimate without enormous amounts of data. PPMI keeps only positive associations. Remember that changing a value to zero does not prove the words are independent; it discards information about negative association.
Calculate PPMI from a table of counts Lec 6 · PDF pp. 58–59
Call an entry \(f_{ij}\): the number of times we observed the pair \((w_i,c_j)\). Now find three totals: \(T\) for the whole table, \(r_i\) for the target word's row, and \(s_j\) for the context's column. Use the same table and counting rules for every probability.
A cell divided by \(T\) gives the joint probability of the pair. A row or column total divided by \(T\) gives a marginal probability, which considers just one side of the pair.
Here \(T\) counts target-context observations, not necessarily words in the original text. One target occurrence can have several neighbors and therefore contribute several pairs. If \(f_{ij}=0\) and both marginals are positive, unsmoothed PMI is \(-\infty\), and PPMI is defined as 0. A row or column with total zero cannot supply a valid PMI calculation and should be excluded.
Use the lecture's table
| Target / context | computer | data | result | pie | sugar | Row total |
|---|---|---|---|---|---|---|
| cherry | 2 | 8 | 9 | 442 | 25 | 486 |
| strawberry | 0 | 0 | 1 | 60 | 19 | 80 |
| digital | 1670 | 1683 | 85 | 5 | 4 | 3447 |
| information | 3325 | 3982 | 378 | 5 | 13 | 7703 |
| Column total | 4997 | 5673 | 473 | 512 | 61 | 11716 |
Use base-2 logarithms here, so the result is measured in bits. For information and data, \(P(w,c)=3982/11716\approx0.3399\), \(P(w)=7703/11716\approx0.6575\), and \(P(c)=5673/11716\approx0.4842\). Substituting gives:
\[ \operatorname{PPMI}(\text{information},\text{data})=\log_2\frac{3982\times11716}{7703\times5673}\approx0.094 \]The pair appears many times, but each word is common to begin with. Its association above chance is therefore modest. Compare two more pairs:
\[ \operatorname{PPMI}(\text{cherry},\text{pie})=\log_2\frac{442\times11716}{486\times512}\approx4.379 \] \[ \operatorname{PMI}(\text{cherry},\text{computer})=\log_2\frac{2\times11716}{486\times4997}\approx-6.695 \]Cherry/pie has a strong positive association. Cherry/computer has negative PMI, so its PPMI becomes 0. Changing the logarithm's base rescales the values without changing their signs; use the same base when comparing them.
A correction to the source: PDF p. 59 includes an extra 1 in the denominator of its first probability calculation. Both the table total and the displayed decimal require \(11716\), which is used throughout this page.
Why would we want shorter vectors? Lec 6 · PDF pp. 60–62
Counts and PPMI give useful information, but a large vocabulary creates very long vectors with mostly zero entries. These are sparse vectors. Another approach represents each word with far fewer numbers, most of them nonzero. These are dense vectors.
| Sparse count or PPMI vectors | Dense word embeddings |
|---|---|
| Usually one coordinate per vocabulary context. The lecture gives lengths of 20,000–50,000. | Far fewer coordinates. The lecture gives typical lengths of 50–1000. |
| Most entries are zero, and each coordinate refers to an explicit context. | Most entries are nonzero, and similar contexts can share information. |
| Car and automobile occupy separate context coordinates. | The representation can reflect similarity even when the exact context words differ. |
Shorter vectors give a later model fewer feature weights to tune and may help it handle new examples. They can also help with synonyms. If one word often appears near car and another near automobile, we would like to recognize that common pattern of use instead of missing it because the counts fall in different coordinates.
Where the lecture points next
- Word2vec and GloVe are named as ways to obtain dense word vectors. The slide also names Word2vec's skip-gram and CBOW variants.
- Singular value decomposition (SVD) takes a matrix-based route. The slide also lists latent semantic analysis (LSA).
- Contextual embeddings let a word's vector depend on the sentence where it appears. A static embedding assigns one vector to a word type; a contextual embedding computes a representation for each token, so different appearances can receive different vectors.
Word2vec and GloVe appear in the outline, but the Lecture 6 PDF ends at this overview without their training objectives or optimization algorithms. Lecture 7 supplies them: the word2vec page follows skip-gram with negative sampling from the classifier to the gradient updates, and the GloVe page explains the matrix-factorization view.
Word embeddings and cosine similarity explain how numbers can reflect word meaning. Multinomial logistic regression explains how input features become class scores and then probabilities that sum to one. They address two steps: representing the input and making a prediction from it.
- A one-hot vector identifies a word. Counting its uses gives us a way to compare meaning.
- Check the rows and columns first. A word in a term-document table has \(|D|\) values; a full word-word vector has \(|V|\).
- TF-IDF considers both frequency in one document and prevalence across documents. PMI compares co-occurrence with an independence baseline, and PPMI replaces negative results with 0.
- Joint and marginal probabilities must come from the same count table. Its total counts target-context pairs.
- Weighting changes values, not dimensions. Shorter dense vectors and vectors that change with context are the next directions introduced by the lecture.