Glossary & Formula Sheet

Special markers for the start and end of a sentence. The first word gets a preceding context, and the model has a way to signal that the sentence is finished.

The non-linear function applied to a unit’s weighted sum. Sigmoid, tanh, and ReLU are the standard choices. Without one, stacked layers collapse into a single linear model.

Add the same small count to every possible outcome before turning counts into probabilities. This reserves probability for unseen combinations. Add-one is usually too crude for language models, though this family of methods is useful in models such as Naïve Bayes.

Two ways biased embeddings cause damage. Allocational harm is an unfair distribution of resources or opportunities, such as hiring searches skewed by a “programmer is to man” association. Representational harm is a demeaning association carried by the representation itself, whether or not any decision is made with it.

Try to carry a relation from one pair of words to another. If a is to b as c is to d, look for a word vector near b − a + c. This works for some relations, not every analogy.

Words such as hot and cold have opposite meanings in a particular respect. They may still appear in very similar sentences, so their word vectors can be close.

Represent a document by counting how often each vocabulary word appears. The result keeps word counts but loses word order and the surrounding context.

A number added to the weighted feature total. It can shift the prediction even when every input feature is zero, allowing the decision boundary to move.

Find the probability of a sequence by working from left to right: take the first word’s probability, then multiply by each next word’s probability given everything before it.

A closed-vocabulary setup assumes every test word is already in the vocabulary. An open-vocabulary setup allows unfamiliar words and needs a way to handle them, such as an unknown-word marker.

Choose parameters that give the known correct labels high probability for their inputs. Taking the negative logarithm gives the cross-entropy loss used to train a classifier.

A word can carry an emotional tone as well as a literal meaning. VAD describes pleasantness (valence), emotional intensity (arousal), and a sense of control (dominance).

For a convex loss, a local minimum, if it exists, is also a global minimum. There may be more than one minimizer, and gradient descent still needs suitable step sizes and conditions. Logistic regression has a convex loss; neural-network losses are generally not convex.

Compare the directions of two nonzero vectors, after accounting for their lengths. The score ranges from −1 to 1; a larger score means more similar directions. It is undefined for a zero vector. Lecture 7 adds why it beats the raw dot product: long vectors, which belong to frequent words, would otherwise win.

For an example with one correct class, take the negative log of the probability assigned to that class. A confident correct prediction has a small loss; a confident wrong prediction has a large one.

Learn which label to predict from an input. The model estimates the probability of a label given the input; logistic regression is one example.

Words that appear in similar surroundings often have similar meanings. This lets us learn something about a word by studying where it is used.

An intrinsic evaluation checks a model on its own, using a score such as perplexity. An extrinsic evaluation checks whether it helps with a real task, such as speech recognition. The first is usually easier; the second is closer to the intended use.

Layers of units where information flows from the inputs, through one or more hidden layers, to the outputs. Each hidden unit is a weighted sum followed by an activation function. The output layer is multinomial logistic regression on the hidden vector.

A feature is a measurable clue in the input, such as the number of positive words in a review. Its weight tells the model how strongly, and in which direction, that clue should affect a class score.

A sentence whose opening encourages a reading that later words force us to revise. In “The old man the boat,” man is a verb. These examples show why a short local context can be misleading.

Global Vectors: a static embedding model that starts from the whole word-word co-occurrence matrix and fits target and context vectors whose dot products approximate its logarithm, with a fixed small number of dimensions. It builds on PPMI intuitions and matrix factorization.

A collection of derivatives, one for each parameter. It tells us how the loss changes as the parameters move; its direction gives the steepest local increase, so learning usually steps the other way.

Set aside examples that are not used to fit the weights. Use them to choose settings such as smoothing strength or regularization, while keeping a separate test set for the final evaluation.

A setting chosen outside the weight-learning process, such as the learning rate, batch size, or strength of smoothing and regularization. Use validation data to compare choices.

Mix predictions from models using different amounts of context, such as a unigram, bigram, and trigram model. Nonnegative weights that sum to 1 keep the result a probability distribution; the weights may depend on the context.

Add a cost for large weights. L2 charges for squared weights and tends to shrink them; L1 charges for absolute values and can make some weights exactly zero, effectively selecting features.

A model that assigns probabilities to sequences of words, or predicts the probabilities of the next word from the words already seen.

The number that scales each parameter update. A larger rate takes bigger steps; a rate that is too large can overshoot rather than improve the model.

A lemma is the dictionary form shared by variants such as break and broke. A sense is one meaning of a word. Polysemy means a word has several, usually related, meanings.

Store log probabilities and add them instead of multiplying many tiny probabilities. This makes the arithmetic more stable and helps avoid numbers becoming too small for the computer to represent.

Approximate the next word’s probability using only the last few words, rather than the whole sentence so far. This makes the amount of context manageable.

Choose the model parameters that make the observed data most likely. For a simple count-based word model, this means dividing the relevant count by the total count of possible outcomes.

Use a small group of examples for one update. Compute their gradients, usually average them, and adjust the parameters once for the whole group.

Extend logistic regression to several mutually exclusive labels. Calculate one score for each class, then use softmax to turn the scores into probabilities that sum to 1.

For each real (target, context) pair, draw k words at random to serve as “not a neighbor” examples. Words are sampled by their unigram frequency raised to a power, commonly 0.75, which gives rare words a slightly better chance.

A sequence of n neighboring tokens. An n-gram language model predicts a token from the previous n−1 tokens: unigram uses none, bigram uses one, and trigram uses two.

A vector with one 1 and zeros everywhere else. It can mark a class or a vocabulary word. Different word identities have no shared nonzero coordinate, so the representation alone does not express meaning similarity.

An out-of-vocabulary word is missing from the model’s vocabulary. One approach maps rare training words and unfamiliar test words to the same <UNK> marker, so the model can learn a probability for unknown words.

A model fits the training examples so closely that it performs worse on new examples. It may rely on accidental patterns or noise instead of learning relationships that carry over.

A model whose behavior is controlled by a specified set of parameters, such as weights and a bias. The structure is fixed while training changes the parameter values.

Combine several word vectors into one vector by applying the same rule, such as the average or maximum, to each coordinate. It gives a classifier one input per text but discards word order.

A summary of how much probability a model gives to the test text, adjusted for its length. Lower is better under the same evaluation setup. For equally likely choices, it equals the number of choices.

PMI compares how often two items occur together with how often they would co-occur by independence. It takes the log of that ratio. PPMI replaces negative results with zero; estimates from rare counts can be unreliable.

Two activation functions. ReLU, max(0, z), passes positive values and zeroes negative ones and is the most common choice. Tanh is an S-shaped curve centered at zero with outputs between −1 and 1.

Training on labels that the data supplies for itself, with no human annotation. In word2vec, a word that really appears near the target is the “yes” answer.

A smooth S-shaped function that turns any finite real-valued score into a number between 0 and 1. Binary logistic regression uses it to produce a probability.

The word2vec variant developed in lecture. A logistic regression classifier decides whether a context word belongs near a target word, using the sigmoid of their dot product. Real neighbors are positive examples and randomly sampled words are negative ones.

Turn several class scores into one probability distribution: exponentiate each score, then divide by the sum of those exponentials. For finite scores, the probabilities are positive and sum to 1.

A sparse vector has many zero entries, often because most possible contexts were never observed. A dense word vector usually has fewer coordinates, most of them nonzero, and packs information into that smaller space. Lecture 7 gives typical lengths: 20,000–50,000 for sparse, 50–1000 for dense.

A static embedding gives a vocabulary word the same vector wherever it appears. A contextual embedding changes with the sentence, so different occurrences of the same word can have different vectors.

Update the parameters after one example, or after a small batch, rather than waiting to process the whole training set. Training usually visits examples in a shuffled order.

With the same input and settings, a deterministic system gives the same output. A stochastic system uses randomness, so the output can vary. Sampling from a language model is one example; fixed decoding can instead be deterministic.

Very common words such as the, is, and of. Some preprocessing pipelines remove them when they add little useful signal, but whether to remove them depends on the task.

Synonyms can share a meaning in some contexts. Similar words share important properties, as coffee and tea do. Related words belong together in a situation, as coffee and cup do, without meaning the same kind of thing.

Word2vec keeps two vectors per word, one for its role as the target and one for its role as a context word, in matrices W and C. A common final representation adds the two.

A table with one row per word and one column per document. Each entry records that word’s count in that document. A row describes a word across documents; a column describes one document.

Give a term more weight if it appears often in one document but in relatively few documents overall. TF measures within-document frequency; IDF discounts terms that appear almost everywhere.

One unit produced when text is split for a model. In these word-level examples it is usually a word occurrence, though other tokenizers may use punctuation, word pieces, or characters.

Tokenization splits text into units. Stemming applies rules that shorten word forms, such as the Porter stemmer. Lemmatization finds a dictionary form. These steps organize the input but do not all do the same job.

A type is a distinct vocabulary item; a token is one occurrence of it. After lowercasing and ignoring punctuation, “the cat chased the cat” has three types and five tokens.

A number becomes too small for the computer’s number format to represent properly. Multiplying many small probabilities can cause this; adding log probabilities helps avoid it.

Shrink weights toward zero during learning. With ordinary gradient descent, adding an L2 penalty produces this effect alongside the update from the prediction loss.

Represent a word with a list of numbers. When the vectors are learned from how words are used, similar words can share useful information instead of being treated as unrelated names.

How many words on each side of the target count as context. Small windows (about ±2) find words that behave alike; large windows (about ±5) find words about the same topic. It shapes both sparse and dense vectors.

A table with target words as rows and nearby context words as columns. An entry counts how often that context word appears within the chosen window around the target.

Three routes to dense vectors. Word2vec trains a classifier on word pairs and keeps its weights; it includes skip-gram and CBOW. GloVe factorizes the log co-occurrence matrix. SVD reduces a matrix directly, and LSA applies it to text. Lecture 6 named all three; Lecture 7 developed the first two.

WordNet organizes word senses into groups that express the same concept, called synsets. It also records relations such as one thing being a kind of another, or a part of another.

An unsmoothed count model may give zero probability to a word combination missing from training. If that combination appears in the test text, its log probability is not finite and perplexity becomes infinite.

A few words occur very often, while a great many words are rare. Roughly, frequency falls as a power of a word’s rank in the frequency list.