Lecture 5 reviews these training methods on PDF pages 14–16. The same approach also applies to multinomial logistic regression and softmax.
IV. Adjust the model to reduce its errors Lec 4 · Sep 3
A loss tells us how poor a prediction is, but it does not change the model by itself. Optimization is the process of finding better parameter values. For logistic regression, those parameters are the weights and bias, \(\theta=[w;b]\).
Write the model's prediction as \(\hat y=f(x;\theta)\). We want parameters that make the cross-entropy loss small on average across all \(N\) training examples. The notation \(\arg\min\) means “the parameter values that give the smallest value”:
Gradient descent: take a small step downhill
Imagine trying to reach the bottom of a river canyon. You look around, find the steepest downhill direction, take a step, and look again. Gradient descent does something similar on the model's loss surface. Each location represents a choice of parameters, and its height is the loss.
The shape of this surface affects what we can expect from the method:
| Convex loss, as in logistic regression | Non-convex loss, as in typical neural networks |
|---|---|
| There are no local minima worse than the global minimum. However, several parameter settings can share the minimum. Convergence still requires suitable step sizes and other conditions; convexity alone does not guarantee a unique solution or even a finite parameter value that attains the minimum. | The surface can contain multiple local minima and saddle points, where some directions go up and others go down. The starting point can affect the result. Gradient methods are useful in practice, but we cannot generally promise they will find the global minimum. |
First consider just one weight
Should we increase \(w\) or decrease it? The slope of the loss tells us. If the slope at the current value \(w_1\) is negative, a small increase in \(w\) lowers the loss. If the slope is positive, a small decrease lowers it. In either case, we move against the slope.
With many weights, use a gradient
With several parameters, we need one slope for each parameter. These partial derivatives form a vector called the gradient. It points toward the steepest local increase in loss, so we subtract a multiple of it to move downhill.
The learning rate \(\eta\) controls how large that step is. The superscript \(t\) counts updates:
- A larger learning rate makes a larger change. This may speed up progress, but a step that is too large can overshoot and increase the loss.
- A weight vector \(w\) with \(d\) entries has \(d\) corresponding gradient entries. If we also include the bias, the full parameter vector \([w;b]\) has \(d+1\) entries. With just one weight and one bias, the gradient is an arrow in the \(w\)–\(b\) plane.
- Large models have much longer gradients, but each entry answers the same question: how would a small change in \(\theta_i\) affect the loss while the other parameters stay fixed?
The gradient for logistic regression Lec 4 · Sep 3
Return to sentiment classification. Its cross-entropy is \(L_{CE}(\hat y,y)=-[y\log\sigma(w\cdot x+b)+(1-y)\log\sigma(-(w\cdot x+b))]\). Differentiating it gives a simple expression that we can calculate directly:
The weight gradient is (predicted probability − correct label) × feature value. For a positive-valued feature such as a word count, overestimating the positive class pushes its weight down, while underestimating pushes its weight up. If the feature is zero, this example contributes no loss gradient to that weight. A negative-valued feature reverses the direction, exactly as the multiplication indicates.
Update after one example, or after several? Lec 4 · Sep 3
Full, or batch, gradient descent computes the average gradient over all \(N\) training examples before making an update. This uses all the available evidence, but each step can be expensive.
Stochastic gradient descent (SGD) takes a cheaper step after each example. It visits the examples in random order, predicts an answer, measures the error, and updates the parameters:
function STOCHASTIC-GRADIENT-DESCENT(L(), f(), x, y) returns θ
# L: loss function; f: model with parameters θ
# x: training inputs x(1)..x(N); y: correct labels y(1)..y(N)
θ ← 0 (or initialize randomly)
repeat until the stopping condition is met
for each training pair (x(i), y(i)) in random order:
1. ŷ(i) ← f(x(i); θ) # make a prediction
2. compute L(ŷ(i), y(i)) # measure the error
3. g ← ∇θ L(f(x(i); θ), y(i)) # local direction of steepest increase
4. θ ← θ − η g # take a step in the opposite direction
return θMini-batches provide a middle ground
A mini-batch contains \(m\) examples. Compute their individual gradients, average them, and then update once. In the pseudocode, step 3 therefore uses the sum of the batch's gradients divided by \(m\).
Averaging several examples usually gives a steadier estimate than using one example, while costing much less per update than processing the full dataset. It also works well with hardware that processes many examples in parallel.
| Method | Examples per update | What to expect |
|---|---|---|
| Batch gradient descent | All \(N\) | Uses the exact training-set gradient, but each update can be slow. |
| Stochastic gradient descent | 1 | Cheap, noisy updates. In non-convex problems, the noise can sometimes help the model move out of poor regions. |
| Mini-batch SGD | \(m\), for example 32–512 | A common practical choice that balances computation and a steadier gradient. |
- Training tries to reduce average loss. Gradient descent subtracts the gradient, with the step size controlled by the learning rate.
- Logistic regression has a convex loss. This rules out worse local minima, but convergence and uniqueness still need additional conditions. Neural-network losses are generally non-convex.
- For a logistic-regression weight, the loss gradient is \((\sigma(w\cdot x+b)-y)x_j\): prediction minus truth, multiplied by the feature.
- SGD and mini-batch SGD make large datasets practical to train on. We must also decide when to stop, which connects to overfitting.