Build the mental model
A loss function turns 'how wrong is the model right now' into a single number the network can improve against — `nn.MSELoss`, for example, computes the average squared difference between predicted and actual values for a regression problem, penalizing bigger errors disproportionately more than small ones. On its own, a loss value only tells you how bad things are; it doesn't tell you what to change. That's where an optimizer (`torch.optim.SGD`, `Adam`, etc.) comes in: it takes the gradients autograd already computed via `loss.backward()` and applies an update rule to nudge every parameter in the direction that reduces the loss. Three calls form the heartbeat of every training loop, and their order matters: `loss.backward()` computes gradients and adds them into each parameter's `.grad`; `optimizer.step()` reads those gradients and updates the parameters; `optimizer.zero_grad()` clears `.grad` back to zero. That last step is easy to forget — PyTorch accumulates gradients by default (useful for some advanced techniques), so skipping zero_grad silently mixes gradients from multiple batches together.
Connect it to a real scenario
When the Tutorial Platform's relevance-ranking model is training, its loss function measures how far the model's predicted ranking score is from what learner click data implies the 'correct' ranking should be, and the optimizer is what actually turns that error signal into better weights over thousands of training steps. Getting the forward -> loss -> backward -> step -> zero_grad sequence right, in the right order, on every single step, is the difference between a model that steadily improves and one that never learns or learns the wrong thing entirely.
Try the working example
import torch
import torch.nn as nn
torch.manual_seed(0)
model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.tensor([[2.0]])
y_true = torch.tensor([[10.0]])
# One manual training step
prediction = model(x)
loss_before = loss_fn(prediction, y_true)
print("Loss before step:", loss_before.item())
loss_before.backward() # compute gradients
optimizer.step() # update parameters using gradients
optimizer.zero_grad() # clear gradients for next round
prediction_after = model(x)
loss_after = loss_fn(prediction_after, y_true)
print("Loss after step:", loss_after.item())Prints a positive loss value before the step, then a smaller loss value after the step — algebraically, one SGD update with lr=0.01 on this single-example setup multiplies the loss by 0.81, so the printed 'loss after' is about 19% lower than 'loss before'.5-minute try-it
Run the same manual step three times in a row (forward -> loss -> backward -> step -> zero_grad each time) and print the loss after each one — confirm it keeps decreasing, then try removing optimizer.zero_grad() and see how the behavior changes.
One important caution
Calling optimizer.step() before loss.backward() — step() reads whatever is currently in each parameter's .grad, so calling it too early either uses stale gradients from a previous step or None, silently corrupting training.
Forgetting optimizer.zero_grad() between steps — since PyTorch accumulates gradients into .grad by default, later steps add new gradients on top of old ones, effectively training with an unintended, ever-growing learning rate.
PyTorch Docs — Optimization — Deep Learning