Autograd and Gradients

Build the mental model

Training a neural network is fundamentally an optimization loop that repeatedly asks: for each parameter, how should it change to reduce the loss a little? That question is answered exactly by a derivative — the gradient of the loss with respect to that parameter tells you the direction and steepness of the fastest increase, so moving a parameter a small step in the opposite direction reduces the loss. Computing these derivatives by hand for every new architecture would be impractical: a network with millions of parameters composed through dozens of layers requires the chain rule applied over and over, and one algebra mistake silently breaks training. PyTorch's autograd solves this by recording, as your code runs, every operation performed on tensors with `requires_grad=True` into a computational graph. Calling `.backward()` on a final scalar (like a loss) walks that graph backward, applying the chain rule automatically at each recorded step, and accumulates the result into each tensor's `.grad` attribute. This is what makes arbitrary network architectures trainable without anyone hand-deriving their gradients.

Connect it to a real scenario

When the Tutorial Platform trains a sentiment classifier on feedback comments, autograd is what turns 'the model got this comment's sentiment wrong' into a concrete update for every weight in the network — without it, engineers would need to hand-derive gradients for every layer of every architecture change, making experimentation impossibly slow. Every model in this course, from a one-layer network to a mini transformer in the Projects chapter, relies on the exact mechanism you're about to see: mark tensors as needing gradients, compute something, call `.backward()`, and read off `.grad`.

requires_grad=True tells PyTorch to record operations on a tensor into a computational graph.
Calling .backward() applies the chain rule automatically, walking the graph backward from a scalar output.
The resulting derivative is stored in the tensor's .grad attribute — no manual calculus needed.

Try the working example

python

import torch

x = torch.tensor(2.0, requires_grad=True)
y = x**2 + 3*x  # y = x^2 + 3x

y.backward()  # computes dy/dx and stores it in x.grad

print("x =", x.item())
print("y =", y.item())
print("x.grad =", x.grad.item())  # dy/dx = 2x + 3, so at x=2: 2*2 + 3 = 7

You should see

Prints x = 2.0, y = 10.0 (since 2^2 + 3*2 = 10), and x.grad = 7.0, matching the hand-computed derivative dy/dx = 2x + 3 evaluated at x = 2.

5-minute try-it

Change the expression to y = x**3 - 2*x, compute .backward() at x = 1.0, and verify x.grad matches the hand-computed derivative dy/dx = 3x^2 - 2.

One important caution

Calling .backward() a second time on the same graph without retain_graph=True raises an error — by default PyTorch frees the graph after one backward pass to save memory, since most training loops build a fresh graph every step anyway.

Forgetting requires_grad=True on a leaf tensor (or wrapping it in a plain Python operation that detaches it) silently makes .grad stay None — there's no crash, just a confusing None where a gradient was expected.

PyTorch Docs — Autograd — Deep Learning