Learning Rate Scheduling

Build the mental model

The learning rate scales how big a step the optimizer takes on each parameter update. Set it too high and updates overshoot good minima, causing the loss to diverge or oscillate; set it too low and training crawls, or gets stuck in a shallow local minimum it can't step out of. A single fixed learning rate for the entire run is rarely ideal because these two failure modes trade off differently at different training stages: early on, while the model is far from any good solution, a relatively high rate makes fast progress and cheaply explores the loss landscape; later, as the model approaches convergence, that same high rate causes it to bounce around near a good minimum instead of settling into it, so a smaller, more careful rate is needed. Learning rate scheduling automates this by decaying the rate over time according to a fixed policy — for example StepLR drops it by a multiplicative factor every fixed number of epochs, while CosineAnnealingLR decays it smoothly along a cosine curve. A related technique, warmup, does the opposite at the very start: it holds the rate below the target for the first few steps, since large early gradients (before the model's weights have settled at all) can otherwise cause instability.

Connect it to a real scenario

Training the Tutorial Platform's sentiment classifier from scratch on review data, a fixed learning rate would either overshoot early (wasting the first several epochs bouncing around instead of learning) or crawl throughout (wasting compute budget in a scheduled retraining job). A StepLR schedule that starts at a relatively high rate and drops it every few epochs lets the model quickly pick up coarse patterns like obvious negative words, then fine-tune more carefully on subtler cases — sarcasm, mixed sentiment — as training progresses, all within the same fixed training-time budget the platform allocates for periodic model refreshes.

A learning rate that's too high causes training to diverge or oscillate, while too low causes it to crawl or get stuck in a local minimum — so a single fixed rate is rarely ideal
Scheduling automatically decays the rate as training progresses — StepLR drops it every fixed number of epochs, while CosineAnnealingLR decays it smoothly along a cosine curve
Warmup holds the rate below the target at the very start of training to prevent instability from large early gradients

Try the working example

python

import torch
import torch.nn as nn

model = nn.Linear(4, 2)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.5)

for epoch in range(9):
    optimizer.step()
    current_lr = optimizer.param_groups[0]["lr"]
    print(f"epoch {epoch}: lr = {current_lr:.5f}")
    scheduler.step()

You should see

epoch 0: lr = 0.10000
epoch 1: lr = 0.10000
epoch 2: lr = 0.10000
epoch 3: lr = 0.05000
epoch 4: lr = 0.05000
epoch 5: lr = 0.05000
epoch 6: lr = 0.02500
epoch 7: lr = 0.02500
epoch 8: lr = 0.02500

With step_size=3, the learning rate is multiplied by gamma=0.5 every 3 epochs, halving it at epochs 3 and 6 as shown.

5-minute try-it

Swap the scheduler for torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=9) and rerun — compare how the decay curve differs from StepLR's step-wise drops.

One important caution

Calling scheduler.step() inside the batch loop instead of once per epoch (or before optimizer.step() has actually been used) makes the learning rate decay much faster than intended, since StepLR's step_size counts the number of times scheduler.step() is called

Checking optimizer.defaults['lr'] instead of optimizer.param_groups[0]['lr'] or scheduler.get_last_lr() to read the current rate is a common mistake — defaults holds the original value set at construction and never reflects the scheduler's decay

PyTorch Docs — How to adjust learning rate — Deep Learning