Overfitting and Regularization

Build the mental model

A model's training loss going down tells you it's fitting the training data better — it says nothing about whether that fit will generalize. A model with enough capacity can memorize the exact training examples, including their noise and idiosyncrasies, achieving near-perfect training accuracy while performing barely better than chance on new data — this is overfitting. The only reliable way to detect it is holding out a validation or test set the model never sees during training and watching for the point where training loss keeps falling but validation loss starts rising; that gap is the tell that the model has stopped learning general patterns and started memorizing specifics. Two common regularization techniques fight this directly. Dropout randomly zeroes a fraction of neurons on each forward pass during training, forcing the network to spread useful signal across many neurons instead of relying on any single fragile pathway, which acts like training an ensemble of smaller subnetworks. Weight decay adds a small penalty proportional to the squared magnitude of the weights to the loss function, discouraging any single weight from growing large enough to memorize a specific training example.

Connect it to a real scenario

If the Tutorial Platform's recommendation model is trained only on last month's clicks and evaluated on that same data, it will look nearly perfect — it may have simply memorized which learner clicked which lesson rather than learning what makes a lesson genuinely relevant next. The team must always score it on a separate, held-out batch of more recent activity to catch this before shipping. To keep the model honest during training, they'd add dropout between the hidden layers of the recommender network and a small weight decay term to the optimizer, both of which push the model toward learning broad patterns — like 'learners who finish Python basics tend to want OOP next' — instead of memorizing individual click histories.

• Low training loss alone doesn't prove a model generalizes — always evaluate on a held-out validation/test set the model never trained on.
• Dropout randomly zeroes a fraction of neurons during training, preventing the network from over-relying on any single neuron.
• Weight decay adds an L2 penalty on weight magnitude to the loss, discouraging weights from growing large enough to memorize training data.

Try the working example

import torch
import torch.nn as nn

class SmallNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 8)
        self.dropout = nn.Dropout(p=0.5)
        self.fc2 = nn.Linear(8, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        return self.fc2(x)

model = SmallNet()
x = torch.ones(1, 4)

model.train()  # dropout is active
print("Train mode:", model(x).item(), model(x).item())

model.eval()  # dropout is disabled
print("Eval mode: ", model(x).item(), model(x).item())

In train mode the two forward passes print different numbers because dropout randomly zeroes different neurons each call, while in eval mode both calls print the identical number because dropout is disabled.

Wikipedia — Regularization (mathematics) — Deep Learning