Transfer Learning

Build the mental model

Training a large, accurate network from scratch demands a huge labeled dataset and enormous compute — resources most individual projects don't have. Transfer learning sidesteps this by starting from a model already trained on a large, general dataset, whose early layers have already learned broadly useful, low-level features (edges, textures, basic shapes) that transfer well to almost any visual task, then adapting that model to a new, smaller, more specific task instead of learning everything from zero. There are two main strategies, differing in how much of the pretrained model you retrain. Feature extraction freezes every pretrained layer's weights entirely — setting `requires_grad = False` so no gradients update them — and only trains a newly added final classifier layer on top; this is fast, needs comparatively little new data, and works well when the new task is similar to what the model originally learned. Fine-tuning instead unfreezes some or all of the pretrained layers and continues training them, typically at a much lower learning rate than a from-scratch training run, letting the model adapt its learned features more deeply to the new task at the cost of needing more data and compute to avoid destroying what it already learned.

Connect it to a real scenario

Rather than training a search-relevance ranking model from zero on the Tutorial Platform's relatively small click-through dataset, the team starts from a text model already pretrained on a large general corpus, whose early layers already understand grammar and word meaning broadly. For a first version they freeze those pretrained layers and train only a new final layer to score query-lesson relevance, since their click data is limited and feature extraction avoids overfitting on so little data. Once the platform accumulates months of real usage data, they can switch to fine-tuning, unfreezing the later pretrained layers so the model adapts its language understanding specifically to how learners phrase programming questions.

• Transfer learning starts from a model pretrained on a large general dataset instead of training from scratch, reusing its already-learned low-level features.
• Feature extraction freezes all pretrained weights (`requires_grad = False`) and trains only a new final layer — fast and needs little data.
• Fine-tuning unfreezes some or all pretrained layers and retrains them at a low learning rate — slower and needs more data, but adapts more deeply.

Try the working example

import torch
import torch.nn as nn

class TinyBackbone(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 8, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(8, 16, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        # pretend this was pretrained on a 1000-class dataset
        self.classifier = nn.Linear(16 * 8 * 8, 1000)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        return self.classifier(x)

model = TinyBackbone()

# Freeze all pretrained feature-extraction layers
for param in model.features.parameters():
    param.requires_grad = False

# Replace the final layer for a new 5-class task (trainable by default)
model.classifier = nn.Linear(16 * 8 * 8, 5)

for name, param in model.named_parameters():
    print(name, "trainable" if param.requires_grad else "frozen")

It prints one line per parameter tensor, showing every `features.*` weight and bias as `frozen` and every `classifier.*` weight and bias as `trainable`.

PyTorch Docs — Transfer Learning Tutorial — Deep Learning