Self-Supervised Learning for Vision

Build the mental model

Contrastive self-supervised learning solves a labeling problem that plagues most of computer vision: humans are slow and expensive at drawing boxes or typing captions, but computers are effortless at generating two different-looking versions of the same image. The trick is to manufacture a training signal directly from that fact instead of from human annotation. Take one unlabeled image, apply two independent random augmentations — a crop here, a color shift there, a flip somewhere else — and you get two views that look superficially different but still show the same underlying scene. A shared encoder processes both views into embedding vectors, and the only 'label' the model ever sees is the trivial fact that these two embeddings came from the same source image, so they should end up close together in embedding space, while embeddings from an unrelated image should end up far apart. No person ever decided what the image contains; the supervision is entirely self-generated from the data's own structure.

Connect it to a real scenario

The Tutorial Platform receives thousands of user-uploaded lesson screenshots and diagrams, and manually labeling which pairs are near-duplicates simply isn't feasible. The content moderation team pretrains a self-supervised encoder on the platform's own unlabeled image corpus, then computes an embedding for every new upload and measures its cosine similarity against embeddings of existing images. When the similarity crosses a threshold, the pair gets flagged as a likely duplicate and routed to a review queue — no hand-labeled 'duplicate/not duplicate' dataset ever needed to exist.

Contrastive self-supervised learning uses the fact that 'these two views came from the same image' as its only label, requiring zero human annotation.
Random augmentations turn a single image into a positive pair of views, pushing the shared encoder toward augmentation-invariant embeddings.
In domains where labels are scarce or expensive — satellite imagery, medical scans — self-supervised pretraining followed by fine-tuning on a small labeled set is dramatically more data-efficient than training from scratch.

Try the working example

python

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.transforms as T

torch.manual_seed(0)

# a single synthetic "image" - pretend it's a 32x32 RGB photo
image = torch.rand(3, 32, 32)

# two independent augmentation pipelines applied to the SAME image
augment_1 = T.Compose([
    T.RandomHorizontalFlip(p=1.0),
    T.RandomErasing(p=1.0, scale=(0.05, 0.15)),
])
augment_2 = T.Compose([
    T.ColorJitter(brightness=0.4, contrast=0.4),
    T.RandomRotation(degrees=15),
])

view_1 = augment_1(image)
view_2 = augment_2(image)


class SmallEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(3, 8, kernel_size=3, stride=2, padding=1),   # 32x32 -> 16x16
            nn.ReLU(),
            nn.Conv2d(8, 16, kernel_size=3, stride=2, padding=1),  # 16x16 -> 8x8
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),
        )
        self.project = nn.Linear(16, 8)

    def forward(self, x):
        features = self.conv(x).flatten(1)  # (batch, 16)
        return self.project(features)       # (batch, 8) embedding

encoder = SmallEncoder()

# add a batch dimension: (1, 3, 32, 32) -> encoder -> (1, 8)
embedding_1 = encoder(view_1.unsqueeze(0))
embedding_2 = encoder(view_2.unsqueeze(0))

similarity = F.cosine_similarity(embedding_1, embedding_2)

print("Embedding 1 shape:", embedding_1.shape)
print("Embedding 2 shape:", embedding_2.shape)
print("Cosine similarity between the two views:", similarity.item())

You should see

Prints Embedding 1 shape: torch.Size([1, 8]) and Embedding 2 shape: torch.Size([1, 8]). The cosine similarity is a single float between -1.0 and 1.0 — because torch.manual_seed(0) is set, rerunning the script always prints the exact same number, but since the encoder has random, untrained weights, there's no guarantee that number is close to 1.0; it hasn't actually learned to capture semantic similarity yet.

5-minute try-it

Modify the code to create a second, completely different synthetic image (image_2 = torch.rand(3, 32, 32)), pass one of its augmented views through the same encoder, and compute the cosine similarity between a view of the original image and a view of image_2. Compare that against the similarity between the two views of the original image, and think about what pattern you'd expect to see even with an untrained encoder.

One important caution

Using augmentations that are too aggressive (for example, a RandomErasing scale that's too large) can erase the shared content between the two views, so the encoder learns to match leftover noise patterns instead of the actual object.

Training with only a positive pair and no negative pairs at all lets the encoder cheat by mapping every input to the same constant embedding (representation collapse) — in that failure mode, cosine similarity ends up near 1.0 for almost any two images, not just genuinely related ones.

Wikipedia — Self-supervised learning — Computer Vision