Batch Normalization vs Layer Normalization

Build the mental model

As a deep network trains, the parameters of early layers keep changing with every update, which means the distribution of activations flowing into later layers keeps shifting too — a phenomenon called internal covariate shift. Later layers then have to constantly re-adapt to a moving target instead of learning a stable mapping, which slows convergence and can destabilize training, especially with high learning rates. Batch normalization addresses this directly: for each feature, it subtracts the mean and divides by the standard deviation computed across the current batch, then applies a learnable scale and shift so the layer can still represent any distribution it needs. The catch is that batch normalization's statistics depend on batch composition — with very small batches, or with sequence models where batch entries have different lengths and padding, those statistics become noisy or ill-defined. Layer normalization sidesteps this by normalizing across the features of a single sample instead of across the batch dimension, so its statistics don't depend on batch size or on what other examples happen to be in the same batch, which is why it's the default choice in transformers.

Connect it to a real scenario

The Tutorial Platform's sentiment classifier processes review text a batch at a time, but real traffic sends batches of wildly varying size — one request during a quiet hour, hundreds during a launch. Batch normalization layers would compute their mean/std from whatever batch happens to arrive, so a single-review batch gets unreliable statistics compared to a 200-review batch, making predictions inconsistent. Since the classifier is transformer-based, layer normalization is the natural fit: each review's own token features are normalized independently of every other review in the batch, so a lone review at 2am gets normalized exactly the same way it would inside a batch of 200.

• Internal covariate shift is the problem of activation distributions in later layers constantly shifting because earlier layers keep updating, which slows training
• Batch normalization normalizes each feature across the batch dimension — it depends on batch statistics, which becomes fragile with small batches or variable-length sequences
• Layer normalization normalizes across a single sample's own features instead, so it's entirely independent of batch size, which is why it's the default in transformers

Try the working example

import torch
import torch.nn as nn

torch.manual_seed(0)

batch_size, num_features = 4, 6
x = torch.randn(batch_size, num_features) * 5 + 3

bn = nn.BatchNorm1d(num_features)
ln = nn.LayerNorm(num_features)

bn_out = bn(x)
ln_out = ln(x)

print("input mean/std:", round(x.mean().item(), 3), round(x.std().item(), 3))
print("batchnorm per-feature (column) mean:", bn_out.mean(dim=0))
print("batchnorm per-feature (column) std:", bn_out.std(dim=0, unbiased=False))
print("layernorm per-sample (row) mean:", ln_out.mean(dim=1))
print("layernorm per-sample (row) std:", ln_out.std(dim=1, unbiased=False))

input mean/std: 3.xxx 5.xxx
batchnorm per-feature (column) mean: tensor([~0, ~0, ~0, ~0, ~0, ~0])
batchnorm per-feature (column) std: tensor([~1, ~1, ~1, ~1, ~1, ~1])
layernorm per-sample (row) mean: tensor([~0, ~0, ~0, ~0])
layernorm per-sample (row) std: tensor([~1, ~1, ~1, ~1])

BatchNorm1d normalizes each feature (column) across the 4 samples in the batch, so each column's mean/std lands near 0/1. LayerNorm normalizes each sample (row) across the 6 features instead, so each row's mean/std lands near 0/1 — the two normalize along entirely different dimensions.

Wikipedia — Batch normalization — Deep Learning