Build the mental model
Stack any number of linear (fully-connected) layers together with nothing between them, and the result collapses mathematically into a single linear layer — multiplying matrices repeatedly is still just one matrix multiplication, no matter how many layers you chain. That means a network with a hundred linear layers has the same expressive power as a network with one: it can only ever draw a straight line (or flat hyperplane) through its data, no matter how deep it looks. Activation functions fix this by inserting a nonlinear function between layers, which breaks that collapse and lets the network approximate genuinely curved, complex decision boundaries. ReLU (max(0, x)) is the default choice in most modern networks: it's cheap to compute and its gradient is either 0 or 1, which keeps gradients flowing well during training — but a neuron whose input is always negative gets a permanent zero gradient and stops learning entirely, called 'dying ReLU'. Sigmoid squashes any input into (0, 1), useful for probabilities, but saturates at the extremes, producing near-zero gradients and slowing or stalling learning in deep networks. Tanh is sigmoid's zero-centered cousin, squashing to (-1, 1), which often trains slightly better than sigmoid but shares the same saturation problem.
Connect it to a real scenario
When the Tutorial Platform builds a model to classify learner feedback as positive, neutral, or negative, the input text embeddings interact in complicated, non-linear ways — 'not bad at all' and 'not good at all' differ by one word but mean opposite things, a pattern no stack of linear layers alone could ever separate. Every hidden layer in that sentiment classifier needs a nonlinear activation between it and the next, or the whole network mathematically reduces to one linear layer no matter how many are stacked. The team would pick ReLU for the hidden layers, since it trains fast and avoids vanishing gradients across the network's depth, and reserve sigmoid for the very last layer, where squashing the output into (0, 1) gives a clean 'probability of positive sentiment' the platform can threshold or rank feedback by.
Try the working example
import torch
x = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0])
relu_out = torch.relu(x)
sigmoid_out = torch.sigmoid(x)
tanh_out = torch.tanh(x)
print("Input: ", x)
print("ReLU: ", relu_out)
print("Sigmoid:", sigmoid_out)
print("Tanh: ", tanh_out)Printing the input tensor alongside the ReLU, sigmoid, and tanh outputs shows ReLU zeroing every negative value, sigmoid squashing everything into (0, 1), and tanh squashing everything into (-1, 1).5-minute try-it
Add a fourth line applying `torch.nn.functional.leaky_relu` to the same tensor and compare its output for negative values against plain ReLU's.
One important caution
Using sigmoid in every hidden layer of a deep network, not just the output layer — its saturating gradient makes deep networks extremely slow or impossible to train, which is why ReLU became the default for hidden layers.
Assuming a 'dead' ReLU neuron (always outputting 0) will recover on its own — since its gradient is exactly 0 for negative inputs, standard gradient descent can never push its weights back to positive territory.
Wikipedia — Activation function — Deep Learning