Recurrent Neural Networks

Build the mental model

A CNN or plain feedforward network expects a fixed-size input and processes it all at once, which works for images but breaks down for sequences like text or time series where length varies and the order of elements carries meaning. An RNN instead processes a sequence one element at a time, maintaining a hidden state — a vector summarizing everything seen so far — that gets updated at each step by combining the current input with the previous hidden state, then passed forward to the next step. This lets later elements be interpreted in the context of everything before them, which is exactly what language and time series need: 'bank' means something different after 'river' than after 'money'. The catch is training: gradients must be backpropagated through every time step (backpropagation through time), and across many steps they tend to shrink multiplicatively until they're effectively zero, so a plain RNN struggles to learn dependencies spanning more than a handful of steps — it 'forgets' early context by the time it reaches later ones. LSTM and GRU cells fix this with learned gating mechanisms that explicitly control what information to keep, update, or discard at each step, preserving long-range signal far better.

Connect it to a real scenario

If the Tutorial Platform wants to rank search results by learned relevance using the sequence of words in a learner's query, or predict what a learner is likely to search next based on their recent activity history, it's dealing with ordered sequences where later items depend on earlier context — exactly the RNN's home turf. A plain RNN reading a long query or a long session history might lose track of an important word from early in the sequence by the time it reaches the end. In practice the team would reach for an LSTM or GRU instead of a plain RNN specifically to keep that early context alive across a longer sequence.

An RNN maintains a hidden state that gets updated at each time step, letting it interpret later elements in the context of earlier ones.
Gradients shrink as they're backpropagated through many time steps, making plain RNNs struggle to learn long-range dependencies (the vanishing gradient problem).
LSTM and GRU cells use learned gating mechanisms to control what to remember or forget, fixing the vanishing gradient problem far better than a plain RNN.

Try the working example

python

import torch
import torch.nn as nn

rnn = nn.RNN(input_size=5, hidden_size=8, batch_first=True)

# batch=2 sequences, each 6 time steps long, each step a 5-dim vector
sequence = torch.randn(2, 6, 5)

output, hidden = rnn(sequence)

print("Output shape:", output.shape)  # hidden state at every time step
print("Hidden shape:", hidden.shape)  # final hidden state only

You should see

It prints `Output shape: torch.Size([2, 6, 8])` (the hidden state at every one of the 6 time steps for both sequences) and `Hidden shape: torch.Size([1, 2, 8])` (just the final hidden state, one 8-dim vector per sequence).

5-minute try-it

Change `hidden_size` to 16 and re-run, then check that `output`'s last dimension and `hidden`'s last dimension both become 16.

One important caution

Forgetting `batch_first=True` and then indexing the sequence tensor as (batch, seq_len, features) anyway — by default `nn.RNN` expects (seq_len, batch, features), so omitting the flag silently swaps what the dimensions mean and produces wrong results without an error.

Using `output`'s last time step to represent 'what the sequence means' when `hidden` was actually needed (or vice versa) — for a single-layer unidirectional RNN they coincide, but it's easy to grab the wrong tensor when stacking layers or using bidirectional RNNs, where they diverge.

Wikipedia — Recurrent neural network — Deep Learning