Attention Mechanism

Build the mental model

Before attention, sequence models built on RNNs (or LSTMs/GRUs) had to read an entire input sequence and compress everything it contains into one fixed-size hidden state vector before producing any output. This is a severe information bottleneck: for a long sentence or document, whatever the model learned from the first few tokens has to survive being overwritten step after step by every token that follows, so by the time the model reaches the end, early context is diluted or forgotten entirely. Attention removes this bottleneck by letting the model, at each output step, look back directly at every position of the input rather than relying on a single squeezed-through summary. It does this with three learned projections: a query vector representing what the current step is looking for, and a key vector and value vector for every input position. The query is compared against every key (typically via dot product) to produce a relevance score per position; softmax turns these scores into attention weights that sum to one, and the output is the weighted sum of the value vectors. Distance in the sequence no longer matters — position 1 is exactly as reachable as position 1000.

Connect it to a real scenario

On Thuta's Tutorial Platform, attention is exactly the mechanism a learned lesson-recommendation model would need to decide which of a user's past lessons are actually relevant to recommending the next one — instead of squashing the whole learning history into one summary vector the way an RNN would. A query built from 'what topic is the user on now' attends over keys built from every past lesson the user completed, giving directly related lessons a high weight and unrelated ones near zero, no matter how many lessons or how much time separates them, since attention has no built-in decay based on position or recency.

• Attention lets every output step directly access any input position via learned query/key/value comparisons, removing the fixed-size bottleneck of RNN hidden states
• Attention weights come from softmax over query-key similarity scores, so they always sum to 1 and act as a differentiable, learned 'spotlight' over positions
• Because attention has no built-in notion of sequence order, distance between positions doesn't matter — it can equally connect adjacent or far-apart tokens

Try the working example

import torch
import torch.nn.functional as F

torch.manual_seed(0)

seq_len, d_k = 4, 8
Q = torch.randn(seq_len, d_k)
K = torch.randn(seq_len, d_k)
V = torch.randn(seq_len, d_k)

scores = Q @ K.T / (d_k ** 0.5)
weights = F.softmax(scores, dim=-1)
output = weights @ V

print("attention weights shape:", weights.shape)
print("output shape:", output.shape)
print("weights row sums:", weights.sum(dim=-1))

attention weights shape: torch.Size([4, 4])
output shape: torch.Size([4, 8])
weights row sums: tensor([1.0000, 1.0000, 1.0000, 1.0000])

The weights matrix shows how much each of the 4 query positions attends to each of the 4 key positions, and every row sums to 1 because of softmax. The output shape matches the (seq_len, d_k) shape of Q and V.

Wikipedia — Attention (machine learning) — Deep Learning