Model Evaluation and Metrics

Build the mental model

Accuracy is the fraction of predictions a model gets right, and it is the first metric anyone reaches for — but it silently breaks down when classes are imbalanced. On a spam dataset that is 90% legitimate email, a model that always predicts 'not spam,' having learned nothing, still scores 90% accuracy. Accuracy alone cannot reveal whether a model is actually useful. The confusion matrix — counts of true positives, false positives, true negatives, and false negatives — is the raw material every better metric is built from. Precision (of everything flagged positive, how much was actually correct) and recall (of everything actually positive, how much was caught) each expose a different failure, and they trade off against each other: pushing a model to flag more raises recall but tends to lower precision. Which matters more depends on the cost of each mistake. A spam filter that buries real email is worse than one that lets some spam through, so it should favor precision; a cancer screen that misses a real case is far worse than one flagging a healthy patient for follow-up, so it should favor recall. F1 score, the harmonic mean of both, collapses them into a single number.

Connect it to a real scenario

On the Tutorial Platform, imagine a sentiment classifier that flags harsh comments on lesson discussions for moderator review — most comments are neutral or positive, so the harsh-comment class is naturally rare, exactly the imbalanced setting where accuracy lies. If the platform only checked accuracy, a classifier that never flags anything could look like it works. What actually matters is recall (catching harsh comments so they get reviewed) balanced against precision (not burying moderators in false alarms). Reporting precision, recall, and F1 alongside accuracy — computed from a confusion matrix on a held-out set of labeled comments — is what tells the team whether the classifier is deployment-ready, not just accuracy-ready.

• Accuracy can be high and still meaningless when classes are imbalanced — always check precision and recall too.
• Precision and recall trade off; which one to prioritize depends on whether false positives or false negatives are more costly for the task.
• The confusion matrix (TP, FP, TN, FN) is the source every classification metric is computed from, and F1 score summarizes precision/recall into one number.

Try the working example

import torch

# 100 samples: 90 "not spam" (0), 10 "spam" (1) -- realistic class imbalance
actual = torch.cat([torch.zeros(90, dtype=torch.long), torch.ones(10, dtype=torch.long)])
predicted = torch.zeros(100, dtype=torch.long)

# Model wrongly flags 2 legitimate emails as spam (false positives)
predicted[3] = 1
predicted[7] = 1

# Model correctly catches only 3 of the 10 real spam emails (7 false negatives)
predicted[90] = 1
predicted[91] = 1
predicted[92] = 1

# Confusion matrix counts, computed with tensor comparisons
TP = ((predicted == 1) & (actual == 1)).sum().item()
TN = ((predicted == 0) & (actual == 0)).sum().item()
FP = ((predicted == 1) & (actual == 0)).sum().item()
FN = ((predicted == 0) & (actual == 1)).sum().item()

accuracy = (TP + TN) / actual.numel()
precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)

print(f"Confusion matrix -> TP: {TP}, FP: {FP}, FN: {FN}, TN: {TN}")
print(f"Accuracy:  {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall:    {recall:.4f}")
print(f"F1 Score:  {f1:.4f}")

Confusion matrix -> TP: 3, FP: 2, FN: 7, TN: 88
Accuracy:  0.9100
Precision: 0.6000
Recall:    0.3000
F1 Score:  0.4000
Despite 91% accuracy, the model catches only 30% of actual spam (recall) — precision and recall expose the weakness accuracy hides.

Wikipedia — Precision and recall — Deep Learning