Overfitting & Regularization

Let's Think About This for a Second

Overfitting is a symptom where the model 'memorizes' even the noise and details of the training data, so training accuracy is very high (99%+) but test accuracy drops (70%) — this shows the model's complexity is greater than the actual pattern in the data. Underfitting is the opposite — the model is too simple, so it performs poorly even on the training data. Regularization (L1/L2 penalty) is a technique that 'punishes' model complexity to prevent overfitting — by penalizing coefficients that get too large, it forces the model to prioritize simpler patterns.

Connecting to a Real Scenario

If training accuracy is 98% and test accuracy is 65%, that's a classic overfitting signal — using `Ridge`/`Lasso` (Linear Regression variants with L2/L1 regularization) via `from sklearn.linear_model import Ridge; model = Ridge(alpha=1.0)` lets you increase `alpha` (regularization strength) to further limit model complexity and address the overfitting.

Overfitting = high training accuracy, low test accuracy (the model 'memorized' the data)
Underfitting = both training and test accuracy are low (the model is too simple)
Regularization (L1/L2, Ridge/Lasso) = penalizes model complexity to prevent overfitting

Let's Look at It Together

python

from sklearn.linear_model import Ridge, LinearRegression

# Without regularization — may overfit with many features
model_plain = LinearRegression()
model_plain.fit(X_train, y_train)

# With L2 regularization (Ridge) — penalizes large coefficients
model_ridge = Ridge(alpha=1.0)
model_ridge.fit(X_train, y_train)

print(f"Plain - train: {model_plain.score(X_train, y_train):.2f}, test: {model_plain.score(X_test, y_test):.2f}")
print(f"Ridge - train: {model_ridge.score(X_train, y_train):.2f}, test: {model_ridge.score(X_test, y_test):.2f}")

You should see

Plain - train: 0.98, test: 0.65
Ridge - train: 0.91, test: 0.86

Try It in 5 Minutes

Compare train/test scores between LinearRegression and Ridge on an overfitting-prone dataset (lots of features, few samples) — directly observe Ridge's regularization effect.

A Quick Word of Caution

Collecting more data is often a more effective solution to overfitting than regularization — using a complex model with too little data makes overfitting likely, so you should think about data quantity/quality first.