Hyperparameter Tuning

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand Hyperparameter Tuning without being intimidated by it
Be able to run scikit-learn code yourself
Apply this concept in a real project right away

Let's Think About This for a Second

A Hyperparameter isn't a parameter (coefficient) the model learns from data — it's a setting the developer/data scientist has to specify in advance, before training starts (e.g. `max_depth`, `n_estimators`, `K`). Grid Search systematically tries out every hyperparameter combination (e.g. `max_depth: [3,5,7]` × `n_estimators: [50,100,200]` = 9 combinations), evaluates each combination's performance using cross-validation (from the Intermediate chapter), and automatically selects the best one.

Connecting to a Real Scenario

Writing `from sklearn.model_selection import GridSearchCV; params = {'max_depth': [3,5,7], 'n_estimators': [50,100,200]}; grid = GridSearchCV(RandomForestClassifier(), params, cv=5); grid.fit(X_train, y_train)` evaluates each of the 9 combinations using cross-validation, and checking `grid.best_params_` tells you instantly which combination performed best.

Hyperparameter = a setting you specify before training starts (not something learned from data)
Grid Search = systematically trying out every hyperparameter combination
GridSearchCV = a scikit-learn tool that combines Grid Search + Cross-Validation

Let's Look at It Together

python

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {
    'max_depth': [3, 5, 7],
    'n_estimators': [50, 100, 200]
}

grid = GridSearchCV(RandomForestClassifier(random_state=42), param_grid, cv=5)
grid.fit(X_train, y_train)

print(f"Best parameters: {grid.best_params_}")
print(f"Best cross-validation score: {grid.best_score_:.2f}")

You should see

Best parameters: {'max_depth': 7, 'n_estimators': 200}
Best cross-validation score: 0.89

Try It in 5 Minutes

Try a few `max_depth`/`n_estimators` combinations on a Random Forest using `GridSearchCV` — compare the `best_params_` result against the manual default values.

A Quick Word of Caution

Don't think of hyperparameter tuning as 'magic that's guaranteed to boost model performance' — data quality and feature engineering (from the Intermediate chapter) are often more important than hyperparameter tuning.

Easy traps

Setting too many parameter combinations for Grid Search (5 parameters x 10 values each = 100,000 combinations) — training time can balloon astronomically (consider RandomizedSearchCV as an alternative)
Running hyperparameter tuning directly on the test set — the tuning process should only run on the training data + cross-validation, leaving the test set reserved solely for the final, one-time evaluation

Now Try It Yourself

Try a few `max_depth`/`n_estimators` combinations on a Random Forest using `GridSearchCV` — compare the `best_params_` result against the manual default values.

You'll know it worked when: Best parameters: {'max_depth': 7, 'n_estimators': 200} Best cross-validation score: 0.89