Random Forests & Ensemble Methods

Let's Think About This for a Second

An Ensemble Method is a technique that combines many models (weak learners) to produce a prediction that's more stable/accurate than any individual model — it's similar to the 'wisdom of the crowd' concept (the average of many opinions tends to be more stable than any single opinion). Random Forest is an ensemble algorithm that combines many Decision Trees (from the Intermediate chapter) using the bagging technique (randomly sampling data/features to train each individual tree) — the final prediction is decided by majority vote (classification) or averaging (regression) across all the trees.

Connecting to a Real Scenario

Writing `from sklearn.ensemble import RandomForestClassifier; model = RandomForestClassifier(n_estimators=100, max_depth=5); model.fit(X_train, y_train)` trains 100 trees (`n_estimators=100`), and when predicting on test data, it combines the votes from all 100 trees — this is more stable than a single Decision Tree and has lower overfitting risk.

Ensemble Method = combining many models to get a more stable prediction
Random Forest = an ensemble algorithm that combines many Decision Trees using the bagging technique
n_estimators = the number of trees; the final prediction is decided by majority vote/averaging

Let's Look at It Together

python

from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=5, random_state=42)
tree.fit(X_train, y_train)

forest = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
forest.fit(X_train, y_train)

print(f"Single tree accuracy: {tree.score(X_test, y_test):.2f}")
print(f"Random Forest accuracy: {forest.score(X_test, y_test):.2f}")

You should see

Single tree accuracy: 0.84
Random Forest accuracy: 0.91

Try It in 5 Minutes

Train and compare a single Decision Tree vs. a Random Forest on the same dataset, and confirm the accuracy improvement — also check out `model.feature_importances_` (the importance ranking of each feature).

A Quick Word of Caution

Assuming the Random Forest's default hyperparameters (`n_estimators`, `max_depth`) are 'good enough' without tuning them for your dataset — you should go on to study the Hyperparameter Tuning lesson in the Advanced chapter.