Data Preprocessing Basics

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand Data Preprocessing Basics, without any of the intimidation
Be able to run scikit-learn code yourself
Be able to apply this concept immediately in a real project

Let's think about it this way for a second

There are several ways to handle Missing Values — dropping the row/column (you can lose a lot of data this way), or filling with the mean/median value (imputation) — the right choice depends on the characteristics of the data. Feature Scaling means standardizing the value ranges across columns (features) — for example, if you feed 'age' (0-100) and 'income' (0-1,000,000) into a model at mismatched scales, the income column can end up dominating the weight (for certain algorithms) — StandardScaler/MinMaxScaler are used to bring all features onto a comparable scale.

Let's connect it to a real scenario

In a Pandas DataFrame, `df.isnull().sum()` shows you how many missing values each column has — writing `df['age'].fillna(df['age'].mean())` fills missing age values with the average age. Writing `from sklearn.preprocessing import StandardScaler; scaler = StandardScaler(); X_scaled = scaler.fit_transform(X)` scales all feature columns to mean=0, std=1.

Missing Value handling = dropping (loses data) vs. imputation (mean/median fill)
Feature Scaling = standardizing column value ranges (StandardScaler/MinMaxScaler)
Features with mismatched scales can give certain algorithms an 'unfair weighting'

Let's look at it together

python

import pandas as pd
from sklearn.preprocessing import StandardScaler

df = pd.read_csv("houses.csv")

# Check for missing values
print(df.isnull().sum())

# Fill missing values with the column mean
df['age'] = df['age'].fillna(df['age'].mean())

# Scale numeric features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df[['age', 'income']])

You should see

age        3
income     0
price      0
dtype: int64

5-minute try-it

Load a CSV file (sample data with a few missing values) using Pandas, check the missing values with `isnull().sum()`, then try filling them in with `fillna()`.

A quick word of caution

Feature Scaling matters less for tree-based algorithms (Decision Tree, Random Forest), since they're scale-invariant — but for Linear Regression, KNN, and SVM (covered in the Advanced chapter), scaling can noticeably affect performance.

Easy traps

Calling `fit_transform()` separately on the test data for Feature Scaling — you should only call `fit()` on the training data's scaler, then call `transform()` (not fit) on the test data, to avoid data leakage
Randomly choosing to drop or fill missing values without considering the data's context — you should understand why the data is missing (random vs. systematic) before deciding

Now try it yourself

Load a CSV file (sample data with a few missing values) using Pandas, check the missing values with `isnull().sum()`, then try filling them in with `fillna()`.

You'll know it worked when: age 3 income 0 price 0 dtype: int64