Machine learningChapter 109 of 114
Preprocessing
Scaling numbers, encoding categories and filling gaps.
Why scale
Models that measure distance or fit weights are dominated by whichever column has the largest numbers. Here income swamps age, purely because of its units:
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
# [age, income]
X = [[25, 30000], [26, 31000], [55, 30500], [56, 31500]]
y = ["young", "young", "old", "old"]
new = [[27, 31500]]
raw = KNeighborsClassifier(n_neighbors=1).fit(X, y)
print("unscaled:", raw.predict(new)[0])
scaler = StandardScaler().fit(X)
scaled = KNeighborsClassifier(n_neighbors=1).fit(scaler.transform(X), y)
print("scaled: ", scaled.predict(scaler.transform(new))[0])Output
unscaled: old scaled: young
A 27-year-old was called old. Unscaled, the nearest neighbour is whoever has the closest income, and income differences run into the hundreds while age differences are single digits — so age never got a vote. After scaling, both columns have the same spread and the answer flips.
StandardScaler
Shifts each column to a mean of 0 and a spread of 1:
import numpy as np
from sklearn.preprocessing import StandardScaler
X = np.array([[25.0, 30000.0], [26.0, 31000.0], [55.0, 30500.0], [56.0, 31500.0]])
scaled = StandardScaler().fit_transform(X)
print(scaled.round(2))
print("means:", scaled.mean(axis=0).round(2))
print("stds: ", scaled.std(axis=0).round(2))Output
[[-1.03 -1.34] [-0.97 0.45] [ 0.97 -0.45] [ 1.03 1.34]] means: [0. 0.] stds: [1. 1.]
MinMaxScaler squeezes into 0 to 1 instead, which suits bounded data. RobustScaler uses the median and quartiles, which suits data with outliers.
Trees do not need any of this — they split on one column at a time, so units never compete.
fit on train, transform on both
This is the rule that keeps preprocessing honest:
import numpy as np
from sklearn.preprocessing import StandardScaler
train = np.array([[10.0], [20.0], [30.0]])
test = np.array([[40.0]])
scaler = StandardScaler().fit(train)
print("train:", scaler.transform(train).round(2).ravel())
print("test: ", scaler.transform(test).round(2).ravel())Output
train: [-1.22 0. 1.22] test: [2.45]
The scaler learned its mean and spread from the training data only. The test point lands outside that range, which is exactly right — it is new data.
Encoding categories
Models take numbers. A category column has to become one — but not by numbering the values, which invents an order that is not there:
import numpy as np
from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder
colours = np.array([["red"], ["green"], ["blue"], ["red"]])
print(OrdinalEncoder().fit_transform(colours).ravel())
print(OneHotEncoder(sparse_output=False).fit_transform(colours))Output
[2. 1. 0. 2.] [[0. 0. 1.] [0. 1. 0.] [1. 0. 0.] [0. 0. 1.]]
The ordinal version says green is between blue and red and that red is twice green, which is nonsense. One-hot gives each colour its own column and no false ordering.
Use ordinal encoding only when the order is real — small, medium, large — and say so explicitly:
import numpy as np
from sklearn.preprocessing import OrdinalEncoder
sizes = np.array([["medium"], ["small"], ["large"]])
encoder = OrdinalEncoder(categories=[["small", "medium", "large"]])
print(encoder.fit_transform(sizes).ravel())Output
[1. 0. 2.]
Unseen categories
A category the encoder never saw during training will otherwise raise at exactly the wrong moment:
import numpy as np
from sklearn.preprocessing import OneHotEncoder
train = np.array([["red"], ["green"]])
live = np.array([["blue"]])
strict = OneHotEncoder(sparse_output=False).fit(train)
try:
strict.transform(live)
except ValueError:
print("strict encoder refused the unseen colour")
lenient = OneHotEncoder(sparse_output=False, handle_unknown="ignore").fit(train)
print("lenient:", lenient.transform(live)[0])Output
strict encoder refused the unseen colour lenient: [0. 0.]
All zeros means "none of the colours I know". Whether that is better than failing loudly depends on your situation — but decide it deliberately.
Missing values
import numpy as np
from sklearn.impute import SimpleImputer
X = np.array([[1.0, 10.0], [2.0, np.nan], [3.0, 30.0], [np.nan, 40.0]])
for how in ["mean", "median", "most_frequent"]:
filled = SimpleImputer(strategy=how).fit_transform(X)
print(f"{how:14}", filled[:, 1].round(1))Output
mean [10. 26.7 30. 40. ] median [10. 30. 30. 40.] most_frequent [10. 10. 30. 40.]
Filling with the mean is the default and is rarely the best answer. Often the fact that a value is missing is itself informative — a blank income field is not a random omission — in which case add a "was missing" column rather than quietly papering over it.
Test yourself
3 questionsWhy scale features before k nearest neighbours?
Show the answer
Otherwise the column with the largest numbers decides every distance — Trees do not need it, because they split one column at a time.
Why is ordinal encoding wrong for colours?
Show the answer
It invents an order and a scale that do not exist — One-hot gives each value its own column. Use ordinal only when the order is real, like small, medium, large.
Where should a scaler be fitted?
Show the answer
On the training data only, then applied to both — Fitting before the split leaks the test set's statistics into training and flatters the score.
Pipelines
Chain preprocessing and model into one object that cannot leak.