Machine learningChapter 110 of 114
Pipelines
Chain preprocessing and model into one object that cannot leak.
The problem they solve
Do preprocessing by hand and you have to remember to apply exactly the same steps, fitted on exactly the training data, to every future input. A pipeline makes that structural rather than a matter of discipline:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0, stratify=y
)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
print("accuracy:", round(model.score(X_test, y_test), 3))
print("steps:", [name for name, _ in model.steps])Output
accuracy: 0.982 steps: ['standardscaler', 'logisticregression']
fit fits the scaler on the training data and then the model on the scaled result. predict applies the already fitted scaler and then the model. One object, and no way to get the order wrong.
Why it prevents leakage
Cross-validation with a hand-fitted scaler is quietly wrong, because the scaler has already seen every fold:
Here is the mistake at its most dramatic. The data below is pure noise with random labels — there is nothing whatsoever to learn:
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(0)
X = rng.normal(size=(60, 2000))
y = rng.integers(0, 2, size=60)
# wrong: pick the best 5 columns using the whole dataset, labels included,
# and only then cross-validate
leaked = SelectKBest(f_classif, k=5).fit_transform(X, y)
wrong = cross_val_score(LogisticRegression(), leaked, y, cv=5)
# right: the selection happens inside each fold, seeing only that fold's
# training data
right = cross_val_score(
make_pipeline(SelectKBest(f_classif, k=5), LogisticRegression()), X, y, cv=5
)
print("leaked:", round(wrong.mean(), 2))
print("clean: ", round(right.mean(), 2))Output
leaked: 0.8 clean: 0.63
Eighty percent, from data containing no signal at all. Choosing the five columns that best match the labels used every label in the dataset, so by the time cross-validation ran the answers were already baked into the features.
The clean number is far lower, and what remains above 0.5 is small-sample noise: with sixty examples split five ways, each fold is judged on twelve coin flips. The honest expectation is 0.5, and the leaked figure is thirty points of pure fiction.
Scaling leaks far less than this. Feature selection, imputation and target encoding leak like a sieve. Put every step inside the pipeline and none of them can.
Naming the steps
Pipeline lets you name them, which makes the settings readable:
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
model = Pipeline([
("scale", StandardScaler()),
("classify", LogisticRegression(max_iter=1000)),
])
print([name for name, _ in model.steps])
print(type(model.named_steps["classify"]).__name__)Output
['scale', 'classify'] LogisticRegression
Different columns, different treatment
Real tables mix numbers and categories. ColumnTransformer routes each group to its own preprocessing and glues the results back together:
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
df = pd.DataFrame({
"age": [25, 40, 55, 30, 60, 45],
"income": [30000, 52000, 61000, 33000, 70000, 48000],
"city": ["berlin", "munich", "berlin", "munich", "berlin", "munich"],
})
y = [0, 1, 1, 0, 1, 1]
prepare = ColumnTransformer([
("numbers", StandardScaler(), ["age", "income"]),
("categories", OneHotEncoder(handle_unknown="ignore"), ["city"]),
])
model = Pipeline([("prepare", prepare), ("classify", LogisticRegression())])
model.fit(df, y)
print("columns after preparing:", prepare.fit_transform(df).shape[1])
print("prediction:", model.predict(pd.DataFrame(
[{"age": 28, "income": 31000, "city": "berlin"}]
))[0])Output
columns after preparing: 4 prediction: 0
Two scaled numbers plus two one-hot columns for the cities. The whole thing takes a DataFrame in and gives a prediction out, so deployment is one object rather than a document describing six manual steps.
Tuning the whole thing
Because a pipeline is a model, it goes straight into a search. Use a double underscore to reach a step's setting:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_breast_cancer(return_X_y=True)
model = Pipeline([
("scale", StandardScaler()),
("classify", LogisticRegression(max_iter=5000)),
])
search = GridSearchCV(model, {"classify__C": [0.01, 1.0, 100.0]}, cv=5)
search.fit(X, y)
print("best C:", search.best_params_["classify__C"])
print("best score:", round(search.best_score_, 3))Output
best C: 1.0 best score: 0.981
classify__C means "the C of the step called classify". Because the scaler is inside the pipeline, it is refitted within every fold of every candidate, so the search is honest.
Saving it
import joblib
from sklearn.datasets import load_iris
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X, y)
joblib.dump(model, "model.joblib")
loaded = joblib.load("model.joblib")
print(loaded.predict([[5.1, 3.5, 1.4, 0.2]])[0])
print(type(loaded).__name__)Output
0 Pipeline
The scaler's learned statistics travel with the model. Saving a bare estimator and forgetting its preprocessing is a classic way to have a model that works in your notebook and nowhere else.
Test yourself
2 questionsWhat does a Pipeline guarantee?
Show the answer
Every step is fitted on training data only, and reapplied in the same order — It turns a matter of discipline into something structural.
What does classify__C mean in a grid search?
Show the answer
The C setting of the pipeline step named classify — Double underscore reaches into a step. Because the pipeline refits inside every fold, the search stays honest.
Clustering
Finding groups in data that has no labels at all.