Where It Goes Wrong

A suspiciously good score is a bug report

Almost every model that scores far better than expected is cheating, and almost always the cause is that information about the answer reached the features.

Leakage from the target

The most obvious form, and it happens more than you would think:

import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(0)
n = 300

# whether someone cancels is pure chance here: age and visits say nothing
y = rng.integers(0, 2, n)
df = pd.DataFrame({
    "age": rng.integers(20, 70, n),
    "visits": rng.integers(1, 10, n),
    "cancelled_on": ["2026-01-04" if cancelled else "" for cancelled in y],
})
df["has_cancel_date"] = (df["cancelled_on"] != "").astype(int)

leaky = cross_val_score(DecisionTreeClassifier(random_state=0),
                        df[["age", "visits", "has_cancel_date"]], y, cv=5)
honest = cross_val_score(DecisionTreeClassifier(random_state=0),
                         df[["age", "visits"]], y, cv=5)

print("with the cancellation date:", round(leaky.mean(), 2))
print("without it:               ", round(honest.mean(), 2))

with the cancellation date: 1.0
without it:                0.53

A perfect score on data where the outcome is a coin flip, because "has a cancellation date" is "cancelled". Drop that column and the model correctly reports that it knows nothing.

The question to ask about every feature: would I have had this value at the moment I need the prediction? A cancellation date exists only after the cancellation.

Leakage from preprocessing

Covered in the Pipelines chapter and worth repeating: fitting a scaler, an imputer or a feature selector on all your data before splitting lets the test set influence training. Put every step inside a Pipeline and it cannot happen.

Imbalanced classes

When one class is rare, accuracy stops meaning anything and the model learns to ignore it:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score

rng = np.random.default_rng(0)
X = rng.normal(size=(1000, 5))
y = (rng.random(1000) < 0.02).astype(int)
X[y == 1] += 0.8

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0, stratify=y
)

plain = LogisticRegression(max_iter=1000).fit(X_train, y_train)
weighted = LogisticRegression(max_iter=1000, class_weight="balanced").fit(X_train, y_train)

for name, model in [("plain", plain), ("balanced", weighted)]:
    predicted = model.predict(X_test)
    print(f"{name:9} accuracy {model.score(X_test, y_test):.3f}"
          f"  recall on the rare class {recall_score(y_test, predicted):.2f}")

plain     accuracy 0.980  recall on the rare class 0.00
balanced  accuracy 0.820  recall on the rare class 0.67

The plain model is 98% accurate and never once finds the thing you built it to find. class_weight="balanced" makes the rare class count for more, trading overall accuracy for the recall you actually wanted.

Always look at recall on the class that matters, never accuracy alone.

The data decides what the model learns

A model reproduces the patterns in its training data, including the ones you did not intend:

from sklearn.tree import DecisionTreeClassifier

# past hiring decisions, with a biased history
# features: [years experience, went to the favoured university]
X = [[2, 1], [8, 0], [3, 1], [9, 0], [1, 1], [7, 0]]
y = ["hired", "rejected", "hired", "rejected", "hired", "rejected"]

model = DecisionTreeClassifier(random_state=0).fit(X, y)

print("10 years, wrong university:", model.predict([[10, 0]])[0])
print("1 year, right university:  ", model.predict([[1, 1]])[0])
print("importances:", model.feature_importances_.round(2))

10 years, wrong university: rejected
1 year, right university:   hired
importances: [0. 1.]

The importances say it outright: experience counts for nothing and the university counts for everything. The model learned the bias perfectly, because the bias was the strongest pattern in the data. It is not making a mistake — it is doing exactly what it was asked.

Removing the offending column rarely helps, because other columns stand in for it: a postcode can carry the same information as ethnicity. This is a data and policy problem, and no amount of model tuning solves it.

The world moves

A model assumes tomorrow resembles the data it was trained on. When that stops being true, the score silently degrades:

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)

X_old = rng.normal(loc=0.0, size=(400, 3))
y_old = (X_old[:, 0] > 0).astype(int)
model = LogisticRegression().fit(X_old, y_old)

X_same = rng.normal(loc=0.0, size=(200, 3))
X_shifted = rng.normal(loc=2.5, size=(200, 3))

print("same world: ", round(model.score(X_same, (X_same[:, 0] > 0).astype(int)), 2))
print("shifted:    ", round(model.score(X_shifted, (X_shifted[:, 0] > 2.5).astype(int)), 2))

same world:  0.99
shifted:     0.47

Nothing broke and nothing raised. The inputs moved, the rule the model learned no longer applies, and the only way to know is to keep measuring.

Monitor the score on fresh labelled data, and watch the distribution of the inputs too — that shifts before the score does.

Tuning on the test set

Choose a setting by test score and the test set has quietly joined training. The number it reports is then optimistic, and you have no honest estimate left.

Use cross-validation on the training data to choose, and touch the test set once, at the very end, to report.

The checklist

Before believing a model:

1. Could any feature contain information from after the prediction moment?
2. Is every preprocessing step inside the pipeline?
3. Is the score better than always predicting the majority class?
4. For the class that matters, what is the recall?
5. Did you choose anything at all by looking at the test score?
6. Who is under-represented in the training data?
7. What happens when the world changes, and how would you find out?

A model that survives all seven is worth deploying. Most first attempts do not survive the first one.