Overfitting and Underfitting

The two failures

Underfitting — the model is too simple to capture the pattern. It does

badly on training data and on new data.

Overfitting — the model is complex enough to memorise the training data,

noise included. It does brilliantly on it and badly on anything new.

You diagnose both by comparing the two scores:

Python needs scikit-learn, downloaded on first run

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

# 15% of the labels are deliberately wrong, as real data always is
X, y = make_classification(
    n_samples=400, n_features=20, n_informative=5, flip_y=0.15, random_state=0
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0, stratify=y
)

print("depth  train   test")
for depth in [1, 3, 8, None]:
    model = DecisionTreeClassifier(max_depth=depth, random_state=0)
    model.fit(X_train, y_train)
    label = str(depth) if depth else "full"
    print(f"{label:6} {model.score(X_train, y_train):.2f}    {model.score(X_test, y_test):.2f}")

Output

depth  train   test
1      0.81    0.82
3      0.89    0.84
8      0.99    0.78
full   1.00    0.77

Read the two columns together. At depth 1 the model is too simple to have learned much, and the two scores sit close together. Depth 3 is the sweet spot.

After that the training score climbs to a perfect 1.00 while the test score falls, from 0.84 to 0.77. Those extra branches went on memorising the 15% of labels that are deliberately wrong, and noise does not generalise.

That widening gap between the columns is what overfitting looks like, and it is why you never judge a model on the data it trained on.

One test score is noisy

A single split is one sample. Change the seed and the number moves:

Python needs scikit-learn, downloaded on first run

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)
scores = []

for seed in range(5):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=seed, stratify=y
    )
    model = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
    scores.append(round(model.score(X_test, y_test), 2))

print(scores)
print("spread:", min(scores), "to", max(scores))

Output

[0.97, 0.97, 0.97, 0.9, 0.97]
spread: 0.9 to 0.97

Seven points of spread from nothing but the split. Reporting whichever number you happened to get is how models come to be oversold.

Cross-validation

Split the data into k parts, train on k-1 and test on the one left out, k times. Every example gets used for testing exactly once:

Python needs scikit-learn, downloaded on first run

from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_iris(return_X_y=True)
model = DecisionTreeClassifier(random_state=0)

scores = cross_val_score(model, X, y, cv=5)

print(scores.round(2))
print("mean:", scores.mean().round(3), "std:", scores.std().round(3))

Output

[0.97 0.97 0.9  0.97 1.  ]
mean: 0.96 std: 0.033

Five numbers instead of one, and a standard deviation telling you how much to trust the mean. This is the right way to report a model's performance when data is limited.

Comparing models honestly

Python needs scikit-learn, downloaded on first run

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)

for name, model in [("tree  ", DecisionTreeClassifier(random_state=0)),
                    ("forest", RandomForestClassifier(n_estimators=50, random_state=0))]:
    scores = cross_val_score(model, X, y, cv=5)
    print(f"{name} {scores.mean():.3f} +/- {scores.std():.3f}")

Output

tree   0.917 +/- 0.016
forest 0.963 +/- 0.017

The forest is genuinely better here: the gap between the means is larger than the spread of either. When two means differ by less than their standard deviations, you have not shown anything.

Fixing each one

Underfitting — the model cannot represent the pattern:

use a more flexible model, or let it grow deeper
add features that actually carry information
train longer, if it is the kind of model that iterates

Overfitting — it has learned the noise:

more data is the best fix, and usually the hardest
simplify: shallower trees, fewer features, stronger regularisation
for a random forest, more trees rather than deeper ones

Python needs scikit-learn, downloaded on first run

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = load_breast_cancer(return_X_y=True)

for strength in [0.01, 1.0, 100.0]:
    model = make_pipeline(StandardScaler(), LogisticRegression(C=strength, max_iter=5000))
    scores = cross_val_score(model, X, y, cv=5)
    print(f"C={strength:<7} {scores.mean():.3f}")

Output

C=0.01    0.949
C=1.0     0.981
C=100.0   0.963

C is the inverse of the regularisation strength: small C means a heavily constrained, simpler model. Sweeping a setting like this and reading the cross-validated mean is the everyday shape of model tuning.

The two failures

One test score is noisy

Cross-validation

Comparing models honestly

Fixing each one

Test yourself

How do you recognise overfitting?

What does cross-validation give you that one train/test split does not?

What is the best fix for overfitting when you can get it?

Preprocessing