Machine learningChapter 107 of 114
Evaluating a Model
Accuracy hides more than it shows. What to look at instead.
Accuracy can be worthless
Imagine a disease affecting one person in a hundred. Here is a model that never predicts it:
import numpy as np
from sklearn.metrics import accuracy_score
actual = np.array([0] * 99 + [1])
never = np.zeros(100, dtype=int)
print("accuracy:", accuracy_score(actual, never))
print("times it found the disease:", int(never[actual == 1].sum()))Output
accuracy: 0.99 times it found the disease: 0
Ninety-nine percent accurate and completely useless. Whenever classes are imbalanced, accuracy tells you about the majority class and nothing else.
The confusion matrix
Count each kind of outcome instead:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=0, stratify=data.target
)
model = LogisticRegression(max_iter=5000).fit(X_train, y_train)
predicted = model.predict(X_test)
matrix = confusion_matrix(y_test, predicted)
print(matrix)
print("labels:", [str(n) for n in data.target_names])Output
[[39 3] [ 3 69]] labels: ['malignant', 'benign']
Rows are the truth, columns are the prediction. So:
| predicted malignant | predicted benign | |
|---|---|---|
| actually malignant | 39 correct | 3 missed |
| actually benign | 3 false alarms | 69 correct |
Three tumours called benign that were not. In this setting that is the number that matters, and accuracy of 0.95 never mentioned it.
Precision and recall
Two questions about the errors, and they pull against each other:
- Precision — of the things I flagged, how many were real? High precision
means few false alarms.
- Recall — of the real things, how many did I find? High recall means few
missed.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score, f1_score
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=0, stratify=data.target
)
model = LogisticRegression(max_iter=5000).fit(X_train, y_train)
predicted = model.predict(X_test)
# treat malignant (0) as the class we are trying to catch
print("precision:", round(precision_score(y_test, predicted, pos_label=0), 2))
print("recall: ", round(recall_score(y_test, predicted, pos_label=0), 2))
print("f1: ", round(f1_score(y_test, predicted, pos_label=0), 2))Output
precision: 0.93 recall: 0.93 f1: 0.93
F1 is their harmonic mean, a single number when you need one. It is low unless both are decent, which is the point.
Which to favour depends entirely on the cost of each mistake:
| Situation | Care most about |
|---|---|
| screening for cancer | recall — missing one is far worse than a scare |
| flagging spam | precision — a lost real email is worse than spam getting through |
| recommending a film | precision — nobody minds the ones you did not suggest |
The whole report
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report
data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.3, random_state=0, stratify=data.target
)
model = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
names = [str(n) for n in data.target_names]
print(classification_report(y_test, model.predict(X_test), target_names=names))Output
precision recall f1-score support
setosa 1.00 1.00 1.00 15
versicolor 0.94 1.00 0.97 15
virginica 1.00 0.93 0.97 15
accuracy 0.98 45
macro avg 0.98 0.98 0.98 45
weighted avg 0.98 0.98 0.98 45One call, per class, with the counts. It is the first thing to print for any classifier, and it immediately shows that setosa is trivial while the other two get confused with each other.
Moving the threshold
A classifier gives a probability; turning that into a decision is a choice you make, and the default of 0.5 is only a default:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=0, stratify=data.target
)
model = LogisticRegression(max_iter=5000).fit(X_train, y_train)
malignant_chance = model.predict_proba(X_test)[:, 0]
for threshold in [0.05, 0.5, 0.95]:
flagged = (malignant_chance >= threshold).astype(int)
truth = (y_test == 0).astype(int)
print(f"threshold {threshold} precision {precision_score(truth, flagged):.2f}"
f" recall {recall_score(truth, flagged):.2f}")Output
threshold 0.05 precision 0.76 recall 1.00 threshold 0.5 precision 0.93 recall 0.93 threshold 0.95 precision 1.00 recall 0.86
At 0.05 the model finds every malignant tumour, at the cost of a quarter of its alarms being false. At 0.95 every alarm is real, and it misses one in seven.
There is no correct threshold, only the one that matches what each mistake costs you. For cancer screening you would take the first row without hesitating.
Test yourself
3 questionsA model predicts 'no disease' every time and scores 99% accuracy. What is wrong?
Show the answer
The classes are imbalanced, so accuracy measures the majority class — It never finds the thing it was built to find. Look at recall on the class that matters.
What is recall?
Show the answer
Of the real cases, how many did the model find — The other one is precision. F1 is their harmonic mean, and is low unless both are decent.
Why lower the decision threshold below 0.5?
Show the answer
To catch more real cases, accepting more false alarms — There is no correct threshold, only the one matching what each kind of mistake costs you.
Overfitting and Underfitting
Memorising the training data, or not learning it at all.