Machine learningChapter 105 of 114
Classification
Predicting which category something belongs to.
The task
The label is one of a fixed set. Spam or not. Which of three species. Which digit. Everything in this chapter predicts a category rather than a number.
Three classifiers, one interface
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0, stratify=y
)
models = {
"decision tree": DecisionTreeClassifier(random_state=0),
"k nearest": KNeighborsClassifier(n_neighbors=5),
"logistic": LogisticRegression(max_iter=1000),
}
for name, model in models.items():
model.fit(X_train, y_train)
print(f"{name:14} {model.score(X_test, y_test):.2f}")Output
decision tree 0.97 k nearest 1.00 logistic 1.00
Same four lines for each. Trying several models is cheap, and worth doing before you spend a day tuning one.
Two of them scored perfectly, which should make you suspicious rather than pleased: iris is a small, clean, famously easy dataset and the test set is only 30 flowers. A perfect score there says more about the data than the model.
Decision tree
A series of yes/no questions on single features. Its great virtue is that you can read it:
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text
data = load_iris()
model = DecisionTreeClassifier(max_depth=2, random_state=0)
model.fit(data.data, data.target)
print(export_text(model, feature_names=list(data.feature_names)))Output
|--- petal width (cm) <= 0.80 | |--- class: 0 |--- petal width (cm) > 0.80 | |--- petal width (cm) <= 1.75 | | |--- class: 1 | |--- petal width (cm) > 1.75 | | |--- class: 2
That is the entire model, in four rules a person can check. Very few model types let you do that, and it is worth a lot when someone asks why.
It also tells you which features mattered:
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
data = load_iris()
model = DecisionTreeClassifier(random_state=0).fit(data.data, data.target)
for name, importance in zip(data.feature_names, model.feature_importances_):
print(f"{name:20} {importance:.2f}")Output
sepal length (cm) 0.00 sepal width (cm) 0.01 petal length (cm) 0.06 petal width (cm) 0.92
Petal width does almost all the work. The sepal measurements could be dropped with little loss.
k nearest neighbours
No training as such: it remembers the examples and, for a new one, looks at the k closest and takes a vote.
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0, stratify=y
)
for k in [1, 5, 15, 100]:
model = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)
print(f"k={k:<4} {model.score(X_test, y_test):.2f}")Output
k=1 0.97 k=5 1.00 k=15 1.00 k=100 0.67
k=1 follows every example exactly, noise included. Larger k smooths that out — until it smooths the real pattern away too. At k=100, out of 120 training flowers, almost every neighbourhood contains most of the dataset, so the model answers nearly the same thing every time.
Logistic regression
Despite the name it is a classifier. It fits a weight per feature and squashes the result into a probability:
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
data = load_iris()
model = LogisticRegression(max_iter=1000).fit(data.data, data.target)
print(model.coef_.shape)
print(model.predict_proba([[5.1, 3.5, 1.4, 0.2]])[0].round(2))
print(data.target_names[model.predict([[5.1, 3.5, 1.4, 0.2]])[0]])Output
(3, 4) [0.98 0.02 0. ] setosa
Three classes, four features, so twelve weights. It is fast, it gives calibrated probabilities, and it is the sensible thing to try first on tabular data.
Two classes
The most common shape in practice, and the one where the vocabulary matters:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=0, stratify=data.target
)
model = LogisticRegression(max_iter=5000).fit(X_train, y_train)
print([str(name) for name in data.target_names])
print("accuracy:", round(model.score(X_test, y_test), 2))
print("first five predictions:", model.predict(X_test[:5]))Output
['malignant', 'benign'] accuracy: 0.95 first five predictions: [0 0 0 1 0]
Ninety-five percent sounds good. Whether it is good depends entirely on what the errors are, and which kind of error matters — missing a malignant tumour is not the same as a false alarm. That is the Evaluating chapter.
Test yourself
2 questionsWhat can you do with a decision tree that you cannot do with most models?
Show the answer
Print the whole model and read its rules — export_text prints the entire model. That is worth a great deal when someone asks why.
Which model needs its features scaled?
Show the answer
k nearest neighbours, because it measures distance — Trees split one column at a time, so units never compete. Anything distance-based or weight-based does care.
Regression
Predicting a number rather than a category.