Machine learningChapter 111 of 114
Clustering
Finding groups in data that has no labels at all.
No answers to learn from
Everything so far had a y. Clustering does not: you hand it X and ask what natural groups are in there.
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
X, _ = load_iris(return_X_y=True)
model = KMeans(n_clusters=3, random_state=0, n_init=10).fit(X)
print("labels for the first ten:", model.labels_[:10])
print("cluster sizes:", [int((model.labels_ == c).sum()) for c in range(3)])Output
labels for the first ten: [1 1 1 1 1 1 1 1 1 1] cluster sizes: [62, 50, 38]
Note the underscore: we deliberately threw the species away. The model found three groups from the measurements alone.
The numbers mean nothing
Cluster 0 is not "the first species". The labels are arbitrary names for groups, and they change with the seed:
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
X, _ = load_iris(return_X_y=True)
a = KMeans(n_clusters=3, random_state=0, n_init=10).fit(X).labels_
b = KMeans(n_clusters=3, random_state=7, n_init=10).fit(X).labels_
print("identical labels:", (a == b).all())
print("same grouping:", len(set(zip(a, b))) == 3)Output
identical labels: False same grouping: True
Different numbers, same three groups. Never compare cluster ids across runs; compare which points ended up together.
Did it find the species?
We do have the true species here, so we can check — which is a luxury real clustering never has:
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from collections import Counter
data = load_iris()
labels = KMeans(n_clusters=3, random_state=0, n_init=10).fit(data.data).labels_
for cluster in range(3):
species = Counter(data.target[labels == cluster])
breakdown = ", ".join(
f"{str(data.target_names[s])} {n}" for s, n in sorted(species.items())
)
print(f"cluster {cluster}: {breakdown}")Output
cluster 0: versicolor 48, virginica 14 cluster 1: setosa 50 cluster 2: versicolor 2, virginica 36
Setosa came out perfectly on its own. The other two overlap, and the model split them somewhere that is not quite the species boundary — because nothing told it where that boundary was.
Centres
k-means puts k centres and assigns each point to the nearest, repeatedly, until nothing moves:
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
data = load_iris()
model = KMeans(n_clusters=3, random_state=0, n_init=10).fit(data.data)
print(model.cluster_centers_.round(1))
print(data.feature_names[2])Output
[[5.9 2.7 4.4 1.4] [5. 3.4 1.5 0.2] [6.8 3.1 5.7 2.1]] petal length (cm)
Each row is an average flower for that group. Reading the centres is usually how you work out what a cluster means — here, small petals versus large.
Choosing k
You have to say how many groups to look for, and there is no correct answer. Inertia — the total distance from points to their centre — always falls as k rises, so you look for the bend:
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
X, _ = load_iris(return_X_y=True)
for k in range(1, 7):
model = KMeans(n_clusters=k, random_state=0, n_init=10).fit(X)
print(f"k={k} inertia {model.inertia_:7.1f}")Output
k=1 inertia 681.4 k=2 inertia 152.3 k=3 inertia 78.9 k=4 inertia 57.2 k=5 inertia 46.4 k=6 inertia 39.0
The huge drop is from 1 to 2, then 2 to 3, and after that it flattens. That bend is the "elbow", and it is a judgement call rather than a calculation.
The silhouette score is a second opinion, and it does have a best value:
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
X, _ = load_iris(return_X_y=True)
for k in range(2, 6):
labels = KMeans(n_clusters=k, random_state=0, n_init=10).fit_predict(X)
print(f"k={k} silhouette {silhouette_score(X, labels):.3f}")Output
k=2 silhouette 0.681 k=3 silhouette 0.553 k=4 silhouette 0.498 k=5 silhouette 0.489
It prefers 2, because setosa is so separated that splitting the other two barely helps. Two reasonable methods, two different answers — which is clustering.
It always finds something
Give k-means noise and ask for three groups, and it will give you three groups:
import numpy as np
from sklearn.cluster import KMeans
rng = np.random.default_rng(0)
noise = rng.normal(size=(150, 4))
model = KMeans(n_clusters=3, random_state=0, n_init=10).fit(noise)
print("cluster sizes:", sorted(int((model.labels_ == c).sum()) for c in range(3)))Output
cluster sizes: [46, 50, 54]
Three neat groups, from data with no structure whatsoever. Clustering never tells you whether the groups are real — that is your job, by looking at the centres and asking whether they describe anything meaningful.
Test yourself
3 questionsWhat does clustering need that supervised learning also needs?
Show the answer
Nothing extra; clustering needs no labels at all — You hand it X and ask what groups are in there. There is no y.
Two runs give different cluster numbers for the same points. What does that mean?
Show the answer
Nothing; the labels are arbitrary names for the groups — Never compare cluster ids across runs. Compare which points ended up together.
What does k-means do when given pure noise and asked for three clusters?
Show the answer
Returns three neat clusters anyway — It never tells you whether the groups are real. Reading the centres and asking if they mean anything is your job.
Neural Networks
What a network actually is, and when it beats simpler models.