Machine learningChapter 103 of 114
Features and Labels
Shaping your data into the grid of numbers a model expects.
X is a table, y is a column
Every scikit-learn model wants the same two things:
X— a 2D grid, one row per example and one column per featurey— a 1D sequence with one label per row ofX
import numpy as np
X = np.array([
[5.1, 3.5],
[4.9, 3.0],
[6.2, 3.4],
])
y = np.array([0, 0, 1])
print(X.shape)
print(y.shape)
print(X.shape[0] == y.shape[0])Output
(3, 2) (3,) True
X.shape is (rows, columns). y.shape is (rows,) — one dimension, not a column. Getting those to line up is most of the work of preparing data.
The mismatch you will hit
import numpy as np
from sklearn.tree import DecisionTreeClassifier
X = np.array([[1.0], [2.0], [3.0]])
y = np.array([0, 1])
try:
DecisionTreeClassifier().fit(X, y)
except ValueError as problem:
print(type(problem).__name__)
print("mentions both counts:", "2" in str(problem) and "3" in str(problem))Output
ValueError mentions both counts: True
Three rows of features, two labels. The message names both counts, and the cause is nearly always a filtering step that dropped rows from one and not the other.
From a DataFrame
Real data usually arrives as a table with a column you want to predict. Split it by name:
import pandas as pd
df = pd.DataFrame({
"length": [5.1, 4.9, 6.2, 5.9],
"width": [3.5, 3.0, 3.4, 3.0],
"species": ["a", "a", "b", "b"],
})
X = df[["length", "width"]]
y = df["species"]
print(X.shape, y.shape)
print(list(X.columns))
print(y.tolist())Output
(4, 2) (4,) ['length', 'width'] ['a', 'a', 'b', 'b']
Note the double brackets on df[["length", "width"]]. A single pair gives one column as a 1D Series; the double pair gives a 2D DataFrame, which is what X has to be.
import pandas as pd
df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
print(df["a"].shape)
print(df[["a"]].shape)Output
(2,) (2, 1)
Dropping the target
When you want everything except the label:
import pandas as pd
df = pd.DataFrame({
"length": [5.1, 4.9],
"width": [3.5, 3.0],
"species": ["a", "b"],
})
X = df.drop(columns=["species"])
y = df["species"]
print(list(X.columns))
print(y.tolist())Output
['length', 'width'] ['a', 'b']
Labels can be text
Most scikit-learn models accept string labels directly and remember them:
from sklearn.tree import DecisionTreeClassifier
X = [[1.0], [2.0], [8.0], [9.0]]
y = ["small", "small", "large", "large"]
model = DecisionTreeClassifier(random_state=0).fit(X, y)
print(model.predict([[1.5], [8.5]]))
print(model.classes_)Output
['small' 'large'] ['large' 'small']
classes_ is sorted, which is also the column order of predict_proba. It is worth printing once rather than assuming.
Choosing features
More columns is not better. A feature earns its place if it carries information about the answer that the others do not:
- Useless. A row id, or a column that is the same for everyone.
- Leaking. Anything recorded after the thing you are predicting. A
"cancellation date" cannot help you predict cancellations.
- Duplicated. Height in centimetres and in inches say the same thing twice.
import pandas as pd
df = pd.DataFrame({
"id": [1, 2, 3],
"height_cm": [180, 165, 172],
"height_in": [70.9, 65.0, 67.7],
"country": ["de", "de", "de"],
"bought": [1, 0, 1],
})
for column in df.columns:
unique = df[column].nunique()
print(f"{column:10} {unique} distinct value(s)")Output
id 3 distinct value(s) height_cm 3 distinct value(s) height_in 3 distinct value(s) country 1 distinct value(s) bought 2 distinct value(s)
country is the same for every row, so it can tell the model nothing. id is different for every row, which is just as useless — and worse, a model can memorise it.
Test yourself
3 questionsWhat shape does X have?
Show the answer
Two dimensions: one row per example, one column per feature — y is one dimensional, with one label per row of X. Lining those up is most of preparing data.
What is the difference between df["a"] and df[["a"]]?
Show the answer
The first gives a 1D Series, the second a 2D DataFrame — X has to be 2D, so a single feature column needs the double brackets.
Why is a 'cancellation date' column dangerous when predicting cancellations?
Show the answer
It only exists after the thing you are predicting, so it leaks the answer — Ask of every feature: would I have had this value at the moment I need the prediction?
Train and Test Split
Hold data back, or you cannot tell whether the model learned anything.