Machine learningChapter 106 of 114
Regression
Predicting a number rather than a category.
The task
The label is a number on a continuous scale: a price, a temperature, a duration. The interface is identical to classification — only the model and the way you score it change.
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0
)
model = LinearRegression().fit(X_train, y_train)
print("first three predictions:", model.predict(X_test[:3]).round(1))
print("actual: ", y_test[:3].round(1))Output
first three predictions: [238.5 248.9 164.1] actual: [321. 215. 127.]
The predictions are in the right neighbourhood and none is exactly right. That is normal, and it is why regression is scored by how far off rather than by "correct or not".
The line it fitted
A linear model gives one weight per feature, plus an intercept. Those numbers are the model:
from sklearn.linear_model import LinearRegression
# hours studied -> exam score
X = [[1], [2], [3], [4], [5]]
y = [52, 55, 61, 64, 68]
model = LinearRegression().fit(X, y)
print("slope: ", round(model.coef_[0], 2))
print("intercept:", round(model.intercept_, 2))
print("predict 6 hours:", round(model.predict([[6]])[0], 1))Output
slope: 4.1 intercept: 47.7 predict 6 hours: 72.3
Each extra hour is worth about 4.1 marks, and someone studying nothing scores about 47.7. You can read a linear model out loud, which is a large part of why it is still used.
Reading the weights
With several features, the weights say which ones the model leans on — but only if they are on comparable scales:
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
data = load_diabetes()
model = LinearRegression().fit(data.data, data.target)
pairs = sorted(zip(data.feature_names, model.coef_), key=lambda p: -abs(p[1]))
for name, weight in pairs[:4]:
print(f"{name:6} {weight:8.1f}")Output
s1 -792.2 s5 751.3 bmi 519.8 s2 476.7
Scoring
Three numbers, answering different questions:
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0
)
model = LinearRegression().fit(X_train, y_train)
predicted = model.predict(X_test)
print("MAE: ", round(mean_absolute_error(y_test, predicted), 1))
print("RMSE:", round(mean_squared_error(y_test, predicted) ** 0.5, 1))
print("R2: ", round(r2_score(y_test, predicted), 2))Output
MAE: 46.2 RMSE: 58.5 R2: 0.33
- MAE — the average error, in the units of the target. Off by 46.2 on
average. The easiest to explain to anyone.
- RMSE — squares the errors first, so a few large misses hurt much more than
many small ones. Always at least as big as the MAE.
- R² — the fraction of the variation the model explains. 1.0 is perfect,
0 is no better than always guessing the mean, and negative is worse than that.
R² of 0.33 means this model captures about a third of what is going on. For ten routine measurements predicting disease progression a year later, that is a real signal and nowhere near a solution.
R² can be negative
from sklearn.metrics import r2_score
actual = [10, 20, 30, 40]
print("perfect: ", r2_score(actual, [10, 20, 30, 40]))
print("always mean: ", r2_score(actual, [25, 25, 25, 25]))
print("actively bad: ", r2_score(actual, [40, 30, 20, 10]))Output
perfect: 1.0 always mean: 0.0 actively bad: -3.0
model.score() on a regressor returns R², which is why a regression score is not a percentage and cannot be read as one.
When a line is not enough
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0
)
for name, model in [("linear", LinearRegression()),
("forest", RandomForestRegressor(random_state=0))]:
model.fit(X_train, y_train)
print(f"{name:7} R2 {model.score(X_test, y_test):.2f}")Output
linear R2 0.33 forest R2 0.27
The more complicated model did not win. That happens often, and it is why you start with the simple one: it is faster, it is readable, and it is the baseline anything else has to beat.
Test yourself
2 questionsWhat does an R² of 0 mean?
Show the answer
No better than always predicting the mean — R² can also be negative, which means worse than guessing the mean every time.
Why is a large coefficient not proof that a feature matters?
Show the answer
Its size depends on the feature's units unless everything is scaled — Change a column from metres to millimetres and its weight shrinks by a thousand while the model is unchanged.
Evaluating a Model
Accuracy hides more than it shows. What to look at instead.