Machine learningChapter 112 of 114
Neural Networks
What a network actually is, and when it beats simpler models.
One in scikit-learn
A multi-layer perceptron is the classic neural network, and it has the same interface as everything else:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0, stratify=y
)
model = make_pipeline(
StandardScaler(),
MLPClassifier(hidden_layer_sizes=(16,), max_iter=2000, random_state=0),
)
model.fit(X_train, y_train)
print("accuracy:", round(model.score(X_test, y_test), 3))Output
accuracy: 0.965
The StandardScaler is not optional. Networks are trained by gradient descent, and unscaled features make the gradients wildly uneven.
What is inside
Layers of numbers. Each layer multiplies its input by a matrix of weights, adds a bias, and passes the result through a non-linear function:
from sklearn.datasets import load_breast_cancer
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
X = StandardScaler().fit_transform(X)
model = MLPClassifier(hidden_layer_sizes=(16, 8), max_iter=2000, random_state=0)
model.fit(X, y)
print("layer shapes:", [w.shape for w in model.coefs_])
print("total weights:", sum(w.size for w in model.coefs_))
print("activation:", model.activation)Output
layer shapes: [(30, 16), (16, 8), (8, 1)] total weights: 616 activation: relu
Thirty features in, through sixteen units, then eight, then one output. 616 numbers, all found by training. Nobody chose any of them, and nobody can read them.
Why the non-linearity matters
Stack two linear layers with nothing between them and you have a linear model with extra steps. The activation function is what makes depth worth anything:
import numpy as np
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import LogisticRegression
# exclusive or: no straight line can separate this
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])
linear = LogisticRegression().fit(X, y)
network = MLPClassifier(hidden_layer_sizes=(8,), max_iter=5000, random_state=1).fit(X, y)
print("logistic regression:", linear.score(X, y))
print("neural network: ", network.score(X, y))Output
logistic regression: 0.5 neural network: 1.0
XOR is the standard demonstration. A line cannot separate those four points, and a hidden layer can bend the boundary until it does.
Size and training
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
X, y = load_breast_cancer(return_X_y=True)
for size in [(4,), (32,), (64, 32)]:
model = make_pipeline(
StandardScaler(),
MLPClassifier(hidden_layer_sizes=size, max_iter=2000, random_state=0),
)
scores = cross_val_score(model, X, y, cv=5)
print(f"{str(size):10} {scores.mean():.3f}")Output
(4,) 0.977 (32,) 0.975 (64, 32) 0.977
Barely any difference — the smallest network matches the largest. On thirty tidy features and 569 rows, a bigger network has nothing more to find — and logistic regression scored 0.981 in the Overfitting chapter, with a fraction of the compute and weights you can read.
When a network is worth it
Neural networks win where the features are not given to you and have to be discovered: pixels, audio samples, raw text. A photograph has no "column that means cat", and a network builds one from edges and textures.
| Data | Reach for |
|---|---|
| tabular rows and columns | gradient boosting, or logistic regression |
| images | a convolutional network |
| text | a transformer |
| audio | a convolutional or transformer network |
| small data of any kind | something simpler |
On ordinary spreadsheet data, tree ensembles usually beat neural networks. That is not folklore; it is the repeated result of benchmark after benchmark.
Deep learning
"Deep" just means many layers. The ideas above do not change — what changes is scale, and that scale needs tools scikit-learn does not provide:
python -m pip install torch
python -m pip install tensorflowimport torch
from torch import nn
model = nn.Sequential(
nn.Linear(30, 16),
nn.ReLU(),
nn.Linear(16, 1),
nn.Sigmoid(),
)
print(sum(p.numel() for p in model.parameters()), "parameters")Output, from a real run elsewhere
513 parameters
That is the same shape of model as the scikit-learn one, written explicitly. PyTorch gives you the training loop, GPU support and automatic differentiation; MLPClassifier is the version where all of that is decided for you.
Test yourself
2 questionsWhy does a network need a non-linear activation function?
Show the answer
Without one, stacked layers collapse into a single linear model — XOR is the standard demonstration: no straight line separates it, and one hidden layer can.
For ordinary rows-and-columns data, what usually wins?
Show the answer
Gradient boosting or logistic regression — Networks win where the features must be discovered: pixels, audio, raw text.
Saving and Using a Model
Get a trained model out of your notebook and into something that runs.