Overfitting: how to spot it and what to do about it

You have trained your first model and the training loss is low. Great, right? It depends. The question that matters is a different one: has it learned something that works on new data, or has it simply memorised the data you gave it?

What it means when your model memorises

Picture someone preparing for an exam by learning last year’s answers by heart. If this year’s paper is identical, they will score full marks. If a single question changes, they fall apart. They haven’t learned the subject: they have memorised the exam.

That is overfitting. The scikit-learn documentation illustrates it with an extreme case: a model that simply repeated the labels of the samples it has already seen would get a perfect score and yet predict nothing useful on new data. The same page calls it a methodological mistake to learn the parameters of a prediction function and then test it on the same data.

That gives us the golden rule: to know whether a model generalises, you have to measure it on data it has not seen during training. I won’t repeat here how the weights get adjusted; if you need a refresher on gradient descent, it is covered in the beginner’s guide to neural networks. What interests us here is what to do when that adjustment goes too far.

Splitting the data: training, validation and test

The usual practice is to hold out part of the data as a test set, according to scikit-learn’s cross-validation guide. But that alone is not enough, and the reason is subtle.

If you try several configurations (number of layers, learning rate, whatever it may be) and keep the one that scores best on the test set, knowledge of the test set “leaks” into the model. Without realising it, you have tuned your choice to that data, and the metrics no longer reflect the model’s real ability to generalise.

The solution is three sets:

Training: used to adjust the weights.
Validation: used to compare configurations and decide when to stop.
Test: used only once, at the end, when the experiment looks good, for the final evaluation.

A warning for later: if your data comes in groups (for example, several samples from each person), a sufficiently flexible model can learn features specific to each subject and fail on new subjects. Scikit-learn offers GroupKFold, which prevents the same group from appearing in both training and test and makes it possible to detect that kind of overfitting. It’s enough for now that you keep it on your radar.

What overfitting looks like in the loss curves

While you train, record the training loss and the validation loss at the end of each epoch (one full pass over the training data) and plot both. The typical pattern is this:

Illustrative diagram with no real values. The training curve falls throughout all the epochs. The validation curve falls at first, reaches a minimum around the middle, and then rises again, so the gap between the two curves widens.validation minimumepochslosstrainingvalidation

This behaviour is described in a paper on early stopping: in high-capacity models, the training loss falls throughout the process, while the test loss saturates at some point and then starts to rise again.

To read the plot, look for two signals:

The point where validation stops improving. From there on, whatever the model keeps gaining on the training set no longer translates into anything useful.
The growing gap between the two curves. Some difference is normal; what should put you on alert is that it widens while validation stalls or gets worse.

One honest caveat: the sources consulted do not set a numerical threshold for “too big a gap”. It is a judgement you make by looking at the shape of the curves, not a magic number. And the drawing above is a diagram, not data from a real experiment.

A minimal example to see it for yourself

You don’t need a neural network to see the phenomenon. A high-degree polynomial fitted to a few noisy points is a model with a lot of capacity, and it behaves the same way. This code uses only NumPy:

import numpy as np

rng = np.random.default_rng(0)

# Little data: a smooth curve plus noise
x = np.linspace(-1, 1, 30)
y = np.sin(3 * x) + rng.normal(0, 0.2, size=x.shape)

# Shuffle and split: 20 for training, 10 for validation
orden = rng.permutation(len(x))
idx_train, idx_val = orden[:20], orden[20:]

def error_cuadratico_medio(coef, idx):
    prediccion = np.polyval(coef, x[idx])
    return np.mean((prediccion - y[idx]) ** 2)

# Higher degree = more capacity to stick to the data
errores_val = {}
for grado in (1, 3, 5, 9, 15):
    coef = np.polyfit(x[idx_train], y[idx_train], grado)
    err_train = error_cuadratico_medio(coef, idx_train)
    err_val = error_cuadratico_medio(coef, idx_val)
    print(f"degree {grado:2d} | train {err_train:.4f} | val {err_val:.4f}")
    errores_val[grado] = err_val
assert errores_val[15] > errores_val[5], "degree 15 should overfit"

Step by step:

np.random.default_rng(0) fixes the seed so the result is reproducible.
We generate 30 points from a sine curve with noise and split them at random: 20 for training and 10 for validation. The model never fits on the validation points.
np.polyfit finds the coefficients of the polynomial that best fits the training data; np.polyval evaluates it.
The loop raises the degree, that is, the model’s capacity, and measures the error on both partitions.

Run it and take a look. What you should expect to see, in qualitative terms, is that the training error keeps falling as the degree goes up, while the validation error improves at first and ends up getting worse at the highest degrees. With so little data, the details depend on the seed and the split; if you don’t see it first time, change the seed or reduce the training data. The degree-1 model is the opposite case: too simple to capture the curve (underfitting).

What to do: L1 and L2 regularisation, and dropout

A common defence is to add to the function being minimised a term that penalises the parameter vector w, normally with the l1 or l2 norm, and that limits its magnitude, as the authors of the early stopping paper explain. The idea is that huge weights are the typical way of sticking to the noise; if they become costly, the model tends towards more moderate solutions.

L2 adds a cost proportional to the square of the weights. In neural networks it is also called weight decay, and the TensorFlow tutorial on overfitting states that it is mathematically exactly the same thing.
L1 pushes weights towards exactly zero and favours a sparse model. L2 penalises the weights without making them sparse, because its penalty tends to zero for small weights; the tutorial cites this as one of the reasons L2 is more common.

If you use PyTorch, watch out for this: torch.optim.Adam has weight_decay=0 by default, so no L2 penalty is applied unless you switch it on:

import torch

# 'modelo' is your network; you choose the weight_decay value
optimizador = torch.optim.Adam(modelo.parameters(), lr=1e-3, weight_decay=1e-4)

The documentation also states that with decoupled_weight_decay=True the optimiser is equivalent to AdamW. There is no single weight_decay value that works for everything: it is a hyperparameter, and you choose it by comparing validation curves.

Dropout

Dropout attacks the problem from a different angle. During training, it randomly sets to zero a fraction of a layer’s outputs; the rate is usually set between 0.2 and 0.5. At test time nothing is dropped, and the layer’s output values are scaled to compensate for the fact that more units are active than during training.

The key idea, according to the original paper by Srivastava, Hinton and colleagues, is that dropping units at random prevents them from co-adapting too much, that is, from depending on one another in a rigid way. The authors report that it improves performance on supervised tasks in vision, speech recognition, document classification and computational biology.

It comes at a price: a network with dropout usually takes 2 to 3 times longer to train than a standard one with the same architecture, largely because the parameter updates are very noisy. It is a tool with benefits and costs, not a switch that fixes overfitting.

Early stopping: stop training in time

If validation starts to get worse, the most direct fix is to stop training. Early stopping halts the optimiser when a criterion that you define is met. The authors of the paper cited above regard monitoring the loss on a validation set as the “gold standard”, with one drawback: it leaves you with less effective data to train on.

The most common criterion uses patience: the number of consecutive epochs without improvement that you tolerate before stopping. A 2024 paper on overfitting describes that, on stopping, the epoch with the lowest validation loss is returned. The logic fits in a single function:

def parada_temprana(perdidas_val, paciencia):
    """Goes through the validation losses epoch by epoch.
    Returns (epoch at which it stops, epoch of the best model)."""
    mejor_epoca = 0
    mejor_perdida = float("inf")
    for epoca, perdida in enumerate(perdidas_val):
        if perdida < mejor_perdida:
            mejor_perdida = perdida
            mejor_epoca = epoca
        elif epoca - mejor_epoca >= paciencia:
            return epoca, mejor_epoca
    return len(perdidas_val) - 1, mejor_epoca
print(parada_temprana([1.0, 0.8, 0.7, 0.72, 0.75, 0.74, 0.8], paciencia=3))
assert parada_temprana([1.0, 0.8, 0.7, 0.72, 0.75, 0.74, 0.8], paciencia=3) == (5, 2)

It keeps the best loss seen so far and its epoch; if paciencia epochs go by without beating it, it cuts off and tells you which epoch to keep. In a real training run you would also save the weights from that best epoch so you can restore them.

Patience is not a minor detail. In the example in Figure 2a of that paper, with a patience of 20 epochs training stops at epoch 70 and epoch 50 is returned; with a patience of 40 it stops at epoch 140, but returns epoch 100, which has a lower loss. It is one specific curve from that paper, not a general rule, but it shows that the value you choose changes the model you end up with.

If you work with Keras, there is already the tf.keras.callbacks.EarlyStopping callback, which the TensorFlow tutorial uses to avoid long, unnecessary training runs. The same 2024 paper notes that the TensorFlow implementation is the most widespread and that PyTorch Ignite also includes its own.

Common mistakes

Evaluating on the training data and celebrating the score. This is exactly the methodological mistake that scikit-learn describes.
Tuning hyperparameters by looking at the test set. Every decision based on it contaminates it; that is what validation is for.
Assuming there is regularisation. In torch.optim.Adam, weight_decay is 0 by default.
Choosing patience blindly. Too short and you may stop too early; too long and you waste epochs. Look at the curve.
Forgetting the cost of validation. Setting data aside to watch the loss is useful, but it leaves less for training.
Mixing groups between training and test. If several samples come from the same person or source, split them by group (GroupKFold), or your validation will be too optimistic.

Practice

Take the polynomial example and plot the training and validation errors against the degree with Matplotlib. Find the degree where validation bottoms out.
Reduce the training points to 12 and repeat. Does overfitting set in earlier?
If you build a small network, train it with no defences at all, save both losses for each epoch, and compare with another version that switches on L2 (weight_decay) first and then dropout or early stopping. Compare the shape of the curves, not just the final number.
Extra challenge: pass the list of validation losses you saved to parada_temprana, with patience values of 5, 10 and 20, and see at which epoch the best model ends up.

To understand where dropout comes from and the research around it, the natural next step is Kia’s articles: she covers what has been discovered, while I stick to how it works on the inside. And if you want to strengthen the foundations, the introduction to deep learning shows you where all of this fits.

Sources

Image: geralt on Pixabay

Translated and adapted by Ollie.

Overfitting: how to spot it and what to do when your model memorises