What is Overfitting?

Overfitting happens when a machine learning model fits the training data too closely, capturing not only the real patterns but also the noise and quirks of that specific sample. The result is a model that performs very well in training and poorly on new data.

The most useful analogy is the student who memorizes the answers to last year's exam instead of understanding the subject: a perfect score on the practice test they have already seen, and a failing grade on the real one.

How to spot it

The telltale sign is the divergence between the error curves. While the training error keeps falling, the validation error plateaus and then starts to rise — the point where it turns is where the model stopped learning and started memorizing.

What causes it

• A model too complex for the amount of available data — spare parameters left over to memorize examples.
• Insufficient training data, or data that poorly represents the real variety.
• Training for too long, past the point where the model had already generalized.
• Data leakage — information from the test set influencing training, artificially inflating performance.
• Irrelevant features that offer noise for the model to latch onto.

Techniques to fight it

Each technique attacks the problem from a different angle:

• More data — the most effective solution when feasible; it makes memorization harder.
• L1 and L2 regularization — penalizes large coefficients, keeping the model simpler. The basis of Ridge and Lasso regression.
• Cross-validation — evaluates on multiple splits of the data, giving a more reliable estimate.
• Early stopping — halts training when the validation error stops improving.
• Dropout — in neural networks, randomly disables neurons during training, preventing rigid dependencies.
• Data augmentation — generates variations of existing data (rotation, cropping, noise) to broaden diversity.

The opposite extreme: underfitting

Fighting overfitting with excessive zeal leads to the inverse problem. In underfitting, the model is too simple to capture the patterns and performs poorly in both training and validation. The balance between the two is known as the bias-variance trade-off: simple models have high bias, complex models have high variance, and the goal is the middle ground. Regularization techniques applied to regression are detailed in types of linear regression.