Author(s): Ali Oraji

Originally published on Towards AI.

Overfitting is when a neural network (or any ML model) captures noise and characteristics of the training dataset rather than the underlying patterns. It excels at training performance but fails to generalize to unseen data.

Think of it as overspecialization where the model becomes like a parrot, repeating what it memorized, rather than a thinker that understands.

Imagine a student studying for a math exam:

• A good student learns the underlying formulas and concepts (generalization). They can solve problems they’ve never seen before.
• An overfitting student memorizes the exact answers to every question in the textbook (memorization). When given a new, slightly different problem on the exam, they fail completely because it doesn’t match what they memorized.

In the context of neural networks, an overfit model performs exceptionally well on the data it was trained on (high training accuracy) but fails miserably when exposed to new, unseen data (low validation/test accuracy). This is the hallmark of overfitting.

Why Does Overfitting Happen?

1. Excessive Model Complexity:

• Deep/wide networks with millions of parameters have enormous capacity.
• They can memorize the training data completely, including outliers.
• Analogy: using a rocket to deliver a pizza. overkill 😀

2. Insufficient or Imbalanced Data

• Small datasets make it trivial for a large model to memorize.
• Class imbalance can worsen this: the model may “memorize” the dominant class.

3. Excessive Training (Too Many Epochs)

• After the generalizable structure is learned, the model keeps chasing smaller loss values by fitting noise.

4. Noisy or Irrelevant Features

• False correlations, mislabeled data, or irrelevant columns mislead the network into learning non-generalizable rules.

Symptoms of Overfitting

Training accuracy climbs → nearly perfect.

Validation/test accuracy stalls or declines.

Training loss continues decreasing, but validation loss diverges.

Model confidence is high on training examples, but erratic on unseen samples.

Methods to Fix Overfitting

1. Data Centric Approaches

Collect More Data: Bigger, more diverse datasets dilute noise. (Easiest in principle, hardest in practice.)

Data Augmentation: Create new examples by transformations (rotation, noise injection, synonym replacement). Forces robustness to variations.

If getting more data is not feasible, you can artificially create more data from your existing dataset. This teaches the model that slight variations of an image are still the same object, making it more robust.

How (for Images):

• Rotate, flip (horizontally/vertically), crop, or zoom the images.
• Change brightness, contrast, or color saturation.
• Add random noise.

How (for Text):

• Back-translation: Translate a sentence to another language and then back to the original.
• Synonym replacement: Replace words with their synonyms.
• Implementation: Deep learning frameworks like TensorFlow and PyTorch have built-in layers for data augmentation that can be added directly to your model pipeline.

# Example of Data Augmentation in Keras (TensorFlow)
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential

data_augmentation = Sequential(
 [
 layers.RandomFlip("horizontal"),
 layers.RandomRotation(0.1),
 layers.RandomZoom(0.1),
 layers.RandomContrast(0.2),
 ]
)

# You can then add this `data_augmentation` layer as the first layer in your model.

2. Model Centric Approaches

Simplify the Architecture: Reduce layers/neurons → constrain capacity.

Regularization:
L1 (Lasso): Shrinks weights, encourages sparsity. Adds a penalty equal to the absolute value of the weights. This can force some weights to become exactly zero, effectively performing feature selection and making the model sparser.
L2 (Ridge / Weight Decay): Prevents excessively large weights. Adds a penalty equal to the square of the weights. This encourages all weights to be small and close to zero, but they rarely become exactly zero. It’s the most common type of regularization.

python
# Example of L2 Regularization in Keras
from tensorflow.keras import layers, regularizers

# The lambda value is passed as the argument
layer = layers.Dense(
 64,
 activation='relu',
 kernel_regularizer=regularizers.l2(0.001) # 0.001 is the lambda value
)

Dropout: Randomly deactivates neurons during training → prevents co adaptation.
Batch Normalization: Adds stability, slight regularization through mini-batch noise.

python
# Example of Dropout in Keras
from tensorflow.keras import layers, Sequential

model = Sequential([
 layers.Dense(128, activation='relu', input_shape=(...)),
 layers.Dropout(0.5), # Drops 50% of neurons from the previous layer
 layers.Dense(64, activation='relu'),
 layers.Dropout(0.3), # Drops 30% of neurons
 layers.Dense(10, activation='softmax')
])

3. Training Centric Approaches

Early Stopping: Stop training when validation loss no longer improves → “freeze” the model at its sweet spot.

This is a straightforward and highly effective method.

How it Works: You monitor the model’s performance on the validation set during training. If the validation performance (e.g. validation loss) stops improving or starts getting worse for a certain number of consecutive epochs (called “patience”), you stop the training process.

Why it Works: It directly stops the training at the “Good Fit” point in the graph, right before significant overfitting begins.

python
# Example of Early Stopping in Keras
from tensorflow.keras.callbacks import EarlyStopping

# Stop training when validation loss hasn't improved in 10 epochs
early_stopping_callback = EarlyStopping(
 monitor='val_loss',
 patience=10,
 restore_best_weights=True # Restores model weights from the epoch with the best val_loss
)

# Pass the callback to the model's fit method
# model.fit(..., callbacks=[early_stopping_callback])

Cross-Validation: Ensures model performance is consistent across different data splits.

Learning Rate Scheduling: Reduces step size progressively, avoiding overfitting to noise late in training.

A Practical Anti-Overfitting Recipe

1. Always hold out validation/test sets.
2. Use augmentation (images/text/audio) aggressively.
3. Start small → increase model size only if underfitting.
4. Add Dropout + L2 as default.
5. Enable Early Stopping callback.
6. Iterate systematically, not blindly.

Overfitting is one of the most common challenges in training neural networks, but it is also one of the most preventable. By recognizing the early warning signs, like the widening gap between training and validation performance, you can intervene before your model becomes a memorization machine.

The key lies in balance: building models that are powerful enough to capture the true patterns in data but disciplined enough to ignore the noise. With practical techniques such as data augmentation, regularization, dropout, and early stopping, we can guide our networks toward generalization rather than perfectionism.

In the end, the goal of any neural network is not to ace the training set but to thrive in the real world, making reliable predictions on data it has never seen before.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI