# Deep Learning's Top Pitfalls: Why Your Model Struggles (And How to Fix It!)

#39 Feb 24, 2025 Training

https://www.aibutsimple.com/p/deep-learning-s-top-pitfalls-why-your-model-struggles-and-how-to-fix-it

This week, we’ll be going over some common pitfalls in deep learning—and, more importantly, how to fix them for better-performing models. From vanishing gradients to overfitting, we’ll break down key challenges that can negatively impact model performance and explore how to solve them.

Before getting into today’s issue, we’re proud to announce that we’re partnering with 1440 Media for this sponsored segment:

---

### Fact-based news without bias awaits. Make 1440 your choice today.

Overwhelmed by biased news? Cut through the clutter and get straight facts with your daily 1440 digest. From politics to sports, join millions who start their day informed.

[Sign up now!](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_humans&_bhiiv=opp_2d84cf84-49e2-4243-8439-bafc7897f658_1b75ca79)

---

### Vanishing Gradient

Imagine you’re shining a flashlight through a stack of tinted sunglasses, tinted such that each pair of sunglasses dims the light a little.

If you stack 10 pairs, the beam becomes extremely dim. But imagine if you stacked 20, 30, or even 50 pairs. The final beam would become so weak that you couldn’t see anything.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/347c13e2-ddff-4db3-9bc3-4e59cc57831c/Copy_of_Copy_of_Copy_of_positional_emb__2_.png)

Here, the flashlight is the "learning signal" in a neural network, and the sunglasses represent the network’s layers. By the time the signal reaches the early layers (the first pairs of sunglasses), it’s too small to adjust a network’s weights meaningfully (the light beam is too dim to be seen). This is a classic case of the vanishing gradient problem.

The math reason behind why it happens is tiny numbers multiplying together. For instance, if each layer in the network reduces the input signal by half (in other words, halves the input value), after just 10 layers, the signal is multiplied by 0.510 = 0.0009, giving extremely small values.

When it comes to neural networks, activation functions such as sigmoid and tanh are the main causes. Older functions like sigmoid or tanh squish input values into small ranges, making their gradients—the key component needed for the optimization algorithm, the process that changes the network’s parameters—even smaller.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/95d6d800-da83-4693-8444-a49522a20b7f/88a3674d-0d19-4721-9782-a5156243fab6.png)

Sigmoid Activation Function & Its Derivative

Once the gradient is small, it becomes very difficult to nudge weights and biases enough for the model to meaningfully “learn”.

The vanishing gradient problem is also very common in standard RNNs, since gradients shrink or explode exponentially over long sequences (which there are a lot of in NLP tasks).

#### How Do We Fix It?

A typical solution is simply to swap the activation function. Instead of sigmoid or tanh, we can use ReLU or ReLU-based activation functions since they don’t shrink gradients. This is why we see so many ReLU-based activation functions being used in modern networks.

Another way of promoting gradient flow is to use residual (skip) connections. The connections add shortcuts so data can flow directly between layers, bypassing certain layers—preventing the data from vanishing.

Residual connections come from the Residual Network, commonly known as the ResNet. If you want to learn more about the ResNet, I’ve linked an older issue that explains it below!

- [Modern Convolutional Neural Network Architectures (Intermediate)](https://www.aibutsimple.com/p/modern-convolutional-neural-network-architectures?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-s-top-pitfalls-why-your-model-struggles-and-how-to-fix-it)

For RNNs, to mitigate the vanishing gradient problem, many new architectures—such as LSTMs and GRUs—have been developed. We covered LSTMs and GRUs in detail in [this previous issue](https://www.aibutsimple.com/p/recurrent-neural-networks-revisited?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-s-top-pitfalls-why-your-model-struggles-and-how-to-fix-it)—check it out!

Additionally, transformers significantly mitigate the vanishing and exploding gradient problems through their self-attention mechanisms and residual connections.

### Overfitting

---

When training neural networks, it is common practice to split your dataset into three parts: a training set, a validation set, and a testing set.

However, when the model trains on the training data set for too long or when the model is too complex, it can start to learn the “noise” (irrelevant information) within the dataset.

This is what we call overfitting. When a model learns the training data too well, including its noise, it results in poor performance on new and unseen data—it cannot generalize well. The model stops learning useful patterns and starts memorizing everything.

- If a model cannot generalize well to new data, it won’t be accurate on the classification or prediction tasks it was intended for.

Imagine a student who only memorizes answers to practice tests (a model trains very long on the training dataset). They ace every homework quiz (accurate on the train set) but panic when the final exam asks the same questions but with different twists (poor performance on new data). That’s overfitting!

#### How To Tell If A Model Is Overfitting

When you train the model and note down its performance over its training cycles, if it performs extremely well on the training data but poorly on the validation or test data, it might be overfitting.

We usually measure the performance of a model by its loss (given by a loss function) over a period of time known as epochs.

A quick benchmark: Low training error but high test error.

Another indicator of overfitting is when the training loss decreases over time while the validation loss makes a plateau or increases.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/27bbae9a-5348-4f55-9618-9bcb0afcf85b/image.png)

Why does it happen? Overfitting can be caused by overly complex models (models that have way too many parameters, too many layers, or a complex structure), noisy data, insufficient training samples, or training a model for too long on the same training samples.

#### How Do We Fix It?

An easy way to fix overfitting is to stop training the model before the validation or test error starts increasing.

Another way is to simplify the model by using fewer layers/neurons. Your model might be too complex or have too many parameters for your task!

We can feed the model more varied data and better examples to reduce overfitting. If there’s not enough data to go around, we can use techniques such as data augmentation to create new samples of old training data.

Finally, we can use regularization to mitigate overfitting, which we’ll discuss later on!

### Underfitting

---

A neural network underfits when it fails to learn the underlying patterns in the training data. Some models learn so poorly that they are no better than a guess or a baseline model.

Why does it happen? Underfitting typically occurs when the model is too simple to represent the data. For instance, the network might not have enough layers or neurons to capture the complexities of the data.

Another reason a model might underfit is that it might not have been trained for enough epochs (training cycles) to learn the underlying patterns.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/982d31fd-5d12-4320-aeed-c46a67fd6caf/image.png)

A quick benchmark: High training and test errors.

#### How Do We Fix It?

One way to mitigate underfitting is to add more layers or neurons to the network if the model is too simple.

If the model hasn’t trained for long enough, we can also increase the training time, as in train the model for more epochs.

The model might also benefit from gathering more data to learn the features from.

#### Overfitting And Underfitting

Overfitting and underfitting are closely linked in the training cycle. As we train a model, the parameters are initially random, causing it to underfit the data. The more the model learns correctly from the data, the more both the training and validation loss decrease, showing progress.

As we train, the model will reach a point that nets the lowest validation loss before reaching a plateau and increasing.

This point is what we call the best fit, where we ideally should stop training the model—a technique known as “early stopping”.

If we continue to keep training the model, it will most likely start overfitting and learning the noise in the training dataset.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/160800c5-6f06-4a6b-a771-6fea0a02dad2/image.png)

### Not Using Regularization

---

Regularization is a set of techniques used in machine learning and deep learning to prevent overfitting and improve the generalization of a given model. Not using regularization can leave lots of potential improvement in model accuracy untapped.

Regularization usually works by adding additional information or constraints to the model. For instance, a “penalty term” is added to the loss function when the model is training, which prevents the model from having large weights, lowering the complexity.

Some common regularization methods used frequently in deep learning are L1 and L2 regularization, dropout, early stopping, and additional regularization methods tailored to specific model architectures such as transformers or CNNs.

L1 regularization adds a penalty term to the loss function that is equal to the absolute value of the model’s weights. The model is being encouraged to keep the weights small.

L2 regularization adds a penalty that is equal to the square of the magnitude of weights. This encourages the model to distribute weights more evenly and prevents any single weight from becoming too large.

- This helps in reducing overfitting by smoothing the model and making it less sensitive to noise in the training data.

Dropout randomly sets a number of input neuron values to 0 during training to prevent overfitting.

Early stopping stops training when performance on a validation set starts to degrade.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/24aa1a7e-4e06-4b80-ac08-f8efe2a660be/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__1_.png)

The cost function with L1 regularization is plotted on the left; the absolute value function forms a diamond-like shape.

L2 regularization makes the cost function circular and parabolic due to the squaring used in the regularization term.

### Not Using Normalization

---

Over the years, researchers have come up with different methods to accelerate and stabilize the learning process. Normalization is one technique that proved to be very effective in doing this.

In machine learning, normalization is usually done once at the start of the process to center the data to improve the fit of the model.

In deep learning, given the increasing complexity and depth of deep learning models, it is common to normalize the data within the network itself at various points between layers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3445a516-4c38-4f87-981b-18f74c478d99/image.png)

Normalization vs. No Normalization

Commonly used normalization techniques in modern deep learning include:

- Batch Normalization
- Layer Normalization
- Instance Normalization

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/26d8a0c3-381b-426e-94e0-87933e746729/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png)

To understand our normalization techniques, let’s go over some terminology. Accounting for batch size, input data is usually 2, 3, or 4 dimensional:

- For simple tabular data, the dimensions are (N x W), where N is the batch size and W is the number of features.
- For grayscale images, the dimensions are (N x H x W); each image has a width W and a height H.
- For RGB images, the dimensions are (N x H x W x C), since the image has 3 channels (C) for red, green, and blue.

#### Batch Normalization

Batch norm works by normalizing data over a batch. But what does this specifically mean? Looking above, imagine each of the piles of squares as a book.

Each book (with dimensions H x W x C) has multiple pages (C), with each page having height and width H x W.

Now, think of a row of N books on a bookshelf as a batch of images (N x C x H x W).

To apply batch norm, we pick the first page of each book and normalize it. We then repeat this process for the rest of the C - 1 pages.

This normalizes data across the batch but for each feature or channel (C) separately.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a328dcbf-369b-43be-b1d0-08e66809b86e/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px_.png)

#### Layer Normalization

Layer norm normalizes the input data over the channels. This way, the normalization is not dependent on the batch.

Continuing our analogy, each book (H x W x C) has multiple pages (C), with each page having height and width H x W.

To apply layer norm, we pick up the first book (H x W x C) and normalize it. We then repeat this process for the rest of the N - 1 books.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c7eeac2a-aafa-4d3f-966f-5d989f436df5/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___1_.png)

#### Instance Normalization

Instance norm involves normalizing across both the batch and the channels.

Instance norm involves normalizing a single page (blue shade) of size H x W across both the channel (C) and batch (N) dimensions.

To apply instance norm, we normalize on the first page of the first book (H x W). We then repeat this process for the rest of the C - 1 pages for N - 1 books.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/219c6a2e-44a4-4a78-b23c-a64b39fa4948/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px___720_x_350_px___2_.png)

Want to learn more about normalization and its applications? Check out our issue on normalization, found [here](https://www.aibutsimple.com/p/normalization-in-deep-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-s-top-pitfalls-why-your-model-struggles-and-how-to-fix-it).

### Choosing The Wrong Activation Functions

---

Activation functions are mathematical functions in deep learning models placed between layers, playing a critical role in the performance and capabilities of neural networks.

#### Why Do We Need Activation Functions?

They introduce non-linearity into the model, enabling it to learn and represent complex patterns in data.

Without activation functions, neural networks would be limited to linear mappings, simplifying boundaries to lines regardless of how many layers are used.

No matter how many layers you add, if all they do is apply more multiplications, the result is still linear. This is not complex enough to represent many real-world problems!

Some commonly used activation functions include:

- ReLU — The golden standard in deep learning. Simple but powerful, its variants are used in almost all domains of deep learning.
  - Leaky ReLU — Variant of ReLU, allows small negative values.
  - Softplus — Smooth approximation of ReLU.
  - ELU, GELU — Advanced variants used in newer architectures.
- Sigmoid, Tanh — Older activation functions that squish values into small ranges.
- Softmax — Used to transform values into a probability distribution, usually at an output.

Want to learn more about activation functions? In a previous issue, we explored a wider range of activation functions in detail; check it out below!

- [All About Activation Functions (Beginner)](https://www.aibutsimple.com/p/all-about-activation-functions?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-s-top-pitfalls-why-your-model-struggles-and-how-to-fix-it)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e951a4aa-b78f-45fd-8616-5946897c6606/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__720_x_600_px_.png)

#### Using The Wrong Activation Functions

Activation functions play a critical role in the performance of neural networks. Choosing the wrong activation functions for specific use cases can impede learning and slow convergence—in some cases, it can even break the model!

Example 1: Sigmoid in Deep Neural Networks

Consider a neural network with 10 layers that uses the sigmoid activation function in hidden layers.

Problem: The sigmoid activation function squishes inputs to a range of (0, 1), and its gradient becomes tiny for very large or very small inputs. This is a classic case of the vanishing gradient problem.

Impact: Early layers receive almost no gradient during backpropagation; the changes to the parameters in earlier layers are negligible. The network stops learning after a few epochs.

What’s The Fix?: Replace sigmoid with ReLU or Leaky ReLU to mitigate the vanishing gradient problem.

Example 2: ReLU Given Sparse Data (Little Positive Values)

Consider a network that uses the ReLU activation function in a layer where most inputs are negative.

Problem: All negative outputs become zero. Little information will be passed between layers, affecting backpropagation and gradient flow.

Impact: The layer stops contributing to the learning process of adjusting parameters.

What’s The Fix?: Use Leaky ReLU, which allows small gradients for negatives, or ELU, which smoothly handles negative values.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-s-top-pitfalls-why-your-model-struggles-and-how-to-fix-it):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-learning-s-top-pitfalls-why-your-model-struggles-and-how-to-fix-it)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team