Deep Learning's Top Pitfalls: Why Your Model Struggles (And How to Fix It!)

This week, we’ll be going over some common pitfalls in deep learning—and, more importantly, how to fix them for better-performing models. From vanishing gradients to overfitting, we’ll break down key challenges that can negatively impact model performance and explore how to solve them.

Vanishing Gradient

Imagine you’re shining a flashlight through a stack of tinted sunglasses, tinted such that each pair of sunglasses dims the light a little.

If you stack 10 pairs, the beam becomes extremely dim. But imagine if you stacked 20, 30, or even 50 pairs. The final beam would become so weak that you couldn’t see anything.

Here, the flashlight is the "learning signal" in a neural network, and the sunglasses represent the network’s layers. By the time the signal reaches the early layers (the first pairs of sunglasses), it’s too small to adjust a network’s weights meaningfully (the light beam is too dim to be seen). This is a classic case of the vanishing gradient problem.

The math reason behind why it happens is tiny numbers multiplying together. For instance, if each layer in the network reduces the input signal by half (in other words, halves the input value), after just 10 layers, the signal is multiplied by 0.5¹⁰ = 0.0009, giving extremely small values.

When it comes to neural networks, activation functions such as sigmoid and tanh are the main causes. Older functions like sigmoid or tanh squish input values into small ranges, making their gradients—the key component needed for the optimization algorithm, the process that changes the network’s parameters—even smaller.

Once the gradient is small, it becomes very difficult to nudge weights and biases enough for the model to meaningfully “learn”.

The vanishing gradient problem is also very common in standard RNNs, since gradients shrink or explode exponentially over long sequences (which there are a lot of in NLP tasks).

How Do We Fix It?

A typical solution is simply to swap the activation function. Instead of sigmoid or tanh, we can use ReLU or ReLU-based activation functions since they don’t shrink gradients. This is why we see so many ReLU-based activation functions being used in modern networks.

Another way of promoting gradient flow is to use residual (skip) connections. The connections add shortcuts so data can flow directly between layers, bypassing certain layers—preventing the data from vanishing.