Normalization In Deep Learning

Training effective deep neural networks (DNNs) is a challenging process with many interconnected elements.

Over the years, researchers have come up with different methods to accelerate and stabilize the learning process. Normalization is one technique that proved to be very effective in doing this.

In machine learning (ML), normalization is used to center the data to improve the fit of the model. In ML processes, normalization is usually done at the start of the process, normalizing the entire dataset once before feeding it to the model—what we refer to as “pre-processing”.

We then use the normalized data to fit the model, resulting in a higher overall training efficiency.

However, in deep learning, this process is a little different. In recent years, given the increasing complexity and depth of deep learning models, it is common to normalize the data within the network itself at various points between layers.

In this issue, we will explore 3 widely used types of between-layer normalization techniques that speed up training and improve convergence.