All About Activation Functions

Activation functions are mathematical functions in deep learning models placed between layers, playing a critical role in the performance and capabilities of neural networks.

• They introduce non-linearity into the model, enabling it to learn and represent complex patterns in data.

• It allows models to perform popular deep learning tasks like image recognition, natural language processing (NLP), and more recently, text generation with LLMs.

Without activation functions, neural networks would be limited to linear mappings, simplifying their boundaries to lines regardless of how many layers are used. Activation functions enable networks to model complicated non-linear relationships.

Mathematically speaking, an activation function is an equation that determines the output of a neuron given an input or set of inputs.

The activation function “decides” whether a neuron should be activated or not by calculating the weighted sum of inputs and biases.

Where are activation functions used in neural networks?

Activation functions are used in a network’s hidden layers to add non-linearity and to mathematically transform the data.

Each and every neuron in the hidden layers of a neural network applies an activation function.

To further understand the activation function’s importance in the hidden layer, here’s how a hidden layer is composed:

Here, the linear transformation portion will differ for what type of model is being used.

• In Multi-Layer Perceptrons (MLPs), the linear transformation will be the perceptron’s pre-activation (Wx+b).

• In CNNs, it will take the form of the convolutional layer (convolution of Wx +b).

• In RNNs, it will be the hidden state pre-activation (a form of Wx + b including timesteps).

Above, the math representation of this hidden layer is for a MLP with the pre-activation (z) represented using vectors. W is the weight matrix, b is the bias vector, and x is the input vector.

Activation functions are also used at the output layer. The activation function used in the output layer depends on the type of problem since it transforms the network's output into a desired format.

If the problem is a regression problem, we would typically use a linear activation function and simply take the output from the last hidden layer as the numerical prediction.

If the problem is a classification problem:

• For multiple classes (multi-class), we will often use the Softmax activation function, generating a probability distribution over all possible classes.

• For binary classification, we can use the Sigmoid activation function.