Logistic Regression, Mathematically Explained

Imagine you want to predict whether a student passes (or fails) an exam based on how many hours they studied. Your outcome is categorical—either 1 (pass) or 0 (fail).

We need a function that always stays between 0 and 1 to represent a valid probability. This is where the logistic function (or the sigmoid function) is used.

The logistic/sigmoid function takes any real number and “squeezes” it into the range [0,1]. It is defined as:

Its graph looks like this:

Its end behaviors are the following:

In machine learning applications, this function is the heart of a very popular binary classifier—logistic regression.

In logistic regression, we model the probability of the positive or true class (y=1) with the logistic or sigmoid function.

• For instance, an input of 0 (z=0) will result in a 0.5 probability from σ(z).

We can then interpret this probability into a classification using a threshold value, which we’ll discuss more later.

We want this model to be able to learn from data that we give it, allowing it to predict probabilities that will accurately classify samples. In other words, the model needs parameters that it can learn (e.g., β₀, β₁).

To do this, we wrap a linear combination of inputs (β₀ + β₁x₁ + … + β_nx_n) inside a logistic function.

We choose to use a linear model for the logistic regression since it is easy to compute and also easy to interpret—it’s what came naturally for the process.

• The portion that is actually “learned” is the linear combination, which is structurally similar to multiple linear regression. The combination is passed through a logistic/sigmoid function to produce a probability.

What about neural networks? Logistic regression can actually be considered a neural network—it is represented by a singular neuron having a sigmoid activation with no hidden layers.

When we actually feed our logistic regression model input data, σ(z) becomes σ(β₀ + β₁x₁ + … + β_nx_n), where n represents the number of input features.

Thus, the probability (which we denote as p(x)) that a sample belongs to a given class (y=1) is:

Above, Pr(y = 1 | x) is read as the probability of a sample belonging to a given class given the input sequence x.