# Logistic Regression, Mathematically Explained

#33 Jan 6, 2025 Foundation

https://www.aibutsimple.com/p/logistic-regression-mathematically-explained

Imagine you want to predict whether a student passes (or fails) an exam based on how many hours they studied. Your outcome is categorical—either 1 (pass) or 0 (fail).

We need a function that always stays between 0 and 1 to represent a valid probability. This is where the logistic function (or the sigmoid function) is used.

The logistic/sigmoid function takes any real number and “squeezes” it into the range [0,1]. It is defined as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cad81690-868a-453c-941d-e639789835bd/image.png)

Its graph looks like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/44199f81-96c4-411f-8f8e-d19801535b31/output.png)

Its end behaviors are the following:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d5f433bc-e06d-487c-a394-0430b789b6e8/image.png)

In machine learning applications, this function is the heart of a very popular binary classifier—logistic regression.

In logistic regression, we model the probability of the positive or true class (y=1) with the logistic or sigmoid function.

- For instance, an input of 0 (z=0) will result in a 0.5 probability from σ(z).

We can then interpret this probability into a classification using a threshold value, which we’ll discuss more later.

It is recommended to have some mathematical background of multivariable calculus, probability and math notation. Without it, it may be more difficult to follow along.

We want this model to be able to learn from data that we give it, allowing it to predict probabilities that will accurately classify samples. In other words, the model needs parameters that it can learn (e.g., β0, β1).

To do this, we wrap a linear combination of inputs (β0 + β1x1 + … + βnxn) inside a logistic function.

We choose to use a linear model for the logistic regression since it is easy to compute and also easy to interpret—it’s what came naturally for the process.

- The portion that is actually “learned” is the linear combination, which is structurally similar to multiple linear regression. The combination is passed through a logistic/sigmoid function to produce a probability.

---

What about neural networks? Logistic regression can actually be considered a neural network—it is represented by a singular neuron having a sigmoid activation with no hidden layers.

---

When we actually feed our logistic regression model input data, σ(z) becomes σ(β0 + β1x1 + … + βnxn), where n represents the number of input features.

Thus, the probability (which we denote as p(x)) that a sample belongs to a given class (y=1) is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/95a1fe48-8d40-4c57-9798-265c20b00ffd/image.png)

Above, Pr(y = 1 | x) is read as the probability of a sample belonging to a given class given the input sequence x.

The probability p(x) can also be viewed as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/68400fec-269b-4946-97ee-b6ec93b9f8e0/image.png)

If p(x) is near 1, the model believes a data sample belonging to a class (y=1) is very likely; if p(x) is near 0, a data sample belonging to a class (y=1) is very unlikely.

Now, we need to estimate the parameters β=(β0,β1,…,βn). To do that, we use Maximum Likelihood Estimation (MLE).

For a single training example (x(i), y(i)) (where the superscript i represents the sample number):

- If y(i) =1, we want the probability p(x(i) ) to be as large as possible.
- If y(i) =0, we want 1−p(x(i) ) to be as large as possible.

Each observed y(i) is treated as a draw from a [Bernoulli distribution](https://www.cuemath.com/data/bernoulli-distribution/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=logistic-regression-mathematically-explained) with a success probability p(x(i)).

- A Bernoulli random variable can take only two values (commonly 1 or 0). This matches our typical binary classification outcome: “pass” or “fail”.
- Therefore, if y(i) is either 1 or 0, using a Bernoulli model is natural.

Using a Bernoulli model, the probability of observing y(i) given features x(i) and parameters β is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/666ed4f5-a1f7-4adf-99e9-3ff9be48d4ce/image.png)

Because the samples are assumed independent, the likelihood (joint probability) of all m training points in our dataset is the product of the individual Bernoulli probabilities:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/33cc687b-6ffe-471a-a05b-5a50989e7625/image.png)

Above, the big “n-looking” symbol represents repeated multiplications of different elements (called the pi or product notation), similar to how a sum represents repeated additions of different elements.

We usually work with the log-likelihood (which is just the log of the likelihood) because products can get very small, and logs turn products into sums:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/da262815-67b8-4fdb-bba4-83980f42b342/image.png)

However, in optimization, we minimize a cost/loss function rather than maximize a likelihood. So we take the negative of the log-likelihood:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f123b87a-4a4c-401a-8e5b-d4797a401b38/image.png)

This cost function is also called the cross-entropy loss in classification, a very popular loss function in deep learning. Maximum Likelihood Estimation leads to the cross-entropy loss.

If you want to read about the difference between a loss, cost, and objective function and why we use them interchangeably, we have an issue detailing this [here](https://www.aibutsimple.com/p/optimization-algorithms-in-deep-learning-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=logistic-regression-mathematically-explained).

Now, our goal is to find β that minimizes J(β). To find the parameter values β that minimize J(β), a common approach is gradient descent.

In gradient descent, we would initially randomly pick some β and use it along with our input to generate an output.

We then use this output to calculate the gradient of the loss function, which gives us the derivative of the loss with respect to each individual parameter βj.

We update the parameters in the opposite direction of the gradient. To obtain the new parameter values, we subtract the gradient multiplied by a learning rate (which is a small constant).

We repeatedly update the parameters until the loss plateaus and the updates become very small.

But the model still needs a way to convert a probability to a classification if it wants to generate a relevant output for binary classification.

So, while the output of logistic regression is a probability p(x), the final classification often uses a threshold.

Typically:

- If p(x)≥0.5, we predict y=1.
- If p(x)<0.5, we predict y=0.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/11772ced-d6a6-4d29-96ba-38801d8640ae/image.png)

To clarify, the actual decision boundaries on the data separating the classes in logistic regression are still linear despite the nonlinearity of the logistic function.

---

### Worked Example

We’ll demonstrate the logistic regression process using a simple example while showing the computations. Let’s use a small dataset of 2 features and 3 entries.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd7c491e-71e7-44c5-b509-f1828a1c737c/image.png)

Above, we only have one entry with a true label (y = 1).

Using this small batch, we perform the forward pass. We define the logistic regression model as shown below (as seen previously):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4f737e9e-9a34-4a6e-aa89-45f9115a4077/image.png)

Where β0​ is the intercept (bias term), and β1​ and β2​ are coefficients for x1​ and x2​, respectively.

Using the negative log-likelihood (or cross-entropy loss), the cost function comes out to be:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/db915617-d333-458d-8b0d-5edae22c5221/image.png)

Let’s initialize our parameters to 0 to simplify the computations:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c83d593b-04fa-46dc-8b09-ad6c20c8419c/image.png)

With β0=β1=β2=0, the linear term β0+β1x1+β2x2=0 for every data point. This means that the predicted probability for every data point will end up being 0.5.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/95a2150d-f399-4f43-80bb-e87c1b9edddf/image.png)

Using this probability and our data samples, we can compute the loss given these parameters.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/054a4322-93e4-4097-87ee-01ddbb7f5967/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Input_to_Hidden__720_x_600_px___900_x_750_px_.png)

We calculate this loss to measure how well the model performs; it’s used for comparison in further iterations.

After calculating the loss, we calculate the gradient for gradient descent.

The chain rule shows that the derivative of the cross-entropy loss (−[yln⁡(p)+(1−y)ln⁡(1−p)]) with respect to β0 + β1x1 + β2x2 (which we denote as z) is p(x)−y.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/624375eb-f47c-4815-bdfb-6f1cbe5ef3f7/image.png)

Using this, the derivative of J(β) with respect to an individual parameter βj is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f80bd233-5a84-4652-88f5-19fcff04f7f8/p_x___1_.png)

The individual partial derivatives are shown:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/94bc4a75-d0ab-4b9a-adc3-7fd742e55eb0/Your_paragraph_text.png)

To calculate the new parameter values, we use the gradient descent update rule. The update rule is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1be8334e-966e-49ce-a563-ead971b682b5/image.png)

Above, α represents the learning rate, a small constant controlling how quickly the model learns. We will do one update with a learning rate of 0.1.

To perform the update, we first compute the partial derivatives for each individual sample, using p(x), y, and x for β1 and β2. Then, we update the parameters based on the update rule.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8609da3d-7303-4ab8-9db6-059c4fc44ec4/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Input_to_Hidden__720_x_600_px___900_x_750_px_.png)

With more iterations, these parameters will continue to be adjusted, gradually improving the model's performance.

Over time, the adjustments will lead to the parameters reaching values that allow the model to classify data more accurately.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=logistic-regression-mathematically-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=logistic-regression-mathematically-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team