# Linear Regression, Mathematically Explained

#26 Nov 18, 2024 Foundation

https://www.aibutsimple.com/p/linear-regression-mathematically-explained

In this issue, we'll explore Linear Regression, one of the simplest and most accessible algorithms in data science and machine learning.

Linear regression is a type of supervised machine learning algorithm, meaning that the data fed to the algorithm is labeled (there are true target values along with their inputs).

- Regression means to predict a numerical value instead of a class, like in classification.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4c9ad591-7093-494b-beed-1adc8130bc8b/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__4_.png)

Supervised machine learning algorithms rely on mathematical transformations using linear algebra and calculus to analyze patterns in labeled data.

These algorithms (we’ll call them models for now) are fed input data and are asked to produce relevant outputs.

- For instance, if we’re given the square footage of a house, could the model predict a reasonable price for the house?

The training occurs when the model’s parameter values are changed, leading to a better fit line or curve (which gives more accurate predictions).

In the training process, the model uses what’s known as a loss function. Loss functions represent the distance or inaccuracy of the model’s prediction compared to the actual value.

The model determines the best fitting line, plane, or curve by optimizing the loss function through different optimization algorithms, minimizing the loss it produces.

So, if the loss is high, then the numerical distance between the predicted value and the actual value is large, which is bad.

If the loss is low, the distance between the predicted value and the actual value is small, which is good.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/21435e79-93a3-41e9-848d-4edfb840313f/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Add_a_heading__3_.png)

---

If you have an interest in AI topics outside of deep learning and machine learning, feel free to check TheTechOasis out:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/publication/logo/9b355c4a-4dbc-4f53-b676-a166fee6812c/Isotipo_negro.png)

Sponsored

TheWhiteBox by Nacho de GregorioThe newsletter to stay ahead of the curve in AI

Subscribe

---

Before getting into the mathematics of linear regression, please read the issue below on optimization algorithms to further understand loss, cost, and objective functions, along with how optimization works in general:

- [Optimization Algorithms, Mathematically Explained (Intermediate)](https://www.aibutsimple.com/p/optimization-algorithms-in-deep-learning-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-regression-mathematically-explained)

Basic linear algebra knowledge (matrix multiplication, transpose) and a basic knowledge of multivariable calculus (partial derivatives, gradients, chain rule, matrix calculus) are also recommended.

### Linear Regression Math

---

Mathematically, linear regression is an algorithm that models the linear relationship between a target variable and one or more input variables by fitting a line to given data.

It finds the line (or hyperplane) of best fit that minimizes the error (or loss) between predicted and actual values.

The algorithm uses either a direct mathematical solution or gradient descent for optimization, minimizing the Mean Squared Error (MSE) or Sum of Squared Errors (SSE) cost functions.

Here’s the equation for Simple Linear Regression; linear regression consisting of one input feature (a 2-dimensional line):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c70ba54e-1856-4e5b-9040-24ea7706f95b/Untitled_design__30_.png)

Here, yi is the true observed value for the i-th observation, β0 is a parameter representing the y-intercept of a line, β1 is a parameter representing the slope coefficient, xi is the i-th input, and ϵi is the error term for the i-th observation (which represents the difference between the observed and predicted value).

- ϵi is used only in this notation to be mathematically correct since it’s using an equal sign (exactly equal).

Keep in mind that this notation for the equation factors in the error within the equation itself since it considers the true and observed value (based off of the label).

The algorithm’s predicted value is only the β0 + β1xi part, and it can be found by subtracting the error ϵi from the actual value:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6c61a426-45ff-47d1-9876-7d3024ae0afb/image.png)

In addition to Simple Linear Regression, there’s also Multiple Linear Regression. It consists of multiple input features and can go up to very high dimensions:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6b6669c7-5069-4ca5-aebf-66670d762700/image.png)

Above, βj​ is the parameter coefficient for the j-th feature, xij​ is the input of the j-th feature (or variable) for each observation i, and ϵi is still the error term for the i-th observation.

- Here, p represents the total number of features (independent variables or rows) in the input data.

Multiple linear regression works like simple linear regression but in higher dimensions, often finding parameter values to fit the best [hyperplane](https://en.wikipedia.org/wiki/Hyperplane?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-regression-mathematically-explained).

It might look something like this for 2 input features (3d), but the dimensions increase with more input variables.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/27aab7da-43e0-4e13-885d-9a1c306de8f5/image.png)

We can convert these equations of simple and multiple linear regression for individual observations into a single equation in vector form containing all values for all observations.

The vector form of an equation is simply an equation represented by vectors, matrices, or tensors. In vector form, multiplications turn into matrix multiplications.

The vector form of the true observed value (y) is shown below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bf9794dd-fe62-42ca-9755-5416de3fd7aa/image.png)

Above, β is a vector of βp parameter values, X is a matrix of input data, and ϵ is the error term vector.

This means that the predicted value, or ŷ will be:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1d8790ac-3507-4a3c-894d-53b36b241064/image.png)

Where Xβ is a matrix multiplication of matrix X with dimensions (p+1)×n and vector β with dimensions n×1, resulting in a size of (p+1)×1.

- The +1 accounts for the intercept term (β0), which is included by adding a column of ones to X.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f4e37218-6964-42b0-b408-7f2b143f4c47/image.png)

The goal of linear regression is to approximate (or learn) the optimal values of β by minimizing the Sum of Squared Errors (SSE) (or MSE) between the observed values and the values predicted by the model.

The SSE function is defined below:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1c2df270-5878-40a5-994a-661cf05c1163/image.png)

Here, ŷi represents the predicted value for the i-th observation and yi represents the observed value for the i-th observation.

- It sums these differences or errors up, hence the name Sum of Squared Errors.

In vector form, SSE simplifies to the equation shown below, where y is the real value vector, X is a matrix of input data, and β is a vector of parameters.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ec70bc09-b3ae-4f6f-99d8-dbf8c1a700c6/image.png)

Above, the squared difference turns into the difference multiplied by its transpose, and the sum is eliminated due to the use of vectors and matrices.

Now, how do we solve for the values in the β vector (or the βp parameters in non-vector form) that will minimize this cost?

We can either:

- Use the ordinary least squares method (OLS) using linear algebra and a little bit of calculus to directly solve for the β vector.
- Use an optimization algorithm like gradient descent to solve for the parameter values, iteratively adjusting the parameters until they converge.

In practice, most libraries use the OLS method under the hood since they can directly solve for β.

However, it is also possible to use algorithms like gradient descent to iteratively adjust the parameter values if the data is large or certain mathematical requirements (for the matrices and vectors) are not met.

#### Ordinary Least Squares (OLS)

To find the value of β that minimizes the SSE, we need to take the partial derivative of the SSE with respect to β.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e62a1458-9355-4182-996b-00ce066eb269/image.png)

We set the derivative of SSE with respect to β to zero and solve for β to find the critical points (points where the SSE is at a minimum or maximum at a given β).

Finally, we solve for β, computed as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7512f394-bf48-4171-9e9d-cc019636cfb7/image.png)

Note that this makes the assumption that XTX is invertible (if there is a square matrix that is its inverse).

- To check if a matrix is invertible, we check if the [determinant](https://en.wikipedia.org/wiki/Determinant?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-regression-mathematically-explained) of the matrix is not equal to 0. If it is, then it isn’t invertible.

Now if you have no math background, just take what you saw at face value; a general intuition should suffice. If you want to read the mathematics behind why this is, consider [upgrading](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-regression-mathematically-explained) for more features!

If the data is too large or XTX isn’t invertible, we can use other optimization methods like gradient descent to find the optimal values of β.

#### Gradient Descent

To find the optimal values in the β vector using gradient descent, we iteratively nudge the parameter values in the direction that minimizes the loss.

If you aren’t already familiar with gradient descent, you can find an explanation in the issue linked at the start of this issue.

For gradient descent, our loss or cost function will be a little different. We’ll use the Mean Squared Error (MSE) cost function instead of SSE:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/282d579a-e01b-4964-9ec5-6058b36854ce/image.png)

Above, n is the size of the dataset or batch, y is the target value, X is the input data matrix, and β are the parameters (parameter vector), which are the coefficients of the linear equation.

The difference between SSE and MSE is that MSE averages the squared errors over all samples in the dataset or batch.

We take the gradient of the cost function or all the partial derivatives of the loss with respect to the parameter vector β (in this case, it would be the partial derivatives for β0 up to βp).

In vector form, the gradient would look something like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9daf0b4d-4144-4951-a880-879cecc41d0b/image.png)

The gradient descent algorithm will update β after every cycle through the dataset or batch by subtracting the learning rate (α) times the gradient of the loss with respect to β at a timestep k (which changes every iteration):

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b9103226-82ed-43a4-a5c5-ed29d8583d42/image.png)

Intuitively, the new values for β (β(k+1)) are computed by subtracting a term from the old values for β (β(k)).

Using gradient descent, the learning process will look something like this:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2493def2-cd7d-4362-921c-8f172fc4766d/0__9P3eQX6rq3CRebW.gif)

For linear regression to be effective, it makes the assumption that the relationship between the independent variables and the dependent variable is linear.

If this is not the case, try another algorithm for better predictions. Don’t try to fit a line to a curve!

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-regression-mathematically-explained):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Feedback, inquiries, advertising? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=linear-regression-mathematically-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team