# Boosting Models in Machine Learning, Simply Explained

#80 Dec 8, 2025 Foundation

https://www.aibutsimple.com/p/boosting-models-in-machine-learning-simply-explained

In machine learning, ensembling is a very useful technique to improve model generalization and performance by combining multiple weaker models to create a stronger one.

Gradient boosting is a key ensembling technique that achieves state-of-the-art results in regression and classification, powering models like XGBoost, LightGBM, and CatBoost.

Gradient boosting is so performant; models implementing this technique have dominated Kaggle competitions and supervised real-world applications for many years.

It is important to note that gradient boosting is a [supervised algorithm](https://www.aibutsimple.com/p/supervised-machine-learning-simplified?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained). For unsupervised applications, gradient boosting would not work—we would have to look at other ensembling techniques specific to unsupervised models.

The "gradient" in gradient boosting refers to its use of gradient descent to iteratively improve predictions. Gradient boosting builds multiple weak models into a strong model sequentially, with each new model correcting the errors of the previous models.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/139efd22-a1ba-4532-a810-825897324fbc/Gradient_Boosting_Algorithm.png)

Before we continue, to get the most out of this issue, you should have a brief understanding of:

- Decision trees, knowing that they split data based on features to make predictions and that deep trees typically overfit while shallow trees underfit.
- Loss functions, the mathematical way we define how "wrong" a model is (ex., Mean Squared Error).
- The concept of a gradient, understanding that a gradient points in the direction of steepest ascent, and so the negative gradient points toward the steepest descent (the fastest way to minimize error).

---

### AI that actually handles customer service. Not just chat.

[![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/eced2e2d-071e-409d-b0ad-9ec2ce6e1a41/Beehiiv_Ad_Network_Campaign_-__CX_Retail__1200x600_product__1_.jpg)](https://www.gladly.ai/get-started/?utm_campaign=parent-2024-08-demo-request-evergreen-demo-demandg&utm_source=beehiiv&utm_medium=content-syndication&_bhiiv=opp_fcd2aa0d-eee4-463f-964b-270fbcf35e16_20cc6b9f)

Most AI tools chat. [Gladly](https://www.gladly.ai/get-started/?utm_campaign=parent-2024-08-demo-request-evergreen-demo-demandg&utm_source=beehiiv&utm_medium=content-syndication&_bhiiv=opp_fcd2aa0d-eee4-463f-964b-270fbcf35e16_20cc6b9f) actually resolves. Returns processed. Tickets routed. Orders tracked. FAQs answered. All while freeing up your team to focus on what matters most — building relationships. [See the difference.](https://www.gladly.ai/get-started/?utm_campaign=parent-2024-08-demo-request-evergreen-demo-demandg&utm_source=beehiiv&utm_medium=content-syndication&_bhiiv=opp_fcd2aa0d-eee4-463f-964b-270fbcf35e16_20cc6b9f)

[Find out how](https://www.gladly.ai/get-started/?utm_campaign=parent-2024-08-demo-request-evergreen-demo-demandg&utm_source=beehiiv&utm_medium=content-syndication&_bhiiv=opp_fcd2aa0d-eee4-463f-964b-270fbcf35e16_20cc6b9f)

---

### How Does Gradient Boosting Work?

Imagine you're trying to solve a typical supervised learning problem like predicting house prices based on features like square footage and location. A single model might perform decently but might not provide the complexity or accuracy required for the problem.

Gradient boosting iteratively refines predictions by reducing the errors or residuals of previous models, resulting in a more accurate overall prediction.

Mathematically, it tries to find some function F(x) that maps input features x to a target y while minimizing a loss function L(y, F(x)).

This loss function can vary for both classification and regression.

Regression losses include mean squared error (MSE), mean absolute error (MAE), root mean squared error (RMSE), and many other variations.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2018d5bb-b778-47f1-afae-ff143052168c/Gradient_Boosting_Loss_Functions.png)

Classification losses are mainly cross-entropy variants: log loss for binary classification and cross entropy for multi-class classification.

If you want to learn more about cross-entropy and loss functions, please consult the previous issues below, where we deep dive into the topic:

- [KL Divergence and Cross-Entropy, Simply Explained (Intermediate)](https://www.aibutsimple.com/p/kl-divergence-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained)

Gradient boosting constructs this function F(x) as a weighted sum of T weak learners ht(x), which are typically decision trees but can be other models.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4f09ba71-6de0-4a65-9333-e868f6f9defd/image.png)

Here, Ftotal(x) is the total ensemble model after T steps, ht(x) is the new base learner or weak model, and γt​ is the step size or learning rate for each model.

The model starts with an initial prediction F0(x), usually a constant like the mean of the true labels (y) for regression. It then iteratively adds trees to reduce the loss—the sum is obtained after many iterations of trees, typically after a stopping criterion is met.

Notice that the base learner is multiplied with the learning rate. This is a key feature of gradient boosting called shrinkage.

Shrinkage scales the contribution of each new model using the learning rate rather than scaling the gradient.

- Smaller learning rates mean the contribution of each tree is smaller, reducing the risk of overfitting, but requiring more trees to achieve the same performance.
- Larger learning rates mean each tree has a higher impact. This can potentially lead to overfitting.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2a480bb1-571e-4287-87ba-c8799a879fde/Gradient_Boosting_Shrinkage_Learning_Rates__1_.png)

Essentially, instead of asking, "How much should I change a weight to reduce loss?" gradient boosting asks, "What new function (model) can I add to my current ensemble to reduce the overall loss?"

### Gradient Descent for Gradient Boosting

If you recall standard linear regression, we use gradient descent to find the optimal parameters (weights and bias) that minimize a loss function.

We calculate the derivative of the loss with respect to a parameter and move the parameter in the opposite direction. In gradient boosting, the optimization process stays relatively similar.

However, unlike traditional gradient descent, which optimizes parameters in a numerical space, gradient boosting performs gradient descent in function space.

Instead of updating numerical parameters θ, we update the function F(x).

To be more precise, in each iteration, gradient boosting computes the gradient of the loss function with respect to predictions, called the pseudo-residuals, and then trains a new weak model F(x) to minimize these pseudo-residuals.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/42b5be4b-dec1-4d8c-b401-e40701d94bb3/Gradient_Boosting_Function_Space_Pseudo_Residuals.png)

We can describe the gradient update with a version of the same previous total prediction equation! Notice how it looks very similar to a gradient descent update step.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b1c1c90a-7fbd-462c-8acc-323d8125e839/image.png)

It is important to note that during training, the features are the initial supervised dataset features x; however, the target is not the supervised target feature (e.g., the house price in dollars).

The target is simply the pseudo-residual we just calculated; the models minimize this pseudo-residual generated from our loss function, which will increase the overall accuracy.

In each iteration, we only take a small step in the direction suggested by the new tree. The "shrinkage" strategy mentioned earlier slows down the learning process but significantly improves generalization to new data.

### Regularization for Gradient Boosting

To prevent overfitting, gradient boosting incorporates regularization techniques. The first technique is shrinkage, as mentioned previously. The learning rate γt scales down the contribution of each tree.

We can also constrain the tree, limiting tree depth, number of leaves, or minimum child weight. Finally, we can also subsample the data, using a fraction of data or features at each iteration.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained)

---

### Applications of Gradient Boosting: XGBoost

[XGBoost (eXtreme Gradient Boosting)](https://arxiv.org/pdf/1603.02754?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained) is effectively a more optimized implementation of the gradient boosting we just discussed. It performs extremely well at real-world tasks and dominates Kaggle competitions.

XGBoost implements additional techniques to make gradient boosting scalable, fast, and accurate. It contains all the necessities for production—it cleverly optimizes hardware and memory.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ea639032-63cf-4403-82b0-b74471985796/XGBoost.png)

So how is XGBoost actually different from regular gradient boosting?

#### Regularization

One difference is that in plain gradient boosting, the goal is simply to minimize the loss function, while XGBoost changes the objective function (the minimization target function) to include regularization directly.

The objective function at step t is defined below, with the left side being the standard loss function (L) and the right side being the regularization term (Ω).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/936fd838-3498-42e8-8895-6b31277d17dc/image.png)

The regularization of XGBoost also penalizes the complexity of the trees as it builds them, penalizing the number of leaves and discouraging the tree from growing too complex.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/688d0eb4-9b53-41d7-842e-1b95da27d136/image.png)

How XGBoost’s regularization works

In addition, XGBoost regulates leaf weights using L1/L2 regularization, making it so that any single leaf has a lower chance of giving a highly weighted prediction, reducing overfitting.

#### Approximate Algorithm

For massive datasets, checking every possible split point is too slow. To solve this problem, XGBoost introduces the Approximate Algorithm, or Weighted Quantile Sketch.

In this algorithm, the model looks at the distribution of the gradients (errors) and divides the data into buckets based on where the errors are highest.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a2e6381f-a8be-4132-a7b8-890a613cb301/image.png)

To speed up computation and reduce memory usage, the model only checks a certain number of candidate split points that represent the distribution well. This makes it feasible to train on datasets that are larger and don’t fit into memory.

#### Advanced Techniques and Hardware Optimization

XGBoost also takes advantage of more advanced tree techniques like tree pruning, which grows the tree to a set maximum depth and then removes nodes backwards, and sparsity awareness, which learns a default direction for missing data for every split to speed up missing data handling.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/65c160da-c7ad-4f55-ba24-60154866edc3/image.png)

Sparsity awareness algorithm

XGBoost also optimizes on the hardware level: it uses parallelization to speed up computation. Generally, we cannot build trees in parallel since the algorithm is sequential (model 2 needs model 1), so how does XGBoost parallelize?

The answer is that XGBoost parallelizes node splitting. In the paper, they call this the “cache-aware algorithm.” In a standard decision tree, finding the best split requires scanning every feature value for every data point to calculate the [information gain](https://www.aibutsimple.com/p/supervised-ml-decision-trees?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained) metric.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4a251b83-7163-4bcd-9dc2-603735d70f67/image.png)

Cache-aware algorithm (blue) vs. basic algorithm (red)

XGBoost pre-sorts data and stores it in blocks, then uses multiple processor cores to calculate the splits for different features at the same time.

### Algorithm Summary

The gradient boosting algorithm loop can be summarized as:

1. Initialize the base model.
2. Compute pseudo-residuals for the current model at each step.
3. Train a weak learner to those pseudo-residuals.
4. Find the optimal learning rate for shrinkage.
5. Update the model by adding the weighted weak learner.

Beyond plain gradient boosting, we can use more complicated algorithms like XGBoost, which utilize regularization and other optimizations to speed up computation and improve performance.

Overall, boosting models are very performant on supervised tasks. In our [PRO article](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained), we dive directly into coding some boosting models in Python.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=boosting-models-in-machine-learning-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team