Boosting Models in Machine Learning, Simply Explained

In machine learning, ensembling is a very useful technique to improve model generalization and performance by combining multiple weaker models to create a stronger one.

Gradient boosting is a key ensembling technique that achieves state-of-the-art results in regression and classification, powering models like XGBoost, LightGBM, and CatBoost.

Gradient boosting is so performant; models implementing this technique have dominated Kaggle competitions and supervised real-world applications for many years.

It is important to note that gradient boosting is a supervised algorithm. For unsupervised applications, gradient boosting would not work—we would have to look at other ensembling techniques specific to unsupervised models.

The "gradient" in gradient boosting refers to its use of gradient descent to iteratively improve predictions. Gradient boosting builds multiple weak models into a strong model sequentially, with each new model correcting the errors of the previous models.

Before we continue, to get the most out of this issue, you should have a brief understanding of:

• Decision trees, knowing that they split data based on features to make predictions and that deep trees typically overfit while shallow trees underfit.

• Loss functions, the mathematical way we define how "wrong" a model is (ex., Mean Squared Error).

• The concept of a gradient, understanding that a gradient points in the direction of steepest ascent, and so the negative gradient points toward the steepest descent (the fastest way to minimize error).