Issue #74 Foundation Edwin Dong, Ebrahim Jahangiri
Ensemble Models, Simply Explained
Free editionPRO edition
This week’s PRO issue is accompanied by a Colab notebook testing the three main ensembling techniques: bagging, boosting, and stacking. As always, the code in this issue can be executed on a normal home PC and does not require a large amount of VRAM.
Imagine you’re watching a boxing competition. The bell for the final round rings, and it goes to the judges’ scorecard. You notice that there is a panel of three judges. Why is it that there are usually at least three judges and never just one?
Well, judges can be biased, tired, or make errors and miss important details. Accordingly, we have a panel of judges to form a collective decision.
Combining their final scores, we get a final decision that is more balanced and more fair than using only one individual judge’s score.
In essence, this is exactly what ensembling does in machine learning. The individual judgments of multiple models to make decisions are all considered or combined in order to make one final, more accurate decision.
Every AI model learns differently. One model could potentially be overfit, meaning it has memorized the training data too closely, while another might underfit, meaning it has oversimplified the true complexity of the data.
Want to learn more about overfitting/underfitting? We go over this concept in another issue, linked here.
Combining their outputs, either by averaging, voting, or stacking them, means individual mistakes can potentially be cancelled out, to an extent—the overall performance will increase.
Hence, just like the case in boxing when we can have judges to offer more than one perspective, models can also consider all outputs to make a final decision that is more accurate. Ensemble models can generalize better than singular models.
Ensembling is effectively a method to reduce variance, which is a measure of how much uncertainty a specific model contains. By combining several models that have errors in different directions, these uncertainties are smoothed out, and a more stable prediction range is output.
In simplest terms, ensembling does not make any individual model smarter, but it makes a collection of models less wrong together.
In the above diagram, the red, blue, and green waves indicate noisy models, whereas the bold, black line showcases the weighted average of the three colored models.
This edition is for PRO members.
