# Supervised ML: Decision Trees

#51 May 19, 2025 Foundation

https://www.aibutsimple.com/p/supervised-ml-decision-trees

Imagine you are planning a weekend picnic. You might wonder if it’s sunny and if it’s too hot to stay outdoors. Perhaps, if it’s cloudy, you might wonder if it’s windy or raining. You go through a sequence of questions, and it guides you towards a final decision—whether to go for a picnic or to stay home.

By breaking more complex decisions into a series of simpler questions, decision trees provide an intuitive and mathematical way to make predictions.

Decision trees are foundational models used in supervised learning for regression and classification. They are also referred to as CARTs, the classification and regression tree algorithm.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/137e219e-d278-4296-a8a8-6b14406917a6/image.png)

---

### Get Your Free ChatGPT Productivity Bundle

[Mindstream](https://www.mindstream.news/subscribe?utm_medium=incentivized-signup&utm_source=beehiivads&utm_campaign=XIAEVJHTGJ&utm_term=Version-D&_bhiiv=opp_ebcf3646-43d5-4358-b79a-cf2e5e9efbad_8c74c9cb) brings you 5 essential resources to master ChatGPT at work. This free bundle includes decision flowcharts, prompt templates, and our 2025 guide to AI productivity.

Our team of AI experts has packaged the [most actionable ChatGPT hacks](https://www.mindstream.news/subscribe?utm_medium=incentivized-signup&utm_source=beehiivads&utm_campaign=XIAEVJHTGJ&utm_term=Version-D&_bhiiv=opp_ebcf3646-43d5-4358-b79a-cf2e5e9efbad_8c74c9cb) that are actually working for top marketers and founders. Save hours each week with these proven workflows.

It's completely free when you subscribe to our daily AI newsletter.

[Subscribe to get instant access](https://www.mindstream.news/subscribe?utm_medium=incentivized-signup&utm_source=beehiivads&utm_campaign=XIAEVJHTGJ&utm_term=Version-D&_bhiiv=opp_ebcf3646-43d5-4358-b79a-cf2e5e9efbad_8c74c9cb)

---

Decision trees are categorized based on the type of target variable they predict.

Classification trees are used when the target variable is categorical—“Will it rain?” The resulting prediction will be a class.

Following the same logic, regression trees are used when the target variable is numerical—“How much rainfall?” The prediction will be a numerical value.

Both types of trees are built the same way but handle the output differently: classification trees use majority voting to predict classes, while regression trees use the mean or median of all the probable values.

The components of a decision tree form a tree-like structure that makes them simple to interpret:

- The root node represents the entire dataset; it’s the starting point of the decision process. Decision nodes are tests on features (“Is it raining?”), while branches represent the outcomes of the tests (“Yes” or “No”). The branches are represented as arrows in the diagram below.
- Finally, leaf nodes are the final predictions or decisions, whether they are classes or numerical values.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2da6a8b7-6152-44ff-ac4f-e3f573163e8d/Copy_of_Copy_of_Copy_of_Copy_of_CV_illust.png)

### Why Decision Trees?

The decision tree’s structure is similar to human decision-making. Comparable to flowcharts, their tree structure is intuitive and can be visualized as a simple diagram with straightforward logic, making them easy to learn and implement.

- Decision trees are a popular foundational machine learning algorithm and, in most cases, are used to introduce machine learning concepts to those who don’t already have an advanced technical background.

They require no feature scaling or normalization, unlike algorithms like SVMs or neural networks, which require more data processing.

Furthermore, they are effective for more complex non-linear tasks and handle outliers decently well for their simplicity.

### Decision Tree Algorithm

Building a decision tree involves recursively splitting the dataset (which creates a branch) into subsets based on the feature that best separates the data according to the target variable.

In other words, the decision tree splits variables into different subgroups, where the data points in each subgroup are more similar to each other (more homogenous) in terms of the target variable.

For instance, if we want to classify a target variable as either blue or red, we can split the dataset based on features (like size) into subgroups where each resulting group contains a more dominant class—either more blues than reds or more reds than blues.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f878558f-ba81-4d3c-851e-8c380674c46c/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__1_.png)

To start the tree-building process, we need to select the best feature to split at every point. The algorithm evaluates each feature to determine which one best reduces uncertainty in the data.

Two common metrics that we use to select the feature are information gain and Gini impurity—we typically just choose one to use.

#### Entropy

Before explaining information gain, it is crucial to understand what entropy is.

Entropy is a metric to measure the impurity in a given feature; it specifies randomness in data. If you know a bit about statistics, entropy can be roughly thought of as how much variance the data has. For example, if we had a bag of marbles:

- A dataset of only blue marbles would have very low (zero) entropy.
- A dataset of assorted blue, yellow, and red marbles would have relatively high entropy.

So we want a feature to split that decreases the entropy—the higher the entropy, the harder it is to draw any conclusion.

Mathematically, entropy is calculated as:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f9e8f5de-9b23-4a44-bbd0-565a796216bb/image.png)

Here, S represents our dataset, pi is the probability of randomly picking an element of class i (proportions of instances), and c represents the total number of classes.

To better understand entropy, let’s extend our marble example. If we have a dataset of 3 blue marbles, 2 red marbles, and 1 yellow marble:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e9cf8352-9922-4765-acdf-e5130dfbab9a/image.png)

We know that pb is equal to 3/6, which simplifies to 1/2; pr = 2/6 = 1/3; and py = 1/6. We can then calculate our entropy value of our dataset:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3c48af17-d235-4ddb-9b40-92fbdf810a3e/image.png)

#### Information Gain

Information gain measures the reduction in entropy after splitting. Higher information gain means more entropy removed. It can be described using this equation:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f9808982-d30d-48b0-89d8-aa5a09aeb0dc/image.png)

Where Sv is the subset where the feature has value v, |Sv| is the number of data points in subset Sv, and |S| is the total number of data points in the dataset. |Sv| and |S| act as weights for a weighted sum.

Let’s continue our marble example from before. Let’s say that this time, we have 3 blue marbles and 3 red marbles, and we need to classify them as blue or red.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/99ee6ac9-d914-4100-a3d7-cacfebf7bac2/Copy_of_Copy_of_Copy_of_CV_illust__5_.png)

Before the split, we had 3 blues and 3 reds, so the entropy was:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/00c4d3e7-423a-45e4-ac52-d9d59cdbd638/image.png)

If we look at the left side, with the nonoptimal split, we have two entropy values—one for the left and one for the right. The right branch has 2 reds, so Eright = 0, since all of the marbles are the same color. The left branch has 3 blues and 1 red, and its entropy is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4261e985-0da3-430f-b3e0-b65b24ade8ae/image.png)

Now that we have the entropies for both sides (branches) of the nonoptimal split, we can determine the effectiveness of the split by weighing the entropy of each branch by how many elements it has.

Since the left branch has 4 marbles and the right branch has 2, we weight them by about 0.667 and about 0.333, respectively.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/98464024-2326-4af1-b0ee-7fd3e5304b38/image.png)

So for the nonoptimal split value v=2, the information gain is about 0.459.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/44fcac5a-a9e3-4836-99ca-de76b9062ddc/image.png)

On the other hand, it turns out that the optimal split has an information gain of 1. Why is this? Well, both Eright and Eleft are 0 since the left and right subgroups only have one color of marbles in them. Therefore, Esplit is also equal to 0, and IG = 1 - 0 = 1.

This is intuitive, since as the overall entropy decreases, we would expect the information gain to increase—the more entropy or randomness that is removed, the more “information” we gain.

#### Gini Impurity

Alternatively, we can use the Gini index to quantify the quality of a split. Similar to entropy, the Gini index measures impurity, with lower values indicating better splits. To choose a feature to split, the feature with the lowest Gini index is chosen.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f9bcf1ac-51ad-4835-ab34-d3a7d0c1f0a4/image.png)

Above, pi represents the probability of choosing an element of class i, and c is the total number of classes.

Let’s say we have our 6 marbles again: 3 blue and 3 red. We may have an imperfect split or an optimal one.

If we split the dataset non-optimally the same way we did before, our left subgroup will have 3 blue marbles and 1 red marble. The probability of choosing a red marble is 1/4, and the probability of choosing a blue one is 3/4, the same as before.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/758ddaa6-4c37-474e-9356-6eb388a7c3ed/image.png)

Like our previous example, our right branch contains only red marbles. Therefore, it will have a Gini impurity of 0, since the sum of probabilities equals 1.

If we split the dataset optimally, both right and left Gini impurities will equal 0, representing a low impurity. To obtain the overall Gini impurity, we average the Gini impurities of both the right and left subgroups using weighted values based on the number of entries in each subgroup.

---

### Seeking impartial news? Meet 1440.

Every day, 3.5 million readers turn to [1440](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_impartial&_bhiiv=opp_355da45a-ec37-454a-886e-05d704d62693_1b75ca79) for their factual news. We sift through 100+ sources to bring you a complete summary of politics, global events, business, and culture, all in a brief 5-minute email. Enjoy an impartial news experience.

[Join for free today!](https://l.join1440.com/bh?utm_source=beehiiv&utm_medium=cpc&utm_campaign=ai_but_simple_XIAEVJHTGJ&utm_content=prospecting_impartial&_bhiiv=opp_355da45a-ec37-454a-886e-05d704d62693_1b75ca79)

---

#### Splitting the Dataset

After choosing our splitting metric, we divide the data into subsets based on the chosen feature’s values, whether the feature is numerical or categorical.

- For categorical features, each unique value or a combination of values can represent a potential split. For example, “Is it raining?” results in a yes or a no split.
- For numerical features, potential split points are typically considered between sorted unique values of the feature. For instance, “Size > 5?” results in a split based on whether the size is greater or less than 5.

For each possible split across all features, the Gini impurity or information gain of the resulting child nodes is calculated.

As previously shown, a weighted average of the impurity of the child nodes is often used, weighted by the proportion of data points in each child node.

- In the end, the feature and the corresponding split point that result in the greatest information gain or the largest reduction in Gini impurity are selected.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7b33399f-88a7-46c5-8d97-d9f2cf049321/iris-TD-5.png)

The best feature and split point are chosen recursively at each node until a stopping criterion is met, which is either when:

- All data points in a subgroup belong to the same class.
- The maximum tree depth is reached.
- Further splits do not improve the model significantly.

#### Obtaining the Prediction

The terminal or leaf nodes of the decision tree are either assigned a class label for classification or a numerical value for regression.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/55773d9d-8f2f-4fd4-aefa-52eb2b61592c/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__1_.png)

If it’s a classification problem, the class with the most data points at the node will be chosen. For regression, an average of all the values of the data points is chosen.

After the whole tree is built through recursive partitioning, we can make predictions on new data by following the branches from the root to a leaf based on the split criteria at each node.

---

In computer science terminology, decision trees follow a top-down, greedy approach known as recursive binary splitting.

- A greedy algorithm is one that selects the best option at each step. In this case, the best split is taken at each step, instead of looking ahead to pick a split that may be more optimal in the long term.

---

#### Pruning

Decision trees are great introductory models, but they suffer quite a bit from instability and overfitting. To mitigate overfitting, a common technique applied to decision trees is pruning. Pruning decreases the size of the decision tree without reducing accuracy.

Due to the greedy nature of the decision tree algorithm, often times when we build the tree, we may end up with unnecessary nodes that don’t add much value.

This is exactly where pruning comes in—we cut down these nodes to make the tree simpler and more optimal.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d1aa47a4-0022-4425-85f0-802073a6040b/image.png)

There are two main types of pruning:

Pre-pruning stops the tree from growing too large by setting limits, like maximum depth or minimum data points needed at each step. We decide early on how big the tree should get.

Post-pruning builds the tree fully first, then removes nodes and branches after. There are many different types of post-pruning, including cost complexity pruning, reduced error pruning, and more.

- Cost complexity pruning assigns a cost to each subtree based on accuracy and complexity, pruning the subtree with the lowest cost.
- Reduced error pruning removes branches if they don’t affect the overall accuracy too much, and the process typically requires an additional validation dataset.

#### Disadvantages of Decision Trees

Some disadvantages of decision trees include instability (small changes in data can result in very different trees) and a limited capability for regression and classification since it’s such a simple model.

Additionally, the recursive process for building trees for very large datasets with thousands of features can be time-consuming and computationally expensive. The greedy splitting process finds locally optimal solutions, which may not be globally optimal.

Finally, they are decently susceptible to overfitting, and their decision boundaries can look very complicated, leading to poor generalization. Techniques like pruning or setting maximum values for depth can reduce the effects.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/42311006-7f4f-4d90-97a3-b1b65cbf64bc/image.png)

A decision tree’s overfit decision boundary

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=supervised-ml-decision-trees):

Sushant Waidande

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=supervised-ml-decision-trees)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=supervised-ml-decision-trees)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team