Decision trees are a type of predictive model that breaks down a dataset into smaller subsets based on the values of predictor variables. They can be used for regression by measuring the reduction in standard deviation when splitting on variables. The algorithm works top-down, recursively splitting the data on the variable that produces the greatest standard deviation reduction until reaching a minimum number of observations or coefficient of variation. Leaf nodes then contain the average target value for that subset of the data.

1. Map > Data Science > Predic ng the Future > Modeling > Regression > Decision Tree
Decision Tree - Regression
Decision tree builds regression or classifica on models in the form of a tree structure. It breaks down a dataset into
smaller and smaller subsets while at the same me an associated decision tree is incrementally developed. The final
result is a tree with decision nodes and leaf nodes. A decision node (e.g., Outlook) has two or more branches (e.g.,
Sunny, Overcast and Rainy), each represen ng values for the a ribute tested. Leaf node (e.g., Hours Played) represents
a decision on the numerical target. The topmost decision node in a tree which corresponds to the best predictor called
root node. Decision trees can handle both categorical and numerical data.
Decision Tree Algorithm
The core algorithm for building decision trees called ID3 by J. R. Quinlan which employs a top-down, greedy search
through the space of possible branches with no backtracking. The ID3 algorithm can be used to construct a decision
tree for regression by replacing Informa on Gain with Standard Devia on Reduc on.
Standard Devia on
A decision tree is built top-down from a root node and involves par oning the data into subsets that contain instances
with similar values (homogenous). We use standard devia on to calculate the homogeneity of a numerical sample. If
the numerical sample is completely homogeneous its standard devia on is zero.
a) Standard devia on for one a ribute:
Standard Devia on (S) is for tree building (branching).
Coefficient of Devia on (CV) is used to decide when to stop branching. We can use Count (n) as well.
Average (Avg) is the value in the leaf nodes.
b) Standard devia on for two a ributes (target and predictor):
Use machine
Intelligence to
improve efficiency
and boost profit
AD S V IA CA RB ON
Decision Tree Regression https://www.saedsayad.com/decision_tree_reg.htm
1 of 4 08-04-2021, 05:05 pm

2. Standard Devia on Reduc on
The standard devia on reduc on is based on the decrease in standard devia on a er a dataset is split on an a ribute.
Construc ng a decision tree is all about finding a ribute that returns the highest standard devia on reduc on (i.e., the
most homogeneous branches).
Step 1: The standard devia on of the target is calculated.
Standard devia on (Hours Played) = 9.32
Step 2: The dataset is then split on the different a ributes. The standard devia on for each branch is calculated. The
resul ng standard devia on is subtracted from the standard devia on before the split. The result is the standard
devia on reduc on.
Step 3: The a ribute with the largest standard devia on reduc on is chosen for the decision node.
Step 4a: The dataset is divided based on the values of the selected a ribute. This process is run recursively on the non-
leaf branches, un l all data is processed.
Use machine
Intelligence to
improve efficiency
and boost profit
AD S V IA C A RB ON
Decision Tree Regression https://www.saedsayad.com/decision_tree_reg.htm
2 of 4 08-04-2021, 05:05 pm

3. In prac ce, we need some termina on criteria. For example, when coefficient of devia on (CV) for a branch becomes
smaller than a certain threshold (e.g., 10%) and/or when too few instances (n) remain in the branch (e.g., 3).
Step 4b: "Overcast" subset does not need any further spli ng because its CV (8%) is less than the threshold (10%). The
related leaf node gets the average of the "Overcast" subset.
Step 4c: However, the "Sunny" branch has an CV (28%) more than the threshold (10%) which needs further spli ng.
We select "Windy" as the best best node a er "Outlook" because it has the largest SDR.
Because the number of data points for both branches (FALSE and TRUE) is equal or less than 3 we stop further
branching and assign the average of each branch to the related leaf node.
Use machine
Intelligence to
improve efficiency
and boost profit
AD S V IA C A RB ON
Decision Tree Regression https://www.saedsayad.com/decision_tree_reg.htm
3 of 4 08-04-2021, 05:05 pm

4. Step 4d: Moreover, the "rainy" branch has an CV (22%) which is more than the threshold (10%). This branch needs
further spli ng. We select "Windy" as the best best node because it has the largest SDR.
Because the number of data points for all three branches (Cool, Hot and Mild) is equal or less than 3 we stop further
branching and assign the average of each branch to the related leaf node.
When the number of instances is more than one at a leaf node we calculate the average as the final value for the
target.
Exercise
Try to invent a new algorithm to construct a decision tree from data using MLR instead of average at the leaf node.
Use machine
Intelligence to
improve efficiency
and boost profit
AD S V IA C A RB ON
Decision Tree Regression https://www.saedsayad.com/decision_tree_reg.htm
4 of 4 08-04-2021, 05:05 pm