Decision tree random forest classifier

2. Glucose Age Diabetes
78 26 No
85 31 No
89 21 No
100 32 No
103 33 No
107 31 Yes
110 30 No
115 29 Yes
126 27 No
115 32 Yes
116 31 Yes
118 31 Yes
183 50 Yes
189 59 Yes
197 53 Yes
Glucose
Age Glucose
75 < G < 90
G > 90
20 < Age <= 31
Y-0, N-3
Prediction
No (Majority)
100 <= G <= 110
G > 110
Y-1, N-3
Prediction
No (Majority)
Age
30 <= Age < 34
Age
Glucose
110 < G < 127 G > 180
Age
Y-4, N-1
Prediction
Yes (Majority)
Age >= 50
Y-3, N-0
Prediction
Yes (Majority)
A few sample observation on diabetes
result along with glucose and age are
given.
Attempting a decision tree prediction model
30 < Age < 34
Root of the Tree
Branch
Leaf

3.  In the last slide example, the first split divided the into groups of 12 and 3.
 What if decision tree is creating a split for each sample?
 Then for the samples from train data, model can do accurate predictions.
 But for other data, model may perform worse.
 It is better to do the branch split with a minimum number of samples greater
than 1. (Default value is 2)
 We can control minimum sample split count of decision tree model with help
of “min_samples_split” argument.
Glucose
75 < G < 90
G > 90
No. of
samples = 3
No. of
samples = 12
 The samples count in last splits (leaf of decision tree) is also important for the
model
 Default value for minimum sample count in leaf of decision tree is 1.
 We can control the minimum count of samples in a decision tree model with
help of “min_samples_leaf” argument.

4.  Entropy is the measures of impurity, disorder or uncertainty in a bunch of
examples.
 In a decision tree it is the impurity in the split.
 Entropy value ranges from 0 to 1. Maximum impurity represents 1.
Entropy H(s) = -P(Yes) * log2(P(Yes)) -P(No) * log2(P(No))
Possible split with 6 samples (label “Yes” or “No”) with entropy in each split are shown below.
Entropy in case of more than 2 labels
Gini
Gini is another method of
impurity measure in the
decision tree split.
Gini = 1- (P(Yes)^2 + P(No)^2)
In case of more than 2 labels
Gini = 1 - ∑ (Pi)^2
No. of “Yes” No. of “No” P(Yes) P(No) Entropy Notes
0 6 0 1 0 pure split
1 5 0.17 0.83 0.65
2 4 0.33 0.67 0.92
3 3 0.5 0.5 1 maximum imprity
4 2 0.67 0.33 0.92
5 1 0.83 0.17 0.65
6 0 1 0 0 pure split
Gini
0
0.28
0.44
0.50
0.44
0.28
0

5. Information Gain helps to measure the reduction in entropy or surprise by splitting a dataset according to a given value of a random variable.
Information Gain IG(S, a) = H(S) – H(S | a)
IG(S, a) = information for the dataset S for the variable a
H(S) = the entropy for the dataset before any change
H(S | a) is the conditional entropy for the dataset given the variable a.
A larger information gain suggests a lower entropy group or groups of samples.
Overall Entropy = -(8/15)*log2(8/15) – (7/15) * log2(7/15) = 0.996
Feature - Gender:
Entropy of ‘Male’ = -(4/8)*log2(4/8)-(4/8)*log2(4/8) = 1
Entropy of ‘Female’ = -(4/7)*log2(4/7)-(3/7)*log2(3/7) = 0.985
Weighted entropy = (8/15)*1 + (7/15)*0.985 = 0.993
Information Gain for gender feature = 0.996 – 0.993 = 0.003
Information Gain for exercise feature = 0.996 – 0.884 = 0.112
Gender Exercise Diabetes
Male Regular No
Female Irregular No
Male Regular No
Male Regular No
Female No No
Male Irregular Yes
Female No No
Female Regular Yes
Male Regular No
Female Regular Yes
Female Regular Yes
Female Irregular Yes
Male Irregular Yes
Male No Yes
Male Irregular Yes

7. A Random Forest model creates many decision trees and combine the output.
x1 x2 x3 x4 x5 y
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
x1 x2 x3
1
2
3
4
x4 x5 y
13
14
15
16
17
x3 x4 x5 y
1
9
10
11
12
13
x1 x2 x3 y
1
2
3
4
Sample-1
Sample-2
Sample-3
Sample-n
Decision
Tree-1
Decision
Tree-2
Decision
Tree-3
Decision
Tree-n
Combined
output
M
A
J
O
R
I
T
Y
Creating multiple models and
combining the output is called
Bagging.

8. Number of trees in Random Forest can be managed by “n_estimators” argument.
Default number of trees is 100.
Random Forest reduces overfitting in decision trees and helps to improve the
accuracy.