This document describes the random forest algorithm for classification. Random forest creates many decision trees on random subsets of the training data and attributes. Each tree votes for a class, and the class with the most votes across all trees is the random forest prediction. The document outlines the steps to build a random forest: 1) bootstrap samples of data to create training sets, 2) select random subsets of attributes at each node, 3) grow each tree, 4) aggregate trees by majority vote. An example applies random forest to predict if a person will buy a computer based on age, income, student status, and credit rating.

1. Random Forest

2. Training Data
Age Income Student Credit_Rating Buy_NoBuy
Youth High no fair no
Youth High no excellent no
Middle_Aged High no fair yes
Senior Medium no fair yes
Senior Low yes fair yes
Senior Low yes excellent no
Middle_Aged Low yes excellent yes
Youth Medium no fair no
Youth Low yes fair yes
Senior Medium yes fair yes
Youth Medium yes excellent yes
Middle_Aged Medium no excellent yes
Middle_Aged High yes fair yes
Senior Medium no excellent no
To predict if a person will buy a Computer or Not

3. Prediction using Decision Tree
We want to predict the Buy or NoBuy decision of a person, given
that we know his/her:
• Age
• Income
• Student or Not
• Credit_Rating

4. Prediction with Decision Tree
• Decision trees are powerful and popular tools for classification
and prediction.
• We first make a list of attributes that we can measure. In our
case those are Age, Income, Student or Not, & Credit_Rating.
• We then choose a target attribute that we want to predict, in
our case it is the “Buy_NoBuy” decision.

5. Algorithms
Commonly Used Algorithms
• ID3 (Iterative Dichotomiser 3): developed in early 1980s: good
for discrete attributes
• C4.5 (improved from ID3) : Handling both continuous and
discrete attributes
• CART (Classification and Regression Tree) : developed in 1984:
good for continuous and discrete attribute

6. ID3 Algorithm
• Information gain is used to select most useful attribute for
classification/splitting
• To calculate Information Gain, we need to know Entropy
2
1
2 2
( _ ) log
9 9 5 5
log log
14 14 14 14
0.94
c
i i
i
Entropy Buy NoBuy p p

 
  


Buy_NoBuy
no
no
yes
yes
yes
no
yes
no
yes
yes
yes
yes
yes
no
Yes 9
No 5
Total 14

7. ID3 Algorithm
.
( _ , ) ( ) ( )
( ) (2,3) ( ) (4,0)
( ) (3,2)
c Age
Entropy Buy NoBuy Age P c Entropy c
P Youth Entropy P Middle Entropy
P Senior Entropy


 


Weather Buy NoBuy Total
Youth 2 3 5
Middle_Aged 4 0 4
Senior 3 2 5
Total 14
Age Buy_NoBuy
Youth no
Youth no
Middle_Aged yes
Senior yes
Senior yes
Senior no
Middle_Aged yes
Youth no
Youth yes
Senior yes
Youth yes
Middle_Aged yes
Middle_Aged yes
Senior no

8. ID3 Algorithm
2 2 2 2
2 2
5 2 2 3 3 4 4 4 0 0
( _ , ) ( log log ) ( log log )
14 5 5 5 5 14 4 4 4 4
5 3 3 2 2
( log log )
14 5 5 5 5
0.694
Entropy Buy NoBuy Age      
  

( _ , )
( _ ) ( _ , )
0.94 0.694 0.246
InformationGain Buy NoBuy Age
Entropy Buy NoBuy Entropy Buy NoBuy Age


  

9. ID3 Algorithm
Similarly:
( _ , ) 0.246
( _ , ) 0.029
( _ , ) 0.151
( _ , _ ) 0.048
InformationGain Buy NoBuy Age
InformationGain Buy NoBuy Income
InformationGain Buy NoBuy Student
InformationGain Buy NoBuy Credit Rating




Highest Information
Gain
Attribute with highest information gain (here Age), will be selected as
splitting attribute.

11. Final Decision Tree Using ID3
Using this tree, we can predict that a young person who is also a
student will buy a computer

12. Random Forest
• First proposed by Tin Kam Ho of Bell Labs in 1995.
• Random forest is an ensemble/group classifier that consists of
a large number of decision trees.
• Each Decision Tree gives their predicted value, but the final
prediction is made by a majority vote.

13. Step 1
• Take a random sample of size N with replacement from the
data (bootstrap sample).
Selected Age Income Student Credit_Rating Buy_NoBuy
X Youth High no fair no
X Middle_Aged High no fair yes
X Senior Low yes excellent no
X Middle_Aged Low yes excellent yes
X Senior Medium yes fair yes
X Youth Medium yes excellent yes
.
.
Nth Senior Medium no excellent no

14. Step 2
• At each node, take a random sample of attributes of size m
(without replacement). M being total number of attributes,
such that m<M.
• Generally m=sqrt(M)
• Let’s say Age &
Credit_Rating are the
attributes selected
Selected X X Prediction
Age Income Student Credit_Rating Buy_NoBuy
X Youth High no fair no
X Middle_Aged High no fair yes
X Senior Low yes excellent no
X Middle_Aged Low yes excellent yes
X Senior Medium yes fair yes
X Youth Medium yes excellent yes
.
.
Nth Senior Medium no excellent no

15. Information Gain
• From Age
• From Credit_Rating

16. Step 3
• Construct a split by using the m attributes selected in Step 2,
• Let’s say “Age” is selected for the split, can
be done by Information Gain method.
Selected X X Prediction
Age Credit_Rating Buy_NoBuy
X Youth fair no
X Middle_Aged fair yes
X Senior excellent no
X Middle_Aged excellent yes
X Senior fair yes
X Youth excellent yes
.
.
X Senior excellent no
Age
Youth Senior
Middle

17. Step 4
• Repeat Steps 2 and 3 for each subsequent split until the tree is
complete.
• Say, for Age = Youth, let Income
& Credit_Rating are the
attributes selected at random.
Selected X X X Prediction
Age Income Credit_Rating Buy_NoBuy
X Youth High fair no
X
X
X
X
X Youth Medium excellent yes
.
.
Nth
Age
Youth Senior
Middle

18. Information Gain
• From Income
• From Credit_Rating

19. Step 4
• Out of Income & Credit_Rating, say Income is selected for the
split, as in step 3, using information Gain method.
Selected X X X Prediction
Age Income Credit_Rating Buy_NoBuy
X Youth High fair no
X
X
X
X
X Youth Medium excellent yes
.
.
Nth
Age
Income
Youth Senior
Middle
High Medium

20. Step 5
• Repeat steps 1 to 4 to create a large number of decision trees,
let’s say we create 4 trees.
• Make prediction using each decision tree.
• Make final prediction by a majority vote over the set of trees.

21. Prediction
• Predict using random forest if a young student with low
income and fair credit rating will buy a computer or not.
Tree # Predicted (Buy_NoBuy)
1 Buy
2 Buy
3 NoBuy
4 Buy
Final Prediction on the basis of
majority vote
Buy

22. Supply Chain Example