This document discusses decision trees and random forests for classification problems. It explains that decision trees use a top-down approach to split a training dataset based on attribute values to build a model for classification. Random forests improve upon decision trees by growing many de-correlated trees on randomly sampled subsets of data and features, then aggregating their predictions, which helps avoid overfitting. The document provides examples of using decision trees to classify wine preferences, sports preferences, and weather conditions for sport activities based on attribute values.

1. From decision trees to
random forests
Viet-Trung Tran

2. Decision tree learning
• Supervised learning
• From a set of measurements,
– learn a model
– to predict and understand a phenomenon

3. Example 1: wine taste preference
• From physicochemical properties (alcohol, acidity,
sulphates, etc)
• Learn a model
• To predict wine taste preference (from 0 to 10)
P.
Cortez,
A.
Cerdeira,
F.
Almeida,
T.
Matos
and
J.
Reis,
Modeling
wine
preferences
by
data
mining
from
physicochemical
proper@es,
2009

4. Observation
• Decision tree can be interpreted as set of
IF...THEN rules
• Can be applied to noisy data
• One of popular inductive learning
• Good results for real-life applications

5. Decision tree representation
• An inner node represents an attribute
• An edge represents a test on the attribute of
the father node
• A leaf represents one of the classes
• Construction of a decision tree
– Based on the training data
– Top-down strategy

6. Example 2: Sport preferene

7. Example 3: Weather & sport practicing

8. Classification
• The classification of an unknown input vector is done by
traversing the tree from the root node to a leaf node.
• A record enters the tree at the root node.
• At the root, a test is applied to determine which child node
the record will encounter next.
• This process is repeated until the record arrives at a leaf
node.
• All the records that end up at a given leaf of the tree are
classified in the same way.
• There is a unique path from the root to each leaf.
• The path is a rule which is used to classify the records.

9. • The data set has five attributes.
• There is a special attribute: the attribute class is the class
label.
• The attributes, temp (temperature) and humidity are
numerical attributes
• Other attributes are categorical, that is, they cannot be
ordered.
• Based on the training data set, we want to find a set of rules
to know what values of outlook, temperature, humidity and
wind, determine whether or not to play golf.

10. • RULE 1 If it is sunny and the humidity is not above 75%,
then play.
• RULE 2 If it is sunny and the humidity is above 75%, then
do not play.
• RULE 3 If it is overcast, then play.
• RULE 4 If it is rainy and not windy, then play.
• RULE 5 If it is rainy and windy, then don't play.

11. Splitting attribute
• At every node there is an attribute associated with
the node called the splitting attribute
• Top-down traversal
– In our example, outlook is the splitting attribute at root.
– Since for the given record, outlook = rain, we move to the
rightmost child node of the root.
– At this node, the splitting attribute is windy and we find
that for the record we want classify, windy = true.
– Hence, we move to the left child node to conclude that
the class label Is "no play".

14. Decision tree construction
• Identify the splitting attribute and splitting
criterion at every level of the tree
• Algorithm
– Iterative Dichotomizer (ID3)

15. Iterative Dichotomizer (ID3)
• Quinlan (1986)
• Each node corresponds to a splitting attribute
• Each edge is a possible value of that attribute.
• At each node the splitting attribute is selected to be the
most informative among the attributes not yet considered in
the path from the root.
• Entropy is used to measure how informative is a node.

17. Splitting attribute selection
• The algorithm uses the criterion of information gain
to determine the goodness of a split.
– The attribute with the greatest information gain is taken
as the splitting attribute, and the data set is split for all
distinct values of the attribute values of the attribute.
• Example: 2 classes: C1, C2, pick A1 or A2

18. Entropy – General Case
• Impurity/Inhomogeneity measurement
• Suppose X takes n values, V1, V2,… Vn, and
P(X=V1)=p1, P(X=V2)=p2, … P(X=Vn)=pn
• What is the smallest number of bits, on average, per
symbol, needed to transmit the symbols drawn from
distribution of X? It’s
E(X) = p1 log2 p1 – p2 log2 p2 – … pnlog2pn
• E(X) = the entropy of X
)(log
1
2 i
n
i
i pp∑=
−=

19. Example: 2 classes

21. Information gain

23. • Gain(S,Wind)?
• Wind = {Weak, Strong}
• S = {9 Yes &5 No}
• Sweak = {6 Yes & 2 No | Wind=Weak}
• Sstrong = {3 Yes &3 No | Wind=Strong}

24. Example: Decision tree learning
• Choose splitting attribute for root among {Outlook,
Temperature, Humidity, Wind}?
– Gain(S, Outlook) = ... = 0.246
– Gain(S, Temperature) = ... = 0.029
– Gain(S, Humidity) = ... = 0.151
– Gain(S, Wind) = ... = 0.048

25. • Gain(Ssunny,Temperature) = 0,57
• Gain(Ssunny, Humidity) = 0,97
• Gain(Ssunny, Windy) =0,019

26. Over-fitting example
• Consider adding noisy training example #15
– Sunny, hot, normal, strong, playTennis = No
• What effect on earlier tree?

27. Over-fitting

28. Avoid over-fitting
• Stop growing when data split not statistically
significant
• Grow full tree then post-prune
• How to select best tree
– Measure performance over training tree
– Measure performance over separate validation
dataset
– MDL minimize
• size(tree) + size(misclassifications(tree))

29. Reduced-error pruning
• Split data into training and validation set
• Do until further pruning is harmful
– Evaluate impact on validation set of pruning
each possible node
– Greedily remove the one that most improves
validation set accuracy

30. Rule post-pruning
• Convert tree to equivalent set
of rules
• Prune each rule independently
of others
• Sort final rules into desired
sequence for use

31. Issues in Decision Tree Learning
• How deep to grow?
• How to handle continuous attributes?
• How to choose an appropriate attributes selection
measure?
• How to handle data with missing attributes values?
• How to handle attributes with different costs?
• How to improve computational efficiency?
• ID3 has been extended to handle most of these.
The resulting system is C4.5 (http://cis-
linux1.temple.edu/~ingargio/cis587/readings/id3-c45.html)

32. Decision tree – When?

33. References
• Data mining, Nhat-Quang Nguyen, HUST
• http://www.cs.cmu.edu/~awm/10701/slides/
DTreesAndOverfitting-9-13-05.pdf

34. RANDOM FORESTS
Credits: Michal Malohlava @Oxdata

35. Motivation
• Training sample of points
covering area [0,3] x [0,3]
• Two possible colors of
points

36. • The model should be able to predict a color of a
new point

37. Decision tree

38. How to grow a decision tree
• Split rows in a given
node into two sets with
respect to impurity
measure
– The smaller, the more
skewed is distribution
– Compare impurity of
parent with impurity of
children

39. When to stop growing tree
• Build full tree or
• Apply stopping criterion - limit on:
– Tree depth, or
– Minimum number of points in a leaf

40. How to assign leaf
value?
• The leaf value is
– If leaf contains only one point
then its color represents leaf
value
• Else majority color is picked, or
color distribution is stored

41. Decision tree
• Tree covered whole area by rectangles
predicting a point color

42. Decision tree scoring
• The model can predict a point color based
on its coordinates.

43. Over-fitting
• Tree perfectly represents training data (0%
training error), but also learned about noise!

44. • And hence poorly predicts a new point!

45. Handle over-fitting
• Pre-pruning via stopping criterion!
• Post-pruning: decreases complexity of
model but helps with model generalization
• Randomize tree building and combine trees
together

46. Randomize #1- Bagging

47. Randomize #1- Bagging

48. Randomize #1- Bagging
• Each tree sees only sample of training data
and captures only a part of the information.
• Build multiple weak trees which vote
together to give resulting prediction
– voting is based on majority vote, or weighted
average

49. Bagging - boundary
• Bagging averages many trees, and produces
smoother decision boundaries.

50. Randomize #2 - Feature selection
Random forest

51. Random forest - properties
• Refinement of bagged trees; quite popular
• At each tree split, a random sample of m features is drawn,
and only those m features are considered for splitting.
Typically
• m=√p or log2(p), where p is the number of features
• For each tree grown on a bootstrap sample, the error rate
for observations left out of the bootstrap sample is
monitored. This is called the “out-of-bag” error rate.
• Random forests tries to improve on bagging by “de-
correlating” the trees. Each tree has the same expectation

52. Advantages of Random Forest
• Independent trees which can be built in
parallel
• The model does not overfit easily
• Produces reasonable accuracy
• Brings more features to analyze data variable
importance, proximities, missing values
imputation

53. Out of bag points and validation
• Each tree is built over
a sample of training
points.
• Remaining points are
called “out-of-
bag” (OOB).
These
points
are
used
for
valida@on
as
a
good
approxima@on
for
generaliza@on
error.
Almost
iden@cal
as
N-­‐fold
cross
valida@on.