This document discusses decision trees and their use for classification. It provides examples to illustrate key concepts: - Decision trees classify instances by sorting them down the tree from root to leaf node, where each leaf represents a classification outcome. Nodes test attribute values and branches represent test outcomes. - An example decision tree classifies whether to play golf based on weather attributes like temperature and humidity. It generates rules like "if sunny and humidity below 75% then play." - Classification accuracy is measured by how many test instances the tree correctly classifies. Information gain is used to select the most informative attribute to split on at each node, improving classification.

1. Decision Tree

2. Decision Tree
• Decision tree is the most powerful and popular tool for
classification and prediction.
• A Decision tree is a flowchart like tree structure, where each
internal node denotes a test on an attribute, each branch
represents an outcome of the test, and each leaf node (terminal
node) holds a class label.

3. Introduction
• The set of records available for developing
classification methods (decision tree) is divided into
two disjoint subsets – a training set and a test set.
• The attributes of the records are categorise into two
types:
– Attributes whose domain is numerical are called numerical
attributes.
– Attributes whose domain is not numerical are called the
categorical attributes.

4. Decision Tree Example
• The data set has five attributes.

5. Decision Tree Example
• The data set has five attributes.
• There is a special attribute: the attribute class is the class label.
• The attributes, temp (temperature) and humidity are
numerical attributes.
• Other attributes are categorical, that is, they cannot be
ordered.
• Based on the training data set, we want to find a set of rules to
know what values of outlook, temperature, humidity and
wind, determine whether or not to play golf.

6. Decision Tree Example

7. Decision Tree Example
• We have five leaf nodes.
• In a decision tree, each leaf node represents a rule.
• We have the following rules corresponding to the tree given in
Figure.
• RULE 1: If it is sunny and the humidity is not above 75% then
play.
• RULE 2: If it is sunny and the humidity is above 75%, then do
not play.
• RULE 3: If it is overcast, then play.
• RULE 4: If it is rainy and not windy, then play.
• RULE 5: If it is rainy and windy, then don't play

8. Classification
• The classification of an unknown input vector is done by
traversing the tree from the root node to a leaf node.
• A record enters the tree at the root node.
• At the root, a test is applied to determine which child node the
record will encounter next.
• This process is repeated until the record arrives at a leaf node.
• All the records that end up at a given leaf of the tree are
classified in the same way.
• There is a unique path from the root to each leaf.
• The path is a rule which is used to classify the records.

9. Classification
• In our tree, we can carry out the classification for an unknown
record as follows.
• Let us assume, for the record, that we know the values of the
first four attributes (but we the values of the first four
attributes (but we do not know the value of class attribute) as :
outlook= rain; temp = 70; humidity = 65; and windy= true.
• We start from the root node to check the value of the attribute
associated at the root node.

10. Classification
• outlook= rain; temp = 70; humidity = 65; and windy= true.
• We start from the root node to check the value of the attribute
associated at the root node.
• In our example, outlook is the splitting attribute at root.
• Since for the given record, outlook = rain, we move to the
rightmost child node of the root.
• At this node, the splitting attribute is windy and we find that
for the record we want classify, windy = true.
• Hence, we move to the left child node to conclude that the
class label Is "no play".

11. Accuracy of Classification
RULE 1
• If it is sunny and the humidity is not above 75%, then
play.

12. Accuracy of Classification
• The accuracy of the classifier is determined by the percentage
of the test data set that is correctly classified. For example :
RULE 1
– If it is sunny and the humidity is not above 75%, then play.
• We can see that for Rule 1 there are two records of the test
data set satisfying outlook= sunny and humidity < =75, and
only one of these is correctly classified as play.
• Thus, the accuracy of this Rule 1 is 0.5 (or 50%).

13. Advantages of Decision Tree
Classifications
• A decision tree construction process is concerned with
identifying the splitting attributes and splitting criterion at
every level of the tree.
Major strengths are:
• Decision tree able to generate understandable rules.
• They are able to handle both numerical and categorical
attributes.
• They provide clear indication of which fields are most
important for prediction or classification.

14. Shortcomings of Decision Tree
Classifications
Weaknesses are:
• The process of growing a decision tree is computationally
expensive. At each node, each candidate splitting field is
examined before its best split can be found.
• Some decision tree can only deal with binary-valued target
classes.

15. Iterative Dichotomizer (ID3)
Quinlan (1986)
Each node corresponds to a splitting attribute
Each arc is a possible value of that attribute.
• At each node the splitting attribute is selected to be the most
informative among the attributes not yet considered in the path
from the root.
• Entropy is used to measure how informative is a node.

16. Iterative Dichotomizer (ID3)
The algorithm uses the criterion of information gain to
determine the goodness of a split.
The attribute with the greatest information gain is taken as the
splitting attribute, and the data set is split for all distinct values
of the attribute.

17. Attribute Selection Measure: Information
Gain (ID3/C4.5)

18. Entropy
• Entropy measures the homogeneity (purity) of a
set of examples.
• It gives the information content of the set in terms
of the class labels of the examples.
• Consider that you have a set of examples, S with
two classes, P and N. Let the set have p instances
for the class P and n instances for the class N.
• So the total number of instances we have is t = p +
n. The view [p, n] can be seen as a class
distribution of S.

19. Entropy

20. Entropy
• The entropy for a completely pure set is 0 and is 1 for a set
with equal occurrences for both the classes.
i.e. Entropy[14,0] = - (14/14).log2(14/14) - (0/14).log2(0/14)
= -1.log2(1) - 0.log2(0)
= -1.0 - 0
= 0
i.e. Entropy[7,7] = - (7/14).log2(7/14) - (7/14).log2(7/14)
= - (0.5).log2(0.5) - (0.5).log2(0.5)
= - (0.5).(-1) - (0.5).(-1)
= 0.5 + 0.5
= 1

21. Information

22. Information
• Example:
• I(p, n) = I(9, 5) ?
• Here there are 9+5 =14 samples out of which
9 are positive and 5 are negative.
Thus:
• I(9,5) = -(9/14)log2
(9/14) – (5/14)log2
(5/14)
• = 0.4102 + 0.5304
• = 0.9406

24. Information Gain
Gain (age) = I(p.n) – E(age)
= 0.940-0.694
= 0.246

25. • Therefore, out of these 4 attributes (age,
income, students, credit rating), age will be
selected as splitting attribute at level 0