This document provides an overview of decision trees, including: - Decision trees can classify data quickly, achieve accuracy similar to other models, and are simple to understand. - A decision tree has root, internal, and leaf nodes organized in a top-down structure to partition data based on attribute tests. - To classify a record, the attribute tests are applied from the root node down until a leaf node is reached, which assigns the record's class. - Decision trees require attribute-value data, predefined target classes, and sufficient training data to learn the model.

1. By
Dr.Ramkumar.T
ramkumar.thirunavukarasu@vit.ac.in
Decision Tree
1

2. Decision Tree
• Relatively fast compared to other classification
models
• Obtain similar and sometimes better accuracy
compared to other models
• Simple and easy to understand
• Can be converted into simple and easy to
understand classification rules
2

3. Decision Tree
3

4. Decision Tree
• The tree has three types of nodes.
• A root node - that has no incoming edges
and one or more outgoing edges.
• Internal nodes - Each of which has exactly
one incoming edge and two or more
outgoing edges.
• Leaf or terminal nodes - Each of which
has exactly one incoming edge and no
outgoing edges.
4

5. How Decision Tree Works ?
• Classifying a test record is straightforward once a
decision tree has been constructed.
• Starting from the root node, we apply the test
condition to the record and follow the appropriate
branch based on the outcome of the test.
• This will lead us either to another internal node,
for which a new test condition is applied, or to a
leaf node.
• The class label associated with the leaf node is
then assigned to the record.
5

6. Requirements of Decision Tree
• Attribute-value description - object or case must
be expressible in terms of a fixed collection of
properties or attributes (e.g., hot, mild, cold).
• Predefined classes (target values)- the target
function has discrete output values (Boolean or
multiclass)
• Sufficient data- Enough training cases should be
provided to learn the model.
6

7. How Decision Tree Inducted?
• Tree is constructed in a top-down recursive
divide-and-conquer manner
• At start, all the training examples are at the root
• Attributes are categorical (if continuous-valued,
they are discretized in advance)
• Attributes are selected on the basis of a heuristic
or statistical measure (e.g., information gain)
• Examples are partitioned recursively based on
selected attributes
7

8. Decision Tree Induction – Measures
• Entropy or Information Theory - One of the techniques
for selecting an attribute to split a node.
• “If you have uncertainty – you have Information”
• Information is defined as “-pilogpi” where „pi‟ is the
probability of some event.
• Information of any event that is likely to have several
possible outcomes is given by
• Information Gain – It is the measure of how good an attribute
is for predicting the class of each of the training data.
8

9. DT – Measures (ID3 Algorithm)
• Expected information (entropy) needed to
classify a tuple in D:
• Information needed (after using A to split D
into v partitions) to classify D:
 Information gained by branching on
attribute A
9
(D)
Entropy
Entropy(D)
Gain(A) A



10. 10
Gain Ratio Measure (C4.5)
 Information gain measure is biased towards attributes with a
large number of values
 C4.5 (a successor of ID3) uses gain ratio to overcome the
problem (normalization to information gain)
GainRatio(A) = Gain(A)/SplitEntropy(A)
 Ex.
Gain_ratio(income) = 0.029/0.926 = 0.031
 The attribute with the maximum gain ratio is selected as the
splitting attribute
)
|
|
|
|
(
log
|
|
|
|
)
( 2
1 D
D
D
D
D
py
SplitEntro
j
v
j
j
A 

 

926
.
0
)
14
4
(
log
14
4
)
14
6
(
log
14
6
)
14
4
(
log
14
4
)
( 2
2
2 







D
py
SplitEntro A

11. Training Data
11