CS8080_IRT_UNIT - III T5 DECISION TREES.pdf

P1WU
UNIT – III: CLASSIFICATION
Topic 5: DECISION TREES
AALIM MUHAMMED SALEGH COLLEGE OF ENGINEERING
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
SEMESTER – VIII
PROFESSIONAL ELECTIVE – IV
CS8080- INFORMATION RETRIEVAL TECHNIQUES

UNIT III : TEXT CLASSIFICATION AND CLUSTERING
1.A Characterization of Text
Classification
2. Unsupervised Algorithms:
Clustering
3. Naïve Text Classification 4.
Supervised Algorithms
5. Decision Tree
6. k-NN Classifier
7. SVM Classifier
8. Feature Selection or
Dimensionality Reduction
9. Evaluation metrics
10. Accuracy and Error
11. Organizing the classes
12. Indexing and Searching
13. Inverted Indexes
14. Sequential Searching
15. Multi-dimensional
Indexing
SEMESTER – VIII

DECISION TREES
SEMESTER – VIII

INTRODUCTION TO DECISION TREES
• What is a decision tree?
• A decision tree is a structure that includes a root node,
branches, and leaf nodes.
a) Each internal node denotes a test on an attribute,
b) each branch denotes the outcome of a test, and
c) each leaf node holds a class label.
• The topmost node in the tree is the root node.
SEMESTER – VIII

• ID3, C4.5, and CART adopt a greedy (i.e., non-backtracking)
approach in which decision trees are constructed in a top-
down recursive divide-and-conquer manner.
• Most algorithms for decision tree induction also follow such a
top-down approach, which starts with a training set of
tuples and their associated class labels.
• The training set is recursively partitioned into smaller
subsets as the tree is being built.
SEMESTER – VIII

• Decision tree induction is the learning of decision trees from
class-labeled training tuples.
• A decision tree is a flowchart-like tree structure, where each
internal node (nonleaf node) denotes a test on an attribute,
• each branch represents an outcome of the test, and
• each leaf node (or terminal node) holds a class label.
• The top most node in a tree is the root node.
SEMESTER – VIII

•A decision tree is a tree where
• internal node = a test on an attribute
• tree branch = an outcome of the test
• leaf node = class label or class distribution
SEMESTER – VIII

Benefits of Decision Trees
•The benefits of having a decision tree are as
follows −
a) It does not require any domain knowledge.
b) It is easy to comprehend.
c) The learning and classification steps of a decision tree are
simple and fast.
SEMESTER – VIII

Brief History of Decision Trees
SEMESTER – VIII
CLS (Hunt etal. 1966)--- cost driven
ID3 (Quinlan, 1986 MLJ) --- Information-driven
C4.5 (Quinlan, 1993) --- Gain ratio + Pruning ideas
CART (Breiman et al. 1984) --- Gini Index

Elegance of Decision Trees
SEMESTER – VIII

Structure of Decision Trees
• If x1 > a1 & x2 > a2, then it’s A class
• C4.5, CART, two of the most widely used
• Easy interpretation, but accuracy generally unattractive
SEMESTER – VIII
Leaf nodes
Internal nodes
Root node
A
B
B A
A
x1
x2
x4
x3
> a1
> a2

Example of Decision Tree
SEMESTER – VIII

Another Example of Decision Tree
SEMESTER – VIII

Decision Tree classification Tasks
SEMESTER – VIII

5/16/2022 Data Mining: Concepts and Techniques 15
Apply Model to Test Data
Refund
MarSt
TaxInc
YES
NO
NO
NO
Yes No
Married
Single, Divorced
< 80K > 80K
Refund Marital
Status
Taxable
Income Cheat
No Married 80K ?
10
Test Data
Assign Cheat to “No”

Refund
MarSt
TaxInc
YES
NO
NO
NO
Yes No
Married
Single, Divorced
< 80K > 80K
Refund Marital
Status
Taxable
Income Cheat
No Married 80K ?
10
Test Data

Refund
MarSt
TaxInc
YES
NO
NO
NO
Yes No
Married
Single, Divorced
< 80K > 80K
Refund Marital
Status
Taxable
Income Cheat
No Married 80K ?
10
Test Data
Assign Cheat to “No”

Decision Tree Classification Task
Apply
Model
Induction
Deduction
Learn
Model
Model
Tid Attrib1 Attrib2 Attrib3 Class
1 Yes Large 125K No
2 No Medium 100K No
3 No Small 70K No
4 Yes Medium 120K No
5 No Large 95K Yes
6 No Medium 60K No
7 Yes Large 220K No
8 No Small 85K Yes
9 No Medium 75K No
10 No Small 90K Yes
10
Tid Attrib1 Attrib2 Attrib3 Class
11 No Small 55K ?
12 Yes Medium 80K ?
13 Yes Large 110K ?
14 No Small 95K ?
15 No Large 67K ?
10
Test Set
Tree
Induction
algorithm
Training Set
Decision Tree

Constructing a Decision Tree
SEMESTER – VIII

• Two phases of decision tree generation:
1. tree construction
• at start, all the training examples at the root
• partition examples based on selected attributes
• test attributes are selected based on a heuristic or a statistical measure
2. tree pruning
• identify and remove branches that reflect noise or outliers
SEMESTER – VIII

• Basic step:
Determination of the root node of the tree and
the root node of its sub-trees
• Most Discriminatory Feature
• Every feature can be used to partition the training data
• If the partitions contain a pure class of training instances, then this feature is
most discriminatory
SEMESTER – VIII

Constructing a Decision Tree:- Example of Partitions
• Categorical feature
• Number of partitions of the training data is equal to the number of values of
this feature
• Numerical feature
• Two partitions
SEMESTER – VIII

Decision Tree Induction Algorithm
• A machine researcher named J. Ross Quinlan in 1980 developed a
decision tree algorithm known as ID3 (Iterative Dichotomiser).
• Later, he presented C4.5, which was the successor of ID3. ID3 and C4.5
adopt a greedy approach.
• In this algorithm, there is no backtracking; the trees are constructed
in a top-down recursive divide-and-conquer manner.
• Generating a decision tree form training tuples of data partition D
SEMESTER – VIII

Algorithm : Generate_decision_tree
• Input:
• Data partition, D, which is a set of training tuples and their
associated class labels.
• attribute_list, the set of candidate attributes.
• Attribute selection method, a procedure to determine the splitting
criterion that best partitions that the data tuples into individual
classes. This criterion includes a splitting_attribute and either a
splitting point or splitting subset.
Output: A Decision Tree
SEMESTER – VIII

Algorithm : Generate_decision_tree
• Method
1) create a node N;
2) if tuples in D are all of the same class, C then
3) return N as leaf node labeled with class C;
4) if attribute_list is empty then
5) return N as leaf node with labeled with majority class in D;//majority voting
6) apply attribute_selection_method(D, attribute_list) to find the best splitting_criterion;
7) label node N with splitting_criterion;
8) if splitting_attribute is discrete-valued and multiway splits allowed then // not restricted to binary trees
9) attribute_list = attribute_list - splitting attribute; // remove splitting attribute
10) for each outcome j of splitting criterion
11) // partition the tuples and grow subtrees for each partition
12) let Dj be the set of data tuples in D satisfying outcome j; // a partition
13) if Dj is empty then
14) attach a leaf labeled with the majority class in D to node N;
else attach the node returned by Generate_decision tree(Dj, attribute list) to node N;
end for
15) return N;
SEMESTER – VIII

Constructing Decision Tree Example :- Weather Forecasting
SEMESTER – VIII

Constructing Decision Tree :- A Simple Dataset
SEMESTER – VIII
9 Play samples
5 Don’t
A total of 14.

SEMESTER – VIII
Outlook Temp Humidity Windy class
Sunny 75 70 true Play
Sunny 80 90 true Don’t
Sunny 85 85 false Don’t
Sunny 72 95 true Don’t
Sunny 69 70 false Play
Overcast 72 90 true Play
Overcast 83 78 false Play
Overcast 64 65 true Play
Overcast 81 75 false Play
Rain 71 80 true Don’t
Rain 65 70 true Don’t
Rain 75 80 false Play
Instance #
1
2
3
4
5
6
7
8
9
10
11
12
13
14

SEMESTER – VIII
2
outlook
windy
humidity
Play
Play
Play
Don’t
Don’t
sunny
overcast
rain
<= 75
> 75 false
true
2
4
3
3

SEMESTER – VIII
Total 14 training
instances
1,2,3,4,5
P,D,D,D,P
6,7,8,9
P,P,P,P
10,11,12,13,14
D, D, P, P, P
Outlook =
sunny
Outlook =
overcast
Outlook =
rain

SEMESTER – VIII
Total 14 training
instances
5,8,11,13,14
P,P, D, P, P
1,2,3,4,6,7,9,10,12
P,D,D,D,P,P,P,D,P
Temperature
<= 70
Temperature
> 70

Constructing Decision Tree Example :-
Decision on Buying a Computer / customer likely to
purchase a computer
SEMESTER – VIII

Constructing Decision Tree Example :-
Decision on Buying a Computer
SEMESTER – VIII
The following decision tree is for the concept buy_computer that
indicates :
Whether a customer at a company is likely to buy a computer or not?
Each internal node represents a test on an attribute.
Each leaf node represents a class.

Constructing Decision Tree :- Training Dataset
SEMESTER – VIII
age income student credit_rating buys_computer
<=30 high no fair no
<=30 high no excellent no
31…40 high no fair yes
>40 medium no fair yes
>40 low yes fair yes
>40 low yes excellent no
31…40 low yes excellent yes
<=30 medium no fair no
<=30 low yes fair yes
>40 medium yes fair yes
<=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
This follows
an
example of
Quinlan’s
ID3 (Playing
Tennis)

Constructing Decision Tree :- Output: A Decision Tree for
“buys_computer”
SEMESTER – VIII
age?
overcast
student? credit rating?
<=30 >40
no yes yes
yes
31..40
fair
excellent
yes
no
From the training dataset , calculate entropy value, which indicates that splitting attribute is: age

A Decision Tree for “buys_computer”
SEMESTER – VIII

SEMESTER – VIII
From the training data set , age= youth has 2 classes based on student attribute

SEMESTER – VIII
based on majority voting in student attribute , RID=3 is grouped under yes group.

SEMESTER – VIII
From the training data set , age= senior has 2 classes based on credit rating.

SEMESTER – VIII
Final Decision Tree

Classification by Decision Tree
SEMESTER – VIII

• A typical decision tree that represents the concept buys
computer, that is, it predicts whether a customer at
AllElectronics is likely to purchase a computer.
• Internal nodes are denoted by rectangles, and leaf nodes are
denoted by ovals.
• Some decision tree algorithms produce only binary trees
(where each internal node branches to exactly two other
nodes), whereas others can produce non binary trees.
SEMESTER – VIII

• “How are decision trees used for classification?”
• Given a tuple, X, for which the associated class label is
unknown, the attribute values of the tuple are tested against
the decision tree.
• A path is traced from the root to a leaf node, which holds the
class prediction for that tuple.
• Decision trees can easily be converted to classification rules.
SEMESTER – VIII

Why are decision tree classifiers so popular?
• The construction of decision tree classifiers does not require any domain
knowledge or parameter setting, and therefore is appropriate for
exploratory knowledge discovery.
• Decision trees can handle high dimensional data.
• Their representation of acquired knowledge in tree form is intuitive and
generally easy to assimilate by humans.
• The learning and classification steps of decision tree induction are simple
and fast.
• In general, decision tree classifiers have good accuracy.
SEMESTER – VIII

Extracting Classification Rules from Trees
• Represent the knowledge in the form of IF-THEN rules
• One rule is created for each path from the root to a leaf
• Each attribute-value pair along a path forms a conjunction
• The leaf node holds the class prediction
• Rules are easier for humans to understand
• Example
IF age = “<=30” AND student = “no” THEN buys_computer = “no”
IF age = “<=30” AND student = “yes” THEN buys_computer = “yes”
IF age = “31…40” THEN buys_computer = “yes”
IF age = “>40” AND credit_rating = “excellent” THEN buys_computer = “yes”
IF age = “<=30” AND credit_rating = “fair” THEN buys_computer = “no”
SEMESTER – VIII

Training Set and Its AVC Sets
SEMESTER – VIII
student Buy_Computer
yes no
yes 6 1
no 3 4
Age Buy_Computer
yes no
<=30 3 2
31..40 4 0
>40 3 2
Credit
rating
Buy_Computer
yes no
fair 6 2
excellent 3 3
age income studentcredit_rating
buys_computer
<=30 high no fair no
<=30 high no excellent no
31…40 high no fair yes
>40 medium no fair yes
>40 low yes fair yes
>40 low yes excellent no
31…40 low yes excellent yes
<=30 medium no fair no
<=30 low yes fair yes
>40 medium yes fair yes
<=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
AVC-set on income
AVC-set on Age
AVC-set on Student
Training Examples
income Buy_Computer
yes no
high 2 2
medium 4 2
low 3 1
AVC-set on
credit_rating

Any Questions?
SEMESTER – VIII

CS8080_IRT_UNIT - III T5 DECISION TREES.pdf

Recommended

Recommended

More Related Content

Similar to CS8080_IRT_UNIT - III T5 DECISION TREES.pdf

Similar to CS8080_IRT_UNIT - III T5 DECISION TREES.pdf (20)

More from AALIM MUHAMMED SALEGH COLLEGE OF ENGINEERING

More from AALIM MUHAMMED SALEGH COLLEGE OF ENGINEERING (13)

Recently uploaded

Recently uploaded (20)

CS8080_IRT_UNIT - III T5 DECISION TREES.pdf