CART example

1. Decision Tree of CART
Dinda、Isaac

2. Decision Tree (1/2)
feature vector (xi)
yi: +1:Yes, -1: No
• Training set
• Learned decision tree
[Note]
Only one feature will be
involved at a node
2

3. Decision Tree (2/2)
• Example:
Outlook Temperature Humidity Windy Play
Rainy Hot High True ?
Ans: no
(A test instance)
3

4. Why use decision tree
• Easy to interpret
• If outlook is sunny and humidity is high, then
we can’t play tennis ball.
• Perform feature selection
• The top few nodes on which the tree is split are
essentially the most important variables within
the dataset and feature selection is completed
automatically
4

5. Why not use decision tree
• Decision Trees do not work well if you have smooth boundaries
5
Smooth boundaries D.T. boundaries

6. Why not use decision tree
• Poor Resolution on Data With Complex Relationships Among the
Variables
6

7. How to generate a classification tree?
7

8. CART (Classification and Regression Trees)
Dataset S include n classes ,Gini(S) define as
pj is probability of value in S belong to class j
CART Briemen（1984） Discrete and
continuous
Gini Index Entire Error Rate
Learning
Method(演算法)
Author(作者) Data Type(資料屬
性)
Splitting Rule(分割
規則)
Pruning Rule(修剪
樹規則)

9. Example
• GiniOutlook(S) =
3
4
* (1 – (
3
3
)2 – (
0
3
)2 ) +
1
4
* (1 – (
1
1
)2 – (
0
1
)2 ) = 0
9
Thresholdsunny rain
PlayNot play
We have two feature Outlook、Temp., and we want to know that will we play tennis?
Thresholdhot cold
PlayNot play
GiniTemp. (S) =
2
4
* (1 – (
1
2
)2 – (
1
2
)2 ) +
2
4
* (1 – (
1
2
)2 – (
1
2
)2 ) =
1
2
Best threshold
Worst threshold
Method’s Blue part is calculate how pure can we get after we cut threshold. Orange part means its weight.

10. Example
DAY Outlook Temp. Play tennis
D1 Sunny Hot NO
D2 Sunny Hot YES
D3 Sunny Mild NO
D4 Sunny Cold YES
D5 Rain Cold YES
Sunny rain
NO 3 0
YES 1 1
Play
Outlook Hot Mild Cold
NO 1 1 0
YES 1 0 2
Play
Temp.
We have two feature Outlook、Temp., and we want to know that will we play tennis?
And using those two features to draw decision tree by using CART method.

11. 0
1
2
3
4
NO YES
Outlook
Sunny rain
0
1
2
3
NO YES
Temp.
Hot,Mild Cold
0
1
2
3
NO YES
Temp.
Hot Mild,Cold
Gini(S{Sunny}) = 1 – (
1
4
)2 – (
3
4
)2 =
3
8
Gini(S{Rain}) = 1 – (
1
1
)2 – (
0
1
)2 = 0
GiniOutlook(S) =
4
5
∗
3
8
+
1
5
∗ 0 =
3
10
Sunny rain
NO 3 0
YES 1 1
Play
Outlook Hot,Mild Cold
NO 2 0
YES 1 2
Play
Temp.
Gini(STempϵ{Hot,Mild}) = 1 – (
2
3
)2 – (
1
3
)2 =
4
9
Gini(STempϵ{Cold}) = 1 – (
2
2
)2 – (
0
2
)2 = 0
GiniTemp.(S) =
3
5
∗
4
9
+
2
5
∗ 0 =
4
15
Hot Mild,Cold
NO 1 1
YES 1 2
Play
Temp.
Gini(STempϵ{Hot}) = 1 – (
1
2
)2 – (
1
2
)2 =
1
2
Gini(STempϵ{Cold,Mild}) = 1 – (
1
3
)2 – (
2
3
)2 =
4
9
GiniTemp.(S) =
2
5
∗
1
2
+
3
5
∗
4
9
=
7
15
Example

12. Example
• GiniOutlook(S) =
3
10
> GiniTemp.(S) =
4
15
We choose small one to be decision tree’s root.
Temp.
[Cold] [Mild, Hot]
Yes
Outlook
Yes No
[Rain] [Sunny]

13. Example
Riding mower classification
13
Obs # Income Lot size Class
1 Middle Middle Owners
2 High Middle Owners
3 High Big Owners
4 High Big Owners
5 Middle Big Owners
Obs # Income Lot size Class
6 Middle Middle Non-owners
7 Low Big Non-owners
8 Middle Middle Non-owners
9 Low Big Non-owners
10 High Middle Non-owners
A riding-mower manufacturer would like to find a way of classifying families in a city into those that
are likely to purchase a riding mower and those who are not likely to buy one. A pilot random sample
of 5 owners and 5 non-owners in the city is undertaken.

14. Example
14
10
12
14
16
18
20
22
24
30 40 50 60 70 80 90 100 110
Lotsize
Income
Owners Non-owners

15. How to split?
• Split criterion: Goodness function
• Used to select the attribute to be split at a tree node
during the tree generation phase
• Goodness function in CART: Gini Index
15

16. Example: Gini(Lot size)
16
14 14.8 16 16.4 16.8 17.2 17.6 18.4 18.8
14.4 15.4 16.2 16.6 17 17.4 18 18.6 19
19.2
Split-point

17. Example
17
10
12
14
16
18
20
22
24
30 40 50 60 70 80 90 100 110
Lotsize
Income
Owners Non-owners
19
17

18. Example: Gini(Lot size)
Split-point = 17
• D1 = Lot size ≤ 17, D2 = Lot size >
17
Split-point = 19
• D1 = Lot size ≤ 19, D2 = Lot size >
19
18
999.0
666.0333.0
19
8
19
11
1
24
19
5
4
5
1
1
24
5
)(
24
19
)(
24
5
)(
2222
21




























































 DGiniDGinisizeLotGini
5.0
25.025.0
12
9
12
3
1
24
12
12
3
12
9
1
24
12
)(
24
12
)(
24
12
)(
2222
21




























































 DGiniDGinisizeLotGini
Distance is narrow Distance is wide

19. Why use impurity instead of error as
goodness function?
• The main objective of decision tree is to find pure node containing
only one class
19
Error rate: 25%

20. Example: Decision Tree
20
19
Owner
Non-
owner
Lot size ≤19 Lot size>19Lot size