This document discusses decision tree algorithms including ID3. It provides examples of applying decision tree algorithms like ID3 to sample datasets with attributes like size, color, and shape to classify examples. The document shows calculating entropy, information gain, and using information gain to select the best attribute to split on at each node when building a decision tree.

1. Machine Learning – ECERSICE 2
1399/09/30
Amir Shokri

2. Exercise 2 – Decision Tree
Amir Shokri
A B C RESULT
F F F F
F F T F
F T F T
F T T F
T F F T
T F T T
T T F F
T T T F
(A AND B) OR (B AND C) XOR (A OR B) A
B B
false true
false
truefalse
true
c
false
true

3. Exercise 2 – Decision Tree
Amir Shokri
A B C RESULT
F F F T
F F T T
F T F T
F T T F
T F F T
T F T T
T T F T
T T T T
(A AND B AND C) OR (C NAND B) A
B
false true
false
true
c
false
true

4. Exercise 2 – Decision Tree
A B C D RESULT
F F F F F
F F F T F
F F T F F
F F T T F
F T F F F
F T F T F
F T T F F
F T T T T
T F F F F
T F F T F
T F T F F
T F T T T
T T F F F
T T F T F
T T T F F
T T T T T
(A OR B) AND (B OR C) AND (C AND D)
A
B
B
false
true
false
true
false
true
c
false
true
D
false
c
false D
false
cfalse
true
true
D
false
true
Amir Shokri

5. Exercise 2 – Decision Tree
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
size
shape
big small
circle
triangle

6. Exercise 2 – FIND-S
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
h0 = < ∅ , ∅ , ∅ >
h1 = < ∅ , ∅ , ∅ >
h2 = < big, red, triangle >
h3 = < big, red, triangle >
h4 = < big, red, triangle >

7. Exercise 2 – Candidate elimination
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
S0 = < ∅ , ∅ , ∅ >
G0 = < ?, ?, ? >

8. Exercise 2 – Candidate elimination
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
S1 = < ∅ , ∅ , ∅ >
G1 = < small, ?, ?> < ?, blue, ? > < ?, ? ,triangle >

9. Exercise 2 – Candidate elimination
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
S3 = < big, red, triangle >
G3 = < ?, ? ,triangle >

10. Exercise 2 – Candidate elimination
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
S4 = < big, red, triangle >
G4 = < ?, ? ,triangle >

11. Exercise 2 – ID3
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
Entropy :
Entropy(S) :
Gain(S, A):
෍
𝑖=1
−𝑝𝑖 ∗ log2(𝑝𝑖)
(−𝑝+ log2 𝑝+ ) − (−𝑝− log2 𝑝− )
𝐸𝑛𝑡𝑟𝑜𝑝ℎ𝑦 𝑆 − ෍
𝑣=𝑉𝑎𝑙𝑢𝑒𝑠(𝐴)
𝑆 𝑣
𝑆
𝐸𝑛𝑡𝑟𝑜𝑝ℎ𝑦(𝑆 𝑣)

12. Exercise 2 – ID3
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
σ𝑖=1 −𝑝𝑖 ∗ log2 𝑝𝑖 = −
3
4
log2
3
4
−
1
4
log2
1
4
= 0.307 + 0.5 = 0.87
Entropy

13. Exercise 2 – ID3
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
Gain size:
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 𝑠𝑚𝑎𝑙𝑙 => −
2
4
0
2
log
0
2
+
2
2
log
2
2
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 𝑏𝑖𝑔 => −
2
4
1
2
log
1
2
+
1
2
log
1
2
𝐺𝑎𝑖𝑛 𝑆, size = 0.87 − 0 + 0.5 = 0.37

14. Exercise 2 – ID3
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
Gain color:
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 𝑟𝑒𝑑 => −
3
4
2
3
log
2
3
+
1
3
log
1
3
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 𝑏𝑙𝑢𝑒 => −
1
4
1
1
log
1
1
+
0
1
log
0
1
𝐺𝑎𝑖𝑛 𝑆, color = 0.87 − 0.29 + 0 = 0.58

15. Exercise 2 – ID3
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
Gain shape:
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 𝑐𝑖𝑟𝑐𝑙𝑒 => −
3
4
0
3
log
0
3
+
3
3
log
3
3
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 𝑡𝑟𝑖𝑎𝑛𝑔𝑙𝑒 => −
1
4
1
1
log
1
1
+
0
1
log
0
1
𝐺𝑎𝑖𝑛 𝑆, shape = 0.87 − 0 + 0 = 0.87

16. Exercise 2 – ID3
Amir Shokri
Example Size Color Shape Class/Label
1 big red circle No
2 big red triangle Yes
3 small red circle No
4 big red triangle No
5 small blue circle No
shape
circle triangle