ID3 Algorithm semnan university amirsh.nll@gmail.com

1. ID3 (Decision tree)
Amir Shokri
1399/09/25

2. Concept learning
Entropy :
Entropy(S) :
Gain(S, A):
Amir Shokri
෍
𝑖=1
−𝑝𝑖 ∗ log2(𝑝𝑖)
(−𝑝+ log2 𝑝+ ) − (−𝑝− log2 𝑝− )
𝐸𝑛𝑡𝑟𝑜𝑝ℎ𝑦 𝑆 − ෍
𝑣=𝑉𝑎𝑙𝑢𝑒𝑠(𝐴)
𝑆 𝑣
𝑆
𝐸𝑛𝑡𝑟𝑜𝑝ℎ𝑦(𝑆 𝑣)

3. ID3
Amir Shokri
PlayWindyHumidityTemperatureOutlookDay
YesFalseHighHotOvercast1
YesFalseHighMildRainy2
NOFalseNormalCoolRainy3
NoFalseHighMildSunny4
YesFalseHighMildOvercast5
NoTrueNormalCoolSunny6
YesTrueNormalHotSunny7
YesFalseHighCoolRainy8
YesFalseHighCoolSunny9
YesTrueNormalCoolOvercast10
YesTrueHighHotSunny11
YesTrueHighHotRainy12

4. ID3
Amir Shokri
෍
𝑖=1
−𝑝𝑖 ∗ log2 𝑝𝑖 = −
3
12
log2
3
12
−
9
12
log2
9
12
= 0.811
Entropy

5. ID3
Amir Shokri
Gain outlook:
Gain(S, outlook) = 0.811 − (0 +0.27 +0.40) = 0.141
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(𝑜𝑣𝑒𝑟𝑐𝑎𝑠𝑡) => −
3
12
3
3
log
3
3
+
0
3
log
0
3
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(𝑟𝑎𝑖𝑛𝑦) => −
4
12
3
4
log
3
4
+
1
4
log
1
4
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(𝑠𝑢𝑛𝑛𝑦) => −
5
12
3
5
log
3
5
+
2
5
log
2
5
𝐺𝑎𝑖𝑛 𝑆, outlook = 0.811 − 0 + 0.27 + 0.4 = 0.141

6. ID3
Amir Shokri
Gain Temperature:
Gain(S, Temperature) = 0.811 −(0 +0.4 +0.29) = 0.121
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(hot) => −
4
12
4
4
log
4
4
+
0
4
log
0
4
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(mild) => −
3
12
2
3
log
2
3
+
1
3
log
1
3
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(cool) => −
5
12
3
5
log
3
5
+
2
5
log
2
5
𝐺𝑎𝑖𝑛 𝑆, Temperature = 0.811 − 0 + 0.4 + 0.29 = 0.121

7. ID3
Amir Shokri
Gain Humidity:
Gain(S, Humidity) = 0.811 −(0.33 + 0.36) = 0.121
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 high => −
8
12
7
8
log
7
8
+
1
8
log
1
8
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 normal => −
4
12
2
3
log
2
3
+
2
3
log
2
3
𝐺𝑎𝑖𝑛 𝑆, Humidity = 0.811 − 0.33 + 0.36 = 0.121

8. ID3
Amir Shokri
Gain Windy:
Gain(S, Windy) = 0.811 −(0.3 + 0.3) = 0.011
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 false => −
7
12
5
7
log
5
7
+
2
7
log
2
7
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 true => −
5
12
4
5
log
4
5
+
1
5
log
1
5
𝐺𝑎𝑖𝑛 𝑆, Windy = 0.811 − 0.5 + 0.3 = 0.011

9. ID3
Temperature
yes
mild
Entropy(s) = 0.918
cool
Entropy(s) = 0.971
hot
? ?

10. ID3
Temperature
yes
mild
Entropy(s) = 0.918
coolhot
yes
outlook
rainy
sunny overcast
yes yes yes
?

11. ID3
Amir Shokri
Gain outlook: (temperature = cool)
Gain(temperature, outlook) = 0.918 − (0 +0.4 +0.4) = 0.118
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(𝑜𝑣𝑒𝑟𝑐𝑎𝑠𝑡) => −
1
5
1
1
log
1
1
+
0
1
log
0
1
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(𝑟𝑎𝑖𝑛𝑦) => −
2
5
1
2
log
1
2
+
1
2
log
1
2
𝑒𝑛𝑡𝑟𝑜𝑝𝑦(𝑠𝑢𝑛𝑛𝑦) => −
2
5
1
2
log
1
2
+
1
2
log
1
2
𝐺𝑎𝑖𝑛 temperature, outlook = 0.918 − 0 + 0.4 + 0.4 = 0.118

12. ID3
Amir Shokri
Gain Humidity: (temperature = cool)
Gain(temperature, Humidity) = 0.918 −(0 + 0.36) = 0.121
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 high => −
2
5
2
2
log
2
2
+
0
2
log
0
2
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 normal => −
3
5
1
3
log
1
3
+
2
3
log
2
3
𝐺𝑎𝑖𝑛 temperature, Humidity = 0.918 − 0 + 0.55 = 0.368

13. ID3
Amir Shokri
Gain Windy: (temperature = cool)
Gain(temperature, Windy) = 0.918 −(0.4 + 0.5) = 0.018
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 true => −
2
5
1
2
log
1
2
+
1
2
log
1
2
𝑒𝑛𝑡𝑟𝑜𝑝𝑦 false => −
3
5
2
3
log
2
3
+
1
3
log
1
3
𝐺𝑎𝑖𝑛 temperature, Windy = 0.918 − 0.4 + 0.5 = 0.018

14. ID3
Temperature
yes
mild
Entropy(s) = 0.918
coolhot
Humidity
yes
high Normaloutlook
rainy
sunny overcast
yes yes yes
outlook
sunny overcast
no no yes
rainy

15. References
Machine learning book - Thomas Mitchell
Amir Shokri