SlideShare a Scribd company logo
Decision Tree, Entropy
Md Saeed Siddik
Khaza Moinuddin Mazumder
Decision Tree
A decision tree is a decision support tool that
uses a tree and their possible consequences.
Decision Tree is a flow-chart like structure in which
internal node represents test on an attribute
each branch represents outcome of test
each leaf node represents class label (decision taken
after computing all attributes)
03/10/2013DT and Entropy2
Consists of DT
03/10/2013DT and Entropy3
 A decision tree consists of 3 types of nodes:
1.Decision nodes
2.Chance nodes
3.End nodes
Types of variables in DT
Four types of tree can generated from a variables.
Those are..
03/10/2013DT and Entropy4
Terminal
.
Both are Left side
/
Both are Right side

Separated in Both side
/
Decision Table
03/10/2013DT and Entropy5
Evidence Action Author Thread Length
e1 skip known new long
e2 read unknown new short
e3 skip unknown old long
e4 skip known old long
e5 read known new short
e6 skip known old long
Author
Length
Skip
Rea
d
Thread
read skip
Decision Tree
03/10/2013DT and Entropy6
Decision
03/10/2013DT and Entropy7
 Known ∧ Long ⇒ Skip
 Known ∧ Short ⇒ Read
 Unknown ∧ New ⇒ Read
 Unknown ∧ Old ⇒ Skip
Entropy
Entropy is a measure of the uncertainty in a random
variable
The term Entropy, usually refers to the Shannon
entropy, which quantifies the expected value of the
information contained in a message.
Given a random variable ‘v’ with value Vk , the entropy
of x is defined by
k
kk
vPvPvH )(log)()( 2
03/10/2013DT and Entropy8
Entropy Measurement Unit
03/10/2013DT and Entropy9
 bit
 {0,1}
 Based on 2
 nat
 Also known as nit or nepit
 Logarithmic unit, based on e
 1 nat = 1.44 bit = 0.434 ban
 ban
 Also known as hartley or a dit (short for decimal digit)
 Logarithmic unit, based on 10
 Introduced by Alan Turing and I J Good
 1 ban = 3.32 bits = 2.30 nats
Entropy
03/10/2013DT and Entropy10
 Given the Boolean random variable with
probability q, (1-q)
)1(log)1(log)( 22
qqqqqB
Entropy for n+p variables
03/10/2013DT and Entropy11
if we consider we have n+p examples
Where p is positive and n is negative.
qp
n
qp
n
qp
p
qp
p
qp
p
B
2
log
2
log
)(
Reminder
03/10/2013DT and Entropy12
The Expected Entropy (EH) or Reminder remaining
after trying attribute A (with branches i = 1,2.....,k)
is :
d
k kk
kkk
pn
p
B
pn
pn
Ader
1
)()(minRe
Information Gain (IG)
03/10/2013DT and Entropy13
Information Gain is a non-symmetric measure of
the difference between two probability
distributions P and Q.
)(minRe)()( Ader
np
p
BAGain
Calculate the root
03/10/2013DT and Entropy14
 Choose the attribute with highest gain.

More Related Content

More from Saeed Siddik

Quantile Quantile Plot qq plot
Quantile Quantile Plot qq plot  Quantile Quantile Plot qq plot
Quantile Quantile Plot qq plot
Saeed Siddik
 
Comparative analysis on different DES model
Comparative analysis on different DES modelComparative analysis on different DES model
Comparative analysis on different DES model
Saeed Siddik
 
Connect dell equallogic storage to linux instance
Connect dell equallogic storage to linux instanceConnect dell equallogic storage to linux instance
Connect dell equallogic storage to linux instance
Saeed Siddik
 
Comparison between VMware and Open Stack Cloud
Comparison between VMware and Open Stack CloudComparison between VMware and Open Stack Cloud
Comparison between VMware and Open Stack Cloud
Saeed Siddik
 
Deadlock in distribute system by saeed siddik
Deadlock in distribute system by saeed siddikDeadlock in distribute system by saeed siddik
Deadlock in distribute system by saeed siddik
Saeed Siddik
 
MIS Case Study
MIS Case StudyMIS Case Study
MIS Case Study
Saeed Siddik
 
Birth & death information automation
Birth & death information automationBirth & death information automation
Birth & death information automation
Saeed Siddik
 

More from Saeed Siddik (7)

Quantile Quantile Plot qq plot
Quantile Quantile Plot qq plot  Quantile Quantile Plot qq plot
Quantile Quantile Plot qq plot
 
Comparative analysis on different DES model
Comparative analysis on different DES modelComparative analysis on different DES model
Comparative analysis on different DES model
 
Connect dell equallogic storage to linux instance
Connect dell equallogic storage to linux instanceConnect dell equallogic storage to linux instance
Connect dell equallogic storage to linux instance
 
Comparison between VMware and Open Stack Cloud
Comparison between VMware and Open Stack CloudComparison between VMware and Open Stack Cloud
Comparison between VMware and Open Stack Cloud
 
Deadlock in distribute system by saeed siddik
Deadlock in distribute system by saeed siddikDeadlock in distribute system by saeed siddik
Deadlock in distribute system by saeed siddik
 
MIS Case Study
MIS Case StudyMIS Case Study
MIS Case Study
 
Birth & death information automation
Birth & death information automationBirth & death information automation
Birth & death information automation
 

Recently uploaded

High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 

Recently uploaded (20)

High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 

Decision Tree and entropy

  • 1. Decision Tree, Entropy Md Saeed Siddik Khaza Moinuddin Mazumder
  • 2. Decision Tree A decision tree is a decision support tool that uses a tree and their possible consequences. Decision Tree is a flow-chart like structure in which internal node represents test on an attribute each branch represents outcome of test each leaf node represents class label (decision taken after computing all attributes) 03/10/2013DT and Entropy2
  • 3. Consists of DT 03/10/2013DT and Entropy3  A decision tree consists of 3 types of nodes: 1.Decision nodes 2.Chance nodes 3.End nodes
  • 4. Types of variables in DT Four types of tree can generated from a variables. Those are.. 03/10/2013DT and Entropy4 Terminal . Both are Left side / Both are Right side Separated in Both side /
  • 5. Decision Table 03/10/2013DT and Entropy5 Evidence Action Author Thread Length e1 skip known new long e2 read unknown new short e3 skip unknown old long e4 skip known old long e5 read known new short e6 skip known old long
  • 7. Decision 03/10/2013DT and Entropy7  Known ∧ Long ⇒ Skip  Known ∧ Short ⇒ Read  Unknown ∧ New ⇒ Read  Unknown ∧ Old ⇒ Skip
  • 8. Entropy Entropy is a measure of the uncertainty in a random variable The term Entropy, usually refers to the Shannon entropy, which quantifies the expected value of the information contained in a message. Given a random variable ‘v’ with value Vk , the entropy of x is defined by k kk vPvPvH )(log)()( 2 03/10/2013DT and Entropy8
  • 9. Entropy Measurement Unit 03/10/2013DT and Entropy9  bit  {0,1}  Based on 2  nat  Also known as nit or nepit  Logarithmic unit, based on e  1 nat = 1.44 bit = 0.434 ban  ban  Also known as hartley or a dit (short for decimal digit)  Logarithmic unit, based on 10  Introduced by Alan Turing and I J Good  1 ban = 3.32 bits = 2.30 nats
  • 10. Entropy 03/10/2013DT and Entropy10  Given the Boolean random variable with probability q, (1-q) )1(log)1(log)( 22 qqqqqB
  • 11. Entropy for n+p variables 03/10/2013DT and Entropy11 if we consider we have n+p examples Where p is positive and n is negative. qp n qp n qp p qp p qp p B 2 log 2 log )(
  • 12. Reminder 03/10/2013DT and Entropy12 The Expected Entropy (EH) or Reminder remaining after trying attribute A (with branches i = 1,2.....,k) is : d k kk kkk pn p B pn pn Ader 1 )()(minRe
  • 13. Information Gain (IG) 03/10/2013DT and Entropy13 Information Gain is a non-symmetric measure of the difference between two probability distributions P and Q. )(minRe)()( Ader np p BAGain
  • 14. Calculate the root 03/10/2013DT and Entropy14  Choose the attribute with highest gain.