22PCOAM16 ML Unit 3 Session 20 ID3 Algorithm and working.pptx

14/04/2025 1
Department of Computer Science & Engineering (SB-ET)
III B. Tech -I Semester
MACHINE LEARNING
SUBJECT CODE: 22PCOAM16
AcademicY
ear: 2023-2024
by
Dr. M.Gokilavani
GNITC
Department of CSE (SB-ET)

14/04/2025 Department of CSE (SB-ET) 2
22PCOAM16 MACHINE LEARNING
UNIT – III
Syllabus
Learning with Trees – Decision Trees – Constructing Decision Trees –
Classification and Regression Trees – Ensemble Learning – Boosting –
Bagging – Different ways to Combine Classifiers – Basic Statistics –
Gaussian Mixture Models – Nearest Neighbor Methods – Unsupervised
Learning – K means Algorithms

14/04/2025 3
TEXTBOOK:
• Stephen Marsland, Machine Learning - An Algorithmic Perspective, Second Edition,
Chapman and Hall/CRC.
• Machine Learning and Pattern Recognition Series, 2014.
REFERENCES:
• Tom M Mitchell, Machine Learning, First Edition, McGraw Hill Education, 2013.
• Ethem Alpaydin, Introduction to Machine Learning 3e (Adaptive Computation and
Machine
No of Hours Required: 13
Department of CSE (SB-ET)
UNIT - III LECTURE – 20

ID3 Algorithm
• The ID3 (Iterative Dichotomiser 3) algorithm was introduced by Ross
Quinlan in 1986.
• It became a key development in the evolution of decision tree algorithms,
influencing advanced models like C4.5 and CART.
• The algorithm’s main contribution was its innovative use of entropy and
information gain for selecting the most informative attributes when
splitting data.
UNIT - III LECTURE - 20

Purpose and Functionality
The primary purpose of the ID3 algorithm is to construct a decision tree for
classification tasks. It does this by:
• Evaluating each attribute in the dataset to determine its potential to reduce
uncertainty (measured using entropy).
• Selecting the attribute with the highest information gain to create splits that
maximize classification accuracy.
• Repeating the process recursively on smaller subsets until the tree fully
classifies the data.
The ID3 algorithm is particularly effective with categorical data and is
considered a foundational method in machine learning for its simplicity and
logical approach.

Steps in ID3
• The ID3 algorithm constructs a decision tree by recursively splitting the
dataset based on the attribute that provides the highest information gain.
Step 1: Calculate the Entropy of the Dataset
• Entropy measures the impurity or randomness in the dataset.
The formula for entropy is:
• where piis the proportion of instances belonging to class i.

Steps in ID3
Step 2: Compute Information Gain for Each Attribute
• Information Gain is the reduction in entropy after splitting the dataset
based on an attribute.
The formula for information gain is:
Here,
Sv is the subset of S for which attribute A has value V .

Steps in ID3
Step 3: Select the Attribute with the Highest Information Gain
• Choose the attribute that most effectively reduces uncertainty and use it as
the decision node.
Step 4: Split the Dataset
• Partition the dataset into subsets based on the selected attribute’s values.
• Assign branches for each possible outcome of the attribute.

Steps in ID3
Step 5: Recursively Apply the Process
• Repeat steps 1 to 4 for each subset, excluding the previously used attribute.
• Continue until one of the following termination conditions is met:
• All instances in a subset belong to the same class.
• There are no remaining attributes to split.
• The dataset is empty.

Example : ID3 Dataset
Imagine a dataset with attributes like Outlook, Temperature, Humidity, and Wind. The target variable
is whether to Play Tennis (Yes/No).

Steps in ID3
1. Calculate Entropy of the Dataset
• The dataset contains records of days when tennis was played or not.
Calculate the entropy for the target variable Play Tennis:
• The formula for entropy is:
In given Example, a dataset has 10 instances, with 6 labeled “Yes” and 4
labeled “No”.

Steps in ID3
2. Compute Information Gain for Each Attribute
• For each attribute (Outlook, Temperature, etc.), compute the
information gain. For example, splitting by Outlook might give subsets
like Sunny, Overcast, and Rainy, each with its entropy. Combine these
to calculate the overall entropy reduction.
• The formula for information gain is:
• If splitting the dataset by an attribute reduces the overall entropy from 0.97
to 0.58, the information gain is:

Steps in ID3
3. Select the Attribute with Highest Information Gain
• Outlook provides the highest information gain. It is selected as the
root node.
4. Split the Dataset
• Partition the dataset based on the values of Outlook.
For example:
• Sunny days may split further based on Humidity.
• Rainy days may split further based on Wind.

Steps in ID3
5. Repeat the Process Recursively
• Apply the same steps to the subsets until all records are classified or
no further splits are possible.
6.Visualization

Role in ID3 Algorithm
• Calculate Entropy: Compute the entropy for the dataset.
• Evaluate Attributes: Compute the information gain for each attribute.
• Select Attribute: The attribute with the highest information gain becomes
the decision node.

Advantage of ID3
• Simplicity and Interpretability
• Efficient Handling of Categorical Data
• Greedy Approach
• Foundation for Advanced Algorithms
• Versatility Across Applications

Disadvantage of ID3
• Over fitting
• Difficulty Handling Continuous Data
• Bias Towards Multi-Valued Attributes
• Lack of Pruning Mechanisms
• Scalability Issues

Topics to be covered in next session 21
• Classification and Regression Tree Algorithm (CART)
Thank you!!!

22PCOAM16 ML Unit 3 Session 20 ID3 Algorithm and working.pptx

More Related Content

Similar to 22PCOAM16 ML Unit 3 Session 20 ID3 Algorithm and working.pptx

More from Guru Nanak Technical Institutions

Recently uploaded

22PCOAM16 ML Unit 3 Session 20 ID3 Algorithm and working.pptx