Decision trees are a type of supervised learning algorithm that can be used for classification and regression. They work by creating a tree structure where each branch represents a choice between alternatives and each leaf node represents a classification decision. Decision trees are appropriate for problems where instances are represented by attribute-value pairs and the target function has discrete output values. Information gain is used to determine which attribute to use at each decision node by selecting the attribute that reduces entropy, or uncertainty, the most. Issues with decision trees include overfitting the training data if the tree is grown too deeply, handling continuous attributes, and dealing with missing data values. Overfitting occurs when a model aligns too closely to the training data and loses its ability to generalize to new

1. Machine Learning with
DECISION TREES
by
19BQ1A0125

2. Agenda
● What are Decision trees?
Appropriate problems for DTrees
Information gain , Entropy
Issues with DTrees
Overfitting
●
●
●
●

3. What are Decision trees?
● A decision tree is a tree in which each branch
node represents a choice between a number
of alternatives, and each leaf node represents
a decision.
A type of supervised learning algorithm.
●

4. Example

5. Appropriate problems for DTrees
● Instances are represented by attribute-value pairs
● The target function has discrete output values
● The training data may contain errors
● The training data may have missing attribute values

6. Which node can be described easily?

7. ➔ Less impure node requires less information to describe it.
More impure node requires more information.
➔
===> Information theory is a measure to
define this degree of disorganization in a
system known as Entropy.
So, we can conclude

8. Entropy - measuring
homogeneity of a learning set
● Entropy is a measure of the uncertainty about a source of
messages.
Given a collection S, containing positive and negative
examples of some target concept, the entropy of S relative to
this classification.
●
where, pi is the proportion of S belonging to class i

9. ● Entropy is 0 if all the members
of S belong to the same class.
● Entropy is 1 when the collection
contains an equal no. of +ve
and -ve examples.
● Entropy is between 0 and 1 if
the collection contains unequal
no. of +ve and -ve examples.

10. Information gain
● Decides which attribute goes into a decision node.
To minimize the decision tree depth, the attribute with the
most entropy reduction is the best choice!
●

11. ISSUES
● How deeply to grow the decision tree?
Handling continuous attributes
Choosing an appropriate attribute selection
measure
Handling training data with missing attribute
values
●
●
●

12. Overfitting in Decision Trees
● If a decision tree is fully grown, it may lose
some generalization capability.
This is a phenomenon known as overfitting.
●

13. Overfitting
A hypothesis overfits the training examples if
some other hypothesis that fits the training
examples less well actually performs better
over the entire distribution of instances (i.e,
including instances beyond the training
examples)

14. .
Overfitting is an error that occurs in data modeling
as a result of a particular function aligning too
closely to a minimal set of data points.
Financial professionals are at risk of overfitting a
model based on limited data and ending up with
results that are flawed.
When a model has been compromised by
overfitting, the model may lose its value as a
predictive tool for investing.
A data model can also be underfitted, meaning it is
too simple, with too few data points to be effective.

15. Causes of Overfitting
● Overfitting Due to Presence of Noise -
Mislabeled instances may contradict the class
labels of other similar records.
Overfitting Due to Lack of Representative
Instances - Lack of representative instances
in the training data can prevent refinement of
the learning algorithm.
●

16. Overfitting Due to Noise: An
Example

17. Overfitting Due to Lack of
Samples

18. Overfitting Due to Lack of
Samples
● Although the model’s training error
is zero, its error rate on the test set
is 30%.
● Humans, elephants, and dolphins
are misclassified because the
decision tree classifies all
warmblooded vertebrates that do not
hibernate as non-mammals.

19. A good model must not only fit the
training data well
but also accurately classify records
it has never seen.

20. Thank You