This document provides an overview of machine learning and the ID3 decision tree algorithm. It discusses key concepts in machine learning including supervised and unsupervised learning. It also describes how the ID3 algorithm builds decision trees from training examples by selecting attributes that provide the most information gain at each step, resulting in a hypothesis space search. The document notes that the ID3 algorithm has biases like preferring shorter trees and attributes with high information gain near the root.

1. Artificial Intelligence Lecture 15 Introduction to Machine Learning

2. Overview Machine Learning ID3 Decision Tree Algorithm Discussions

3. Machine Learning Supervised Learning Training examples consist of pairs of input vectors, and desired outputs Unsupervised Learning Training examples do not contain hints about correct outputs Usually used to identify unusual structures in data

4. Inductive Learning Basics Inferring a boolean/real-valued function from training examples A training example is a pair of (x, f(x)) x is the input f(x) is the output of the function applied to x

5. Hypothesis Any function that approximates the given set of examples Bias : preference for one hypothesis beyond mere consistency (a) (b) (c) (d)

6. Hypothesis Space A set of all hypotheses consistent with data denoted by H:{H 1 , H 2 , …, H n } Inductive learning is searching for a good hypothesis in the hypothesis space Occam’s razor: prefer the simplest hypothesis consistent with data Inductive Learning Assumption : Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved examples .

7. ID3 Decision Tree Algorithm

8. Training Examples

9. Decision Tree for PlayTennis ? ? PlayTennis Wind Humidity Temperature Outlook Weak High Hot Overcast Weak High Hot Sunny

10. Decision Tree Representation Decision Tree Each internal node tests an attribute Each branch takes an attribute value Each leaf node predict a class label Disjunction of conjunctions of a set of attribute values

11. Entropy

12. Information Gain

13. ID3 Algorithm

14. Training Examples

15. Selecting the Best Attribute

17. Hypothesis Space Search by ID3

18. Discussions

19. Hypothesis Space Search by ID3 Hypothesis space is complete Target function surely in there … Outputs a single hypothesis (Which one?) Cannot determine how many alternatives No back tracking Local minima … Use statistical properties of all training data at each step in search Robust to noisy data

20. Inductive Bias in ID3 Note the hypothesis space H is the power set of instances X. Unbiased? Not really. Preference for short trees, and for those with high information gain attributes near the root Bias is a preference for some hypotheses, rather than a restriction of hypothesis space H, e.g. target concept is not in H.

21. Summary Machine Learning ID3 Decision Tree Algorithm Discussions