The document discusses concept learning and the general-to-specific ordering of hypotheses. It describes how concept learning can be framed as a search problem through a hypothesis space to find the hypothesis that best fits training examples. The Find-S algorithm performs a specific-to-general search to find the most specific hypothesis, while the Candidate-Elimination algorithm computes the version space by iteratively updating the sets of most specific and most general hypotheses consistent with the data. The Candidate-Elimination algorithm provides a framework for concept learning but may not be robust to noisy data or situations where the target concept is not expressible in the hypothesis space. Inductive bias, such as the assumption that the target concept exists in the hypothesis space

1. Concept Learning and
the General-to-Specific Ordering
이 종우
자연언어처리연구실

2. Concept Learning
• Concepts or Categories
– “birds”
– “car”
– “situations in which I should study more in
order to pass the exam”
– Concept
• some subset of objects or events defined over a
larger set, or a boolean valued function defined over
this larger set.

3. – Learning
• inducing general functions from specific training
examples
– Concept Learning
• acquiring the definition of a general category given
a sample of positive and negative training examples
of the category

4. A Concept Learning Task
• Target Concept
– “days on which Aldo enjoys water sport”
• Hypothesis
– vector of 6 constraints (Sky, AirTemp, Humidity,
Wind, Water, Forecast, EnjoySport )
– Each attribute (“?”, single value or “0”)
– e.g. <?, Cold, High, ?, ?, ?>

5. Instance Sky AirTemp Humidity Wind Water Forecast EnjoySport
A Sunny Warm Normal Strong Warm Same No
B Sumny Warm High Strong Warm Same Yes
C Rainy Cold High Strong Warm Change No
D Sunny Warm High Strong Cool Change Yes
Training examples for the target concept EnjoySport

6. • Given :
– instances (X): set of iterms over which the concept
is defined.
– target concept (c) : c : X → {0, 1}
– training examples (positive/negative) : <x,c(x)>
– training set D: available training examples
– set of all possible hypotheses: H
• Determine :
– to find h(x) = c(x) (for all x in X)

7. Inductive Learning Hypothesis
• Inductive Learning Hypothesis
– Any good hypothesis over a sufficiently large
set of training examples will also approximate
the target function. well over unseen examples.

8. Concept Learning as Search
• Issue of Search
– to find training examples hypothesis that best fits
training examples
• Kinds of Space in EnjoySport
– 3*2*2*2*2 = 96: instant space
– 5*4*4*4*4 = 5120: syntactically distinct
hypotheses within H
– 1+4*3*3*3*3 = 973: semantically distinct
hypotheses

9. • Search Problem
– efficient search in hypothesis
space(finite/infinite)

10. General-to-Specific Ordering of
Hypotheses
• Hypotheses의 General-to-Specific Ordering
– x satisfies h ⇔ h(x)=1
– more_general_than_or_equal_to relations
• <Sunny,?,?,Strong,?,?> ≦ g <Sunny,?,?,?,?,?>
– more_general_than_or_equal_to relations
)]
1
)
(
(
)
1
)[(
( 





 x
h
h
X
x
h
h j
k
k
g
j
)
(
)
( j
g
k
k
g
j
k
g
j h
h
h
h
h
h 






11. – partial order (reflexive,antisymmetric,transitive)
Concept Learning as Search

12. Find-S: Finding a Maximally
Specific Hypothesis
• algorithm
• 1. Initialize h to the most specific hypothesis in H
• 2. For each positive training example x
• For each attribute constraint ai in h
– If the constraint ai is satisfied by x
– then do nothing
– else replace ai in h by the next more general constraint that is
satisfied by x
• 3. Output hypothesis h
• Property
• guaranteed to output the most specific hypothesis
• no way to determine unique hypothesis
• not cope with inconsistent errors or noises

13. Find-S:Finding a Maximally
Specific Hypothesis(2)

14. Version Spaces and the Candidate-
Elimination Algorithm
– output all hypotheses consistent with the training
examples.
– perform poorly with noisy training data.
• Representation
– Consistent(h,D) ⇔(∀<x,c(x)>  D) h(x) = c(x)
– VSH,D ⇔ {h  H | Consistent(h,D)}
• List-Then-Eliminate Algorithm
– lists all hypotheses -> remove inconsistent ones.
– Appliable to finite H

15. Version Spaces and the Candidate-
Elimination Algorithm(2)
• More Compact Representation for Version Spaces
– general boundary G
– specific boundary S
– Version Space redefined with S and G
)]}
,
'
(
)
'
)[
'
(
)
,
(
|
{ D
s
Consistent
g
g
H
g
D
s
Consistent
H
g
G g 






)]}
,
'
(
)
'
)[(
'
(
)
,
(
|
{ D
s
Consistent
s
s
H
s
D
s
Consistent
H
s
S g 






)}
)(
)(
(
|
{
, S
h
g
G
g
S
s
H
h
VS g
g
D
H 








16. Version Spaces and the Candidate-
Elimination Algorithm(3)

17. Version Spaces and the Candidate-
Elimination Algorithm(4)
• Condidate-Elimination Learning Algorithm
• Initialize G to the set of maximally general hypotheses in H
• Initialize S to the set of maximally specific hypotheses in H
• For each training example d, do
• If d is a positive example
• Remove from G any hypothesis inconsistent with d
• For each hypothesis s in S that is not consistent with d
• Remove s from S
• Add to S all minimal generalizations h of s such that
• h is consistent with d, and some member of G is more general
• than h
• Remove from S any hypothesis that is more general than another
• hypothesis in S

18. Version Spaces and the Candidate-
Elimination Algorithm(5)
• If d is a negative example
• Remove from S any hypothesis inconsistent with d
• For each hypothesis g in G that is not consistent with d
• Remove g from G
• Add to G all minimal specializations h of g such that
• h is consistent with d, and some member of S is more specific
than h
• Remove from G any hypothesis that is less general than another
hypothesis in G

19. Version Spaces and the Candidate-
Elimination Algorithm(6)
• Illustrative Example

20. Version Spaces and the Candidate-
Elimination Algorithm(7)

21. Version Spaces and the Candidate-
Elimination Algorithm(8)

22. Version Spaces and the Candidate-
Elimination Algorithm(9)

23. Remarks on Version Spaces and
Candidate-Elimination
• Will the Candidate-Elimination Algorithm
Converge to the Correct Hypothesis?
– Prerequisite
– 1. No error in training examples
– 2. Hypothesis exists which correctly describes c(x).
– S and G boundary sets converge to an empty set =>
no hypothesis in H consistent with observed examples.
• What Training Example Should the Learner
Request Next?
– Negative one specifies G , positive one generalizes S.
– optimal query satisfy half the hypotheses.

24. Remarks on Version Spaces and
Candidate-Elimination(2)
• How Can Partially Learned Concepts Be Used?
Instance Sky AirTemp Humidity Wind Water Forecast EnjoySport
A Sunny Warm Normal Strong Cool Change ?
B Rainy Cold Normal Light Warm Same ?
C Sunny Warm Normal Light Warm Same ?
D Sunny Cold Normal Strong Warm Same ?
A : classified to positive
B : classified to negative
C : 3 positive , 3 negative
D : 2 positive, 4 negative

25. Inductive Bias
• A Biased Hypothesis Space
Example Sky AirTemp Humidity Wind Water Forecast EnjoySport
1 Sunny Warm Normal Strong Cool Change Yes
2 Cloudy Warm Normal Strong Cool Change Yes
3 Rainy Warm Normal Strong Cool Change No
- zero hypothesis in the version space
- caused by only conjunctive hypothesis

26. Inductive Bias(2)
• An Unbiased Learner
– Power set of X : set of all subsets of a set X
• number of size of power set : 2|X|
– e.g. <Sunny,?,?,?,?,?> ∨ <Cloudy,?,?,?,?,?>
– new problem : unable to generalize beyond the
observed examples.
• Observed examples are only unambiguously classified.
• Voting results in no majority or minority.

27. Inductive Bias(3)
• The Futility of Bias-Free Learning
– no inductive bias => cannot classify unseen data
reasonably
– inductive bias of L : any minimal set of assertions B
such that
– inductive bias of Candidate-Elimination algorithm
• c ∈ H
– advantage of introducing inductive bias
• generalizing beyond the observed data
• allows comparison of different learners
)]
,
(
)
)[(
( c
i
i
c
i D
x
L
x
D
B
X
x 





28. Inductive Bias(4)
• e.g
– Rote-learner : no inductive bias
– Candidate-Elimination algo : c ∈ H => more strong
– Find-S : c ∈ H and that all are negative unless not
proved positive

29. Inductive Bias(5)

30. Summary
• Concept learning can be cast as a problem of searching
through a large predefined space of potential hypotheses.
• General-to-specific partial ordering of hypotheses provides
a useful structure for search.
• Find-S algorithm performs specific-to-general search to
find the most specific hypothesis.
• Candidate-Elimination algorithm computes version space
by incrementally computing the sets of maximally specific
(S) and maximally general (G) hypotheses.
• S and G delimit the entire set of hypotheses consistent with
the data.

31. • Version spaces and Candidate-Elimination algorithm
provide a useful conceptual framework for studying
concept learning.
• Candidate-Elimination algorithm not robust to noisy data
or to situations where the unknown target concept is not
expressible in the provided hypothesis space.
• Inductive bias in Candidate-Elimination algorithm is that
target concept exists in H
• If hypothesis space be enriched so that there is a every
possible hypothesis, that would remove the ability to
classify any instance beyond the observed examples.