A Theory of the Learnable
Leslie Valiant
Dhruv Gairola
Computational Complexity, Michael Soltys
gairold@mcmaster.ca ; dhru...
Overview

1

Learning

2

Contribution

3

PAC learning
Sample complexity
Boolean functions
k-decision lists

4

Conclusio...
Learning

Humans can learn.
Machine learning (ML) : learning from data; knowledge acquisition
w/o explicit programming.
Ex...
Modelling supervised Learning

Given training set of labelled examples, learning algorithm generates a
hypothesis (candida...
Contribution

With high probability an (efficient) learning algorithm will find a
hypothesis that is approximately identical ...
PAC learning

Goal : show that after training, with high probability, all good
hypothesis will be approximately correct.
N...
PAC learning (2)

Hypothesis hg ∈ H is approximately correct if :
error (hg ) ≤ where
error(h) = P(h(x) = f (x)| x drawn f...
PAC learning (3)

P(hb agrees with 1 example) ≤ (1 − ).
P(hb agrees with N examples) ≤ (1 − )N .
P(Hb contains a good hypo...
Sample complexity

N ≥ ( 1 )(ln 1 + ln|H|)
δ
If you train the learning algo with Xtraining of size N, then the
returned hy...
Why boolean functions?

Because boolean functions can represent concepts, which is what we
commonly want machines to learn...
Boolean functions

Boolean functions are of the form f : {0, 1}n → {0, 1} where n are
the number of literals.
n

Let H = {...
k-decision lists

A single decision list (DL) is a representation of a single boolean
function. DL is not PAC-learnable ei...
k-decision lists (2)

What is |H| for k-DL i.e., what is |k-DL(n)| where n is number of
literals?
k
k
After calculations, ...
Conclusion

PAC learning : with high
probability an (efficient)
learning algorithm will find a
hypothesis that is
approximate...
References

Carla Gomes, Cornell, Foundations of AI notes

Dhruv Gairola (McMaster Univ.)

A Theory of the Learnable

Nove...
Upcoming SlideShare
Loading in...5
×

A Theory of the Learnable; PAC Learning

577

Published on

PAC Learning, Leslie Valiant

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
577
On Slideshare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Theory of the Learnable; PAC Learning

  1. 1. A Theory of the Learnable Leslie Valiant Dhruv Gairola Computational Complexity, Michael Soltys gairold@mcmaster.ca ; dhruvgairola.blogspot.ca November 13, 2013 Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 1 / 15
  2. 2. Overview 1 Learning 2 Contribution 3 PAC learning Sample complexity Boolean functions k-decision lists 4 Conclusion Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 2 / 15
  3. 3. Learning Humans can learn. Machine learning (ML) : learning from data; knowledge acquisition w/o explicit programming. Explore computational models for learning. Use models to get insights about learning. Use models to develop new learning algorithms. Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 3 / 15
  4. 4. Modelling supervised Learning Given training set of labelled examples, learning algorithm generates a hypothesis (candidate function). Run hypothesis on test set to check how good it is. But how good really? Maybe training and test data consists of bad examples so the hypothesis doesn’t generalize well. Insight : Introduce probabilities to measure degree of certainty and correctness. Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 4 / 15
  5. 5. Contribution With high probability an (efficient) learning algorithm will find a hypothesis that is approximately identical to the hidden target function. Intuition : A hypothesis built from a large amount of training data is unlikely to be wrong i.e., Probably approximately correct (PAC). Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 5 / 15
  6. 6. PAC learning Goal : show that after training, with high probability, all good hypothesis will be approximately correct. Notation : X : set of all possible examples D : distribution from which examples are drawn H : set of all possible hypothesis N : |Xtraining | f : target function Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 6 / 15
  7. 7. PAC learning (2) Hypothesis hg ∈ H is approximately correct if : error (hg ) ≤ where error(h) = P(h(x) = f (x)| x drawn from D) Bad hypothesis : error (hb ) > P(hb disagrees with 1 example) > Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 7 / 15
  8. 8. PAC learning (3) P(hb agrees with 1 example) ≤ (1 − ). P(hb agrees with N examples) ≤ (1 − )N . P(Hb contains a good hypothesis) ≤ |Hb |(1 − )N ≤ |H|(1 − )N . Lets say |H|(1 − )N ≤ δ. ... N ≥ ( 1 )(ln 1 + ln|H|) δ This expresses sample complexity. Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 8 / 15
  9. 9. Sample complexity N ≥ ( 1 )(ln 1 + ln|H|) δ If you train the learning algo with Xtraining of size N, then the returned hypothesis is PAC because there exists a probability (1 − δ) that this hypothesis will have an error of at most (approximately). e.g., if you want smaller and smaller δ, you need more N’s (more examples). Lets look at example of H : boolean functions. Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 9 / 15
  10. 10. Why boolean functions? Because boolean functions can represent concepts, which is what we commonly want machines to learn. Concepts are predicates e.g., isMaleOrFemale(height). Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 10 / 15
  11. 11. Boolean functions Boolean functions are of the form f : {0, 1}n → {0, 1} where n are the number of literals. n Let H = {all boolean functions on n literals} ∴ |H| = 22 Substituting H into sample complexity expression gives O(2n ) i.e., boolean functions are not PAC-learnable. Can we restrict size of H? Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 11 / 15
  12. 12. k-decision lists A single decision list (DL) is a representation of a single boolean function. DL is not PAC-learnable either. A single DL consists of a series of tests. e.g. if f1 then return b1 ; elseif f2 then return b2 ; ... elseif fn return bn ; A single DL corresponds to a single hypothesis. Apply restriction : A k-decision list is a decision list where each test is a conjunction of at most k literals. Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 12 / 15
  13. 13. k-decision lists (2) What is |H| for k-DL i.e., what is |k-DL(n)| where n is number of literals? k k After calculations, |k-DL(n)| = 2O(n log (n )) Substitute |k-DL(n)| into sample complexity expression : N ≥ 1 (ln 1 + O(nk log (nk ))) δ δ Sample complexity is poly! What about learning complexity? There are efficient algorithms for learning k-decision lists! (e.g., greedy algorithm) We have polynomial sample complexity and efficient k-DL algorithms ∴ k-DL is PAC learnable! Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 13 / 15
  14. 14. Conclusion PAC learning : with high probability an (efficient) learning algorithm will find a hypothesis that is approximately identical to the hidden target hypothesis. k-DL is PAC learnable. Computational learning theory : concerned with the analysis of ML algorithms and covers a lot of fields. Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 14 / 15
  15. 15. References Carla Gomes, Cornell, Foundations of AI notes Dhruv Gairola (McMaster Univ.) A Theory of the Learnable November 13, 2013 15 / 15
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×