Published in ECML 2010 @inproceedings{DBLP:conf/pkdd/RoletT10, author = {Philippe Rolet and Olivier Teytaud}, title = {Complexity Bounds for Batch Active Learning in Classification}, booktitle = {ECML/PKDD (3)}, year = {2010}, pages = {293-305}, ee = {http://dx.doi.org/10.1007/978-3-642-15939-8_19}, crossref = {DBLP:conf/pkdd/2010-3}, bibsource = {DBLP, http://dblp.uni-trier.de} } @proceedings{DBLP:conf/pkdd/2010-3, editor = {Jos{\'e} L. Balc{\'a}zar and Francesco Bonchi and Aristides Gionis and Mich{\`e}le Sebag}, title = {Machine Learning and Knowledge Discovery in Databases, European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part III}, booktitle = {ECML/PKDD (3)}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, volume = {6323}, year = {2010}, isbn = {978-3-642-15938-1}, ee = {http://dx.doi.org/10.1007/978-3-642-15939-8}, bibsource = {DBLP, http://dblp.uni-trier.de} }

1. Complexity bounds for
batch active learning
P. Rolet, O. Teytaud
What is active learning ?
What is batch active learning ?
Upper bounds on the speed­up
Lower bounds on the speed­up
Good cases
Conclusions, take­home messages

2. Complexity bounds for
batch active learning
What is active learning ?
What is batch active learning ?
Upper bounds on the speed­up
Lower bounds on the speed­up
Good cases
Conclusions, take­home messages

3. What is active learning ?
Supervised learning:
I have examples x1,x2,x3,...,xn
I have labels y1,y2,...,yn
They are randomly drawn.
I try to build f such that
E (f(x) – y)2 is small.

4. What is active learning ?
Active supervised learning:
I choose x1
I get y1
I choose x2
I get y2
… until xn,yn.
I try to build f such that
E (f(x) – y)2 is small.

5. What is active learning ?
Typical application
I build a prototype of a car (x1)
I test it and get caracteristics (y1)
I build a prototype of a car (x2)
I test it and get caracteristics (y2)
… until xn,yn.
I try to build f such that
E (f(x) – y)2 is small.
I can predict the caracteristics of a new car
==> Very related to optimization

6. What is active learning ?
Other typical applications
Learning value functions in reinforcement learning (==>
really efficient in moderate dimension)
Modelizing the effect of parameters on a big numerical
code

7. Complexity bounds for
batch active learning
What is active learning ?
What is batch active learning ?
Upper bounds on the speed­up
Lower bounds on the speed­up
Good cases
Conclusions, take­home messages

8. Batch active learning ?
Active supervised learning (reminder):
I choose x1
I get y1 Non-batch
I choose x2 case
(active
I get y2 learning)
… until xn,yn.
I try to build f such that
E (f(x) – y)2 is small.

9. Batch active learning ?
Batch active supervised learning:
I choose x1,x2,...,x
I get y1,x2,...,x
I choose x+1,x+2,...,x2 Batch
I get y+1,y+2,...,y2 case
(active
… until xn,yn. learning)
I try to build f such that
E (f(x) – y)2 is small.

10. Batch active learning ?
Batch active supervised learning:
I choose x (1­)
I get y (1­)
I choose x (+1 ­ 2) Batch
I get y(+1 ­ 2) case
(active
… until x(1­n),y(1­n). learning)
I try to build f such that
E (f(x) – y)2 is small.

11. Batch active learning
Advantage: parallelism.
Drawbacks: not necessarily  times faster.
Question: how much faster ?

12. Notations
A bit tedious, sorry. Necessary I guess.
Target
function

13. Notations
A bit tedious, sorry. Necessary I guess.
Approximation

14. Notations
A bit tedious, sorry. Necessary I guess.
Cost

15. Notations
A bit tedious, sorry. Necessary I guess.
Cost for the best
possible
algorithm

16. Complexity bounds for
batch active learning
What is active learning ?
What is batch active learning ?
Upper bounds on the speed­up
Lower bounds on the speed­up
Good cases
Conclusions, take­home messages

17. Upper bound 1: triviality.
First, in the best case, it's linear:

18. Upper bound 2: VC-dim.
V=VC-dim
To be compared with:

19. Upper bound 2: VC-dim.
Proof (sketch of the formal proof in the paper):
- VC-dimension V ==> at most K = Shattering(,V)
possible different combinations.
- Therefore at most K possible cases,
so log2(K) bits of information
- Speed-up at most log2(K)
To be compared with:

20. Upper bound 2: VC-dim.
Proof (sketch of the formal proof in the paper):
- VC-dimension V ==> at most K = Shattering(,V)
possible different combinations.
- Therefore at most K possible cases,
so log2(K) bits of information
- Speed-up at most log2(K)
To be compared with:

21. Upper bound 2: VC-dim.
Proof (sketch of the formal proof in the paper):
Logarithmic speed-up
- VC-dimension V ==> at most K = Shattering(,V)
+
possible different combinations.
a factor V has
disappeared!
- Therefore at most K possible cases,
so log2(K) bits of information
Linear speed-up until V
- Speed-up at most log2(K)
(in some cases)
and then logarithmic
To be compared with:

22. Complexity bounds for
batch active learning
What is active learning ?
What is batch active learning ?
Upper bounds on the speed­up
Lower bounds on the speed­up
Good cases
Conclusions, take­home messages

23. Lower bounds

24. Lower bounds
i.e. speed-up logarithmic, starting at any .
So:
- choose  such that your favorite algorithm
has linear speed-up (if possible)
- then apply theorem above for extending to '

25. Complexity bounds for
batch active learning
What is active learning ?
What is batch active learning ?
Upper bounds on the speed­up
Lower bounds on the speed­up
Good cases
Conclusions, take­home messages

26. Good cases
Then,
and

27. Complexity bounds for
batch active learning
What is active learning ?
What is batch active learning ?
Upper bounds on the speed­up
Lower bounds on the speed­up
Good cases
Conclusions, take­home messages

28. Conclusions
You can always have a logarithmic speed­up in batch
active learning
Asymptotically, it will never be better than
logarithmic.
Sometimes, linear until V and then logarithmic.