1. An Introduction to Boosting Yoav Freund Banter Inc.

4. Generative modeling Voice Pitch Probability mean1 var1 mean2 var2

5. Discriminative approach Voice Pitch No. of mistakes

6. Ill-behaved data Voice Pitch Probability mean1 mean2 No. of mistakes

7. Traditional Statistics vs. Machine Learning Data Estimated world state Predictions Actions Statistics Decision Theory Machine Learning

8. Comparison of methodologies Mismatch problems Performance measure Goal Model Misclassifications Outliers Misclassification rate Likelihood Classification rule Probability estimates Discriminative Generative

9. Boosting

10. A weak learner weak learner A weak rule h h Weighted training set (x1, y1 , w1 ),(x2, y2 , w2 ) … (xn, yn , wn ) instances x1,x2,x3,…,xn labels y1,y2,y3,…,yn The weak requirement: Feature vector Binary label Non-negative weights sum to 1

11. The boosting process Final rule: Sign [ ] h1   h2   hT   weak learner h1 (x1,y1, 1/n ), … (xn,yn, 1/n ) weak learner h2 (x1,y1, w1 ), … (xn,yn, wn ) h3 (x1,y1, w1 ), … (xn,yn, wn ) h4 (x1,y1, w1 ), … (xn,yn, wn ) h5 (x1,y1, w1 ), … (xn,yn, wn ) h6 (x1,y1, w1 ), … (xn,yn, wn ) h7 (x1,y1, w1 ), … (xn,yn, wn ) h8 (x1,y1, w1 ), … (xn,yn, wn ) h9 (x1,y1, w1 ), … (xn,yn, wn ) hT (x1,y1, w1 ), … (xn,yn, wn )

16. Margins view Project Prediction = + - + + + + + + - - - - - - - w Correct Mistakes Margin Cumulative # examples Mistakes Correct Margin =

17. Adaboost et al. Loss Correct Margin Mistakes Brownboost Logitboost Adaboost = 0-1 loss

20. Alternating Trees Joint work with Llew Mason

21. Decision Trees X>3 Y>5 -1 +1 -1 +1 -1 -1 no yes yes no X Y 3 5

22. Decision tree as a sum -0.2 -0.2 X Y Y>5 +0.2 -0.3 yes no X>3 -0.1 no yes +0.1 +0.1 -0.1 +0.2 -0.3 +1 -1 -1 sign

23. An alternating decision tree -0.2 +0.7 X Y +0.1 -0.1 +0.2 -0.3 sign Y>5 +0.2 -0.3 yes no X>3 -0.1 no yes +0.1 Y<1 0.0 no yes +0.7 +1 -1 -1 +1

25. Adtree for Cleveland heart-disease diagnostics problem

26. Cross-validated accuracy 0.8% 16.5% 16 Boost Stumps 0.5% 20.2% 446 C5.0 + boosting 0.5% 27.2% 27 C5.0 0.6% 17.0% 6 ADtree Test error variance Average test error Number of splits Learning algorithm

27. Boosting and over-fitting

28. Curious phenomenon Boosting decision trees Using < 10,000 training examples we fit > 2,000,000 parameters

29. Explanation using margins Margin 0-1 loss

30. Explanation using margins Margin 0-1 loss No examples with small margins!!

31. Experimental Evidence

32. Theorem For any convex combination and any threshold No dependence on number of weak rules that are combined!!! Schapire, Freund, Bartlett & Lee Annals of stat. 98 Probability of mistake Fraction of training example with small margin Size of training sample VC dimension of weak rules

33. Suggested optimization problem Margin

34. Idea of Proof

35. Applications

37. Academic research % test error rates 7.4 2.95 3.5 11.8 16.5 Boosting ~40% ~60% 74% 46% 39% Error reduction 11.3, 12.1, 13.4 Reuters 8 Reuters 4 Letter Promoters Cleveland Database 5.8, 6.0, 9.8 13.8 (DT) 22.0 (DT) 27.2 (DT) Other

40. Weak rules generated by “boostexter” Calling card Collect call Third party Weak Rule Category Word occurs Word does not occur

44. Alternating tree for “buizocity”

45. Alternating Tree (Detail)

46. Precision/recall graphs Score Accuracy