Upcoming SlideShare
×

# Introduction to Bag of Little Bootstrap

3,010 views

Published on

Reading group presentation on Bag of Little Bootstrap (BLB)

6 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
3,010
On SlideShare
0
From Embeds
0
Number of Embeds
108
Actions
Shares
0
64
0
Likes
6
Embeds 0
No embeds

No notes for slide

### Introduction to Bag of Little Bootstrap

1. 1. ML-IR Discussion: Bag of Little Bootstrap (BLB)
2. 2. Recap: - Recap - Why bootstrap - What is bootstrap - Bag of Little Bootstrap (BLB) - Guarantees - Examples
3. 3. Recap: Population Our Sample
4. 4. Estimate the median!
5. 5. Estimate the median!
6. 6. Asymptotic Approach Theory has it:
7. 7. Asymptotic Approach Theory has it: ?
8. 8. Asymptotic Approach 95% Confidence Interval
9. 9. Problems with the asymptotic Approach: - Density “f” is hard to estimate - Sample size demand is much larger than the mean for Central Limit theorem to kick in - True median unknown
10. 10. Solution: When theory is too hard… Let’s empirically estimate theoretical truth!
11. 11. Empirical Approach: Ideal Population Sample Over and Over again!
12. 12. Empirical Approach: Ideal Population Sample Over and Over again! Median Est 1 Median Est 2
13. 13. Empirical Approach: Ideal
14. 14. Empirical Approach: Ideal 95% of sample medians
15. 15. Similar Enough? Population Our Sample
16. 16. Empirical Approach: Bootstrap Efron Tibshirani (1993) Our Sample Draw with replacement n samples Median Est* 1 Median Est* 2
17. 17. Empirical Approach: Bootstrap
18. 18. Empirical Approach: Bootstrap 95% of sample medians
19. 19. Empirical Approach: Bootstrap Used for: - Bias estimation - Variance - Confidence intervals Main benefits: - Automatic - Flexible - Fast convergence (Hall, 1992)
20. 20. Key: There are 3 distributions Population
21. 21. Key: There are 3 distributions Population Approximate distribution Actual Sample
22. 22. Key: There are 3 distributions Population Approximate distribution Actual Sample Approximate distribution Bootstrap Samples
23. 23. Key: There are 3 distributions Population Approximate distribution Actual Sample Approximate distribution Bootstrap Samples Approximate the approximation - Is there bias? - What’s the variance? - etc.
24. 24. No free meals: - Bootstrapping requires re-sampling the entire population B times - Each sample is size n - Sampling m < n will violate the sample size properties - Original sample size cannot be too small - “Pre-asymptopia” cases
25. 25. Hope - Resample expects .632n unique samples Sample less – m out of n bootstrap is possible with analytical adjustments. (Bickel 1997)
26. 26. Hope - Resample expects .632n unique samples Sample less – m out of n bootstrap is possible with analytical adjustments. (Bickel 1997) Intuition: Need less than all n values for each bootstrap.
27. 27. Hope - Resample expects .632n unique samples Sample less – m out of n bootstrap is possible with analytical adjustments. (Bickel 1997) Intuition: Need less than all n values for each bootstrap. Problem: - Analytical adjustment is not as automatic as desirable - m out of n bootstrap is sensitive to choices of m
28. 28. Bag of Little Bootstrap - Sample without replacement the sample s times into sizes of b
29. 29. Bag of Little Bootstrap - Sample without replacement the sample s times into sizes of b - Resample each until sample size is n, r times.
30. 30. Bag of Little Bootstrap - Med 1 Med r Sample without replacement the sample s times into sizes of b - Resample each until sample size is n, r times. - Compute the median for each
31. 31. Bag of Little Bootstrap - Med 1 Med r Sample without replacement the sample s times into sizes of b - Resample each until sample size is n, r times. - Compute the median for each - Compute the confidence interval for each
32. 32. Bag of Little Bootstrap - Med 1 Med r Sample without replacement the sample s times into sizes of b - Resample each until sample size is n, r times. - Compute the median for each - Compute the confidence interval for each
33. 33. Bag of Little Bootstrap - Med 1 Med r - Sample without replacement the sample s times into sizes of b - Resample each until sample size is n, r times. - Compute the median for each - Compute the confidence interval for each Take average of each upper and lower point for the confidence interval
34. 34. Bag of Little Bootstrap Klein et al. 2012 Computational Gains: - Each sample only has b unique values! - Can sample a b-dimensional multinomial with n trials. - Scales in b instead of n - Easily parallelizable
35. 35. Bag of Little Bootstrap Klein et al. 2012 Computational Gains: - Each sample only has b unique values! - Can sample a b-dimensional multinomial with n trials. - Scales in b instead of n - Easily parallelizable If b=n^(0.6), a dataset of size 1TB: - Bootstrap storage demands ~ 632GB - BLB storage demands ~ 4GB
36. 36. Bag of Little Bootstrap Theoretical guarantees: - Consistency - Higher order correctness - Fast convergence rate (same as bootstrap)
37. 37. Performance b = n^(gamma), 0.5<= gamma <=1 These choices of gamma ensures bootstrap convergence rates.
38. 38. Performance b = n^(gamma), 0.5<= gamma <=1 These choices of gamma ensures bootstrap convergence rates. Relative error of confidence interval width of logistic regression coefficients (Klein et al. 2012)
39. 39. Performance b = n^(gamma), 0.5<= gamma <=1 These choices of gamma ensures bootstrap convergence rates. Relative error of confidence interval width of logistic regression coefficients (Klein et al. 2012) Gamma residuals t-distr residuals
40. 40. Performance vs Time
41. 41. Selecting Hyperparameters • b, the number of unique samples for each little bootstrap • s, the number of size b samples w/o replacement • r, the number of multinomials to draw
42. 42. Selecting Hyperparameters • b, the number of unique samples for each little bootstrap • s, the number of size b samples w/o replacement • r, the number of multinomials to draw b: the larger the better s, r: adaptively increase this until a convergence has been reached. (Median doesn’t change)
43. 43. Bag of Little Bootstrap Main benefits: - Computationally friendly - Maintains most statistical properties of bootstrap - Flexibility - More robust to choice of b than older methods
44. 44. Reference • Efron, Tibshirani (1993) An Introduction to the Bootstrap • Kleiner et al. (2012) A Scalable Bootstrap for Massive Data Thanks!