Upcoming SlideShare
×

# Reading Efron's 1979 paper on bootstrap

5,479 views

Published on

seminar talk of Marco Brandi, Nov. 26, 2012

4 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
5,479
On SlideShare
0
From Embeds
0
Number of Embeds
2,484
Actions
Shares
0
41
0
Likes
4
Embeds 0
No embeds

No notes for slide

### Reading Efron's 1979 paper on bootstrap

1. 1. INTRODUCTION DESCRIPTION OF METHODSBOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP Bootstrap Methods: Another Look at the Jackknife Marco Brandi TSI-EuroBayes Student University Paris Dauphine26 November 2012 / Reading Seminar on Classics Marco Brandi Bootstrap Methods: Another Look at the Jackknife
2. 2. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP "To pull oneself up by one is bootstrap"Rudolph Erich Raspe Marco Brandi Bootstrap Methods: Another Look at the Jackknife
3. 3. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOUTLINE 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
4. 4. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
5. 5. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPRESENTING THE PROBLEM X = (X1 , . . . , Xn ) Xi ∼ F with F completely unspeciﬁed Marco Brandi Bootstrap Methods: Another Look at the Jackknife
6. 6. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPRESENTING THE PROBLEM X = (X1 , . . . , Xn ) Xi ∼ F with F completely unspeciﬁed GOAL ⇓ Given R(X, F ) estimate R on the basis of x = (x1 , . . . , xn ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
7. 7. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINTRODUCTION JACKKNIFE METHOD θ(F ) parameter of interest and t(X) its estimator R(X, F ) = t(X) − θ(F ) ˆ t(X)−Bias(t)−θ(F ) R(X, F ) = ˆ(t))1/2 (Var ˆ ˆ Bias(t) and Var (t) are obtained recomputing t(·) n times , each time removing one component of X Marco Brandi Bootstrap Methods: Another Look at the Jackknife
8. 8. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
9. 9. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
10. 10. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
11. 11. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n X∗ boostrap sample Marco Brandi Bootstrap Methods: Another Look at the Jackknife
12. 12. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n X∗ boostrap sample R∗ ˆ = R(X∗ , F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
13. 13. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
14. 14. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) ˆ Xi∗ = 1 x = θ(F ) ¯ Xi∗ =0 1−x ¯ ⇓ ˆ ¯ R ∗ = R(X∗ , F ) = X ∗ − x ¯ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
15. 15. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) ˆ Xi∗ = 1 x = θ(F ) ¯ Xi∗ =0 1−x ¯ ⇓ ˆ ¯ R ∗ = R(X∗ , F ) = X ∗ − x ¯ ¯ E∗ (X ∗ − x ) = 0 ¯ ¯ Var∗ (X ∗ − x ) = x (1 − x )/n ¯ ¯ ¯ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
16. 16. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPROBLEM The complexity on the bootstrap procedure is to calculate the bootstrap distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
17. 17. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPROBLEM The complexity on the bootstrap procedure is to calculate the bootstrap distribution ⇓ 3 methods of calculation are possible Marco Brandi Bootstrap Methods: Another Look at the Jackknife
18. 18. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
19. 19. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
20. 20. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMethod 1 Direct theoretical calculation Marco Brandi Bootstrap Methods: Another Look at the Jackknife
21. 21. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F Marco Brandi Bootstrap Methods: Another Look at the Jackknife
22. 22. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
23. 23. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
24. 24. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1 R(X, F ) = t(X) − θ(F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
25. 25. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
26. 26. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
27. 27. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ R∗ ˆ = R(X∗ , F ) = X(m) − x(m) ∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
28. 28. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ R∗ ˆ = R(X∗ , F ) = X(m) − x(m) ∗ l −1 Pr∗ {R ∗ = x(l) − x(m) } =Pr {Bin(n, ) ≤ m − 1}− n (1) l −Pr {Bin(n, ) ≤ m − 1} n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
29. 29. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(1) for n = 15 and m = 8 l 2 or 14 3 or 13 4 or 12 5 or 11 6 or 10 7 or 9 8 (1) .0003 .0040 .0212 .0627 .1249 .1832 .2073 15 Use E∗ (R ∗ )2 = l=1 [x(l) − x(8) ]2 Pr∗ R ∗ = x(l) − x(8) as an estimate of EF R 2 = EF [t(X) − θ(F )]2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
30. 30. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(2) Results for bootstrap limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
31. 31. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(2) Results for bootstrap limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ) Results for the standard jackknife 2 limn→∞ nVarˆ(R) = (1/4f 2 (θ)) χ2 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
32. 32. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
33. 33. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMETHOD 2 - MONTE CARLO APPROXIMATION Repeat X∗ B times x∗1 , x∗2 , . . . , x∗B ˆ ˆ ˆ R(x∗1 , F ), R(x∗2 , F ), . . . , R(x∗B , F ) is taken as an approximation of the boostrap distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
34. 34. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
35. 35. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] 0.8 Density 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
36. 36. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
37. 37. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density mean = 1.9341 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
38. 38. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density mean = 1.9341 0.4 se = 0.382 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
39. 39. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
40. 40. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] 0.4 Density 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
41. 41. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
42. 42. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density mean = 2.191 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
43. 43. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density mean = 2.191 se = 0.649 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
44. 44. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPR CODE ## s i m u l a t i o n poisson data s e t . seed ( 5 9 2 ) x= r p o i s ( 1 5 , lambda =2) B=10000 ## c r e a t e t h e b o o t s t r a p f u n c t i o n b o o t s t r a p <− f u n c t i o n ( data , nboot , t h e t a , . . . ) { z <− l i s t ( ) datab <− m a t r i x ( sample ( data , s i z e = l e n g t h ( data ) ∗nboot , r e p l a c e =TRUE) , nrow=nboot ) e s t b <− a p p l y ( datab , 1 , t h e t a , . . . ) e s t <− t h e t a ( data , . . . ) z\$ e s t <− e s t z\$ d i s t n <− e s t b z\$ b i a s <− mean ( e s t b)−e s t z\$se <− sd ( e s t b ) z } ## E s t i m a t i n g t h e mean X1= b o o t s t r a p ( x , B , t h e t a =mean ) h i s t ( X1\$ d i s t n , main= " Histogram o f b o o t s t r a p mean " , prob=T , x l a b = " B o o t s t r a p e s t i m a t i o n o f mean " ) mean ( X1\$ d i s t n ) X1\$se Marco Brandi Bootstrap Methods: Another Look at the Jackknife
45. 45. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
46. 46. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMETHOD 3 - RELATIONSHIP WITH THE JACKKNIFE Pi∗ = Ni∗ /n P∗ = (P1 , P2 , . . . , Pn ) ∗ ∗ ∗ E∗ P∗ = e/n Cov∗ P∗ = I/n2 − e e/n3 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
47. 47. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
48. 48. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n 1 R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n) 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
49. 49. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n 1 R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n) 2 . . .  . .  . . . . . . .  ∂R(P∗ )  . . ∂ 2 R(P∗ ) U =  ∂P ∗  V = . . . .   i  ∂Pi∗ ∂Pj∗ .   . . . . . . . . P∗ =e/n . . . P∗ =e/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
50. 50. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPDERIVATION OF BOOTSTRAP EXPECTATION ANDVARIANCE P∗ R(P∗ ) = R n ∗ i=1 Pi eU = 0 eV = −nU eVe = 0 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
51. 51. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPDERIVATION OF BOOTSTRAP EXPECTATION ANDVARIANCE P∗ R(P∗ ) = R n ∗ i=1 Pi eU = 0 eV = −nU eVe = 0 1 1 ¯ E∗ R(P∗ ) = R(e/n) + tr V I/n2 − e e/n3 = R(e/n) + V 2 2n n Var∗ R(P∗ ) = U I/n2 − e e/n3 U = Ui2 /n2 i=1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
52. 52. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS ˆ BiasF θ(F ) ≈ 1 ¯ 2n V ˆ n 2 2 VarF θ(F ) ≈ i=1 Ui /n The results agree with those given by Jaeckel’s inﬁnitesimal jackknife Marco Brandi Bootstrap Methods: Another Look at the Jackknife
53. 53. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
54. 54. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPREGRESSION MODELS Xi = gi (β) + i i ∼F i = 1, . . . , n Having observed X = x we compute the estimate of β n 2 ˆ β = minβ ˆ xi − gi β i=1 ˆ 1 ˆ F : mass at ˆi = xi − gi β n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
55. 55. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP SAMPLE Xi∗ = gi β + ˆ ∗ ∗ ˆ ∼F i i n 2 ˆ β ∗ : minβ xi∗ − gi β ˆ i=1 β ∗1 , β ∗2 , β ∗3 , . . . , β ∗B ˆ ˆ ˆ ˆ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
56. 56. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPLINEAR MODEL gi (β) = ci β CC=G β = G−1 C X has mean β and covariance matrix σF G−1 ˆ 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
57. 57. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPLINEAR MODEL gi (β) = ci β CC=G β = G−1 C X has mean β and covariance matrix σF G−1 ˆ 2 ˆ β ∗ = G−1 C X∗ has boostrap mean and variance E∗ β ∗ = β ˆ ˆ Cov∗ β ∗ = σ 2 G−1 ˆ ˆ 2 n ˆ where σ 2 = ˆ i=1 xi − g β /n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
58. 58. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPJACKKNIFE IN LINEAR REGRESSION Applying the inﬁnitesimal jackknife in a linear regression model, Hinkley derive the approximation of n Cov β ≈ G−1 ˆ ci ci ˆ2 G−1 i i=1 Jackknife methods ignore that the errors i are assumed to have the same distribution for every value of i Marco Brandi Bootstrap Methods: Another Look at the Jackknife
59. 59. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
60. 60. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Marco Brandi Bootstrap Methods: Another Look at the Jackknife
61. 61. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Each BB replications generates a posterior probability for each xi Marco Brandi Bootstrap Methods: Another Look at the Jackknife
62. 62. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Each BB replications generates a posterior probability for each xi 1 The posterior probability of each xi is centered at n but has variability Marco Brandi Bootstrap Methods: Another Look at the Jackknife
63. 63. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
64. 64. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 gl = u(l) − u(l−1) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
65. 65. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 gl = u(l) − u(l−1) Attach the vector (g1 , . . . , gn ) to the data X Marco Brandi Bootstrap Methods: Another Look at the Jackknife
66. 66. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCEPTUAL DIFFERENCE Bayesian Bootstrap Simulates the posterior distribution of the parameter Marco Brandi Bootstrap Methods: Another Look at the Jackknife
67. 67. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCEPTUAL DIFFERENCE Bayesian Bootstrap Simulates the posterior distribution of the parameter Classical Bootstrap Simulates the estimated sampling distribution of a statistic Marco Brandi Bootstrap Methods: Another Look at the Jackknife
68. 68. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB EXAMPLE Dichotomous Example The parameter is θ = Pr {Xi = 1} and let n1 number of Xi = 1 Call P1 the sum of the n1 probabilities assigned to the xi = 1 (g1 , . . . , gn ) ∼ Dirichlet(1, . . . , 1) ⇒ P1 ∼ Beta(n1 , n − n1 ) Note: Beta(n1 , n − n1 ) is the posterior distribution when the prior is P(θ) ∝ [θ(1 − θ)]−1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
69. 69. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
70. 70. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Marco Brandi Bootstrap Methods: Another Look at the Jackknife
71. 71. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Marco Brandi Bootstrap Methods: Another Look at the Jackknife
72. 72. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Is it reasonable to assume a priori independent parameters, constrained only to sum to 1, for these values? Using the gap to simulate the posterior distributions of parameters may no longer work Marco Brandi Bootstrap Methods: Another Look at the Jackknife
73. 73. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Is it reasonable to assume a priori independent parameters, constrained only to sum to 1, for these values? Using the gap to simulate the posterior distributions of parameters may no longer work so.. BB and bootstrap cannot avoid the sensitivity of inference to model assumptions Marco Brandi Bootstrap Methods: Another Look at the Jackknife
74. 74. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCLUSION Knowledge of the context of a data set may make the incorporation of reasonable model constraints obvious and bootstrap may be useful in particular contexts Marco Brandi Bootstrap Methods: Another Look at the Jackknife
75. 75. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCLUSION Knowledge of the context of a data set may make the incorporation of reasonable model constraints obvious and bootstrap may be useful in particular contexts In general "There are no general data analytic panaceas that allow us to pull ourselves up by our bootstraps" Donald Rubin Marco Brandi Bootstrap Methods: Another Look at the Jackknife
76. 76. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
77. 77. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
78. 78. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n From each subset we draw r samples with replacement of dimension n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
79. 79. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n From each subset we draw r samples with replacement of dimension n Compute for each subset the estimator quality assessment (e.g the bias) indicated with ξ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
80. 80. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB IMAGE Marco Brandi Bootstrap Methods: Another Look at the Jackknife
81. 81. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPFINALLY... if we choose b = n0.6 ad we have a dataset of 1TB, the subsamples contains at most 3981 distinct points and have size at most 4GB Like the bootstrap Share bootstrap’s consistency Automatic : without knowledge of the internals θ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
82. 82. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPFINALLY... if we choose b = n0.6 ad we have a dataset of 1TB, the subsamples contains at most 3981 distinct points and have size at most 4GB Like the bootstrap Share bootstrap’s consistency Automatic : without knowledge of the internals θ Beyond the bootstrap Can explicity control b Generally faster than the bootstrap and requires less total computation Marco Brandi Bootstrap Methods: Another Look at the Jackknife
83. 83. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPReferences I B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, Vol. 7, No. 1, (Jan. 1979), pp. 1-26. D.B. Rubin. The Bayesian Bootstrap. The Annals of Statistics, Vol. 9, No.1, pp. 130-134. M. Jordan. The Big Data Bootstrap. Proceedings of the 29th International Conference on Machine Learning (ICML). Marco Brandi Bootstrap Methods: Another Look at the Jackknife
84. 84. INTRODUCTION DESCRIPTION OF METHODSBOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP THANK YOU FOR YOUR ATTENTION Marco Brandi Bootstrap Methods: Another Look at the Jackknife