Reading Efron's 1979 paper on bootstrap
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Reading Efron's 1979 paper on bootstrap

  • 3,911 views
Uploaded on

seminar talk of Marco Brandi, Nov. 26, 2012

seminar talk of Marco Brandi, Nov. 26, 2012

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,911
On Slideshare
2,212
From Embeds
1,699
Number of Embeds
3

Actions

Shares
Downloads
17
Comments
0
Likes
2

Embeds 1,699

http://xianblog.wordpress.com 1,567
https://xianblog.wordpress.com 129
http://webcache.googleusercontent.com 3

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. INTRODUCTION DESCRIPTION OF METHODSBOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP Bootstrap Methods: Another Look at the Jackknife Marco Brandi TSI-EuroBayes Student University Paris Dauphine26 November 2012 / Reading Seminar on Classics Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 2. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP "To pull oneself up by one is bootstrap"Rudolph Erich Raspe Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 3. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOUTLINE 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 4. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 5. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPRESENTING THE PROBLEM X = (X1 , . . . , Xn ) Xi ∼ F with F completely unspecified Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 6. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPRESENTING THE PROBLEM X = (X1 , . . . , Xn ) Xi ∼ F with F completely unspecified GOAL ⇓ Given R(X, F ) estimate R on the basis of x = (x1 , . . . , xn ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 7. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINTRODUCTION JACKKNIFE METHOD θ(F ) parameter of interest and t(X) its estimator R(X, F ) = t(X) − θ(F ) ˆ t(X)−Bias(t)−θ(F ) R(X, F ) = ˆ(t))1/2 (Var ˆ ˆ Bias(t) and Var (t) are obtained recomputing t(·) n times , each time removing one component of X Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 8. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 9. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 10. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 11. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n X∗ boostrap sample Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 12. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n X∗ boostrap sample R∗ ˆ = R(X∗ , F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 13. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 14. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) ˆ Xi∗ = 1 x = θ(F ) ¯ Xi∗ =0 1−x ¯ ⇓ ˆ ¯ R ∗ = R(X∗ , F ) = X ∗ − x ¯ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 15. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) ˆ Xi∗ = 1 x = θ(F ) ¯ Xi∗ =0 1−x ¯ ⇓ ˆ ¯ R ∗ = R(X∗ , F ) = X ∗ − x ¯ ¯ E∗ (X ∗ − x ) = 0 ¯ ¯ Var∗ (X ∗ − x ) = x (1 − x )/n ¯ ¯ ¯ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 16. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPROBLEM The complexity on the bootstrap procedure is to calculate the bootstrap distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 17. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPROBLEM The complexity on the bootstrap procedure is to calculate the bootstrap distribution ⇓ 3 methods of calculation are possible Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 18. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 19. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 20. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMethod 1 Direct theoretical calculation Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 21. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 22. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 23. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 24. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1 R(X, F ) = t(X) − θ(F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 25. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 26. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 27. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ R∗ ˆ = R(X∗ , F ) = X(m) − x(m) ∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 28. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ R∗ ˆ = R(X∗ , F ) = X(m) − x(m) ∗ l −1 Pr∗ {R ∗ = x(l) − x(m) } =Pr {Bin(n, ) ≤ m − 1}− n (1) l −Pr {Bin(n, ) ≤ m − 1} n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 29. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(1) for n = 15 and m = 8 l 2 or 14 3 or 13 4 or 12 5 or 11 6 or 10 7 or 9 8 (1) .0003 .0040 .0212 .0627 .1249 .1832 .2073 15 Use E∗ (R ∗ )2 = l=1 [x(l) − x(8) ]2 Pr∗ R ∗ = x(l) − x(8) as an estimate of EF R 2 = EF [t(X) − θ(F )]2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 30. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(2) Results for bootstrap limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 31. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(2) Results for bootstrap limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ) Results for the standard jackknife 2 limn→∞ nVarˆ(R) = (1/4f 2 (θ)) χ2 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 32. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 33. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMETHOD 2 - MONTE CARLO APPROXIMATION Repeat X∗ B times x∗1 , x∗2 , . . . , x∗B ˆ ˆ ˆ R(x∗1 , F ), R(x∗2 , F ), . . . , R(x∗B , F ) is taken as an approximation of the boostrap distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 34. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 35. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] 0.8 Density 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 36. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 37. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density mean = 1.9341 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 38. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density mean = 1.9341 0.4 se = 0.382 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 39. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 40. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] 0.4 Density 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 41. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 42. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density mean = 2.191 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 43. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density mean = 2.191 se = 0.649 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 44. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPR CODE ## s i m u l a t i o n poisson data s e t . seed ( 5 9 2 ) x= r p o i s ( 1 5 , lambda =2) B=10000 ## c r e a t e t h e b o o t s t r a p f u n c t i o n b o o t s t r a p <− f u n c t i o n ( data , nboot , t h e t a , . . . ) { z <− l i s t ( ) datab <− m a t r i x ( sample ( data , s i z e = l e n g t h ( data ) ∗nboot , r e p l a c e =TRUE) , nrow=nboot ) e s t b <− a p p l y ( datab , 1 , t h e t a , . . . ) e s t <− t h e t a ( data , . . . ) z$ e s t <− e s t z$ d i s t n <− e s t b z$ b i a s <− mean ( e s t b)−e s t z$se <− sd ( e s t b ) z } ## E s t i m a t i n g t h e mean X1= b o o t s t r a p ( x , B , t h e t a =mean ) h i s t ( X1$ d i s t n , main= " Histogram o f b o o t s t r a p mean " , prob=T , x l a b = " B o o t s t r a p e s t i m a t i o n o f mean " ) mean ( X1$ d i s t n ) X1$se Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 45. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 46. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMETHOD 3 - RELATIONSHIP WITH THE JACKKNIFE Pi∗ = Ni∗ /n P∗ = (P1 , P2 , . . . , Pn ) ∗ ∗ ∗ E∗ P∗ = e/n Cov∗ P∗ = I/n2 − e e/n3 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 47. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 48. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n 1 R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n) 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 49. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n 1 R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n) 2 . . .  . .  . . . . . . .  ∂R(P∗ )  . . ∂ 2 R(P∗ ) U =  ∂P ∗  V = . . . .   i  ∂Pi∗ ∂Pj∗ .   . . . . . . . . P∗ =e/n . . . P∗ =e/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 50. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPDERIVATION OF BOOTSTRAP EXPECTATION ANDVARIANCE P∗ R(P∗ ) = R n ∗ i=1 Pi eU = 0 eV = −nU eVe = 0 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 51. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPDERIVATION OF BOOTSTRAP EXPECTATION ANDVARIANCE P∗ R(P∗ ) = R n ∗ i=1 Pi eU = 0 eV = −nU eVe = 0 1 1 ¯ E∗ R(P∗ ) = R(e/n) + tr V I/n2 − e e/n3 = R(e/n) + V 2 2n n Var∗ R(P∗ ) = U I/n2 − e e/n3 U = Ui2 /n2 i=1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 52. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS ˆ BiasF θ(F ) ≈ 1 ¯ 2n V ˆ n 2 2 VarF θ(F ) ≈ i=1 Ui /n The results agree with those given by Jaeckel’s infinitesimal jackknife Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 53. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 54. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPREGRESSION MODELS Xi = gi (β) + i i ∼F i = 1, . . . , n Having observed X = x we compute the estimate of β n 2 ˆ β = minβ ˆ xi − gi β i=1 ˆ 1 ˆ F : mass at ˆi = xi − gi β n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 55. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP SAMPLE Xi∗ = gi β + ˆ ∗ ∗ ˆ ∼F i i n 2 ˆ β ∗ : minβ xi∗ − gi β ˆ i=1 β ∗1 , β ∗2 , β ∗3 , . . . , β ∗B ˆ ˆ ˆ ˆ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 56. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPLINEAR MODEL gi (β) = ci β CC=G β = G−1 C X has mean β and covariance matrix σF G−1 ˆ 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 57. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPLINEAR MODEL gi (β) = ci β CC=G β = G−1 C X has mean β and covariance matrix σF G−1 ˆ 2 ˆ β ∗ = G−1 C X∗ has boostrap mean and variance E∗ β ∗ = β ˆ ˆ Cov∗ β ∗ = σ 2 G−1 ˆ ˆ 2 n ˆ where σ 2 = ˆ i=1 xi − g β /n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 58. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPJACKKNIFE IN LINEAR REGRESSION Applying the infinitesimal jackknife in a linear regression model, Hinkley derive the approximation of n Cov β ≈ G−1 ˆ ci ci ˆ2 G−1 i i=1 Jackknife methods ignore that the errors i are assumed to have the same distribution for every value of i Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 59. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 60. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 61. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Each BB replications generates a posterior probability for each xi Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 62. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Each BB replications generates a posterior probability for each xi 1 The posterior probability of each xi is centered at n but has variability Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 63. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 64. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 gl = u(l) − u(l−1) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 65. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 gl = u(l) − u(l−1) Attach the vector (g1 , . . . , gn ) to the data X Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 66. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCEPTUAL DIFFERENCE Bayesian Bootstrap Simulates the posterior distribution of the parameter Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 67. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCEPTUAL DIFFERENCE Bayesian Bootstrap Simulates the posterior distribution of the parameter Classical Bootstrap Simulates the estimated sampling distribution of a statistic Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 68. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB EXAMPLE Dichotomous Example The parameter is θ = Pr {Xi = 1} and let n1 number of Xi = 1 Call P1 the sum of the n1 probabilities assigned to the xi = 1 (g1 , . . . , gn ) ∼ Dirichlet(1, . . . , 1) ⇒ P1 ∼ Beta(n1 , n − n1 ) Note: Beta(n1 , n − n1 ) is the posterior distribution when the prior is P(θ) ∝ [θ(1 − θ)]−1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 69. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 70. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 71. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 72. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Is it reasonable to assume a priori independent parameters, constrained only to sum to 1, for these values? Using the gap to simulate the posterior distributions of parameters may no longer work Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 73. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Is it reasonable to assume a priori independent parameters, constrained only to sum to 1, for these values? Using the gap to simulate the posterior distributions of parameters may no longer work so.. BB and bootstrap cannot avoid the sensitivity of inference to model assumptions Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 74. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCLUSION Knowledge of the context of a data set may make the incorporation of reasonable model constraints obvious and bootstrap may be useful in particular contexts Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 75. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCLUSION Knowledge of the context of a data set may make the incorporation of reasonable model constraints obvious and bootstrap may be useful in particular contexts In general "There are no general data analytic panaceas that allow us to pull ourselves up by our bootstraps" Donald Rubin Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 76. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 77. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 78. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n From each subset we draw r samples with replacement of dimension n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 79. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n From each subset we draw r samples with replacement of dimension n Compute for each subset the estimator quality assessment (e.g the bias) indicated with ξ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 80. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB IMAGE Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 81. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPFINALLY... if we choose b = n0.6 ad we have a dataset of 1TB, the subsamples contains at most 3981 distinct points and have size at most 4GB Like the bootstrap Share bootstrap’s consistency Automatic : without knowledge of the internals θ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 82. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPFINALLY... if we choose b = n0.6 ad we have a dataset of 1TB, the subsamples contains at most 3981 distinct points and have size at most 4GB Like the bootstrap Share bootstrap’s consistency Automatic : without knowledge of the internals θ Beyond the bootstrap Can explicity control b Generally faster than the bootstrap and requires less total computation Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 83. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPReferences I B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, Vol. 7, No. 1, (Jan. 1979), pp. 1-26. D.B. Rubin. The Bayesian Bootstrap. The Annals of Statistics, Vol. 9, No.1, pp. 130-134. M. Jordan. The Big Data Bootstrap. Proceedings of the 29th International Conference on Machine Learning (ICML). Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  • 84. INTRODUCTION DESCRIPTION OF METHODSBOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP THANK YOU FOR YOUR ATTENTION Marco Brandi Bootstrap Methods: Another Look at the Jackknife