Reading Efron's 1979 paper on bootstrap

5,875 views

Published on

seminar talk of Marco Brandi, Nov. 26, 2012

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,875
On SlideShare
0
From Embeds
0
Number of Embeds
2,689
Actions
Shares
0
Downloads
46
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Reading Efron's 1979 paper on bootstrap

  1. 1. INTRODUCTION DESCRIPTION OF METHODSBOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP Bootstrap Methods: Another Look at the Jackknife Marco Brandi TSI-EuroBayes Student University Paris Dauphine26 November 2012 / Reading Seminar on Classics Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  2. 2. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP "To pull oneself up by one is bootstrap"Rudolph Erich Raspe Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  3. 3. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOUTLINE 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  4. 4. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  5. 5. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPRESENTING THE PROBLEM X = (X1 , . . . , Xn ) Xi ∼ F with F completely unspecified Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  6. 6. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPRESENTING THE PROBLEM X = (X1 , . . . , Xn ) Xi ∼ F with F completely unspecified GOAL ⇓ Given R(X, F ) estimate R on the basis of x = (x1 , . . . , xn ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  7. 7. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINTRODUCTION JACKKNIFE METHOD θ(F ) parameter of interest and t(X) its estimator R(X, F ) = t(X) − θ(F ) ˆ t(X)−Bias(t)−θ(F ) R(X, F ) = ˆ(t))1/2 (Var ˆ ˆ Bias(t) and Var (t) are obtained recomputing t(·) n times , each time removing one component of X Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  8. 8. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  9. 9. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  10. 10. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  11. 11. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n X∗ boostrap sample Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  12. 12. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP METHOD BOOTSTRAP METHOD at x1 , x2 , . . . , xn put mass 1/n ˆ F is the sample probability distribution Xi∗ = xi∗ ˆ Xi∗ ∼ F i = 1, . . . , n X∗ boostrap sample R∗ ˆ = R(X∗ , F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  13. 13. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  14. 14. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) ˆ Xi∗ = 1 x = θ(F ) ¯ Xi∗ =0 1−x ¯ ⇓ ˆ ¯ R ∗ = R(X∗ , F ) = X ∗ − x ¯ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  15. 15. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPSIMPLE EXAMPLE Dichotomous Example θ(F ) = Pr {X = 1} ¯ R(X, F ) = X − θ(F ) ˆ Xi∗ = 1 x = θ(F ) ¯ Xi∗ =0 1−x ¯ ⇓ ˆ ¯ R ∗ = R(X∗ , F ) = X ∗ − x ¯ ¯ E∗ (X ∗ − x ) = 0 ¯ ¯ Var∗ (X ∗ − x ) = x (1 − x )/n ¯ ¯ ¯ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  16. 16. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPROBLEM The complexity on the bootstrap procedure is to calculate the bootstrap distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  17. 17. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPPROBLEM The complexity on the bootstrap procedure is to calculate the bootstrap distribution ⇓ 3 methods of calculation are possible Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  18. 18. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  19. 19. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  20. 20. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMethod 1 Direct theoretical calculation Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  21. 21. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  22. 22. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  23. 23. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  24. 24. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 1ST STEP Initializing the procedure θ(F ) indicate the median of F t(X) = X(m) X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1 R(X, F ) = t(X) − θ(F ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  25. 25. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  26. 26. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  27. 27. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ R∗ ˆ = R(X∗ , F ) = X(m) − x(m) ∗ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  28. 28. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPESTIMATING THE MEDIAN 2ST STEP Formalazing the procedure X∗ = x∗ Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn ) ∗ ∗ ∗ R∗ ˆ = R(X∗ , F ) = X(m) − x(m) ∗ l −1 Pr∗ {R ∗ = x(l) − x(m) } =Pr {Bin(n, ) ≤ m − 1}− n (1) l −Pr {Bin(n, ) ≤ m − 1} n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  29. 29. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(1) for n = 15 and m = 8 l 2 or 14 3 or 13 4 or 12 5 or 11 6 or 10 7 or 9 8 (1) .0003 .0040 .0212 .0627 .1249 .1832 .2073 15 Use E∗ (R ∗ )2 = l=1 [x(l) − x(8) ]2 Pr∗ R ∗ = x(l) − x(8) as an estimate of EF R 2 = EF [t(X) − θ(F )]2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  30. 30. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(2) Results for bootstrap limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  31. 31. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS(2) Results for bootstrap limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ) Results for the standard jackknife 2 limn→∞ nVarˆ(R) = (1/4f 2 (θ)) χ2 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  32. 32. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  33. 33. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMETHOD 2 - MONTE CARLO APPROXIMATION Repeat X∗ B times x∗1 , x∗2 , . . . , x∗B ˆ ˆ ˆ R(x∗1 , F ), R(x∗2 , F ), . . . , R(x∗B , F ) is taken as an approximation of the boostrap distribution Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  34. 34. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  35. 35. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] 0.8 Density 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  36. 36. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  37. 37. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density mean = 1.9341 0.4 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  38. 38. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(1) Xi ∼ Pois(2) i = 1, . . . , 15 Histogram of bootstrap mean t(X) = E [X] B = 10000 0.8 n◦ of bootstrap samples Density mean = 1.9341 0.4 se = 0.382 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Bootstrap estimation of mean Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  39. 39. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  40. 40. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] 0.4 Density 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  41. 41. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  42. 42. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density mean = 2.191 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  43. 43. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPEXAMPLE(2) Histogram of bootstrap variance t(X) = V [X] B = 10000 n◦ of bootstrap samples 0.4 Density mean = 2.191 se = 0.649 0.2 0.0 0 1 2 3 4 5 Bootstrap estimation of variance Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  44. 44. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPR CODE ## s i m u l a t i o n poisson data s e t . seed ( 5 9 2 ) x= r p o i s ( 1 5 , lambda =2) B=10000 ## c r e a t e t h e b o o t s t r a p f u n c t i o n b o o t s t r a p <− f u n c t i o n ( data , nboot , t h e t a , . . . ) { z <− l i s t ( ) datab <− m a t r i x ( sample ( data , s i z e = l e n g t h ( data ) ∗nboot , r e p l a c e =TRUE) , nrow=nboot ) e s t b <− a p p l y ( datab , 1 , t h e t a , . . . ) e s t <− t h e t a ( data , . . . ) z$ e s t <− e s t z$ d i s t n <− e s t b z$ b i a s <− mean ( e s t b)−e s t z$se <− sd ( e s t b ) z } ## E s t i m a t i n g t h e mean X1= b o o t s t r a p ( x , B , t h e t a =mean ) h i s t ( X1$ d i s t n , main= " Histogram o f b o o t s t r a p mean " , prob=T , x l a b = " B o o t s t r a p e s t i m a t i o n o f mean " ) mean ( X1$ d i s t n ) X1$se Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  45. 45. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  46. 46. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPMETHOD 3 - RELATIONSHIP WITH THE JACKKNIFE Pi∗ = Ni∗ /n P∗ = (P1 , P2 , . . . , Pn ) ∗ ∗ ∗ E∗ P∗ = e/n Cov∗ P∗ = I/n2 − e e/n3 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  47. 47. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  48. 48. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n 1 R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n) 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  49. 49. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPUSING TAYLOR EXPANSION ˆ R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n 1 R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n) 2 . . .  . .  . . . . . . .  ∂R(P∗ )  . . ∂ 2 R(P∗ ) U =  ∂P ∗  V = . . . .   i  ∂Pi∗ ∂Pj∗ .   . . . . . . . . P∗ =e/n . . . P∗ =e/n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  50. 50. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPDERIVATION OF BOOTSTRAP EXPECTATION ANDVARIANCE P∗ R(P∗ ) = R n ∗ i=1 Pi eU = 0 eV = −nU eVe = 0 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  51. 51. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPDERIVATION OF BOOTSTRAP EXPECTATION ANDVARIANCE P∗ R(P∗ ) = R n ∗ i=1 Pi eU = 0 eV = −nU eVe = 0 1 1 ¯ E∗ R(P∗ ) = R(e/n) + tr V I/n2 − e e/n3 = R(e/n) + V 2 2n n Var∗ R(P∗ ) = U I/n2 − e e/n3 U = Ui2 /n2 i=1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  52. 52. INTRODUCTION DESCRIPTION OF METHODS METHOD 1 BOOTSTRAP IN REGRESSION MODELS METHOD 2 BAYESIAN BOOTSTRAP METHOD 3 DISCUSSION BAG OF LITTLE BOOTSTRAPRESULTS ˆ BiasF θ(F ) ≈ 1 ¯ 2n V ˆ n 2 2 VarF θ(F ) ≈ i=1 Ui /n The results agree with those given by Jaeckel’s infinitesimal jackknife Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  53. 53. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  54. 54. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPREGRESSION MODELS Xi = gi (β) + i i ∼F i = 1, . . . , n Having observed X = x we compute the estimate of β n 2 ˆ β = minβ ˆ xi − gi β i=1 ˆ 1 ˆ F : mass at ˆi = xi − gi β n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  55. 55. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBOOTSTRAP SAMPLE Xi∗ = gi β + ˆ ∗ ∗ ˆ ∼F i i n 2 ˆ β ∗ : minβ xi∗ − gi β ˆ i=1 β ∗1 , β ∗2 , β ∗3 , . . . , β ∗B ˆ ˆ ˆ ˆ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  56. 56. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPLINEAR MODEL gi (β) = ci β CC=G β = G−1 C X has mean β and covariance matrix σF G−1 ˆ 2 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  57. 57. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPLINEAR MODEL gi (β) = ci β CC=G β = G−1 C X has mean β and covariance matrix σF G−1 ˆ 2 ˆ β ∗ = G−1 C X∗ has boostrap mean and variance E∗ β ∗ = β ˆ ˆ Cov∗ β ∗ = σ 2 G−1 ˆ ˆ 2 n ˆ where σ 2 = ˆ i=1 xi − g β /n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  58. 58. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPJACKKNIFE IN LINEAR REGRESSION Applying the infinitesimal jackknife in a linear regression model, Hinkley derive the approximation of n Cov β ≈ G−1 ˆ ci ci ˆ2 G−1 i i=1 Jackknife methods ignore that the errors i are assumed to have the same distribution for every value of i Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  59. 59. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  60. 60. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  61. 61. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Each BB replications generates a posterior probability for each xi Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  62. 62. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPDEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981) Bayesian Bootstrap In bootstrap we consider sample cdf is population cdf Each BB replications generates a posterior probability for each xi 1 The posterior probability of each xi is centered at n but has variability Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  63. 63. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  64. 64. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 gl = u(l) − u(l−1) Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  65. 65. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB REPLICATION BB replication (n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1 gl = u(l) − u(l−1) Attach the vector (g1 , . . . , gn ) to the data X Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  66. 66. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCEPTUAL DIFFERENCE Bayesian Bootstrap Simulates the posterior distribution of the parameter Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  67. 67. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCEPTUAL DIFFERENCE Bayesian Bootstrap Simulates the posterior distribution of the parameter Classical Bootstrap Simulates the estimated sampling distribution of a statistic Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  68. 68. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBB EXAMPLE Dichotomous Example The parameter is θ = Pr {Xi = 1} and let n1 number of Xi = 1 Call P1 the sum of the n1 probabilities assigned to the xi = 1 (g1 , . . . , gn ) ∼ Dirichlet(1, . . . , 1) ⇒ P1 ∼ Beta(n1 , n − n1 ) Note: Beta(n1 , n − n1 ) is the posterior distribution when the prior is P(θ) ∝ [θ(1 − θ)]−1 Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  69. 69. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  70. 70. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  71. 71. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  72. 72. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Is it reasonable to assume a priori independent parameters, constrained only to sum to 1, for these values? Using the gap to simulate the posterior distributions of parameters may no longer work Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  73. 73. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPINFERENCES PROBLEMS Is it possible that all the values of X have been observed? Is it reasonable to assume a priori independent parameters, constrained only to sum to 1, for these values? Using the gap to simulate the posterior distributions of parameters may no longer work so.. BB and bootstrap cannot avoid the sensitivity of inference to model assumptions Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  74. 74. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCLUSION Knowledge of the context of a data set may make the incorporation of reasonable model constraints obvious and bootstrap may be useful in particular contexts Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  75. 75. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPCONCLUSION Knowledge of the context of a data set may make the incorporation of reasonable model constraints obvious and bootstrap may be useful in particular contexts In general "There are no general data analytic panaceas that allow us to pull ourselves up by our bootstraps" Donald Rubin Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  76. 76. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPOutline 1 INTRODUCTION 2 DESCRIPTION OF METHODS METHOD 1 METHOD 2 METHOD 3 3 BOOTSTRAP IN REGRESSION MODELS 4 BAYESIAN BOOTSTRAP 5 DISCUSSION 6 BAG OF LITTLE BOOTSTRAP Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  77. 77. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  78. 78. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n From each subset we draw r samples with replacement of dimension n Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  79. 79. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB (M. Jordan 2012) When n gets large computational cost is large Expected numbers of distinct points in a resample is ∼ 0.632n BLB Procedure Divide the dataset in s subset of dimension b, with b < n From each subset we draw r samples with replacement of dimension n Compute for each subset the estimator quality assessment (e.g the bias) indicated with ξ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  80. 80. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPBLB IMAGE Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  81. 81. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPFINALLY... if we choose b = n0.6 ad we have a dataset of 1TB, the subsamples contains at most 3981 distinct points and have size at most 4GB Like the bootstrap Share bootstrap’s consistency Automatic : without knowledge of the internals θ Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  82. 82. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPFINALLY... if we choose b = n0.6 ad we have a dataset of 1TB, the subsamples contains at most 3981 distinct points and have size at most 4GB Like the bootstrap Share bootstrap’s consistency Automatic : without knowledge of the internals θ Beyond the bootstrap Can explicity control b Generally faster than the bootstrap and requires less total computation Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  83. 83. INTRODUCTION DESCRIPTION OF METHODS BOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAPReferences I B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, Vol. 7, No. 1, (Jan. 1979), pp. 1-26. D.B. Rubin. The Bayesian Bootstrap. The Annals of Statistics, Vol. 9, No.1, pp. 130-134. M. Jordan. The Big Data Bootstrap. Proceedings of the 29th International Conference on Machine Learning (ICML). Marco Brandi Bootstrap Methods: Another Look at the Jackknife
  84. 84. INTRODUCTION DESCRIPTION OF METHODSBOOTSTRAP IN REGRESSION MODELS BAYESIAN BOOTSTRAP DISCUSSION BAG OF LITTLE BOOTSTRAP THANK YOU FOR YOUR ATTENTION Marco Brandi Bootstrap Methods: Another Look at the Jackknife

×