Your SlideShare is downloading. ×
0
(Approximate) Bayesian Computation as a new        empirical Bayes (something)?                  Christian P. Robert      ...
Advertisment   MCMSki IV to be held in Chamonix Mt Blanc, France, from   Monday, Jan. 6 to Wed., Jan. 8, 2014   All aspect...
OutlineUnavailable likelihoodsABC methodsABC as an inference machineABCelConclusion and perspectives
Intractable likelihood   Case of a well-defined statistical model where the likelihood   function                         (...
Intractable likelihood   Case of a well-defined statistical model where the likelihood   function                         (...
The abc alternative   Approximations to the original B problem       Degrading the precision down to a tolerance ε       R...
The abc alternative   Approximations to the original B problem       Degrading the precision down to a tolerance ε       R...
The abc alternative   Approximations to the original B problem       Degrading the precision down to a tolerance ε       R...
Different worries about abc   Impact on B inference       a mere computational issue (that will eventually end up being    ...
Different worries about abc   Impact on B inference       a mere computational issue (that will eventually end up being    ...
Different worries about abc   Impact on B inference       a mere computational issue (that will eventually end up being    ...
Econom’ections   Similar exploration of simulation-based and approximation   techniques in Econometrics       Simulated me...
Econom’ections   Similar exploration of simulation-based and approximation   techniques in Econometrics       Simulated me...
Indirect inference                                                 ^   Minimise [in θ] a distance between estimators β bas...
Indirect inference (PML vs. PSE)   Example of the pseudo-maximum-likelihood (PML)                ^                β(y) = a...
Indirect inference (PML vs. PSE)   Example of the pseudo-score-estimator (PSE)                                            ...
Consistent indirect inference           “...in order to get a unique solution the dimension of       the auxiliary paramet...
Consistent indirect inference           “...in order to get a unique solution the dimension of       the auxiliary paramet...
Consistent indirect inference           “...in order to get a unique solution the dimension of       the auxiliary paramet...
Approximate Bayesian computation Unavailable likelihoods ABC methods   Genesis of ABC   ABC basics   Advances and interpre...
Genetic background of ABC    skip genetics   ABC is a recent computational technique that only requires being   able to sa...
Demo-genetic inference   Each model is characterized by a set of parameters θ that cover   historical (time divergence, ad...
Demo-genetic inference   Each model is characterized by a set of parameters θ that cover   historical (time divergence, ad...
Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium                             ...
Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium                             ...
Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium                             ...
Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium                             ...
Much more interesting models. . .       several independent locus       Independent gene genealogies and mutations       d...
Intractable likelihood   Missing (too missing!) data structure:                     f (y|θ) =         f (y|G , θ)f (G |θ)d...
Intractable likelihood   Missing (too missing!) data structure:                     f (y|θ) =         f (y|G , θ)f (G |θ)d...
A?B?C?    A stands for approximate    [wrong likelihood /    picture]    B stands for Bayesian    C stands for computation...
A?B?C?    A stands for approximate    [wrong likelihood /    picture]    B stands for Bayesian    C stands for computation...
A?B?C?                                                              ESS=155.6                                             ...
How Bayesian is aBc?  Could we turn the resolution into a Bayesian answer?      ideally so (not meaningfull: requires ∞-ly...
Untractable likelihoodBack to stage zero: what can we dowhen a likelihood function f (y|θ) iswell-defined but impossible / ...
Untractable likelihoodBack to stage zero: what can we dowhen a likelihood function f (y|θ) iswell-defined but impossible / ...
ABC methodology  Bayesian setting: target is π(θ)f (x|θ)  When likelihood f (x|θ) not in closed form, likelihood-free reje...
ABC methodology  Bayesian setting: target is π(θ)f (x|θ)  When likelihood f (x|θ) not in closed form, likelihood-free reje...
ABC methodology  Bayesian setting: target is π(θ)f (x|θ)  When likelihood f (x|θ) not in closed form, likelihood-free reje...
A as A...pproximative   When y is a continuous random variable, strict equality z = y is   replaced with a tolerance zone ...
A as A...pproximative   When y is a continuous random variable, strict equality z = y is   replaced with a tolerance zone ...
ABC algorithm  In most implementations, further degree of A...pproximation:  Algorithm 1 Likelihood-free rejection sampler...
Output  The likelihood-free algorithm samples from the marginal in z of:                                   π(θ)f (z|θ)IA ,...
Output  The likelihood-free algorithm samples from the marginal in z of:                                   π(θ)f (z|θ)IA ,...
Output  The likelihood-free algorithm samples from the marginal in z of:                                   π(θ)f (z|θ)IA ,...
Output  The likelihood-free algorithm samples from the marginal in z of:                                        π(θ)f (z|θ...
Convergence of ABC  What happens when                     → 0?  For B ⊂ Θ, we have               A         f (z|θ)dz      ...
Convergence of ABC  What happens when                     → 0?  For B ⊂ Θ, we have               A         f (z|θ)dz      ...
Convergence (do not attempt!)   ...and the above does not apply to insufficient statistics:   If η(y) is not a sufficient stat...
Convergence (do not attempt!)   ...and the above does not apply to insufficient statistics:   If η(y) is not a sufficient stat...
Convergence (do not attempt!)   ...and the above does not apply to insufficient statistics:   If η(y) is not a sufficient stat...
Convergence (do not attempt!)   ...and the above does not apply to insufficient statistics:   If η(y) is not a sufficient stat...
Comments     Role of distance paramount (because     = 0)     Scaling of components of η(y) is also determinant       matt...
ABC (simul’) advances                         how approximative is ABC?                ABC as knn   Simulating from the pr...
ABC (simul’) advances                         how approximative is ABC?                ABC as knn   Simulating from the pr...
ABC (simul’) advances                         how approximative is ABC?                ABC as knn   Simulating from the pr...
ABC (simul’) advances                         how approximative is ABC?                ABC as knn   Simulating from the pr...
ABC-NP  Better usage of [prior] simulations by adjustement: instead of throwing away θ such that ρ(η(z), η(y)) > , replace...
ABC-NP (regression)   Also found in the subsequent literature, e.g. in        Fearnhead-Prangle (2012)   :   weight direct...
ABC-NP (regression)   Also found in the subsequent literature, e.g. in        Fearnhead-Prangle (2012)   :   weight direct...
ABC-NP (density estimation)   Use of the kernel weights                             Kδ {ρ(η(z(θ)), η(y))}   leads to the N...
ABC-NP (density estimation)   Use of the kernel weights                            Kδ {ρ(η(z(θ)), η(y))}   leads to the NP...
ABC-NP (density estimations)   Other versions incorporating regression adjustments                         i              ...
ABC-NP (density estimations)   Other versions incorporating regression adjustments                         i              ...
ABC-NP (density estimations)   Other versions incorporating regression adjustments                         i              ...
ABC as knn                                    [Biau et al., 2012, arxiv:1207.6461]  Practice of ABC: determine tolerance  ...
ABC as knn                                    [Biau et al., 2012, arxiv:1207.6461]  Practice of ABC: determine tolerance  ...
ABC as knn                                    [Biau et al., 2012, arxiv:1207.6461]  Practice of ABC: determine tolerance  ...
ABC consistency   Provided                 kN / log log N −→ ∞ and kN /N −→ 0    as N → ∞, for almost all s0 (with respect...
ABC consistency   Provided                 kN / log log N −→ ∞ and kN /N −→ 0    as N → ∞, for almost all s0 (with respect...
Rates of convergence   Further assumptions (on target and kernel) allow for precise   (integrated mean square) convergence...
Rates of convergence   Further assumptions (on target and kernel) allow for precise   (integrated mean square) convergence...
ABC inference machine Unavailable likelihoods ABC methods ABC as an inference machine   Error inc.   Exact BC and approxim...
How Bayesian is aBc..?      maybe a convergent method of inference (meaningful?      sufficient? foreign?)      approximatio...
ABCµ  Idea Infer about the error as well as about the parameter:  Use of a joint density                f (θ, |y) ∝ ξ( |y,...
ABCµ  Idea Infer about the error as well as about the parameter:  Use of a joint density                f (θ, |y) ∝ ξ( |y,...
ABCµ  Idea Infer about the error as well as about the parameter:  Use of a joint density                f (θ, |y) ∝ ξ( |y,...
ABCµ details   Multidimensional distances ρk (k = 1, . . . , K ) and errors    k = ρk (ηk (z), ηk (y)), with              ...
ABCµ details   Multidimensional distances ρk (k = 1, . . . , K ) and errors    k = ρk (ηk (z), ηk (y)), with              ...
Wilkinson’s exact BC (not exactly!)   ABC approximation error (i.e. non-zero tolerance) replaced with   exact simulation f...
Wilkinson’s exact BC (not exactly!)   ABC approximation error (i.e. non-zero tolerance) replaced with   exact simulation f...
How exact a BC?  Pros         Pseudo-data from true model and observed data from noisy         model         Interesting p...
How exact a BC?  Pros         Pseudo-data from true model and observed data from noisy         model         Interesting p...
Noisy ABC for HMMs  Specific case of a hidden Markov model                          Xt+1 ∼ Qθ (Xt , ·)                     ...
Noisy ABC for HMMs  Specific case of a hidden Markov model                          Xt+1 ∼ Qθ (Xt , ·)                     ...
Noisy ABC-MLE  Idea: Modify instead the data from the start                        0                      (y1 + ζ1 , . . ....
Consistent noisy ABC-MLE      Degrading the data improves the estimation performances:          Noisy ABC-MLE is asymptoti...
Which summary?  Fundamental difficulty of the choice of the summary statistic when  there is no non-trivial sufficient statist...
Which summary?  Fundamental difficulty of the choice of the summary statistic when  there is no non-trivial sufficient statist...
Which summary for model choice?   Depending on the choice of η(·), the Bayes factor based on this   insufficient statistic, ...
Which summary for model choice?   Depending on the choice of η(·), the Bayes factor based on this   insufficient statistic, ...
Semi-automatic ABC  Fearnhead and Prangle (2012) study ABC and the selection of the  summary statistic in close proximity ...
Summary [of F&P/statistics)      optimality of the posterior expectation                                  E[θ|y]      of t...
Summary [of F&P/statistics)      optimality of the posterior expectation                                  E[θ|y]      of t...
Summary [about summaries]      Choice of summary statistics is paramount for ABC      validation/performances      At best...
Summary [about summaries]      Choice of summary statistics is paramount for ABC      validation/performances      At best...
Summary [about summaries]      Choice of summary statistics is paramount for ABC      validation/performances      At best...
Summary [about summaries]      Choice of summary statistics is paramount for ABC      validation/performances      At best...
Summary [about summaries]      Choice of summary statistics is paramount for ABC      validation/performances      At best...
Empirical likelihood (EL) Unavailable likelihoods ABC methods ABC as an inference machine ABCel   ABC and EL   Composite l...
Empirical likelihood (EL)   Dataset x made of n independent replicates x = (x1 , . . . , xn ) of   some X ∼ F   Generalize...
Empirical likelihood (EL)   Dataset x made of n independent replicates x = (x1 , . . . , xn ) of   some X ∼ F   Generalize...
Convergence of EL [3.4]   Theorem 3.4 Let X , Y1 , . . . , Yn be independent rv’s with common   distribution F0 . For θ ∈ ...
Convergence of EL [3.4]   “...The interesting thing about Theorem 3.4 is what is not there. It                            ...
Raw ABCel sampler   Act as if EL was an exact likelihood                                               [Lazar, 2003]     f...
Raw ABCel sampler   Act as if EL was an exact likelihood                                                      [Lazar, 2003...
Raw ABCel sampler   Act as if EL was an exact likelihood                                                        [Lazar, 20...
Moment condition in population genetics?   EL does not require a fully defined and often complex (hence   debatable) parame...
Moment condition in population genetics?   EL does not require a fully defined and often complex (hence   debatable) parame...
Pairwise composite likelihood   The intra-locus pairwise likelihood                                                       ...
Pairwise likelihood: a simple case                                                      1   Assumptions                   ...
Pairwise likelihood: a simple case                                                      1   Assumptions                   ...
Pairwise likelihood: a simple case                                                      1   Assumptions                   ...
Pairwise likelihood: 2 diverging populations            MRCA                   Then 2 (δ|θ, τ) =                          ...
Pairwise likelihood: 2 diverging populations            MRCA                                   A 2-dim score function     ...
Example: normal posterior            ABCel with two constraints                                  ESS=155.6                ...
Example: normal posterior            ABCel with three constraints                               ESS=300.1                 ...
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
(Approximate) Bayesian computation as a new empirical Bayes (something)?
Upcoming SlideShare
Loading in...5
×

(Approximate) Bayesian computation as a new empirical Bayes (something)?

2,223

Published on

Version of my talk on BC_el given at the workshop on Recent advances in statistical inference, Padova, March 21-23, 2013

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,223
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "(Approximate) Bayesian computation as a new empirical Bayes (something)?"

  1. 1. (Approximate) Bayesian Computation as a new empirical Bayes (something)? Christian P. Robert Universit´ Paris-Dauphine & CREST, Paris e Joint work with K.L. Mengersen and P. Pudlo Workshop on Recent Advances in Statistical Inference, Padova, March 21, 2013
  2. 2. Advertisment MCMSki IV to be held in Chamonix Mt Blanc, France, from Monday, Jan. 6 to Wed., Jan. 8, 2014 All aspects of MCMC++ theory and methodology and applications Parallel (invited and contributed, 2 or 3) sessions: call for proposals still open on website http://www.pages.drexel.edu/∼mwl25/mcmski/
  3. 3. OutlineUnavailable likelihoodsABC methodsABC as an inference machineABCelConclusion and perspectives
  4. 4. Intractable likelihood Case of a well-defined statistical model where the likelihood function (θ|y) = f (y1 , . . . , yn |θ) is (really!) not available in closed form can (easily!) be neither completed nor demarginalised cannot be estimated by an unbiased estimator c Prohibits direct implementation of a generic MCMC algorithm like Metropolis–Hastings
  5. 5. Intractable likelihood Case of a well-defined statistical model where the likelihood function (θ|y) = f (y1 , . . . , yn |θ) is (really!) not available in closed form can (easily!) be neither completed nor demarginalised cannot be estimated by an unbiased estimator c Prohibits direct implementation of a generic MCMC algorithm like Metropolis–Hastings
  6. 6. The abc alternative Approximations to the original B problem Degrading the precision down to a tolerance ε Replacing the likelihood with a non-parametric approximation Summarising the data with insufficient statistics
  7. 7. The abc alternative Approximations to the original B problem Degrading the precision down to a tolerance ε Replacing the likelihood with a non-parametric approximation Summarising the data with insufficient statistics
  8. 8. The abc alternative Approximations to the original B problem Degrading the precision down to a tolerance ε Replacing the likelihood with a non-parametric approximation Summarising the data with insufficient statistics
  9. 9. Different worries about abc Impact on B inference a mere computational issue (that will eventually end up being solved by more powerful computers, &tc, even if too costly in the short term, as for regular Monte Carlo methods) an inferential issue (opening opportunities for new inference machine, with legitimity different than for classical B approach) a Bayesian conundrum (how closely related to the/a B approach? more than recycling B tools? true B with raw data? a new type of empirical Bayes inference?)
  10. 10. Different worries about abc Impact on B inference a mere computational issue (that will eventually end up being solved by more powerful computers, &tc, even if too costly in the short term, as for regular Monte Carlo methods) an inferential issue (opening opportunities for new inference machine, with legitimity different than for classical B approach) a Bayesian conundrum (how closely related to the/a B approach? more than recycling B tools? true B with raw data? a new type of empirical Bayes inference?)
  11. 11. Different worries about abc Impact on B inference a mere computational issue (that will eventually end up being solved by more powerful computers, &tc, even if too costly in the short term, as for regular Monte Carlo methods) an inferential issue (opening opportunities for new inference machine, with legitimity different than for classical B approach) a Bayesian conundrum (how closely related to the/a B approach? more than recycling B tools? true B with raw data? a new type of empirical Bayes inference?)
  12. 12. Econom’ections Similar exploration of simulation-based and approximation techniques in Econometrics Simulated method of moments Method of simulated moments Simulated pseudo-maximum-likelihood Indirect inference [Gouri´roux & Monfort, 1996] e even though motivation is partly-defined models rather than complex likelihoods
  13. 13. Econom’ections Similar exploration of simulation-based and approximation techniques in Econometrics Simulated method of moments Method of simulated moments Simulated pseudo-maximum-likelihood Indirect inference [Gouri´roux & Monfort, 1996] e even though motivation is partly-defined models rather than complex likelihoods
  14. 14. Indirect inference ^ Minimise [in θ] a distance between estimators β based on a pseudo-model for genuine observations and for observations simulated under the true model and the parameter θ. [Gouri´roux, Monfort, & Renault, 1993; e Smith, 1993; Gallant & Tauchen, 1996]
  15. 15. Indirect inference (PML vs. PSE) Example of the pseudo-maximum-likelihood (PML) ^ β(y) = arg max log f (yt |β, y1:(t−1) ) β t leading to arg min ||β(yo ) − β(y1 (θ), . . . , yS (θ))||2 ^ ^ θ when ys (θ) ∼ f (y|θ) s = 1, . . . , S
  16. 16. Indirect inference (PML vs. PSE) Example of the pseudo-score-estimator (PSE) 2 ∂ log f ^ β(y) = arg min (yt |β, y1:(t−1) ) β t ∂β leading to arg min ||β(yo ) − β(y1 (θ), . . . , yS (θ))||2 ^ ^ θ when ys (θ) ∼ f (y|θ) s = 1, . . . , S
  17. 17. Consistent indirect inference “...in order to get a unique solution the dimension of the auxiliary parameter β must be larger than or equal to the dimension of the initial parameter θ. If the problem is just identified the different methods become easier...” Consistency depending on the criterion and on the asymptotic identifiability of θ [Gouri´roux & Monfort, 1996, p. 66] e Which connection [if any] with the B perspective?
  18. 18. Consistent indirect inference “...in order to get a unique solution the dimension of the auxiliary parameter β must be larger than or equal to the dimension of the initial parameter θ. If the problem is just identified the different methods become easier...” Consistency depending on the criterion and on the asymptotic identifiability of θ [Gouri´roux & Monfort, 1996, p. 66] e Which connection [if any] with the B perspective?
  19. 19. Consistent indirect inference “...in order to get a unique solution the dimension of the auxiliary parameter β must be larger than or equal to the dimension of the initial parameter θ. If the problem is just identified the different methods become easier...” Consistency depending on the criterion and on the asymptotic identifiability of θ [Gouri´roux & Monfort, 1996, p. 66] e Which connection [if any] with the B perspective?
  20. 20. Approximate Bayesian computation Unavailable likelihoods ABC methods Genesis of ABC ABC basics Advances and interpretations ABC as knn ABC as an inference machine ABCel Conclusion and perspectives
  21. 21. Genetic background of ABC skip genetics ABC is a recent computational technique that only requires being able to sample from the likelihood f (·|θ) This technique stemmed from population genetics models, about 15 years ago, and population geneticists still contribute significantly to methodological developments of ABC. [Griffith & al., 1997; Tavar´ & al., 1999] e
  22. 22. Demo-genetic inference Each model is characterized by a set of parameters θ that cover historical (time divergence, admixture time ...), demographics (population sizes, admixture rates, migration rates, ...) and genetic (mutation rate, ...) factors The goal is to estimate these parameters from a dataset of polymorphism (DNA sample) y observed at the present time Problem: most of the time, we cannot calculate the likelihood of the polymorphism data f (y|θ)...
  23. 23. Demo-genetic inference Each model is characterized by a set of parameters θ that cover historical (time divergence, admixture time ...), demographics (population sizes, admixture rates, migration rates, ...) and genetic (mutation rate, ...) factors The goal is to estimate these parameters from a dataset of polymorphism (DNA sample) y observed at the present time Problem: most of the time, we cannot calculate the likelihood of the polymorphism data f (y|θ)...
  24. 24. Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium Mutations according to the Simple stepwise Mutation Model (SMM) • date of the mutations ∼ Poisson process with intensity θ/2 over the branches • MRCA = 100 • independent mutations: Sample of 8 genes ±1 with pr. 1/2
  25. 25. Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium Kingman’s genealogy When time axis is normalized, T (k) ∼ Exp(k(k − 1)/2) Mutations according to the Simple stepwise Mutation Model (SMM) • date of the mutations ∼ Poisson process with intensity θ/2 over the branches • MRCA = 100 • independent mutations: ±1 with pr. 1/2
  26. 26. Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium Kingman’s genealogy When time axis is normalized, T (k) ∼ Exp(k(k − 1)/2) Mutations according to the Simple stepwise Mutation Model (SMM) • date of the mutations ∼ Poisson process with intensity θ/2 over the branches • MRCA = 100 • independent mutations: ±1 with pr. 1/2
  27. 27. Neutral model at a given microsatellite locus, in a closedpanmictic population at equilibrium Kingman’s genealogy When time axis is normalized, T (k) ∼ Exp(k(k − 1)/2) Mutations according to the Simple stepwise Mutation Model (SMM) • date of the mutations ∼ Poisson process with intensity θ/2 over the branches Observations: leafs of the tree • MRCA = 100 ^ θ=? • independent mutations: ±1 with pr. 1/2
  28. 28. Much more interesting models. . . several independent locus Independent gene genealogies and mutations different populations linked by an evolutionary scenario made of divergences, admixtures, migrations between populations, etc. larger sample size usually between 50 and 100 genes MRCA τ2 τ1 A typical evolutionary scenario: POP 0 POP 1 POP 2
  29. 29. Intractable likelihood Missing (too missing!) data structure: f (y|θ) = f (y|G , θ)f (G |θ)dG G cannot be computed in a manageable way... The genealogies are considered as nuisance parameters This modelling clearly differs from the phylogenetic perspective where the tree is the parameter of interest.
  30. 30. Intractable likelihood Missing (too missing!) data structure: f (y|θ) = f (y|G , θ)f (G |θ)dG G cannot be computed in a manageable way... The genealogies are considered as nuisance parameters This modelling clearly differs from the phylogenetic perspective where the tree is the parameter of interest.
  31. 31. A?B?C? A stands for approximate [wrong likelihood / picture] B stands for Bayesian C stands for computation [producing a parameter sample]
  32. 32. A?B?C? A stands for approximate [wrong likelihood / picture] B stands for Bayesian C stands for computation [producing a parameter sample]
  33. 33. A?B?C? ESS=155.6 ESS=75.93 ESS=76.87 4 2.0 3 Density Density Density 1.0 2 1.0 1 0.0 0.0 0 A stands for approximate −0.5 0.0 θ ESS=91.54 0.5 1.0 −0.4 −0.2 0.0 θ ESS=108.4 0.2 0.4 −0.4 −0.2 θ ESS=85.13 0.0 0.2 4 0.0 1.0 2.0 3.0 0.0 1.0 2.0 3.0 3 [wrong likelihood / Density Density Density 2 1 0 picture] −0.6 −0.4 −0.2 θ ESS=149.1 0.0 0.2 −0.4 0.0 θ ESS=96.31 0.2 0.4 0.6 −0.2 0.0 θ ESS=83.77 0.2 0.4 0.6 4 2.0 3 2.0 Density Density Density 2 1.0 1.0 B stands for Bayesian 1 0.0 0.0 0 −0.5 0.0 0.5 1.0 −0.4 0.0 0.2 0.4 0.6 −0.6 −0.4 −0.2 0.0 0.2 0.4 θ ESS=155.7 θ ESS=92.42 θ ESS=95.01 0.0 1.0 2.0 3.0 2.0 3.0 C stands for computation Density Density Density 1.0 1.5 0.0 0.0 [producing a parameter −0.5 θ 0.0 ESS=139.2 0.5 −0.4 0.0 θ ESS=99.33 0.2 0.4 0.6 −0.4 θ 0.0 ESS=87.28 0.2 0.4 0.6 2.0 0.0 1.0 2.0 3.0 3 sample] Density Density Density 2 1.0 1 0.0 0 −0.6 −0.2 0.2 0.6 −0.4 −0.2 0.0 0.2 0.4 −0.2 0.0 0.2 0.4 0.6
  34. 34. How Bayesian is aBc? Could we turn the resolution into a Bayesian answer? ideally so (not meaningfull: requires ∞-ly powerful computer approximation error unknown (w/o costly simulation) true Bayes for wrong model (formal and artificial) true Bayes for noisy model (much more convincing) true Bayes for estimated likelihood (back to econometrics?) illuminating the tension between information and precision
  35. 35. Untractable likelihoodBack to stage zero: what can we dowhen a likelihood function f (y|θ) iswell-defined but impossible / toocostly to compute...? MCMC cannot be implemented! shall we give up Bayesian inference altogether?!
  36. 36. Untractable likelihoodBack to stage zero: what can we dowhen a likelihood function f (y|θ) iswell-defined but impossible / toocostly to compute...? MCMC cannot be implemented! shall we give up Bayesian inference altogether?! or settle for an almost Bayesian inference/picture...?
  37. 37. ABC methodology Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: Foundation For an observation y ∼ f (y|θ), under the prior π(θ), if one keeps jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y, then the selected θ ∼ π(θ|y) [Rubin, 1984; Diggle & Gratton, 1984; Tavar´ et al., 1997] e
  38. 38. ABC methodology Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: Foundation For an observation y ∼ f (y|θ), under the prior π(θ), if one keeps jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y, then the selected θ ∼ π(θ|y) [Rubin, 1984; Diggle & Gratton, 1984; Tavar´ et al., 1997] e
  39. 39. ABC methodology Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: Foundation For an observation y ∼ f (y|θ), under the prior π(θ), if one keeps jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y, then the selected θ ∼ π(θ|y) [Rubin, 1984; Diggle & Gratton, 1984; Tavar´ et al., 1997] e
  40. 40. A as A...pproximative When y is a continuous random variable, strict equality z = y is replaced with a tolerance zone ρ(y, z) where ρ is a distance Output distributed from def π(θ) Pθ {ρ(y, z) < } ∝ π(θ|ρ(y, z) < ) [Pritchard et al., 1999]
  41. 41. A as A...pproximative When y is a continuous random variable, strict equality z = y is replaced with a tolerance zone ρ(y, z) where ρ is a distance Output distributed from def π(θ) Pθ {ρ(y, z) < } ∝ π(θ|ρ(y, z) < ) [Pritchard et al., 1999]
  42. 42. ABC algorithm In most implementations, further degree of A...pproximation: Algorithm 1 Likelihood-free rejection sampler for i = 1 to N do repeat generate θ from the prior distribution π(·) generate z from the likelihood f (·|θ ) until ρ{η(z), η(y)} set θi = θ end for where η(y) defines a (not necessarily sufficient) statistic
  43. 43. Output The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }. The idea behind ABC is that the summary statistics coupled with a small tolerance should provide a good approximation of the posterior distribution: π (θ|y) = π (θ, z|y)dz ≈ π(θ|y) . ...does it?!
  44. 44. Output The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }. The idea behind ABC is that the summary statistics coupled with a small tolerance should provide a good approximation of the posterior distribution: π (θ|y) = π (θ, z|y)dz ≈ π(θ|y) . ...does it?!
  45. 45. Output The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }. The idea behind ABC is that the summary statistics coupled with a small tolerance should provide a good approximation of the posterior distribution: π (θ|y) = π (θ, z|y)dz ≈ π(θ|y) . ...does it?!
  46. 46. Output The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }. The idea behind ABC is that the summary statistics coupled with a small tolerance should provide a good approximation of the restricted posterior distribution: π (θ|y) = π (θ, z|y)dz ≈ π(θ|η(y)) . Not so good..! skip convergence details!
  47. 47. Convergence of ABC What happens when → 0? For B ⊂ Θ, we have A f (z|θ)dz f (z|θ)π(θ)dθ ,y B π(θ)dθ = dz B A ,y ×Θ π(θ)f (z|θ)dzdθ A ,y A ,y ×Θ π(θ)f (z|θ)dzdθ B f (z|θ)π(θ)dθ m(z) = dz A ,y m(z) A ,y ×Θ π(θ)f (z|θ)dzdθ m(z) = π(B|z) dz A ,y A ,y ×Θ π(θ)f (z|θ)dzdθ which indicates convergence for a continuous π(B|z).
  48. 48. Convergence of ABC What happens when → 0? For B ⊂ Θ, we have A f (z|θ)dz f (z|θ)π(θ)dθ ,y B π(θ)dθ = dz B A ,y ×Θ π(θ)f (z|θ)dzdθ A ,y A ,y ×Θ π(θ)f (z|θ)dzdθ B f (z|θ)π(θ)dθ m(z) = dz A ,y m(z) A ,y ×Θ π(θ)f (z|θ)dzdθ m(z) = π(B|z) dz A ,y A ,y ×Θ π(θ)f (z|θ)dzdθ which indicates convergence for a continuous π(B|z).
  49. 49. Convergence (do not attempt!) ...and the above does not apply to insufficient statistics: If η(y) is not a sufficient statistics, the best one can hope for is π(θ|η(y)) , not π(θ|y) If η(y) is an ancillary statistic, the whole information contained in y is lost!, the “best” one can “hope” for is π(θ|η(y)) = π(θ) Bummer!!!
  50. 50. Convergence (do not attempt!) ...and the above does not apply to insufficient statistics: If η(y) is not a sufficient statistics, the best one can hope for is π(θ|η(y)) , not π(θ|y) If η(y) is an ancillary statistic, the whole information contained in y is lost!, the “best” one can “hope” for is π(θ|η(y)) = π(θ) Bummer!!!
  51. 51. Convergence (do not attempt!) ...and the above does not apply to insufficient statistics: If η(y) is not a sufficient statistics, the best one can hope for is π(θ|η(y)) , not π(θ|y) If η(y) is an ancillary statistic, the whole information contained in y is lost!, the “best” one can “hope” for is π(θ|η(y)) = π(θ) Bummer!!!
  52. 52. Convergence (do not attempt!) ...and the above does not apply to insufficient statistics: If η(y) is not a sufficient statistics, the best one can hope for is π(θ|η(y)) , not π(θ|y) If η(y) is an ancillary statistic, the whole information contained in y is lost!, the “best” one can “hope” for is π(θ|η(y)) = π(θ) Bummer!!!
  53. 53. Comments Role of distance paramount (because = 0) Scaling of components of η(y) is also determinant matters little if “small enough” representative of “curse of dimensionality” small is beautiful! the data as a whole may be paradoxically weakly informative for ABC
  54. 54. ABC (simul’) advances how approximative is ABC? ABC as knn Simulating from the prior is often poor in efficiency Either modify the proposal distribution on θ to increase the density of x’s within the vicinity of y ... [Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007] ...or by viewing the problem as a conditional density estimation and by developing techniques to allow for larger [Beaumont et al., 2002] .....or even by including in the inferential framework [ABCµ ] [Ratmann et al., 2009]
  55. 55. ABC (simul’) advances how approximative is ABC? ABC as knn Simulating from the prior is often poor in efficiency Either modify the proposal distribution on θ to increase the density of x’s within the vicinity of y ... [Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007] ...or by viewing the problem as a conditional density estimation and by developing techniques to allow for larger [Beaumont et al., 2002] .....or even by including in the inferential framework [ABCµ ] [Ratmann et al., 2009]
  56. 56. ABC (simul’) advances how approximative is ABC? ABC as knn Simulating from the prior is often poor in efficiency Either modify the proposal distribution on θ to increase the density of x’s within the vicinity of y ... [Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007] ...or by viewing the problem as a conditional density estimation and by developing techniques to allow for larger [Beaumont et al., 2002] .....or even by including in the inferential framework [ABCµ ] [Ratmann et al., 2009]
  57. 57. ABC (simul’) advances how approximative is ABC? ABC as knn Simulating from the prior is often poor in efficiency Either modify the proposal distribution on θ to increase the density of x’s within the vicinity of y ... [Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007] ...or by viewing the problem as a conditional density estimation and by developing techniques to allow for larger [Beaumont et al., 2002] .....or even by including in the inferential framework [ABCµ ] [Ratmann et al., 2009]
  58. 58. ABC-NP Better usage of [prior] simulations by adjustement: instead of throwing away θ such that ρ(η(z), η(y)) > , replace θ’s with locally regressed transforms θ∗ = θ − {η(z) − η(y)}T β ^ [Csill´ry et al., TEE, 2010] e ^ where β is obtained by [NP] weighted least square regression on (η(z) − η(y)) with weights Kδ {ρ(η(z), η(y))} [Beaumont et al., 2002, Genetics]
  59. 59. ABC-NP (regression) Also found in the subsequent literature, e.g. in Fearnhead-Prangle (2012) : weight directly simulation by Kδ {ρ(η(z(θ)), η(y))} or S 1 Kδ {ρ(η(zs (θ)), η(y))} S s=1 [consistent estimate of f (η|θ)] Curse of dimensionality: poor estimate when d = dim(η) is large...
  60. 60. ABC-NP (regression) Also found in the subsequent literature, e.g. in Fearnhead-Prangle (2012) : weight directly simulation by Kδ {ρ(η(z(θ)), η(y))} or S 1 Kδ {ρ(η(zs (θ)), η(y))} S s=1 [consistent estimate of f (η|θ)] Curse of dimensionality: poor estimate when d = dim(η) is large...
  61. 61. ABC-NP (density estimation) Use of the kernel weights Kδ {ρ(η(z(θ)), η(y))} leads to the NP estimate of the posterior expectation i θi Kδ {ρ(η(z(θi )), η(y))} i Kδ {ρ(η(z(θi )), η(y))} [Blum, JASA, 2010]
  62. 62. ABC-NP (density estimation) Use of the kernel weights Kδ {ρ(η(z(θ)), η(y))} leads to the NP estimate of the posterior conditional density i ˜ Kb (θi − θ)Kδ {ρ(η(z(θi )), η(y))} i Kδ {ρ(η(z(θi )), η(y))} [Blum, JASA, 2010]
  63. 63. ABC-NP (density estimations) Other versions incorporating regression adjustments i ˜ Kb (θ∗ − θ)Kδ {ρ(η(z(θi )), η(y))} i i Kδ {ρ(η(z(θi )), η(y))} In all cases, error E[^ (θ|y)] − g (θ|y) = cb 2 + cδ2 + OP (b 2 + δ2 ) + OP (1/nδd ) g c var(^ (θ|y)) = g (1 + oP (1)) nbδd
  64. 64. ABC-NP (density estimations) Other versions incorporating regression adjustments i ˜ Kb (θ∗ − θ)Kδ {ρ(η(z(θi )), η(y))} i i Kδ {ρ(η(z(θi )), η(y))} In all cases, error E[^ (θ|y)] − g (θ|y) = cb 2 + cδ2 + OP (b 2 + δ2 ) + OP (1/nδd ) g c var(^ (θ|y)) = g (1 + oP (1)) nbδd [Blum, JASA, 2010]
  65. 65. ABC-NP (density estimations) Other versions incorporating regression adjustments i ˜ Kb (θ∗ − θ)Kδ {ρ(η(z(θi )), η(y))} i i Kδ {ρ(η(z(θi )), η(y))} In all cases, error E[^ (θ|y)] − g (θ|y) = cb 2 + cδ2 + OP (b 2 + δ2 ) + OP (1/nδd ) g c var(^ (θ|y)) = g (1 + oP (1)) nbδd [standard NP calculations]
  66. 66. ABC as knn [Biau et al., 2012, arxiv:1207.6461] Practice of ABC: determine tolerance as a quantile on observed distances, say 10% or 1% quantile, = N = qα (d1 , . . . , dN ) Interpretation of ε as nonparametric bandwidth only approximation of the actual practice [Blum & Fran¸ois, 2010] c ABC is a k-nearest neighbour (knn) method with kN = N N [Loftsgaarden & Quesenberry, 1965]
  67. 67. ABC as knn [Biau et al., 2012, arxiv:1207.6461] Practice of ABC: determine tolerance as a quantile on observed distances, say 10% or 1% quantile, = N = qα (d1 , . . . , dN ) Interpretation of ε as nonparametric bandwidth only approximation of the actual practice [Blum & Fran¸ois, 2010] c ABC is a k-nearest neighbour (knn) method with kN = N N [Loftsgaarden & Quesenberry, 1965]
  68. 68. ABC as knn [Biau et al., 2012, arxiv:1207.6461] Practice of ABC: determine tolerance as a quantile on observed distances, say 10% or 1% quantile, = N = qα (d1 , . . . , dN ) Interpretation of ε as nonparametric bandwidth only approximation of the actual practice [Blum & Fran¸ois, 2010] c ABC is a k-nearest neighbour (knn) method with kN = N N [Loftsgaarden & Quesenberry, 1965]
  69. 69. ABC consistency Provided kN / log log N −→ ∞ and kN /N −→ 0 as N → ∞, for almost all s0 (with respect to the distribution of S), with probability 1, kN 1 ϕ(θj ) −→ E[ϕ(θj )|S = s0 ] kN j=1 [Devroye, 1982] Biau et al. (2012) also recall pointwise and integrated mean square error consistency results on the corresponding kernel estimate of the conditional posterior distribution, under constraints p kN → ∞, kN /N → 0, hN → 0 and hN kN → ∞,
  70. 70. ABC consistency Provided kN / log log N −→ ∞ and kN /N −→ 0 as N → ∞, for almost all s0 (with respect to the distribution of S), with probability 1, kN 1 ϕ(θj ) −→ E[ϕ(θj )|S = s0 ] kN j=1 [Devroye, 1982] Biau et al. (2012) also recall pointwise and integrated mean square error consistency results on the corresponding kernel estimate of the conditional posterior distribution, under constraints p kN → ∞, kN /N → 0, hN → 0 and hN kN → ∞,
  71. 71. Rates of convergence Further assumptions (on target and kernel) allow for precise (integrated mean square) convergence rates (as a power of the sample size N), derived from classical k-nearest neighbour regression, like 4 when m = 1, 2, 3, kN ≈ N (p+4)/(p+8) and rate N − p+8 4 when m = 4, kN ≈ N (p+4)/(p+8) and rate N − p+8 log N 4 when m > 4, kN ≈ N (p+4)/(m+p+4) and rate N − m+p+4 [Biau et al., 2012, arxiv:1207.6461] Drag: Only applies to sufficient summary statistics
  72. 72. Rates of convergence Further assumptions (on target and kernel) allow for precise (integrated mean square) convergence rates (as a power of the sample size N), derived from classical k-nearest neighbour regression, like 4 when m = 1, 2, 3, kN ≈ N (p+4)/(p+8) and rate N − p+8 4 when m = 4, kN ≈ N (p+4)/(p+8) and rate N − p+8 log N 4 when m > 4, kN ≈ N (p+4)/(m+p+4) and rate N − m+p+4 [Biau et al., 2012, arxiv:1207.6461] Drag: Only applies to sufficient summary statistics
  73. 73. ABC inference machine Unavailable likelihoods ABC methods ABC as an inference machine Error inc. Exact BC and approximate targets summary statistic ABCel Conclusion and perspectives
  74. 74. How Bayesian is aBc..? maybe a convergent method of inference (meaningful? sufficient? foreign?) approximation error unknown (w/o simulation) pragmatic B (there is no other solution!) many calibration issues (tolerance, distance, statistics) the NP side should be incorporated into the whole B picture the approximation error should also be part of the B inference to ABCel
  75. 75. ABCµ Idea Infer about the error as well as about the parameter: Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ (θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ) Warning! Replacement of ξ( |y, θ) with a non-parametric kernel approximation. [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
  76. 76. ABCµ Idea Infer about the error as well as about the parameter: Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ (θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ) Warning! Replacement of ξ( |y, θ) with a non-parametric kernel approximation. [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
  77. 77. ABCµ Idea Infer about the error as well as about the parameter: Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ (θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ) Warning! Replacement of ξ( |y, θ) with a non-parametric kernel approximation. [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
  78. 78. ABCµ details Multidimensional distances ρk (k = 1, . . . , K ) and errors k = ρk (ηk (z), ηk (y)), with 1 k ∼ ξk ( |y, θ) ≈ ξk ( |y, θ) = ^ K [{ k −ρk (ηk (zb ), ηk (y))}/hk ] Bhk b then used in replacing ξ( |y, θ) with mink ξk ( |y, θ) ^ ABCµ involves acceptance probability π(θ , ) q(θ , θ)q( , ) mink ξk ( |y, θ ) ^ π(θ, ) q(θ, θ )q( , ) mink ξk ( |y, θ) ^
  79. 79. ABCµ details Multidimensional distances ρk (k = 1, . . . , K ) and errors k = ρk (ηk (z), ηk (y)), with 1 k ∼ ξk ( |y, θ) ≈ ξk ( |y, θ) = ^ K [{ k −ρk (ηk (zb ), ηk (y))}/hk ] Bhk b then used in replacing ξ( |y, θ) with mink ξk ( |y, θ) ^ ABCµ involves acceptance probability π(θ , ) q(θ , θ)q( , ) mink ξk ( |y, θ ) ^ π(θ, ) q(θ, θ )q( , ) mink ξk ( |y, θ) ^
  80. 80. Wilkinson’s exact BC (not exactly!) ABC approximation error (i.e. non-zero tolerance) replaced with exact simulation from a controlled approximation to the target, convolution of true posterior with kernel function π(θ)f (z|θ)K (y − z) π (θ, z|y) = , π(θ)f (z|θ)K (y − z)dzdθ with K kernel parameterised by bandwidth . [Wilkinson, 2008] Theorem The ABC algorithm based on the assumption of a randomised observation y = y + ξ, ξ ∼ K , and an acceptance probability of ˜ K (y − z)/M gives draws from the posterior distribution π(θ|y).
  81. 81. Wilkinson’s exact BC (not exactly!) ABC approximation error (i.e. non-zero tolerance) replaced with exact simulation from a controlled approximation to the target, convolution of true posterior with kernel function π(θ)f (z|θ)K (y − z) π (θ, z|y) = , π(θ)f (z|θ)K (y − z)dzdθ with K kernel parameterised by bandwidth . [Wilkinson, 2008] Theorem The ABC algorithm based on the assumption of a randomised observation y = y + ξ, ξ ∼ K , and an acceptance probability of ˜ K (y − z)/M gives draws from the posterior distribution π(θ|y).
  82. 82. How exact a BC? Pros Pseudo-data from true model and observed data from noisy model Interesting perspective in that outcome is completely controlled Link with ABCµ and assuming y is observed with a measurement error with density K Relates to the theory of model approximation [Kennedy & O’Hagan, 2001] Cons Requires K to be bounded by M True approximation error never assessed
  83. 83. How exact a BC? Pros Pseudo-data from true model and observed data from noisy model Interesting perspective in that outcome is completely controlled Link with ABCµ and assuming y is observed with a measurement error with density K Relates to the theory of model approximation [Kennedy & O’Hagan, 2001] Cons Requires K to be bounded by M True approximation error never assessed
  84. 84. Noisy ABC for HMMs Specific case of a hidden Markov model Xt+1 ∼ Qθ (Xt , ·) Yt+1 ∼ gθ (·|xt ) where only y0 is observed. 1:n [Dean, Singh, Jasra, & Peters, 2011] Use of specific constraints, adapted to the Markov structure: y1 ∈ B(y1 , ) × · · · × yn ∈ B(yn , ) 0 0
  85. 85. Noisy ABC for HMMs Specific case of a hidden Markov model Xt+1 ∼ Qθ (Xt , ·) Yt+1 ∼ gθ (·|xt ) where only y0 is observed. 1:n [Dean, Singh, Jasra, & Peters, 2011] Use of specific constraints, adapted to the Markov structure: y1 ∈ B(y1 , ) × · · · × yn ∈ B(yn , ) 0 0
  86. 86. Noisy ABC-MLE Idea: Modify instead the data from the start 0 (y1 + ζ1 , . . . , yn + ζn ) [ see Fearnhead-Prangle ] noisy ABC-MLE estimate arg max Pθ Y1 ∈ B(y1 + ζ1 , ), . . . , Yn ∈ B(yn + ζn , ) 0 0 θ [Dean, Singh, Jasra, & Peters, 2011]
  87. 87. Consistent noisy ABC-MLE Degrading the data improves the estimation performances: Noisy ABC-MLE is asymptotically (in n) consistent under further assumptions, the noisy ABC-MLE is asymptotically normal increase in variance of order −2 likely degradation in precision or computing time due to the lack of summary statistic [curse of dimensionality]
  88. 88. Which summary? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistics Loss of statistical information balanced against gain in data roughening Approximation error and information loss remain unknown Choice of statistics induces choice of distance function towards standardisation
  89. 89. Which summary? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistics Loss of statistical information balanced against gain in data roughening Approximation error and information loss remain unknown Choice of statistics induces choice of distance function towards standardisation
  90. 90. Which summary for model choice? Depending on the choice of η(·), the Bayes factor based on this insufficient statistic, η π1 (θ1 )f1η (η(y)|θ1 ) dθ1 B12 (y) = , π2 (θ2 )f2η (η(y)|θ2 ) dθ2 is either consistent or inconsistent [X, Cornuet, Marin, & Pillai, 2012] Consistency only depends on the range of Ei [η(y)] under both models [Marin, Pillai, X, & Rousseau, 2012]
  91. 91. Which summary for model choice? Depending on the choice of η(·), the Bayes factor based on this insufficient statistic, η π1 (θ1 )f1η (η(y)|θ1 ) dθ1 B12 (y) = , π2 (θ2 )f2η (η(y)|θ2 ) dθ2 is either consistent or inconsistent [X, Cornuet, Marin, & Pillai, 2012] Consistency only depends on the range of Ei [η(y)] under both models [Marin, Pillai, X, & Rousseau, 2012]
  92. 92. Semi-automatic ABC Fearnhead and Prangle (2012) study ABC and the selection of the summary statistic in close proximity to Wilkinson’s proposal ABC considered as inferential method and calibrated as such randomised (or ‘noisy’) version of the summary statistics ˜ η(y) = η(y) + τ derivation of a well-calibrated version of ABC, i.e. an algorithm that gives proper predictions for the distribution associated with this randomised summary statistic
  93. 93. Summary [of F&P/statistics) optimality of the posterior expectation E[θ|y] of the parameter of interest as summary statistics η(y)! [requires iterative process] use of the standard quadratic loss function (θ − θ0 )T A(θ − θ0 ) recent extension to model choice, optimality of Bayes factor B12 (y) [F&P, ISBA 2012, Kyoto]
  94. 94. Summary [of F&P/statistics) optimality of the posterior expectation E[θ|y] of the parameter of interest as summary statistics η(y)! [requires iterative process] use of the standard quadratic loss function (θ − θ0 )T A(θ − θ0 ) recent extension to model choice, optimality of Bayes factor B12 (y) [F&P, ISBA 2012, Kyoto]
  95. 95. Summary [about summaries] Choice of summary statistics is paramount for ABC validation/performances At best, ABC approximates π(. | η(y)) Model selection feasible with ABC [with caution!] For estimation, consistency if {θ; µ(θ) = µ0 } = θ0 when Eθ [η(y)] = µ(θ) For testing consistency if {µ1 (θ1 ), θ1 ∈ Θ1 } ∩ {µ2 (θ2 ), θ2 ∈ Θ2 } = ∅ [Marin et al., 2011]
  96. 96. Summary [about summaries] Choice of summary statistics is paramount for ABC validation/performances At best, ABC approximates π(. | η(y)) Model selection feasible with ABC [with caution!] For estimation, consistency if {θ; µ(θ) = µ0 } = θ0 when Eθ [η(y)] = µ(θ) For testing consistency if {µ1 (θ1 ), θ1 ∈ Θ1 } ∩ {µ2 (θ2 ), θ2 ∈ Θ2 } = ∅ [Marin et al., 2011]
  97. 97. Summary [about summaries] Choice of summary statistics is paramount for ABC validation/performances At best, ABC approximates π(. | η(y)) Model selection feasible with ABC [with caution!] For estimation, consistency if {θ; µ(θ) = µ0 } = θ0 when Eθ [η(y)] = µ(θ) For testing consistency if {µ1 (θ1 ), θ1 ∈ Θ1 } ∩ {µ2 (θ2 ), θ2 ∈ Θ2 } = ∅ [Marin et al., 2011]
  98. 98. Summary [about summaries] Choice of summary statistics is paramount for ABC validation/performances At best, ABC approximates π(. | η(y)) Model selection feasible with ABC [with caution!] For estimation, consistency if {θ; µ(θ) = µ0 } = θ0 when Eθ [η(y)] = µ(θ) For testing consistency if {µ1 (θ1 ), θ1 ∈ Θ1 } ∩ {µ2 (θ2 ), θ2 ∈ Θ2 } = ∅ [Marin et al., 2011]
  99. 99. Summary [about summaries] Choice of summary statistics is paramount for ABC validation/performances At best, ABC approximates π(. | η(y)) Model selection feasible with ABC [with caution!] For estimation, consistency if {θ; µ(θ) = µ0 } = θ0 when Eθ [η(y)] = µ(θ) For testing consistency if {µ1 (θ1 ), θ1 ∈ Θ1 } ∩ {µ2 (θ2 ), θ2 ∈ Θ2 } = ∅ [Marin et al., 2011]
  100. 100. Empirical likelihood (EL) Unavailable likelihoods ABC methods ABC as an inference machine ABCel ABC and EL Composite likelihood Illustrations Conclusion and perspectives
  101. 101. Empirical likelihood (EL) Dataset x made of n independent replicates x = (x1 , . . . , xn ) of some X ∼ F Generalized moment condition model EF h(X , φ) = 0, where h is a known function, and φ an unknown parameter Corresponding empirical likelihood n Lel (φ|x) = max pi p i=1 for all p such that 0 pi 1, i pi = 1, i pi h(xi , φ) = 0. [Owen, 1988, Bio’ka, & Empirical Likelihood, 2001]
  102. 102. Empirical likelihood (EL) Dataset x made of n independent replicates x = (x1 , . . . , xn ) of some X ∼ F Generalized moment condition model EF h(X , φ) = 0, where h is a known function, and φ an unknown parameter Corresponding empirical likelihood n Lel (φ|x) = max pi p i=1 for all p such that 0 pi 1, i pi = 1, i pi h(xi , φ) = 0. [Owen, 1988, Bio’ka, & Empirical Likelihood, 2001]
  103. 103. Convergence of EL [3.4] Theorem 3.4 Let X , Y1 , . . . , Yn be independent rv’s with common distribution F0 . For θ ∈ Θ, and the function h(X , θ) ∈ Rs , let θ0 ∈ Θ be such that Var(h(Yi , θ0 )) is finite and has rank q > 0. If θ0 satisfies E(h(X , θ0 )) = 0, then Lel (θ0 |Y1 , . . . , Yn ) −2 log → χ2 (q) n−n in distribution when n → ∞. [Owen, 2001]
  104. 104. Convergence of EL [3.4] “...The interesting thing about Theorem 3.4 is what is not there. It ^ includes no conditions to make θ a good estimate of θ0 , nor even conditions to ensure a unique value for θ0 , nor even that any solution θ0 exists. Theorem 3.4 applies in the just determined, over-determined, and under-determined cases. When we can prove that our estimating ^ equations uniquely define θ0 , and provide a consistent estimator θ of it, then confidence regions and tests follow almost automatically through Theorem 3.4.”. [Owen, 2001]
  105. 105. Raw ABCel sampler Act as if EL was an exact likelihood [Lazar, 2003] for i = 1 → N do generate φi from the prior distribution π(·) set the weight ωi = Lel (φi |xobs ) end for return (φi , ωi ), i = 1, . . . , N Output weighted sample of size N
  106. 106. Raw ABCel sampler Act as if EL was an exact likelihood [Lazar, 2003] for i = 1 → N do generate φi from the prior distribution π(·) set the weight ωi = Lel (φi |xobs ) end for return (φi , ωi ), i = 1, . . . , N Performance evaluated through effective sample size  2 N  N  ESS = 1 ωi ωj   i=1 j=1
  107. 107. Raw ABCel sampler Act as if EL was an exact likelihood [Lazar, 2003] for i = 1 → N do generate φi from the prior distribution π(·) set the weight ωi = Lel (φi |xobs ) end for return (φi , ωi ), i = 1, . . . , N More advanced algorithms can be adapted to EL: E.g., adaptive multiple importance sampling (AMIS) of Cornuet et al. to speed up computations [Cornuet et al., 2012]
  108. 108. Moment condition in population genetics? EL does not require a fully defined and often complex (hence debatable) parametric model Main difficulty Derive a constraint EF h(X , φ) = 0, on the parameters of interest φ when X is made of the genotypes of the sample of individuals at a given locus E.g., in phylogeography, φ is composed of dates of divergence between populations, ratio of population sizes, mutation rates, etc. None of them are moments of the distribution of the allelic states of the sample
  109. 109. Moment condition in population genetics? EL does not require a fully defined and often complex (hence debatable) parametric model Main difficulty Derive a constraint EF h(X , φ) = 0, on the parameters of interest φ when X is made of the genotypes of the sample of individuals at a given locus c h made of pairwise composite scores (whose zero is the pairwise maximum likelihood estimator)
  110. 110. Pairwise composite likelihood The intra-locus pairwise likelihood j 2 (xk |φ) 2 (xk , xk |φ) i = i<j 1 n with xk , . . . , xk : allelic states of the gene sample at the k-th locus The pairwise score function j φ log 2 (xk |φ) φ log 2 (xk , xk |φ) i = i<j Composite likelihoods are often much narrower than the original likelihood of the model Safe(r) with EL because we only use position of its mode
  111. 111. Pairwise likelihood: a simple case 1 Assumptions 2 (δ|θ) =√ ρ (θ)|δ| 1 + 2θ sample ⊂ closed, panmictic with θ population at equilibrium ρ(θ) = √ 1 + θ + 1 + 2θ marker: microsatellite mutation rate: θ/2 Pairwise score function ∂θ log 2 (δ|θ) = i j 1 |δ| if xk et xk are two genes of the − + √ sample, 1 + 2θ θ 1 + 2θ j 2 (xk , xk |θ) i depends only on i j δ = xk − xk
  112. 112. Pairwise likelihood: a simple case 1 Assumptions 2 (δ|θ) =√ ρ (θ)|δ| 1 + 2θ sample ⊂ closed, panmictic with θ population at equilibrium ρ(θ) = √ 1 + θ + 1 + 2θ marker: microsatellite mutation rate: θ/2 Pairwise score function ∂θ log 2 (δ|θ) = i j 1 |δ| if xk et xk are two genes of the − + √ sample, 1 + 2θ θ 1 + 2θ j 2 (xk , xk |θ) i depends only on i j δ = xk − xk
  113. 113. Pairwise likelihood: a simple case 1 Assumptions 2 (δ|θ) =√ ρ (θ)|δ| 1 + 2θ sample ⊂ closed, panmictic with θ population at equilibrium ρ(θ) = √ 1 + θ + 1 + 2θ marker: microsatellite mutation rate: θ/2 Pairwise score function ∂θ log 2 (δ|θ) = i j 1 |δ| if xk et xk are two genes of the − + √ sample, 1 + 2θ θ 1 + 2θ j 2 (xk , xk |θ) i depends only on i j δ = xk − xk
  114. 114. Pairwise likelihood: 2 diverging populations MRCA Then 2 (δ|θ, τ) = +∞ e−τθ √ ρ(θ)|k| Iδ−k (τθ). τ 1 + 2θ k=−∞ where In (z) nth-order modified POP a POP b Bessel function of the first Assumptions kind τ: divergence date of pop. a and b θ/2: mutation rate i j Let xk and xk be two genes coming resp. from pop. a and b i j Set δ = xk − xk .
  115. 115. Pairwise likelihood: 2 diverging populations MRCA A 2-dim score function ∂τ log 2 (δ|θ, τ) = −θ+ τ θ 2 (δ − 1|θ, τ) + 2 (δ + 1|θ, τ) 2 2 (δ|θ, τ) POP a POP b ∂θ log 2 (δ|θ, τ) = 1 q(δ|θ, τ) Assumptions −τ − + + 1 + 2θ 2 (δ|θ, τ) τ 2 (δ − 1|θ, τ) + 2 (δ + 1|θ, τ) τ: divergence date of 2 2 (δ|θ, τ) pop. a and b θ/2: mutation rate where i j q(δ|θ, τ) := Let xk andxk be two genes ∞ e−τθ ρ (θ) coming resp. from pop. a and √ |k|ρ(θ)|k| Iδ−k (τθ) 1 + 2θ ρ(θ) b k=−∞ i j Set δ = xk − xk .
  116. 116. Example: normal posterior ABCel with two constraints ESS=155.6 ESS=75.93 ESS=76.87 4 2.0 3 Density Density Density 1.0 2 1.0 1 0.0 0.0 0 −0.5 0.0 0.5 1.0 −0.4 −0.2 0.0 0.2 0.4 −0.4 −0.2 0.0 0.2 θ ESS=91.54 θ ESS=108.4 θ ESS=85.13 4 0.0 1.0 2.0 3.0 0.0 1.0 2.0 3.0 3 Density Density Density 2 1 0 −0.6 −0.4 −0.2 0.0 0.2 −0.4 0.0 0.2 0.4 0.6 −0.2 0.0 0.2 0.4 0.6 θ ESS=149.1 θ ESS=96.31 θ ESS=83.77 4 2.0 3 2.0 Density Density Density 2 1.0 1.0 1 0.0 0.0 −0.5 0.0 0.5 1.0 −0.4 0.0 0.2 0.4 0.6 0 −0.6 −0.4 −0.2 0.0 0.2 0.4 θ ESS=155.7 θ ESS=92.42 θ ESS=95.01 0.0 1.0 2.0 3.0 2.0 3.0 Density Density Density 1.0 1.5 0.0 0.0 −0.5 0.0 0.5 −0.4 0.0 0.2 0.4 0.6 −0.4 0.0 0.2 0.4 0.6 θ ESS=139.2 θ ESS=99.33 θ ESS=87.28 2.0 0.0 1.0 2.0 3.0 3 Density Density Density 2 1.0 1 0.0 0 −0.6 −0.2 0.2 0.6 −0.4 −0.2 0.0 0.2 0.4 −0.2 0.0 0.2 0.4 0.6 Sample sizes are of 25 (column 1), 50 (column 2) and 75 (column 3) observations
  117. 117. Example: normal posterior ABCel with three constraints ESS=300.1 ESS=205.5 ESS=179.4 3.0 2.0 Density Density Density 1.0 1.5 1.0 0.0 0.0 0.0 −0.4 0.0 0.4 0.8 −0.6 −0.2 0.0 0.2 0.4 −0.2 0.0 0.2 0.4 θ ESS=265.1 θ ESS=250.3 θ ESS=134.8 4 4 2.0 Density Density Density 3 3 2 1.0 2 1 1 0.0 0 0 −0.3 −0.2 −0.1 0.0 0.1 −0.6 −0.4 −0.2 0.0 0.2 −0.4 −0.2 0.0 0.1 θ ESS=331.5 θ ESS=167.4 θ ESS=136.5 2.0 4 3 3 Density Density Density 1.0 2 2 1 1 0.0 0 −0.8 −0.4 0.0 0.4 −0.9 −0.7 −0.5 −0.3 0 −0.4 −0.2 0.0 0.2 θ ESS=322.4 θ ESS=202.7 θ ESS=166 0.0 1.0 2.0 3.0 4 2.0 3 Density Density Density 2 1.0 1 0.0 0 −0.2 0.0 0.2 0.4 0.6 0.8 −0.4 −0.2 0.0 0.2 0.4 −0.4 −0.2 0.0 0.2 θ ESS=263.7 θ ESS=190.9 θ ESS=165.3 3.0 3 2.0 Density Density Density 2 1.5 1.0 1 0.0 0.0 0 −1.0 −0.6 −0.2 −0.4 −0.2 0.0 0.2 0.4 0.6 −0.5 −0.3 −0.1 0.1 Sample sizes are of 25 (column 1), 50 (column 2) and 75 (column 3) observations
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×