ABC Methods for Bayesian Model Choice                 ABC Methods for Bayesian Model Choice                               ...
ABC Methods for Bayesian Model Choice  IntroductionIntroductory notions      Introduction      Approximate Bayesian comput...
ABC Methods for Bayesian Model Choice  IntroductionThe ABC of [Bayesian] statistics                 ”What meaning the like...
ABC Methods for Bayesian Model Choice  IntroductionThe ABC of Bayesian statistics                 ”Bayes’ Theorem can also...
ABC Methods for Bayesian Model Choice  IntroductionThe ABC of Bayesian statistics                 ”Bayes’ Theorem can also...
ABC Methods for Bayesian Model Choice  IntroductionA few details                 ”The main argument that p is not a [rando...
ABC Methods for Bayesian Model Choice  IntroductionA few details                 ”Laplace, who had a laissez-faire attitud...
ABC Methods for Bayesian Model Choice  IntroductionA few details                 ”If the number of experiments n is very l...
ABC Methods for Bayesian Model Choice  Introductionvon Mises’ conclusion                 ”We can only hope that statistici...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationApproximate Bayesian computation      Introduction ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsRegular Bayesian computation issues ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsUntractable likelihoods      Cases w...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsIllustrations      Example     Stoch...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsIllustrations      Example      Pott...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsIllustrations      Example      Infe...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsIllustrations      Example   Coalesc...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsThe ABC method      Bayesian setting...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsThe ABC method      Bayesian setting...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsThe ABC method      Bayesian setting...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsWhy does it work?!      The proof is...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsEarlier occurrence             ‘Baye...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsA as approximative      When y is a ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsA as approximative      When y is a ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsABC algorithm      Algorithm 1 Likel...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsOutput      The likelihood-free algo...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsOutput      The likelihood-free algo...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsPima Indian benchmark               ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsMA example      Consider the MA(q) m...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsMA example (2)      ABC algorithm th...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsMA example (2)      ABC algorithm th...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsComparison of distance impact      E...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsComparison of distance impact       ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsComparison of distance impact       ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsABC advances      Simulating from th...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsABC advances      Simulating from th...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsABC advances      Simulating from th...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     ABC basicsABC advances      Simulating from th...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABC-NP    Better usage of [prior]...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABC-MCMC      Markov chain (θ(t) ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABC-MCMC      Markov chain (θ(t) ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABC-MCMC (2)      Algorithm 2 Lik...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupWhy does it work?      Acceptance...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABCµ                       [Ratma...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABCµ                       [Ratma...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABCµ details      Multidimensiona...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABCµ details      Multidimensiona...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABCµ multiple errors             ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABCµ for model choice            ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupQuestions about ABCµ      For eac...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupQuestions about ABCµ      For eac...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupA PMC version      Use of the sam...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupSequential Monte Carlo      SMC i...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupABC-SMC      True derivation of a...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupProperties of ABC-SMC      The AB...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Alphabet soupProperties of ABC-SMC      The AB...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Calibration of ABCWhich summary statistics?   ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Calibration of ABCWhich summary statistics?   ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computation     Calibration of ABCWhich summary statistics?   ...
ABC Methods for Bayesian Model Choice  ABC for model choiceABC for model choice      Introduction      Approximate Bayesia...
ABC Methods for Bayesian Model Choice  ABC for model choice     Model choiceBayesian model choice      Several models M1 ,...
ABC Methods for Bayesian Model Choice  ABC for model choice     Model choiceGeneric ABC for model choice      Algorithm 3 ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Model choiceABC estimates      Posterior probability π(M =...
ABC Methods for Bayesian Model Choice  ABC for model choice     Model choiceABC estimates      Posterior probability π(M =...
ABC Methods for Bayesian Model Choice  ABC for model choice     Model choiceThe Great ABC controversy [# 1?]      On-going...
ABC Methods for Bayesian Model Choice  ABC for model choice     Model choiceThe Great ABC controversy [# 1?]     On-going ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsGibbs random fields      Gibbs distributi...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsGibbs random fields      Gibbs distributi...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsPotts model      Potts model      Vc (y)...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsPotts model      Potts model      Vc (y)...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsBayesian Model Choice      Comparing a m...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsNeighbourhood relations      Choice to b...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsModel index      Computational target:  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsModel index      Computational target:  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsSufficient statistics in Gibbs random fields
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsSufficient statistics in Gibbs random field...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsSufficient statistics in Gibbs random field...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsABC model choice Algorithm      ABC-MC  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsToy example      iid Bernoulli model ver...
ABC Methods for Bayesian Model Choice  ABC for model choice     Gibbs random fieldsToy example (2)                         ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceBack to sufficiency             ‘Suffi...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceBack to sufficiency             ‘Suffi...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceBack to sufficiency             ‘Suffi...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceLimiting behaviour of B12 (T → ∞) ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceLimiting behaviour of B12 (T → ∞) ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceLimiting behaviour of B12 ( → 0)  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceLimiting behaviour of B12 ( → 0)  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceLimiting behaviour of B12 (under s...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceLimiting behaviour of B12 (under s...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choicePoisson/geometric example      Sam...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choicePoisson/geometric discrepancy     ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceFormal recovery      Creating an e...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceFormal recovery      Creating an e...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceFormal recovery      Creating an e...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceMeaning of the ABC-Bayes factor   ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceMeaning of the ABC-Bayes factor   ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceMA(q) divergence                  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceMA(q) divergence                  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceFurther comments             ‘Ther...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceA population genetics evaluation  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceA population genetics evaluation  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceStability of importance sampling  ...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceComparison with ABC      Use of 24...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceComparison with ABC      Use of 15...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceComparison with ABC      Use of 24...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceThe only safe cases      Besides s...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceThe only safe cases      Besides s...
ABC Methods for Bayesian Model Choice  ABC for model choice     Generic ABC model choiceThe only safe cases      Besides s...
Upcoming SlideShare
Loading in...5
×

von Mises lecture, Berlin

871

Published on

Slides on ABC and ABC model choice, updated from Zürich with quotes from von Mises, defending the Bayesian approach versus the likelihood approach.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
871
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

von Mises lecture, Berlin

  1. 1. ABC Methods for Bayesian Model Choice ABC Methods for Bayesian Model Choice Christian P. Robert Universit´ Paris-Dauphine, IuF, & CREST e http://www.ceremade.dauphine.fr/~xian Richard von Mises Lecture 2011, Humboldt–Universit¨t zu a Berlin June 30, 2011
  2. 2. ABC Methods for Bayesian Model Choice IntroductionIntroductory notions Introduction Approximate Bayesian computation ABC for model choice
  3. 3. ABC Methods for Bayesian Model Choice IntroductionThe ABC of [Bayesian] statistics ”What meaning the likelihood may have in any case (...) is inconceivable to me.” — R. von Mises (1957) In a classical (Fisher, 1921) perspective, a statistical model is defined by the law of the observations, also called likelihood n e.g. L(θ|y1 , . . . , yn ) = L(θ|y) = f (yi |θ) i=1 Parameters θ are then estimated based on this function L(θ|y) and on the probabilistic properties of the distribution of the data.
  4. 4. ABC Methods for Bayesian Model Choice IntroductionThe ABC of Bayesian statistics ”Bayes’ Theorem can also be generalized in such a way as to apply to statistical functions.” – R. von Mises (1957) In the Bayesian approach (Bayes, 1763; Laplace, 1773), the parameter is endowed with a probability distribution as well, called the prior distribution and the likelihood becomes a conditional distribution of the data given the parameter, understood as a random variable.
  5. 5. ABC Methods for Bayesian Model Choice IntroductionThe ABC of Bayesian statistics ”Bayes’ Theorem can also be generalized in such a way as to apply to statistical functions.” – R. von Mises (1957) In the Bayesian approach (Bayes, 1763; Laplace, 1773), the parameter is endowed with a probability distribution as well, called the prior distribution and the likelihood becomes a conditional distribution of the data given the parameter, understood as a random variable. Inference then based on the posterior distribution, with density π(θ|y) ∝ π(θ) L(θ|y) Bayes’ Theorem (also called the posterior)
  6. 6. ABC Methods for Bayesian Model Choice IntroductionA few details ”The main argument that p is not a [random] variable but an ‘unknown constant’ does not mean anything to me.” — R. von Mises (1957) The parameter θ does not become a random variable (instead of an unknown constant) in the Bayesian paradygm. Probability calculus is used to quantify the uncertainty about θ as a calibrated quantity.
  7. 7. ABC Methods for Bayesian Model Choice IntroductionA few details ”Laplace, who had a laissez-faire attitude, deduced some simple rules used to an extent quite out of proportion to his modest starting point. Neither does he ever reveal how the sudden transition to real statistical events is to be made.” — R. von Mises (1957) The prior density π(·) is to be understood as a reference measure which, in informative situations, may summarise the available prior information.
  8. 8. ABC Methods for Bayesian Model Choice IntroductionA few details ”If the number of experiments n is very large, then we know that the influence of the initial probabilities is not very considerable.” — R. von Mises (1957) The impact of the prior density π(·) on the resulting inference is real but (mostly) vanishes when the number of observations grows. The only exception is the area of hypothesis testing where both approaches remain unreconcilable.
  9. 9. ABC Methods for Bayesian Model Choice Introductionvon Mises’ conclusion ”We can only hope that statisticians will return to the use of the simple, lucid reasoning of Bayes’ conceptions, and accord to the likelihood theory its proper role.” — R. von Mises (1957) [Berger and Wolpert, The Likelihood Principle, 1987]
  10. 10. ABC Methods for Bayesian Model Choice Approximate Bayesian computationApproximate Bayesian computation Introduction Approximate Bayesian computation ABC basics Alphabet soup Calibration of ABC ABC for model choice
  11. 11. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsRegular Bayesian computation issues When faced with a non-standard posterior distribution π(θ|y) ∝ π(θ)L(θ|y) the standard solution is to use simulation (Monte Carlo) to produce a sample θ1 , . . . , θ T from π(θ|y) (or approximately by Markov chain Monte Carlo methods) [Robert & Casella, 2004]
  12. 12. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsUntractable likelihoods Cases when the likelihood function f (y|θ) is unavailable and when the completion step f (y|θ) = f (y, z|θ) dz Z is impossible or too costly because of the dimension of z c MCMC cannot be implemented!
  13. 13. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsIllustrations Example Stochastic volatility model: for Highest weight trajectories t = 1, . . . , T, 0.4 0.2 yt = exp(zt ) t , zt = a+bzt−1 +σηt , 0.0 −0.2 T very large makes it difficult to −0.4 include z within the simulated 0 200 400 t 600 800 1000 parameters
  14. 14. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsIllustrations Example Potts model: if y takes values on a grid Y of size k n and f (y|θ) ∝ exp θ Iyl =yi l∼i where l∼i denotes a neighbourhood relation, n moderately large prohibits the computation of the normalising constant
  15. 15. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsIllustrations Example Inference on CMB: in cosmology, study of the Cosmic Microwave Background via likelihoods immensely slow to computate (e.g WMAP, Plank), because of numerically costly spectral transforms [Data is a Fortran program] [Kilbinger et al., 2010, MNRAS]
  16. 16. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsIllustrations Example Coalescence tree: in population genetics, reconstitution of a common ancestor from a sample of genes via a phylogenetic tree that is close to impossible to integrate out [100 processor days with 4 parameters] [Cornuet et al., 2009, Bioinformatics]
  17. 17. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsThe ABC method Bayesian setting: target is π(θ)f (x|θ)
  18. 18. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsThe ABC method Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique:
  19. 19. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsThe ABC method Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: ABC algorithm For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y. [Tavar´ et al., 1997] e
  20. 20. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsWhy does it work?! The proof is trivial: f (θi ) ∝ π(θi )f (z|θi )Iy (z) z∈D ∝ π(θi )f (y|θi ) = π(θi |y) . [Accept–Reject 101]
  21. 21. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsEarlier occurrence ‘Bayesian statistics and Monte Carlo methods are ideally suited to the task of passing many models over one dataset’ [Don Rubin, Annals of Statistics, 1984] Note Rubin (1984) does not promote this algorithm for likelihood-free simulation but frequentist intuition on posterior distributions: parameters from posteriors are more likely to be those that could have generated the data.
  22. 22. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsA as approximative When y is a continuous random variable, equality z = y is replaced with a tolerance condition, (y, z) ≤ where is a distance
  23. 23. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsA as approximative When y is a continuous random variable, equality z = y is replaced with a tolerance condition, (y, z) ≤ where is a distance Output distributed from π(θ) Pθ { (y, z) < } ∝ π(θ| (y, z) < )
  24. 24. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsABC algorithm Algorithm 1 Likelihood-free rejection sampler for i = 1 to N do repeat generate θ from the prior distribution π(·) generate z from the likelihood f (·|θ ) until ρ{η(z), η(y)} ≤ set θi = θ end for where η(y) defines a (maybe in-sufficient) statistic
  25. 25. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsOutput The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }.
  26. 26. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsOutput The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }. The idea behind ABC is that the summary statistics coupled with a small tolerance should provide a good approximation of the posterior distribution: π (θ|y) = π (θ, z|y)dz ≈ π(θ|y) .
  27. 27. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsPima Indian benchmark 80 100 1.0 80 60 0.8 60 0.6 Density Density Density 40 40 0.4 20 20 0.2 0.0 0 0 −0.005 0.010 0.020 0.030 −0.05 −0.03 −0.01 −1.0 0.0 1.0 2.0 Figure: Comparison between density estimates of the marginals on β1 (left), β2 (center) and β3 (right) from ABC rejection samples (red) and MCMC samples (black) .
  28. 28. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsMA example Consider the MA(q) model q xt = t+ ϑi t−i i=1 Simple prior: uniform prior over the identifiability zone, e.g. triangle for MA(2)
  29. 29. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsMA example (2) ABC algorithm thus made of 1. picking a new value (ϑ1 , ϑ2 ) in the triangle 2. generating an iid sequence ( t )−q<t≤T 3. producing a simulated series (xt )1≤t≤T
  30. 30. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsMA example (2) ABC algorithm thus made of 1. picking a new value (ϑ1 , ϑ2 ) in the triangle 2. generating an iid sequence ( t )−q<t≤T 3. producing a simulated series (xt )1≤t≤T Distance: basic distance between the series T ρ((xt )1≤t≤T , (xt )1≤t≤T ) = (xt − xt )2 t=1 or between summary statistics like the first q autocorrelations T τj = xt xt−j t=j+1
  31. 31. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsComparison of distance impact Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
  32. 32. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsComparison of distance impact 4 1.5 3 1.0 2 0.5 1 0.0 0 0.0 0.2 0.4 0.6 0.8 −2.0 −1.0 0.0 0.5 1.0 1.5 θ1 θ2 Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
  33. 33. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsComparison of distance impact 4 1.5 3 1.0 2 0.5 1 0.0 0 0.0 0.2 0.4 0.6 0.8 −2.0 −1.0 0.0 0.5 1.0 1.5 θ1 θ2 Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
  34. 34. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsABC advances Simulating from the prior is often poor in efficiency
  35. 35. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsABC advances Simulating from the prior is often poor in efficiency Either modify the proposal distribution on θ to increase the density of x’s within the vicinity of y... [Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
  36. 36. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsABC advances Simulating from the prior is often poor in efficiency Either modify the proposal distribution on θ to increase the density of x’s within the vicinity of y... [Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007] ...or by viewing the problem as a conditional density estimation and by developing techniques to allow for larger [Beaumont et al., 2002]
  37. 37. ABC Methods for Bayesian Model Choice Approximate Bayesian computation ABC basicsABC advances Simulating from the prior is often poor in efficiency Either modify the proposal distribution on θ to increase the density of x’s within the vicinity of y... [Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007] ...or by viewing the problem as a conditional density estimation and by developing techniques to allow for larger [Beaumont et al., 2002] .....or even by including in the inferential framework [ABCµ ] [Ratmann et al., 2009]
  38. 38. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABC-NP Better usage of [prior] simulations by adjustement: instead of throwing away θ such that ρ(η(z), η(y)) > , replace θs with locally regressed θ∗ = θ − {η(z) − η(y)}T β ˆ [Csill´ry et al., TEE, 2010] e ˆ where β is obtained by [NP] weighted least square regression on (η(z) − η(y)) with weights Kδ {ρ(η(z), η(y))} [Beaumont et al., 2002, Genetics]
  39. 39. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABC-MCMC Markov chain (θ(t) ) created via the transition function  θ ∼ Kω (θ |θ(t) ) if x ∼ f (x|θ ) is such that x = y   π(θ )Kω (t) |θ ) θ (t+1) = and u ∼ U(0, 1) ≤ π(θ(t) )K (θ |θ(t) ) ,  ω (θ  (t) θ otherwise,
  40. 40. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABC-MCMC Markov chain (θ(t) ) created via the transition function  θ ∼ Kω (θ |θ(t) ) if x ∼ f (x|θ ) is such that x = y   π(θ )Kω (t) |θ ) θ (t+1) = and u ∼ U(0, 1) ≤ π(θ(t) )K (θ |θ(t) ) ,  ω (θ  (t) θ otherwise, has the posterior π(θ|y) as stationary distribution [Marjoram et al, 2003]
  41. 41. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABC-MCMC (2) Algorithm 2 Likelihood-free MCMC sampler Use Algorithm 1 to get (θ(0) , z(0) ) for t = 1 to N do Generate θ from Kω ·|θ(t−1) , Generate z from the likelihood f (·|θ ), Generate u from U[0,1] , π(θ )Kω (θ(t−1) |θ ) if u ≤ I π(θ(t−1) Kω (θ |θ(t−1) ) A ,y (z ) then set (θ(t) , z(t) ) = (θ , z ) else (θ(t) , z(t) )) = (θ(t−1) , z(t−1) ), end if end for
  42. 42. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupWhy does it work? Acceptance probability that does not involve the calculation of the likelihood and π (θ , z |y) Kω (θ(t−1) |θ )f (z(t−1) |θ(t−1) ) × π (θ(t−1) , z(t−1) |y) Kω (θ |θ(t−1) )f (z |θ ) π(θ ) f (z |θ ) IA ,y (z ) = (t−1) ) f (z(t−1) |θ (t−1) )I (t−1) ) π(θ A ,y (z Kω (θ(t−1) |θ ) f (z(t−1) |θ(t−1) ) × Kω (θ |θ(t−1) ) f (z |θ ) π(θ )Kω (θ(t−1) |θ ) = IA (z ) . π(θ(t−1) Kω (θ |θ(t−1) ) ,y
  43. 43. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABCµ [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS] Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ (θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and x when z ∼ f (z|θ)
  44. 44. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABCµ [Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS] Use of a joint density f (θ, |y) ∝ ξ( |y, θ) × πθ (θ) × π ( ) where y is the data, and ξ( |y, θ) is the prior predictive density of ρ(η(z), η(y)) given θ and x when z ∼ f (z|θ) Warning! Replacement of ξ( |y, θ) with a non-parametric kernel approximation.
  45. 45. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABCµ details Multidimensional distances ρk (k = 1, . . . , K) and errors k = ρk (ηk (z), ηk (y)), with ˆ 1 k ∼ ξk ( |y, θ) ≈ ξk ( |y, θ) = K[{ k −ρk (ηk (zb ), ηk (y))}/hk ] Bhk b ˆ then used in replacing ξ( |y, θ) with mink ξk ( |y, θ)
  46. 46. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABCµ details Multidimensional distances ρk (k = 1, . . . , K) and errors k = ρk (ηk (z), ηk (y)), with ˆ 1 k ∼ ξk ( |y, θ) ≈ ξk ( |y, θ) = K[{ k −ρk (ηk (zb ), ηk (y))}/hk ] Bhk b ˆ then used in replacing ξ( |y, θ) with mink ξk ( |y, θ) ABCµ involves acceptance probability ˆ π(θ , ) q(θ , θ)q( , ) mink ξk ( |y, θ ) ˆ π(θ, ) q(θ, θ )q( , ) mink ξk ( |y, θ)
  47. 47. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABCµ multiple errors [ c Ratmann et al., PNAS, 2009]
  48. 48. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABCµ for model choice [ c Ratmann et al., PNAS, 2009]
  49. 49. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupQuestions about ABCµ For each model under comparison, marginal posterior on used to assess the fit of the model (HPD includes 0 or not).
  50. 50. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupQuestions about ABCµ For each model under comparison, marginal posterior on used to assess the fit of the model (HPD includes 0 or not). Is the data informative about ? [Identifiability] How is the prior π( ) impacting the comparison? How is using both ξ( |x0 , θ) and π ( ) compatible with a standard probability model? [remindful of Wilkinson] Where is the penalisation for complexity in the model comparison? [X, Mengersen & Chen, 2010, PNAS]
  51. 51. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupA PMC version Use of the same kernel idea as ABC-PRC but with IS correction Generate a sample at iteration t by N (t) (t−1) (t−1) πt (θ ) ∝ ˆ ωj Kt (θ(t) |θj ) j=1 modulo acceptance of the associated xt , and use an importance (t) weight associated with an accepted simulation θi (t) (t) (t) ωi ∝ π(θi ) πt (θi ) . ˆ c Still likelihood free [Beaumont et al., Biometrika, 2009]
  52. 52. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupSequential Monte Carlo SMC is a simulation technique to approximate a sequence of related probability distributions πn with π0 “easy” and πT target. Iterated IS as PMC: particles moved from time n to time n via kernel Kn and use of a sequence of extended targets πn˜ n πn (z0:n ) = πn (zn ) ˜ Lj (zj+1 , zj ) j=0 where the Lj ’s are backward Markov kernels [check that πn (zn ) is a marginal] [Del Moral, Doucet & Jasra, Series B, 2006]
  53. 53. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupABC-SMC True derivation of an SMC-ABC algorithm Use of a kernel Kn associated with target π n and derivation of the backward kernel π n (z )Kn (z , z) Ln−1 (z, z ) = πn (z) Update of the weights M m=1 IA n (xm ) in win ∝ wi(n−1) M m=1 IA n−1 (xm i(n−1) ) when xm ∼ K(xi(n−1) , ·) in
  54. 54. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupProperties of ABC-SMC The ABC-SMC method properly uses a backward kernel L(z, z ) to simplify the importance weight and to remove the dependence on the unknown likelihood from this weight. Update of importance weights is reduced to the ratio of the proportions of surviving particles Major assumption: the forward kernel K is supposed to be invariant against the true target [tempered version of the true posterior]
  55. 55. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Alphabet soupProperties of ABC-SMC The ABC-SMC method properly uses a backward kernel L(z, z ) to simplify the importance weight and to remove the dependence on the unknown likelihood from this weight. Update of importance weights is reduced to the ratio of the proportions of surviving particles Major assumption: the forward kernel K is supposed to be invariant against the true target [tempered version of the true posterior] Adaptivity in ABC-SMC algorithm only found in on-line construction of the thresholds t , slowly enough to keep a large number of accepted transitions [Del Moral, Doucet & Jasra, 2009]
  56. 56. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Calibration of ABCWhich summary statistics? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistic [except when done by the experimenters in the field]
  57. 57. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Calibration of ABCWhich summary statistics? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistic [except when done by the experimenters in the field] Starting from a large collection of summary statistics is available, Joyce and Marjoram (2008) consider the sequential inclusion into the ABC target, with a stopping rule based on a likelihood ratio test.
  58. 58. ABC Methods for Bayesian Model Choice Approximate Bayesian computation Calibration of ABCWhich summary statistics? Fundamental difficulty of the choice of the summary statistic when there is no non-trivial sufficient statistic [except when done by the experimenters in the field] Starting from a large collection of summary statistics is available, Joyce and Marjoram (2008) consider the sequential inclusion into the ABC target, with a stopping rule based on a likelihood ratio test. Does not taking into account the sequential nature of the tests Depends on parameterisation Order of inclusion matters.
  59. 59. ABC Methods for Bayesian Model Choice ABC for model choiceABC for model choice Introduction Approximate Bayesian computation ABC for model choice Model choice Gibbs random fields Generic ABC model choice
  60. 60. ABC Methods for Bayesian Model Choice ABC for model choice Model choiceBayesian model choice Several models M1 , M2 , . . . are considered simultaneously for a dataset y and the model index M is part of the inference. Use of a prior distribution. π(M = m), plus a prior distribution on the parameter conditional on the value m of the model index, πm (θ m ) Goal is to derive the posterior distribution of M , challenging computational target when models are complex.
  61. 61. ABC Methods for Bayesian Model Choice ABC for model choice Model choiceGeneric ABC for model choice Algorithm 3 Likelihood-free model choice sampler (ABC-MC) for t = 1 to T do repeat Generate m from the prior π(M = m) Generate θ m from the prior πm (θ m ) Generate z from the model fm (z|θ m ) until ρ{η(z), η(y)} < Set m(t) = m and θ (t) = θ m end for [Toni, Welch, Strelkowa, Ipsen & Stumpf, 2009]
  62. 62. ABC Methods for Bayesian Model Choice ABC for model choice Model choiceABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Early issues with implementation: should tolerances be the same for all models? should summary statistics vary across models? should the distance measure ρ vary as well?
  63. 63. ABC Methods for Bayesian Model Choice ABC for model choice Model choiceABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Early issues with implementation: should tolerances be the same for all models? should summary statistics vary across models? should the distance measure ρ vary as well? Extension to a weighted polychotomous logistic regression estimate of π(M = m|y), with non-parametric kernel weights [Cornuet et al., DIYABC, 2009]
  64. 64. ABC Methods for Bayesian Model Choice ABC for model choice Model choiceThe Great ABC controversy [# 1?] On-going controvery in phylogeographic genetics about the validity of using ABC for testing Against: Templeton, 2008, 2009, 2010a, 2010b, 2010c argues that nested hypotheses cannot have higher probabilities than nesting hypotheses (!)
  65. 65. ABC Methods for Bayesian Model Choice ABC for model choice Model choiceThe Great ABC controversy [# 1?] On-going controvery in phylogeographic genetics about the validity of using ABC for testing Replies: Fagundes et al., 2008, Against: Templeton, 2008, Beaumont et al., 2010, Berger et 2009, 2010a, 2010b, 2010c al., 2010, Csill`ry et al., 2010 e argues that nested hypotheses point out that the criticisms are cannot have higher probabilities addressed at [Bayesian] than nesting hypotheses (!) model-based inference and have nothing to do with ABC...
  66. 66. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsGibbs random fields Gibbs distribution The rv y = (y1 , . . . , yn ) is a Gibbs random field associated with the graph G if 1 f (y) = exp − Vc (yc ) , Z c∈C where Z is the normalising constant, C is the set of cliques of G and Vc is any function also called potential U (y) = c∈C Vc (yc ) is the energy function
  67. 67. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsGibbs random fields Gibbs distribution The rv y = (y1 , . . . , yn ) is a Gibbs random field associated with the graph G if 1 f (y) = exp − Vc (yc ) , Z c∈C where Z is the normalising constant, C is the set of cliques of G and Vc is any function also called potential U (y) = c∈C Vc (yc ) is the energy function c Z is usually unavailable in closed form
  68. 68. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsPotts model Potts model Vc (y) is of the form Vc (y) = θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure
  69. 69. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsPotts model Potts model Vc (y) is of the form Vc (y) = θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure In most realistic settings, summation Zθ = exp{θ T S(x)} x∈X involves too many terms to be manageable and numerical approximations cannot always be trusted
  70. 70. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsBayesian Model Choice Comparing a model with energy S0 taking values in Rp0 versus a model with energy S1 taking values in Rp1 can be done through the Bayes factor corresponding to the priors π0 and π1 on each parameter space exp{θ T S0 (x)}/Zθ 0 ,0 π0 (dθ 0 ) 0 Bm0 /m1 (x) = exp{θ T S1 (x)}/Zθ 1 ,1 π1 (dθ 1 ) 1
  71. 71. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsNeighbourhood relations Choice to be made between M neighbourhood relations m i∼i (0 ≤ m ≤ M − 1) with Sm (x) = I{xi =xi } m i∼i driven by the posterior probabilities of the models.
  72. 72. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsModel index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) , Θm
  73. 73. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsModel index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) , Θm If S(x) sufficient statistic for the joint parameters (M, θ0 , . . . , θM −1 ), P(M = m|x) = P(M = m|S(x)) .
  74. 74. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsSufficient statistics in Gibbs random fields
  75. 75. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsSufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient.
  76. 76. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsSufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient. For Gibbs random fields, 1 2 x|M = m ∼ fm (x|θm ) = fm (x|S(x))fm (S(x)|θm ) 1 = f 2 (S(x)|θm ) n(S(x)) m where n(S(x)) = {˜ ∈ X : S(˜ ) = S(x)} x x c S(x) is therefore also sufficient for the joint parameters
  77. 77. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsABC model choice Algorithm ABC-MC Generate m∗ from the prior π(M = m). ∗ Generate θm∗ from the prior πm∗ (·). Generate x∗ from the model fm∗ (·|θm∗ ). ∗ Compute the distance ρ(S(x0 ), S(x∗ )). Accept (θm∗ , m∗ ) if ρ(S(x0 ), S(x∗ )) < . ∗ Note When = 0 the algorithm is exact
  78. 78. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsToy example iid Bernoulli model versus two-state first-order Markov chain, i.e. n f0 (x|θ0 ) = exp θ0 I{xi =1} {1 + exp(θ0 )}n , i=1 versus n 1 f1 (x|θ1 ) = exp θ1 I{xi =xi−1 } {1 + exp(θ1 )}n−1 , 2 i=2 with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phase transition” boundaries).
  79. 79. ABC Methods for Bayesian Model Choice ABC for model choice Gibbs random fieldsToy example (2) 10 5 5 BF01 BF01 0 ^ ^ 0 −5 −5 −40 −20 0 10 −10 −40 −20 0 10 BF01 BF01 (left) Comparison of the true BF m0 /m1 (x0 ) with BF m0 /m1 (x0 ) (in logs) over 2, 000 simulations and 4.106 proposals from the prior. (right) Same when using tolerance corresponding to the 1% quantile on the distances.
  80. 80. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceBack to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability.’ [Scott Sisson, Jan. 31, 2011, X.’Og]
  81. 81. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceBack to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability.’ [Scott Sisson, Jan. 31, 2011, X.’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm )
  82. 82. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceBack to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability.’ [Scott Sisson, Jan. 31, 2011, X.’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm ) c Potential loss of information at the testing level
  83. 83. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceLimiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior
  84. 84. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceLimiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior As T go to infinity, limit Iρ{η(z),η(y)}≤ π1 (θ 1 )f1 (z|θ 1 ) dz dθ 1 B12 (y) = Iρ{η(z),η(y)}≤ π2 (θ 2 )f2 (z|θ 2 ) dz dθ 2 η Iρ{η,η(y)}≤ π1 (θ 1 )f1 (η|θ 1 ) dη dθ 1 = η , Iρ{η,η(y)}≤ π2 (θ 2 )f2 (η|θ 2 ) dη dθ 2 η η where f1 (η|θ 1 ) and f2 (η|θ 2 ) distributions of η(z)
  85. 85. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceLimiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η , π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2
  86. 86. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceLimiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η , π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 c Bayes factor based on the sole observation of η(y)
  87. 87. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceLimiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic for both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 g1 (y) η = η = B (y) . g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011]
  88. 88. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceLimiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic for both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 g1 (y) η = η = B (y) . g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011] c No discrepancy only when cross-model sufficiency
  89. 89. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choicePoisson/geometric example Sample x = (x1 , . . . , xn ) from either a Poisson P(λ) or from a geometric G(p) Then n S= yi = η(x) i=1 sufficient statistic for either model but not simultaneously Discrepancy ratio g1 (x) S!n−S / i yi ! = g2 (x) 1 n+S−1 S
  90. 90. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choicePoisson/geometric discrepancy η Range of B12 (x) versus B12 (x) B12 (x): The values produced have nothing in common.
  91. 91. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceFormal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011]
  92. 92. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceFormal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] In the Poisson/geometric case, if i xi ! is added to S, no discrepancy
  93. 93. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceFormal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] Only applies in genuine sufficiency settings... c Inability to evaluate loss brought by summary statistics
  94. 94. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceMeaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, X.’Og]
  95. 95. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceMeaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, X.’Og] In the Poisson/geometric case, if E[yi ] = θ0 > 0, η (θ0 + 1)2 −θ0 lim B12 (y) = e n→∞ θ0
  96. 96. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceMA(q) divergence 0.7 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factor equal to 17.71.
  97. 97. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceMA(q) divergence 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21 equal to .004.
  98. 98. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceFurther comments ‘There should be the possibility that for the same model, but different (non-minimal) [summary] statistics (so ∗ different η’s: η1 and η1 ) the ratio of evidences may no longer be equal to one.’ [Michael Stumpf, Jan. 28, 2011, ’Og] Using different summary statistics [on different models] may indicate the loss of information brought by each set but agreement does not lead to trustworthy approximations.
  99. 99. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceA population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter
  100. 100. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceA population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter 24 summary statistics 2 million ABC proposal importance [tree] sampling alternative
  101. 101. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceStability of importance sampling 1.0 1.0 1.0 1.0 1.0 q q 0.8 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.6 q 0.4 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0
  102. 102. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceComparison with ABC Use of 24 summary statistics and DIY-ABC logistic correction 1.0 q q q q q q q q q q q q q q q q qq q q 0.8 q q q q q qq q q q q q q qq q ABC estimates of P(M=1|y) q q q q q q qq qq q qq q q q q q 0.6 q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q qq 0.4 qq q q q q q q q q q 0.2 q q 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Importance Sampling estimates of P(M=1|y)
  103. 103. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceComparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction 1.0 q q q q qq q qq q qq q qqq qq q qq q q q q qq q q q q q q qq q q q q q 0.8 q q q q qq q q q q q q q q q ABC estimates of P(M=1|y) q q q q q q q q q q q q q q 0.6 q q q qq q q q q q q qq 0.4 q q q q q q q q q q q 0.2 q q q q q q q 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Importance Sampling estimates of P(M=1|y)
  104. 104. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceComparison with ABC Use of 24 summary statistics and DIY-ABC logistic correction 1.0 q q q q q q qq q q q qq q q q q q q q q q q q 0.8 qq q q q q q q q q q q q qq qq ABC estimates of P(M=1|y) q q q q q 0.6 q q q q q q q q q q q qq q q q q q q q q q q q q q q 0.4 q q q q q q q q q q q q q q q q q q q q 0.2 q q q q q q 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Importance Sampling estimates of P(M=1|y)
  105. 105. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceThe only safe cases Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010;Sousa et al., 2009]
  106. 106. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceThe only safe cases Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010;Sousa et al., 2009] ...and so does the use of more informal model fitting measures [Ratmann, Andrieu, Richardson and Wiujf, 2009]
  107. 107. ABC Methods for Bayesian Model Choice ABC for model choice Generic ABC model choiceThe only safe cases Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010;Sousa et al., 2009] ...and so does the use of more informal model fitting measures [Ratmann, Andrieu, Richardson and Wiujf, 2009] ...but moment conditions on the summary statistics are actually available to discriminate between “good” and “bad” summary statistics [Marin, Pillai, X. and Rousseau, 2011, in prep.]
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×