ABC Methods for Bayesian Model Choice                 ABC Methods for Bayesian Model Choice                               ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationApproximate Bayesian computation      Approximate B...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationRegular Bayesian computation issues      When faced...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationUntractable likelihoods      Cases when the likelih...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationUntractable likelihoods                            ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationThe ABC method      Bayesian setting: target is π(θ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationThe ABC method      Bayesian setting: target is π(θ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationThe ABC method      Bayesian setting: target is π(θ...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationA as approximative      When y is a continuous rand...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationA as approximative      When y is a continuous rand...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationABC algorithm      Algorithm 1 Likelihood-free reje...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationOutput      The likelihood-free algorithm samples f...
ABC Methods for Bayesian Model Choice  Approximate Bayesian computationOutput      The likelihood-free algorithm samples f...
ABC Methods for Bayesian Model Choice  ABC for model choiceABC for model choice      Approximate Bayesian computation     ...
ABC Methods for Bayesian Model Choice  ABC for model choiceBayesian model choice      Principle      Several models       ...
ABC Methods for Bayesian Model Choice  ABC for model choiceGeneric ABC for model choice      Algorithm 2 Likelihood-free m...
ABC Methods for Bayesian Model Choice  ABC for model choiceABC estimates      Posterior probability π(M = m|y) approximate...
ABC Methods for Bayesian Model Choice  ABC for model choiceABC estimates      Posterior probability π(M = m|y) approximate...
ABC Methods for Bayesian Model Choice  Gibbs random fieldsPotts model      Potts model      Distribution with an energy fun...
ABC Methods for Bayesian Model Choice  Gibbs random fieldsPotts model      Potts model      Distribution with an energy fun...
ABC Methods for Bayesian Model Choice  Gibbs random fieldsNeighbourhood relations      Setup      Choice to be made between...
ABC Methods for Bayesian Model Choice  Gibbs random fieldsModel index      Computational target:               P(M = m|x) ∝...
ABC Methods for Bayesian Model Choice  Gibbs random fieldsModel index      Computational target:               P(M = m|x) ∝...
ABC Methods for Bayesian Model Choice  Gibbs random fieldsSufficient statistics in Gibbs random fields
ABC Methods for Bayesian Model Choice  Gibbs random fieldsSufficient statistics in Gibbs random fields      Each model m has i...
ABC Methods for Bayesian Model Choice  Gibbs random fieldsSufficient statistics in Gibbs random fields      Each model m has i...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceMore about sufficiency             ‘Sufficient statistics for i...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceMore about sufficiency             ‘Sufficient statistics for i...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceMore about sufficiency             ‘Sufficient statistics for i...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceLimiting behaviour of B12 (T → ∞)      ABC approximation   ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceLimiting behaviour of B12 (T → ∞)      ABC approximation   ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceLimiting behaviour of B12 ( → 0)      When         goes to ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceLimiting behaviour of B12 ( → 0)      When         goes to ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceLimiting behaviour of B12 (under sufficiency)      If η(y) su...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceLimiting behaviour of B12 (under sufficiency)      If η(y) su...
ABC Methods for Bayesian Model Choice  Generic ABC model choicePoisson/geometric example      Sample                      ...
ABC Methods for Bayesian Model Choice  Generic ABC model choicePoisson/geometric discrepancy                              ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceFormal recovery      Creating an encompassing exponential f...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceFormal recovery      Creating an encompassing exponential f...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceFormal recovery      Creating an encompassing exponential f...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceMeaning of the ABC-Bayes factor             ‘This is also w...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceMeaning of the ABC-Bayes factor             ‘This is also w...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceMA example                                                 ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceMA example                                                 ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceA population genetics evaluation      Population genetics e...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceA population genetics evaluation      Population genetics e...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceStability of importance sampling                           ...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceComparison with ABC      Use of 24 summary statistics and D...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceComparison with ABC      Use of 15 summary statistics and D...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceComparison with ABC      Use of 15 summary statistics and D...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceThe only safe cases???      Besides specific models like Gib...
ABC Methods for Bayesian Model Choice  Generic ABC model choiceThe only safe cases???      Besides specific models like Gib...
ABC Methods for Bayesian Model Choice  Model choice consistencyABC model choice consistency      Approximate Bayesian comp...
ABC Methods for Bayesian Model Choice  Model choice consistencyThe starting point      Central question to the validation ...
ABC Methods for Bayesian Model Choice  Model choice consistencyThe starting point      Central question to the validation ...
ABC Methods for Bayesian Model Choice  Model choice consistencyA benchmark if toy example      Comparison suggested by ref...
ABC Methods for Bayesian Model Choice  Model choice consistencyA benchmark if toy example      Comparison suggested by ref...
ABC Methods for Bayesian Model Choice  Model choice consistencyA benchmark if toy example      Comparison suggested by ref...
ABC Methods for Bayesian Model Choice  Model choice consistencyA benchmark if toy example      Comparison suggested by ref...
ABC Methods for Bayesian Model Choice  Model choice consistencyFramework      Starting from sample y = (y1 , . . . , yn ) ...
ABC Methods for Bayesian Model Choice  Model choice consistencyFramework      Comparison of          – under M1 , y ∼ F1,n...
ABC Methods for Bayesian Model Choice  Model choice consistencyAssumptions      A collection of asymptotic “standard” assu...
ABC Methods for Bayesian Model Choice  Model choice consistencyAssumptions      A collection of asymptotic “standard” assu...
ABC Methods for Bayesian Model Choice  Model choice consistencyAssumptions      A collection of asymptotic “standard” assu...
ABC Methods for Bayesian Model Choice  Model choice consistencyAssumptions      A collection of asymptotic “standard” assu...
ABC Methods for Bayesian Model Choice  Model choice consistencyAssumptions      A collection of asymptotic “standard” assu...
ABC Methods for Bayesian Model Choice  Model choice consistencyAssumptions      A collection of asymptotic “standard” assu...
ABC Methods for Bayesian Model Choice  Model choice consistencyAsymptotic marginals      Asymptotically, under [A1]–[A5]  ...
ABC Methods for Bayesian Model Choice  Model choice consistencyWithin-model consistency      Under same assumptions, if in...
ABC Methods for Bayesian Model Choice  Model choice consistencyWithin-model consistency      Under same assumptions, if in...
ABC Methods for Bayesian Model Choice  Model choice consistencyBetween-model consistency      Consequence of above is that...
ABC Methods for Bayesian Model Choice  Model choice consistencyBetween-model consistency      Consequence of above is that...
ABC Methods for Bayesian Model Choice  Model choice consistencyConsistency theorem      If           inf{|µ0 − µ2 (θ2 )|; ...
ABC Methods for Bayesian Model Choice  Model choice consistencyConsistency theorem      If           inf{|µ0 − µ2 (θ2 )|; ...
ABC Methods for Bayesian Model Choice  Model choice consistencyConsequences on summary statistics      Bayes factor driven...
ABC Methods for Bayesian Model Choice  Model choice consistencyConsequences on summary statistics      Bayes factor driven...
ABC Methods for Bayesian Model Choice  Model choice consistencyConsequences on summary statistics      Bayes factor driven...
ABC Methods for Bayesian Model Choice  Model choice consistencyConsequences on summary statistics      Bayes factor driven...
ABC Methods for Bayesian Model Choice  Model choice consistencyEmbedded models      When M1 submodel of M2 , and if the tr...
ABC Methods for Bayesian Model Choice  Model choice consistencyEnvoi             happy birthday             gl¨cklich Gebu...
Upcoming SlideShare
Loading in...5
×

Colloquium in honor of Hans Ruedi Künsch

1,447

Published on

Slides of a 40mn talk in

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,447
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Colloquium in honor of Hans Ruedi Künsch

  1. 1. ABC Methods for Bayesian Model Choice ABC Methods for Bayesian Model Choice Christian P. Robert Universit´ Paris-Dauphine, IuF, & CREST e http://www.ceremade.dauphine.fr/~xian Colloquium in Honor of Hans-Ruedi K¨nsch, ETHZ, u Z¨rich, October 4, 2011 u Joint work(s) with Jean-Marie Cornuet, Jean-Michel Marin, Natesh Pillai, & Judith Rousseau
  2. 2. ABC Methods for Bayesian Model Choice Approximate Bayesian computationApproximate Bayesian computation Approximate Bayesian computation ABC for model choice Gibbs random fields Generic ABC model choice Model choice consistency
  3. 3. ABC Methods for Bayesian Model Choice Approximate Bayesian computationRegular Bayesian computation issues When faced with a non-standard posterior distribution π(θ|y) ∝ π(θ)L(θ|y) the standard solution is to use simulation (Monte Carlo) to produce a sample θ1 , . . . , θ T from π(θ|y) (or approximately by Markov chain Monte Carlo methods) [Robert & Casella, 2004]
  4. 4. ABC Methods for Bayesian Model Choice Approximate Bayesian computationUntractable likelihoods Cases when the likelihood function f (y|θ) is unavailable and when the completion step f (y|θ) = f (y, z|θ) dz Z is impossible or too costly because of the dimension of z c MCMC cannot be implemented!
  5. 5. ABC Methods for Bayesian Model Choice Approximate Bayesian computationUntractable likelihoods c MCMC cannot be implemented!
  6. 6. ABC Methods for Bayesian Model Choice Approximate Bayesian computationThe ABC method Bayesian setting: target is π(θ)f (x|θ)
  7. 7. ABC Methods for Bayesian Model Choice Approximate Bayesian computationThe ABC method Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique:
  8. 8. ABC Methods for Bayesian Model Choice Approximate Bayesian computationThe ABC method Bayesian setting: target is π(θ)f (x|θ) When likelihood f (x|θ) not in closed form, likelihood-free rejection technique: ABC algorithm For an observation y ∼ f (y|θ), under the prior π(θ), keep jointly simulating θ ∼ π(θ) , z ∼ f (z|θ ) , until the auxiliary variable z is equal to the observed value, z = y. [Rubin, 1984; Tavar´ et al., 1997] e
  9. 9. ABC Methods for Bayesian Model Choice Approximate Bayesian computationA as approximative When y is a continuous random variable, equality z = y is replaced with a tolerance condition, (y, z) ≤ where is a distance
  10. 10. ABC Methods for Bayesian Model Choice Approximate Bayesian computationA as approximative When y is a continuous random variable, equality z = y is replaced with a tolerance condition, (y, z) ≤ where is a distance Output distributed from π(θ) Pθ { (y, z) < } ∝ π(θ| (y, z) < )
  11. 11. ABC Methods for Bayesian Model Choice Approximate Bayesian computationABC algorithm Algorithm 1 Likelihood-free rejection sampler for i = 1 to N do repeat generate θ from the prior distribution π(·) generate z from the likelihood f (·|θ ) until ρ{η(z), η(y)} ≤ set θi = θ end for where η(y) defines a (maybe in-sufficient) statistic
  12. 12. ABC Methods for Bayesian Model Choice Approximate Bayesian computationOutput The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }.
  13. 13. ABC Methods for Bayesian Model Choice Approximate Bayesian computationOutput The likelihood-free algorithm samples from the marginal in z of: π(θ)f (z|θ)IA ,y (z) π (θ, z|y) = , A ,y ×Θ π(θ)f (z|θ)dzdθ where A ,y = {z ∈ D|ρ(η(z), η(y)) < }. The idea behind ABC is that the summary statistics coupled with a small tolerance should provide a good approximation of the posterior distribution: π (θ|y) = π (θ, z|y)dz ≈ π(θ|y) . [Not garanteed!]
  14. 14. ABC Methods for Bayesian Model Choice ABC for model choiceABC for model choice Approximate Bayesian computation ABC for model choice Gibbs random fields Generic ABC model choice Model choice consistency
  15. 15. ABC Methods for Bayesian Model Choice ABC for model choiceBayesian model choice Principle Several models M1 , M2 , . . . are considered simultaneously for dataset y and model index M central to inference. Use of a prior π(M = m), plus a prior distribution on the parameter conditional on the value m of the model index, πm (θ m ) Goal is to derive the posterior distribution of M, π(M = m|data) a challenging computational target when models are complex.
  16. 16. ABC Methods for Bayesian Model Choice ABC for model choiceGeneric ABC for model choice Algorithm 2 Likelihood-free model choice sampler (ABC-MC) for t = 1 to T do repeat Generate m from the prior π(M = m) Generate θ m from the prior πm (θ m ) Generate z from the model fm (z|θ m ) until ρ{η(z), η(y)} < Set m(t) = m and θ (t) = θ m end for [Grelaud & al., 2009; Toni & al., 2009]
  17. 17. ABC Methods for Bayesian Model Choice ABC for model choiceABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Early issues with implementation: should tolerances be the same for all models? should summary statistics vary across models? incl. their dimension? should the distance measure ρ vary across models?
  18. 18. ABC Methods for Bayesian Model Choice ABC for model choiceABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Extension to a weighted polychotomous logistic regression estimate of π(M = m|y), with non-parametric kernel weights [Cornuet et al., DIYABC, 2009]
  19. 19. ABC Methods for Bayesian Model Choice Gibbs random fieldsPotts model Potts model Distribution with an energy function of the form θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure
  20. 20. ABC Methods for Bayesian Model Choice Gibbs random fieldsPotts model Potts model Distribution with an energy function of the form θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure In most realistic settings, summation Zθ = exp{θ T S(x)} x∈X involves too many terms to be manageable and numerical approximations cannot always be trusted
  21. 21. ABC Methods for Bayesian Model Choice Gibbs random fieldsNeighbourhood relations Setup Choice to be made between M neighbourhood relations m i∼i (0 ≤ m ≤ M − 1) with Sm (x) = I{xi =xi } m i∼i driven by the posterior probabilities of the models.
  22. 22. ABC Methods for Bayesian Model Choice Gibbs random fieldsModel index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) Θm
  23. 23. ABC Methods for Bayesian Model Choice Gibbs random fieldsModel index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) Θm If S(x) sufficient statistic for the joint parameters (M, θ0 , . . . , θM −1 ), P(M = m|x) = P(M = m|S(x)) .
  24. 24. ABC Methods for Bayesian Model Choice Gibbs random fieldsSufficient statistics in Gibbs random fields
  25. 25. ABC Methods for Bayesian Model Choice Gibbs random fieldsSufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient.
  26. 26. ABC Methods for Bayesian Model Choice Gibbs random fieldsSufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient. Explanation: For Gibbs random fields, 1 2 x|M = m ∼ fm (x|θm ) = fm (x|S(x))fm (S(x)|θm ) 1 = f 2 (S(x)|θm ) n(S(x)) m where n(S(x)) = {˜ ∈ X : S(˜ ) = S(x)} x x c S(x) is sufficient for the joint parameters
  27. 27. ABC Methods for Bayesian Model Choice Generic ABC model choiceMore about sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og]
  28. 28. ABC Methods for Bayesian Model Choice Generic ABC model choiceMore about sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm )
  29. 29. ABC Methods for Bayesian Model Choice Generic ABC model choiceMore about sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm ) c Potential loss of information at the testing level
  30. 30. ABC Methods for Bayesian Model Choice Generic ABC model choiceLimiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior
  31. 31. ABC Methods for Bayesian Model Choice Generic ABC model choiceLimiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior As T go to infinity, limit Iρ{η(z),η(y)}≤ π1 (θ 1 )f1 (z|θ 1 ) dz dθ 1 B12 (y) = Iρ{η(z),η(y)}≤ π2 (θ 2 )f2 (z|θ 2 ) dz dθ 2 η Iρ{η,η(y)}≤ π1 (θ 1 )f1 (η|θ 1 ) dη dθ 1 = η , Iρ{η,η(y)}≤ π2 (θ 2 )f2 (η|θ 2 ) dη dθ 2 η η where f1 (η|θ 1 ) and f2 (η|θ 2 ) distributions of η(z)
  32. 32. ABC Methods for Bayesian Model Choice Generic ABC model choiceLimiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2
  33. 33. ABC Methods for Bayesian Model Choice Generic ABC model choiceLimiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 c Bayes factor based on the sole observation of η(y)
  34. 34. ABC Methods for Bayesian Model Choice Generic ABC model choiceLimiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic in both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 g1 (y) η = η = B (y) . g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011]
  35. 35. ABC Methods for Bayesian Model Choice Generic ABC model choiceLimiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic in both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 g1 (y) η = η = B (y) . g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011] c No discrepancy only when cross-model sufficiency
  36. 36. ABC Methods for Bayesian Model Choice Generic ABC model choicePoisson/geometric example Sample x = (x1 , . . . , xn ) from either a Poisson P(λ) or from a geometric G(p) Sum n S= xi = η(x) i=1 sufficient statistic for either model but not simultaneously Discrepancy ratio g1 (x) S!n−S / i xi ! = g2 (x) 1 n+S−1 S
  37. 37. ABC Methods for Bayesian Model Choice Generic ABC model choicePoisson/geometric discrepancy η Range of B12 (x) versus B12 (x): The values produced have nothing in common.
  38. 38. ABC Methods for Bayesian Model Choice Generic ABC model choiceFormal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011]
  39. 39. ABC Methods for Bayesian Model Choice Generic ABC model choiceFormal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] In the Poisson/geometric case, if i xi ! is added to S, no discrepancy
  40. 40. ABC Methods for Bayesian Model Choice Generic ABC model choiceFormal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] Only applies in genuine sufficiency settings... c Inability to evaluate loss brought by summary statistics
  41. 41. ABC Methods for Bayesian Model Choice Generic ABC model choiceMeaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, ’Og]
  42. 42. ABC Methods for Bayesian Model Choice Generic ABC model choiceMeaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, ’Og] In the Poisson/geometric case, if E[yi ] = θ0 > 0, η (θ0 + 1)2 −θ0 lim B12 (y) = e n→∞ θ0
  43. 43. ABC Methods for Bayesian Model Choice Generic ABC model choiceMA example 0.7 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factor equal to 17.71.
  44. 44. ABC Methods for Bayesian Model Choice Generic ABC model choiceMA example 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21 equal to .004.
  45. 45. ABC Methods for Bayesian Model Choice Generic ABC model choiceA population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter
  46. 46. ABC Methods for Bayesian Model Choice Generic ABC model choiceA population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter 24 summary statistics 2 million ABC proposal importance [tree] sampling alternative
  47. 47. ABC Methods for Bayesian Model Choice Generic ABC model choiceStability of importance sampling 1.0 1.0 1.0 1.0 1.0 q q 0.8 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.6 q 0.4 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0
  48. 48. ABC Methods for Bayesian Model Choice Generic ABC model choiceComparison with ABC Use of 24 summary statistics and DIY-ABC logistic correction 1.0 qq qq qq qq qqq qq q q qq q q qq q q q q q q q q q q qq q q qq q q q qq qq qq q q q q qq q q q q q q q q q q q qq q q q qq q q 0.8 q q q q q q q q q q q q qq q q q q q q q qq q q q q q qqq q q qq q ABC direct and logistic q q qq q q q q q q q 0.6 q q q qq q q qq q q q q q q q q q q q q q q q q q q q q q q q q q qq qq q q qq 0.4 qq q q q q q q q q q q q q q q q q q 0.2 q qq q q q q q q q q q q 0.0 qq 0.0 0.2 0.4 0.6 0.8 1.0 importance sampling
  49. 49. ABC Methods for Bayesian Model Choice Generic ABC model choiceComparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction 6 4 q q q q q q q qq 2 q q q q qq ABC direct q q q qq q qq q q q q qq qq q qq q q q q q qq q q q q q qq q q qq q qq q q qq q q q qqq q q q q q q q q 0 qq qq q q q q qq qq q q qq q q q q q q q q −2 −4 −4 −2 0 2 4 6 importance sampling
  50. 50. ABC Methods for Bayesian Model Choice Generic ABC model choiceComparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction qq 6 q q q q q q q q q q q 4 q qq q qq qq q q q q q q q q q q qq q q q q q q q ABC direct and logistic q q q q q q q qq q q q q q q q qq q 2 q q q q q qq q qq q q q qq qq q qq qq q q q qq q q qq qq qq q q qqqq qq q q q qqqqq qq q q q qq q q q q qq q qq q q qqq q q qqqqq q q qq q q qq q q q qq 0 qq qqq q q qq q qqq q q q q q q q q q q q q q q q q q q q q qq q q −2 q q q q q −4 q q q −4 −2 0 2 4 6 importance sampling
  51. 51. ABC Methods for Bayesian Model Choice Generic ABC model choiceThe only safe cases??? Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010; Sousa & al., 2009]
  52. 52. ABC Methods for Bayesian Model Choice Generic ABC model choiceThe only safe cases??? Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010; Sousa & al., 2009] ...and so does the use of more informal model fitting measures [Ratmann & al., 2009]
  53. 53. ABC Methods for Bayesian Model Choice Model choice consistencyABC model choice consistency Approximate Bayesian computation ABC for model choice Gibbs random fields Generic ABC model choice Model choice consistency
  54. 54. ABC Methods for Bayesian Model Choice Model choice consistencyThe starting point Central question to the validation of ABC for model choice: When is a Bayes factor based on an insufficient statistic T (y) consistent?
  55. 55. ABC Methods for Bayesian Model Choice Model choice consistencyThe starting point Central question to the validation of ABC for model choice: When is a Bayes factor based on an insufficient statistic T (y) consistent? T Note: conclusion drawn on T (y) through B12 (y) necessarily differs from the conclusion drawn on y through B12 (y)
  56. 56. ABC Methods for Bayesian Model Choice Model choice consistencyA benchmark if toy example Comparison suggested by referee of PNAS paper: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1 : y ∼ N (θ1 , 1) opposed to model M2 : √ y ∼ L(θ2 , 1/ √2), Laplace distribution with mean θ2 and scale parameter 1/ 2 (variance one).
  57. 57. ABC Methods for Bayesian Model Choice Model choice consistencyA benchmark if toy example Comparison suggested by referee of PNAS paper: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1 : y ∼ N (θ1 , 1) opposed to model M2 : √ y ∼ L(θ2 , 1/ √2), Laplace distribution with mean θ2 and scale parameter 1/ 2 (variance one). Four possible statistics 1. sample mean y (sufficient for M1 if not M2 ); 2. sample median med(y) (insufficient); 3. sample variance var(y) (ancillary); 4. median absolute deviation mad(y) = med(y − med(y));
  58. 58. ABC Methods for Bayesian Model Choice Model choice consistencyA benchmark if toy example Comparison suggested by referee of PNAS paper: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1 : y ∼ N (θ1 , 1) opposed to model M2 : √ y ∼ L(θ2 , 1/ √2), Laplace distribution with mean θ2 and scale parameter 1/ 2 (variance one). 6 5 4 Density 3 2 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 posterior probability
  59. 59. ABC Methods for Bayesian Model Choice Model choice consistencyA benchmark if toy example Comparison suggested by referee of PNAS paper: [X, Cornuet, Marin, & Pillai, Aug. 2011] Model M1 : y ∼ N (θ1 , 1) opposed to model M2 : √ y ∼ L(θ2 , 1/ √2), Laplace distribution with mean θ2 and scale parameter 1/ 2 (variance one). 6 3 5 4 Density Density 2 3 2 1 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.0 0.2 0.4 0.6 0.8 1.0 posterior probability probability
  60. 60. ABC Methods for Bayesian Model Choice Model choice consistencyFramework Starting from sample y = (y1 , . . . , yn ) be the observed sample, not necessarily iid with true distribution y ∼ Pn Summary statistics T (y) = T n = (T1 (y), T2 (y), · · · , Td (y)) ∈ Rd with true distribution T n ∼ Gn .
  61. 61. ABC Methods for Bayesian Model Choice Model choice consistencyFramework Comparison of – under M1 , y ∼ F1,n (·|θ1 ) where θ1 ∈ Θ1 ⊂ Rp1 – under M2 , y ∼ F2,n (·|θ2 ) where θ2 ∈ Θ2 ⊂ Rp2 turned into – under M1 , T (y) ∼ G1,n (·|θ1 ), and θ1 |T (y) ∼ π1 (·|T n ) – under M2 , T (y) ∼ G2,n (·|θ2 ), and θ2 |T (y) ∼ π2 (·|T n )
  62. 62. ABC Methods for Bayesian Model Choice Model choice consistencyAssumptions A collection of asymptotic “standard” assumptions: [A1] There exist a sequence {vn } converging to +∞, an a.c. distribution Q with continuous bounded density q(·), a symmetric, d × d positive definite matrix V0 and a vector µ0 ∈ Rd such that −1/2 n→∞ vn V0 (T n − µ0 ) Q, under Gn and for all M > 0 −d −1/2 sup |V0 |1/2 vn gn (t) − q vn V0 {t − µ0 } = o(1) vn |t−µ0 |<M
  63. 63. ABC Methods for Bayesian Model Choice Model choice consistencyAssumptions A collection of asymptotic “standard” assumptions: [A2] For i = 1, 2, there exist d × d symmetric positive definite matrices Vi (θi ) and µi (θi ) ∈ Rd such that n→∞ vn Vi (θi )−1/2 (T n − µi (θi )) Q, under Gi,n (·|θi ) .
  64. 64. ABC Methods for Bayesian Model Choice Model choice consistencyAssumptions A collection of asymptotic “standard” assumptions: [A3] For i = 1, 2, there exist sets Fn,i ⊂ Θi and constants i , τi , αi >0 such that for all τ > 0, sup Gi,n |T n − µ(θi )| > τ |µi (θi ) − µ0 | ∧ i |θi θi ∈Fn,i −α −αi vn i (|µi (θi ) − µ0 | ∧ i ) with c −τ πi (Fn,i ) = o(vn i ).
  65. 65. ABC Methods for Bayesian Model Choice Model choice consistencyAssumptions A collection of asymptotic “standard” assumptions: [A4] For (u > 0) −1 Sn,i (u) = θi ∈ Fn,i ; |µ(θi ) − µ0 | ≤ u vn if inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0, there exist constants di < τi ∧ αi − 1 such that −d πi (Sn,i (u)) ∼ udi vn i , ∀u vn
  66. 66. ABC Methods for Bayesian Model Choice Model choice consistencyAssumptions A collection of asymptotic “standard” assumptions: [A5] If inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0, there exists U > 0 such that for any M > 0, −d sup sup |Vi (θi )|1/2 vn gi (t|θi ) vn |t−µ0 |<M θi ∈Sn,i (U ) −q vn Vi (θi )−1/2 (t − µ(θi ) = o(1) and πi Sn,i (U ) ∩ ||Vi (θi )−1 || + ||Vi (θi )|| > M lim lim sup =0. M →∞ n πi (Sn,i (U ))
  67. 67. ABC Methods for Bayesian Model Choice Model choice consistencyAssumptions A collection of asymptotic “standard” assumptions: [A1]–[A2] are standard central limit theorems [A3] controls the large deviations of the estimator T n from the estimand µ(θ) [A4] is the standard prior mass condition found in Bayesian asymptotics (di effective dimension of the parameter) [A5] controls more tightly convergence esp. when µi is not one-to-one
  68. 68. ABC Methods for Bayesian Model Choice Model choice consistencyAsymptotic marginals Asymptotically, under [A1]–[A5] mi (t) = gi (t|θi ) πi (θi ) dθi Θi is such that (i) if inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0, Cl vn i ≤ mi (T n ) ≤ Cu vn i d−d d−d and (ii) if inf{|µi (θi ) − µ0 |; θi ∈ Θi } > 0 mi (T n ) = oPn [vn i + vn i ]. d−τ d−α
  69. 69. ABC Methods for Bayesian Model Choice Model choice consistencyWithin-model consistency Under same assumptions, if inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0, the posterior distribution of µi (θi ) given T n is consistent at rate 1/vn provided αi ∧ τi > di .
  70. 70. ABC Methods for Bayesian Model Choice Model choice consistencyWithin-model consistency Under same assumptions, if inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0, the posterior distribution of µi (θi ) given T n is consistent at rate 1/vn provided αi ∧ τi > di . Note: di can be seen as an effective dimension of the model under the posterior πi (.|T n ), since if µ0 ∈ {µi (θi ); θi ∈ Θi }, mi (T n ) ∼ vn i d−d
  71. 71. ABC Methods for Bayesian Model Choice Model choice consistencyBetween-model consistency Consequence of above is that asymptotic behaviour of the Bayes factor is driven by the asymptotic mean value of T n under both models. And only by this mean value! Indeed, if inf{|µ0 − µ2 (θ2 )|; θ2 ∈ Θ2 } = inf{|µ0 − µ1 (θ1 )|; θ1 ∈ Θ1 } = 0 then m1 (T n ) Cl vn 1 −d2 ) ≤ −(d ≤ Cu vn 1 −d2 ) , −(d m2 (T n ) where Cl , Cu = OPn (1), irrespective of the true model. Only depends on the difference d1 − d2
  72. 72. ABC Methods for Bayesian Model Choice Model choice consistencyBetween-model consistency Consequence of above is that asymptotic behaviour of the Bayes factor is driven by the asymptotic mean value of T n under both models. And only by this mean value! Else, if inf{|µ0 − µ2 (θ2 )|; θ2 ∈ Θ2 } > inf{|µ0 − µ1 (θ1 )|; θ1 ∈ Θ1 } = 0 then m1 (T n ) ≥ Cu min vn 1 −α2 ) , vn 1 −τ2 ) , −(d −(d m2 (T n )
  73. 73. ABC Methods for Bayesian Model Choice Model choice consistencyConsistency theorem If inf{|µ0 − µ2 (θ2 )|; θ2 ∈ Θ2 } = inf{|µ0 − µ1 (θ1 )|; θ1 ∈ Θ1 } = 0, T −(d −d ) Bayes factor B12 is O(vn 1 2 ) irrespective of the true model. It is consistent iff Pn is within the model with the smallest dimension
  74. 74. ABC Methods for Bayesian Model Choice Model choice consistencyConsistency theorem If inf{|µ0 − µ2 (θ2 )|; θ2 ∈ Θ2 } = inf{|µ0 − µ1 (θ1 )|; θ1 ∈ Θ1 } = 0, T −(d −d ) Bayes factor B12 is O(vn 1 2 ) irrespective of the true model. It is consistent iff Pn is within the model with the smallest dimension If Pn belongs to one of the two models and if µ0 cannot be attained by the other one : 0 = min (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2) < max (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2) , T then the Bayes factor B12 is consistent
  75. 75. ABC Methods for Bayesian Model Choice Model choice consistencyConsequences on summary statistics Bayes factor driven by the means µi (θi ) and the relative position of µ0 wrt both sets {µi (θi ); θi ∈ Θi }, i = 1, 2. For ABC, this implies the most likely statistics T n are ancillary statistics with different mean values under both models Else, if T n asymptotically depends on some of the parameters of the models, it is quite likely that there exists θi ∈ Θi such that µi (θi ) = µ0 even though model M1 is misspecified
  76. 76. ABC Methods for Bayesian Model Choice Model choice consistencyConsequences on summary statistics Bayes factor driven by the means µi (θi ) and the relative position of µ0 wrt both sets {µi (θi ); θi ∈ Θi }, i = 1, 2. Toy example Laplace versus Gauss: If n T n = n−1 Xi4 i=1 and the true distribution is Laplace with mean 0, so that µ0 = 6. Since under the Gaussian model µ(θ) = 3 + θ4 + 6θ2 √ the value θ∗ = 2 3 − 3 leads to µ0 = µ(θ∗ ) and a Bayes factor associated with such a statistic is not consistent (here d1 = d2 = d = 1).
  77. 77. ABC Methods for Bayesian Model Choice Model choice consistencyConsequences on summary statistics Bayes factor driven by the means µi (θi ) and the relative position of µ0 wrt both sets {µi (θi ); θi ∈ Θi }, i = 1, 2. 8 6 4 2 0 0.0 0.2 0.4 0.6 0.8 1.0 probability Fourth moment
  78. 78. ABC Methods for Bayesian Model Choice Model choice consistencyConsequences on summary statistics Bayes factor driven by the means µi (θi ) and the relative position of µ0 wrt both sets {µi (θi ); θi ∈ Θi }, i = 1, 2. 6 5 4 3 2 1 0 0.0 0.2 0.4 0.6 0.8 1.0 probability Fourth and sixth moments
  79. 79. ABC Methods for Bayesian Model Choice Model choice consistencyEmbedded models When M1 submodel of M2 , and if the true distribution belongs to −(d −d ) the smaller model M1 , Bayes factor is of order vn 1 2 . If summary statistic only informative on a parameter that is the same under both models, i.e if d1 = d2 , then the Bayes factor is not consistent Else, d1 < d2 and Bayes factor is consistent under M1 . If true distribution not in M1 , then Bayes factor is consistent only if µ 1 = µ 2 = µ0
  80. 80. ABC Methods for Bayesian Model Choice Model choice consistencyEnvoi happy birthday gl¨cklich Geburtstag u joyeux anniversaire Hans!!!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×