Christian Robert, profile picture
Uploaded byChristian Robert
PDF, PPTX703 views

ABC in London, May 5, 2011

ABC methods can be used for Bayesian model choice between multiple models. The ABC model choice algorithm samples models from their prior probabilities and parameters from the corresponding prior distributions. Simulated data is generated and accepted if the distance between summary statistics of the simulated and observed data is below a tolerance level. As the number of simulations increases, the ABC approximation of the Bayes factor converges to the true Bayes factor. Sufficient statistics for individual models may not be sufficient for the joint model and parameters.

Embed presentation

Download as PDF, PPTX
ABC Methods for Bayesian Model Choice ABC Methods for Bayesian Model Choice Christian P. Robert Universit´ Paris-Dauphine, IuF, & CREST e http://www.ceremade.dauphine.fr/~xian May 4, 2011
ABC Methods for Bayesian Model Choice ABC for model choice Model choice Gibbs random fields Generic ABC model choice
ABC Methods for Bayesian Model Choice Model choice Bayesian model choice Several models M1 , M2 , . . . are considered simultaneously for a dataset y and the model index M is part of the inference. Use of a prior distribution. π(M = m), plus a prior distribution on the parameter conditional on the value m of the model index, πm (θ m ) Goal is to derive the posterior distribution of M , challenging computational target when models are complex.
ABC Methods for Bayesian Model Choice Model choice Generic ABC for model choice Algorithm 1 Likelihood-free model choice sampler (ABC-MC) for t = 1 to T do repeat Generate m from the prior π(M = m) Generate θ m from the prior πm (θ m ) Generate z from the model fm (z|θ m ) until ρ{η(z), η(y)} < Set m(t) = m and θ (t) = θ m end for [Toni, Welch, Strelkowa, Ipsen & Stumpf, 2009]
ABC Methods for Bayesian Model Choice Model choice ABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Early issues with implementation: should tolerances be the same for all models? should summary statistics vary across models? (incl. their dimension)? should the distance measure ρ vary as well?
ABC Methods for Bayesian Model Choice Model choice ABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Early issues with implementation: then needs to become part of the model this is incompatible with Bayesian model choice same difficulty Extension to a weighted polychotomous logistic regression estimate of π(M = m|y), with non-parametric kernel weights [Cornuet et al., DIYABC, 2009]
ABC Methods for Bayesian Model Choice Model choice The Great ABC controversy On-going controvery in phylogeographic genetics about the validity of using ABC for testing Against: Templeton, 2008, 2009, 2010a, 2010b, 2010c argues that nested hypotheses cannot have higher probabilities than nesting hypotheses (!)
ABC Methods for Bayesian Model Choice Model choice The Great ABC controversy On-going controvery in phylogeographic genetics about the validity of using ABC for testing Replies: Fagundes et al., 2008, Against: Templeton, 2008, Beaumont et al., 2010, Berger et 2009, 2010a, 2010b, 2010c al., 2010, Csill`ry et al., 2010 e argues that nested hypotheses point out that the criticisms are cannot have higher probabilities addressed at [Bayesian] than nesting hypotheses (!) model-based inference and have nothing to do with ABC...
ABC Methods for Bayesian Model Choice Gibbs random fields Gibbs random fields Gibbs distribution The rv y = (y1 , . . . , yn ) is a Gibbs random field associated with the graph G if 1 f (y) = exp − Vc (yc ) , Z c∈C where Z is the normalising constant, C is the set of cliques of G and Vc is any function also called potential U (y) = c∈C Vc (yc ) is the energy function
ABC Methods for Bayesian Model Choice Gibbs random fields Gibbs random fields Gibbs distribution The rv y = (y1 , . . . , yn ) is a Gibbs random field associated with the graph G if 1 f (y) = exp − Vc (yc ) , Z c∈C where Z is the normalising constant, C is the set of cliques of G and Vc is any function also called potential U (y) = c∈C Vc (yc ) is the energy function c Z is usually unavailable in closed form
ABC Methods for Bayesian Model Choice Gibbs random fields Potts model Potts model Vc (y) is of the form Vc (y) = θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure
ABC Methods for Bayesian Model Choice Gibbs random fields Potts model Potts model Vc (y) is of the form Vc (y) = θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure In most realistic settings, summation Zθ = exp{θ T S(x)} x∈X involves too many terms to be manageable and numerical approximations cannot always be trusted
ABC Methods for Bayesian Model Choice Gibbs random fields Bayesian Model Choice Comparing a model with energy S0 taking values in Rp0 versus a model with energy S1 taking values in Rp1 can be done through the Bayes factor corresponding to the priors π0 and π1 on each parameter space exp{θ T S0 (x)}/Zθ 0 ,0 π0 (dθ 0 ) 0 Bm0 /m1 (x) = exp{θ T S1 (x)}/Zθ 1 ,1 π1 (dθ 1 ) 1
ABC Methods for Bayesian Model Choice Gibbs random fields Neighbourhood relations Choice to be made between M neighbourhood relations m i∼i (0 ≤ m ≤ M − 1) with Sm (x) = I{xi =xi } m i∼i driven by the posterior probabilities of the models.
ABC Methods for Bayesian Model Choice Gibbs random fields Model index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) , Θm
ABC Methods for Bayesian Model Choice Gibbs random fields Model index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) , Θm If S(x) sufficient statistic for the joint parameters (M, θ0 , . . . , θM −1 ), P(M = m|x) = P(M = m|S(x)) .
ABC Methods for Bayesian Model Choice Gibbs random fields Sufficient statistics in Gibbs random fields
ABC Methods for Bayesian Model Choice Gibbs random fields Sufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient.
ABC Methods for Bayesian Model Choice Gibbs random fields Sufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient. For Gibbs random fields, 1 2 x|M = m ∼ fm (x|θm ) = fm (x|S(x))fm (S(x)|θm ) 1 = f 2 (S(x)|θm ) n(S(x)) m where n(S(x)) = {˜ ∈ X : S(˜ ) = S(x)} x x c S(x) is therefore also sufficient for the joint parameters
ABC Methods for Bayesian Model Choice Gibbs random fields ABC model choice Algorithm ABC-MC Generate m∗ from the prior π(M = m). ∗ Generate θm∗ from the prior πm∗ (·). Generate x∗ from the model fm∗ (·|θm∗ ). ∗ Compute the distance ρ(S(x0 ), S(x∗ )). Accept (θm∗ , m∗ ) if ρ(S(x0 ), S(x∗ )) < . ∗ Note When = 0 the algorithm is exact
ABC Methods for Bayesian Model Choice Gibbs random fields Toy example iid Bernoulli model versus two-state first-order Markov chain, i.e. n f0 (x|θ0 ) = exp θ0 I{xi =1} {1 + exp(θ0 )}n , i=1 versus n 1 f1 (x|θ1 ) = exp θ1 I{xi =xi−1 } {1 + exp(θ1 )}n−1 , 2 i=2 with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phase transition” boundaries).
ABC Methods for Bayesian Model Choice Gibbs random fields Toy example (2) ^ BF01 −5 0 5 −40 −20 BF01 0 10 ^ BF01 −10 −5 0 5 10 −40 −20 BF01 0 10 (left) Comparison of the true BF m0 /m1 (x0 ) with BF m0 /m1 (x0 ) (in logs) over 2, 000 simulations and 4.106 proposals from the prior. (right) Same when using tolerance corresponding to the 1% quantile on the distances.
ABC Methods for Bayesian Model Choice Generic ABC model choice Back to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og]
ABC Methods for Bayesian Model Choice Generic ABC model choice Back to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm )
ABC Methods for Bayesian Model Choice Generic ABC model choice Back to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm ) c Potential loss of information at the testing level
ABC Methods for Bayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior
ABC Methods for Bayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior As T go to infinity, limit Iρ{η(z),η(y)}≤ π1 (θ 1 )f1 (z|θ 1 ) dz dθ 1 B12 (y) = Iρ{η(z),η(y)}≤ π2 (θ 2 )f2 (z|θ 2 ) dz dθ 2 η Iρ{η,η(y)}≤ π1 (θ 1 )f1 (η|θ 1 ) dη dθ 1 = η , Iρ{η,η(y)}≤ π2 (θ 2 )f2 (η|θ 2 ) dη dθ 2 η η where f1 (η|θ 1 ) and f2 (η|θ 2 ) distributions of η(z)
ABC Methods for Bayesian Model Choice Generic ABC model choice Limiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η , π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2
ABC Methods for Bayesian Model Choice Generic ABC model choice Limiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η , π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 Bayes factor based on the sole observation of η(y)
ABC Methods for Bayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic for both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 = η g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g1 (y) η = B (y) . g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011]
ABC Methods for Bayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic for both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 = η g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g1 (y) η = B (y) . g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011] No discrepancy only when cross-model sufficiency
ABC Methods for Bayesian Model Choice Generic ABC model choice Poisson/geometric example Sample x = (x1 , . . . , xn ) from either a Poisson P(λ) or from a geometric G(p) Sum n S= yi = η(x) i=1 sufficient statistic for either model but not simultaneously Discrepancy ratio g1 (x) S!n−S / i yi ! = g2 (x) 1 n+S−1 S
ABC Methods for Bayesian Model Choice Generic ABC model choice Poisson/geometric discrepancy η Range of B12 (x) versus B12 (x) B12 (x): The values produced have nothing in common.
ABC Methods for Bayesian Model Choice Generic ABC model choice Formal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011]
ABC Methods for Bayesian Model Choice Generic ABC model choice Formal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] In the Poisson/geometric case, if i xi ! is added to S, no discrepancy
ABC Methods for Bayesian Model Choice Generic ABC model choice Formal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] Only applies in genuine sufficiency settings... c Inability to evaluate loss brought by summary statistics
ABC Methods for Bayesian Model Choice Generic ABC model choice The Pitman–Koopman lemma Lemma A necessary and sufficient condition for the existence of a sufficient statistic of a fixed dimension whatever the sample size is that the sampling distribution belongs to an exponential family. [Pitman, 1933; Koopman, 1933]
ABC Methods for Bayesian Model Choice Generic ABC model choice The Pitman–Koopman lemma Lemma A necessary and sufficient condition for the existence of a sufficient statistic of a fixed dimension whatever the sample size is that the sampling distribution belongs to an exponential family. [Pitman, 1933; Koopman, 1933] Provision of fixed support (consider U(0, θ) counterexample)
ABC Methods for Bayesian Model Choice Generic ABC model choice Meaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, ’Og]
ABC Methods for Bayesian Model Choice Generic ABC model choice Meaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, ’Og] In the Poisson/geometric case, if E[yi ] = θ0 > 0, η (θ0 + 1)2 −θ0 lim B12 (y) = e n→∞ θ0
ABC Methods for Bayesian Model Choice Generic ABC model choice MA example Consider the MA(q) model q xt = t+ ϑi t−i i=1 Simple prior: uniform prior over the identifiability zone, e.g. triangle for MA(2)
ABC Methods for Bayesian Model Choice Generic ABC model choice MA example (2) ABC algorithm thus made of 1. picking a new value (ϑ1 , ϑ2 ) in the triangle 2. generating an iid sequence ( t )−q<t≤T 3. producing a simulated series (xt )1≤t≤T
ABC Methods for Bayesian Model Choice Generic ABC model choice MA example (2) ABC algorithm thus made of 1. picking a new value (ϑ1 , ϑ2 ) in the triangle 2. generating an iid sequence ( t )−q<t≤T 3. producing a simulated series (xt )1≤t≤T Distance: basic distance between the series T ρ((xt )1≤t≤T , (xt )1≤t≤T ) = (xt − xt )2 t=1 or between summary statistics like the first q autocorrelations T τj = xt xt−j t=j+1
ABC Methods for Bayesian Model Choice Generic ABC model choice Comparison of distance impact Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
ABC Methods for Bayesian Model Choice Generic ABC model choice Comparison of distance impact 4 1.5 3 1.0 2 0.5 1 0.0 0 0.0 0.2 0.4 0.6 0.8 −2.0 −1.0 0.0 0.5 1.0 1.5 θ1 θ2 Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
ABC Methods for Bayesian Model Choice Generic ABC model choice Comparison of distance impact 4 1.5 3 1.0 2 0.5 1 0.0 0 0.0 0.2 0.4 0.6 0.8 −2.0 −1.0 0.0 0.5 1.0 1.5 θ1 θ2 Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
ABC Methods for Bayesian Model Choice Generic ABC model choice MA(q) divergence 1.0 1.0 1.0 1.0 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factor equal to 17.71.
ABC Methods for Bayesian Model Choice Generic ABC model choice MA(q) divergence 1.0 1.0 1.0 1.0 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21 equal to .004.
ABC Methods for Bayesian Model Choice Generic ABC model choice Further comments ‘There should be the possibility that for the same model, but different (non-minimal) [summary] statistics (so ∗ different η’s: η1 and η1 ) the ratio of evidences may no longer be equal to one.’ [Michael Stumpf, Jan. 28, 2011, ’Og] Using different summary statistics [on different models] may indicate the loss of information brought by each set but agreement does not lead to trustworthy approximations.
ABC Methods for Bayesian Model Choice Generic ABC model choice A population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter
ABC Methods for Bayesian Model Choice Generic ABC model choice A population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter 24 summary statistics 2 million ABC proposal importance [tree] sampling alternative
ABC Methods for Bayesian Model Choice Generic ABC model choice A population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter 24 summary statistics 2 million ABC proposal importance [tree] sampling alternative
ABC Methods for Bayesian Model Choice Generic ABC model choice Stability of importance sampling 1.0 1.0 1.0 1.0 1.0 q q 0.8 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.6 q 0.4 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0
ABC Methods for Bayesian Model Choice Generic ABC model choice Comparison with ABC Use of 24 summary statistics and DIY-ABC logistic correction 1.0 qq qq qq qq qqq qq q q qq q q qq q q q q q q q q q q qq q q qq q q q qq qq qq q q q q qq q q q q q q q q q q q qq q q q qq q q 0.8 q q q q q q q q q q q q qq q q q q q q q qq q q q q q qqq q q qq q ABC direct and logistic q q qq q q q q q q q 0.6 q q q qq q q qq q q q q q q q q q q q q q q q q q q q q q q q q q qq qq q q qq 0.4 qq q q q q q q q q q q q q q q q q q 0.2 q qq q q q q q q q q q q 0.0 qq 0.0 0.2 0.4 0.6 0.8 1.0 importance sampling
ABC Methods for Bayesian Model Choice Generic ABC model choice Comparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction 6 4 q q q q q q q qq 2 q q q q qq ABC direct q q q qq q qq q q q q qq qq q qq q q q q q qq q q q q q qq q q qq q qq q q qq q q q qqq q q q q q q q q 0 qq qq q q q q qq qq q q qq q q q q q q q q −2 −4 −4 −2 0 2 4 6 importance sampling
ABC Methods for Bayesian Model Choice Generic ABC model choice Comparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction qq 6 q q q q q q q q q q q 4 q qq q qq qq q q q q q q q q q q qq q q q q q q q ABC direct and logistic q q q q q q q qq q q q q q q q qq q 2 q q q q q qq q qq q q q qq qq q qq qq q q q qq q q qq qq qq q q qqqq qq q q q qqqqq qq q q q qq q q q q qq q qq q q qqq q q qqqqq q q qq q q qq q q q qq 0 qq qqq q q qq q qqq q q q q q q q q q q q q q q q q q q q q qq q q −2 q q q q q −4 q q q −4 −2 0 2 4 6 importance sampling
ABC Methods for Bayesian Model Choice Generic ABC model choice Comparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction q 2 q q 1 q q q q q q q q q q log−ratio q q q q q qq q q q qqq q q q q q q q qq q q q q q qq qq q qq q q qqqqqq q q qqqq qq qq q qq qq q qq q qq qq q q q qq q qq q q qqq q 0 q q q q qq q q q qq q q q q q q qq q qq q qq qq q qq qqq qq qq q q q q qq q qqqq q q q q q q q q qq qqq q qq qq q q qq q q q q qq q q qq q q q q q q q q qq qqqq q q q q q q q q q −1 q q q q −2 q 0 20 40 60 80 100 index
ABC Methods for Bayesian Model Choice Generic ABC model choice A second population genetics experiment three populations, two divergent 100 gen. ago two scenarios [third pop. recent admixture between first two pop. / diverging from pop. 1 5 gen. ago] In scenario 1, admixture rate 0.7 from pop. 1 100 datasets with 100 diploid individuals per population, 50 independent microsatellite loci. Effective population size of 1000 and mutation rates of 0.0005. 6 parameters: admixture rate (U [0.1, 0.9]), three effective population sizes (U [200, 2000]), the time of admixture/second divergence (U [1, 10]) and time of first divergence (U [50, 500]).
ABC Methods for Bayesian Model Choice Generic ABC model choice Results IS algorithm performed with 100 coalescent trees per particle and 50,000 particles [12 calendar days using 376 processors] Using ten times as many loci and seven times as many individuals degrades the confidence in the importance sampling approximation because of an increased variability in the likelihood. 0.9 0.8 q 0.7 0.6 q 0.5
ABC Methods for Bayesian Model Choice Generic ABC model choice Results Blurs potential divergence between ABC and genuine posterior probabilities because both are overwhelmingly close to one, due to the high information content of the data. 1.0 q qq q qq q q qqq qq q q qq qq q q q qq qq q qq q q q q q 0.8 q Importance Sampling estimates of q P(M=1|y) with 1,000 particles q q 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Importance Sampling estimates of
ABC Methods for Bayesian Model Choice Generic ABC model choice Results Blurs potential divergence between ABC and genuine posterior probabilities because both are overwhelmingly close to one, due to the high information content of the data. 1.0 q q qqq q qqqq q q qqqqq q qqq qq qq qq qq qq q q q qq qqq Importance Sampling estimates of P(M=1|y) with 50,000 particles q q qq q q q qq q q q q q 0.8 q q q q 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0
ABC Methods for Bayesian Model Choice Generic ABC model choice The only safe cases Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010;Sousa et al., 2009]
ABC Methods for Bayesian Model Choice Generic ABC model choice The only safe cases Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010;Sousa et al., 2009] ...and so does the use of more informal model fitting measures [Ratmann, Andrieu, Richardson and Wiujf, 2009]

More Related Content

PDF
Colloquium in honor of Hans Ruedi Künsch
byChristian Robert
 
PDF
BIRS 12w5105 meeting
byChristian Robert
 
PDF
ABC model choice
byChristian Robert
 
PDF
Columbia workshop [ABC model choice]
byChristian Robert
 
PDF
Workshop on Bayesian Inference for Latent Gaussian Models with Applications
byChristian Robert
 
PDF
Edinburgh, Bayes-250
byChristian Robert
 
PDF
seminar at Princeton University
byChristian Robert
 
PDF
von Mises lecture, Berlin
byChristian Robert
 
Colloquium in honor of Hans Ruedi Künsch
byChristian Robert
 
BIRS 12w5105 meeting
byChristian Robert
 
ABC model choice
byChristian Robert
 
Columbia workshop [ABC model choice]
byChristian Robert
 
Workshop on Bayesian Inference for Latent Gaussian Models with Applications
byChristian Robert
 
Edinburgh, Bayes-250
byChristian Robert
 
seminar at Princeton University
byChristian Robert
 
von Mises lecture, Berlin
byChristian Robert
 

What's hot

PDF
Convergence of ABC methods
byChristian Robert
 
PDF
45th SIS Meeting, Padova, Italy
byChristian Robert
 
PDF
Bayesian inference on mixtures
byChristian Robert
 
PDF
ABC short course: survey chapter
byChristian Robert
 
PDF
Likelihood free computational statistics
byPierre Pudlo
 
PDF
Jsm09 talk
byChristian Robert
 
PDF
Considerate Approaches to ABC Model Selection
byMichael Stumpf
 
PDF
random forests for ABC model choice and parameter estimation
byChristian Robert
 
PDF
Reliable ABC model choice via random forests
byChristian Robert
 
PDF
ABC short course: model choice chapter
byChristian Robert
 
PDF
Approximate Bayesian model choice via random forests
byChristian Robert
 
PDF
Savage-Dickey paradox
byChristian Robert
 
PDF
from model uncertainty to ABC
byChristian Robert
 
PDF
Boston talk
byChristian Robert
 
PDF
4th joint Warwick Oxford Statistics Seminar
byChristian Robert
 
PDF
ABC-Gibbs
byChristian Robert
 
PDF
ABC-Gibbs
byChristian Robert
 
PDF
Yes III: Computational methods for model choice
byChristian Robert
 
PDF
ABC short course: final chapters
byChristian Robert
 
PDF
Monte Carlo in Montréal 2017
byChristian Robert
 
Convergence of ABC methods
byChristian Robert
 
45th SIS Meeting, Padova, Italy
byChristian Robert
 
Bayesian inference on mixtures
byChristian Robert
 
ABC short course: survey chapter
byChristian Robert
 
Likelihood free computational statistics
byPierre Pudlo
 
Jsm09 talk
byChristian Robert
 
Considerate Approaches to ABC Model Selection
byMichael Stumpf
 
random forests for ABC model choice and parameter estimation
byChristian Robert
 
Reliable ABC model choice via random forests
byChristian Robert
 
ABC short course: model choice chapter
byChristian Robert
 
Approximate Bayesian model choice via random forests
byChristian Robert
 
Savage-Dickey paradox
byChristian Robert
 
from model uncertainty to ABC
byChristian Robert
 
Boston talk
byChristian Robert
 
4th joint Warwick Oxford Statistics Seminar
byChristian Robert
 
ABC-Gibbs
byChristian Robert
 
ABC-Gibbs
byChristian Robert
 
Yes III: Computational methods for model choice
byChristian Robert
 
ABC short course: final chapters
byChristian Robert
 
Monte Carlo in Montréal 2017
byChristian Robert
 

Viewers also liked

PDF
RSS Read Paper by Mark Girolami
byChristian Robert
 
PPTX
Networks community detection using artificial bee colony swarm optimization
byAboul Ella Hassanien
 
PDF
ABC workshop: 17w5025
byChristian Robert
 
PDF
ABC short course: introduction chapters
byChristian Robert
 
PPTX
ABC Algorithm.
byN Vinayak
 
PPT
Artificial bee colony (abc)
byquadmemo
 
PPTX
ABC, an effective tool for selective harmonic elimination in multilevel inve...
byShridhar kulkarni
 
PPTX
Abc analysis
byakshay_005
 
RSS Read Paper by Mark Girolami
byChristian Robert
 
Networks community detection using artificial bee colony swarm optimization
byAboul Ella Hassanien
 
ABC workshop: 17w5025
byChristian Robert
 
ABC short course: introduction chapters
byChristian Robert
 
ABC Algorithm.
byN Vinayak
 
Artificial bee colony (abc)
byquadmemo
 
ABC, an effective tool for selective harmonic elimination in multilevel inve...
byShridhar kulkarni
 
Abc analysis
byakshay_005
 

Similar to ABC in London, May 5, 2011

PDF
Ab cancun
byChristian Robert
 
ODP
Iwsmbvs
byBob O'Hara
 
PDF
Approximating Bayes Factors
byChristian Robert
 
PDF
Discussion of Fearnhead and Prangle, RSS&lt; Dec. 14, 2011
byChristian Robert
 
PDF
Intro to Approximate Bayesian Computation (ABC)
byUmberto Picchini
 
PDF
3rd NIPS Workshop on PROBABILISTIC PROGRAMMING
byChristian Robert
 
PDF
NBBC15, Reyjavik, June 08, 2015
byChristian Robert
 
PDF
MaxEnt 2009 talk
byChristian Robert
 
PDF
Asymptotics of ABC, lecture, Collège de France
byChristian Robert
 
PDF
Workshop in honour of Don Poskitt and Gael Martin
byChristian Robert
 
PDF
Laplace's Demon: seminar #1
byChristian Robert
 
PDF
Intro to ABC
byMatt Moores
 
PDF
ABC and empirical likelihood
byChristian Robert
 
PDF
Montpellier Math Colloquium
byChristian Robert
 
PDF
Desy
byChristian Robert
 
PDF
San Antonio short course, March 2010
byChristian Robert
 
PDF
Course on Bayesian computational methods
byChristian Robert
 
Ab cancun
byChristian Robert
 
Iwsmbvs
byBob O'Hara
 
Approximating Bayes Factors
byChristian Robert
 
Discussion of Fearnhead and Prangle, RSS&lt; Dec. 14, 2011
byChristian Robert
 
Intro to Approximate Bayesian Computation (ABC)
byUmberto Picchini
 
3rd NIPS Workshop on PROBABILISTIC PROGRAMMING
byChristian Robert
 
NBBC15, Reyjavik, June 08, 2015
byChristian Robert
 
MaxEnt 2009 talk
byChristian Robert
 
Asymptotics of ABC, lecture, Collège de France
byChristian Robert
 
Workshop in honour of Don Poskitt and Gael Martin
byChristian Robert
 
Laplace's Demon: seminar #1
byChristian Robert
 
Intro to ABC
byMatt Moores
 
ABC and empirical likelihood
byChristian Robert
 
Montpellier Math Colloquium
byChristian Robert
 
Desy
byChristian Robert
 
San Antonio short course, March 2010
byChristian Robert
 
Course on Bayesian computational methods
byChristian Robert
 

More from Christian Robert

PDF
Insufficient Gibbs sampling (A. Luciano, C.P. Robert and R. Ryder)
byChristian Robert
 
PDF
The future of conferences towards sustainability and inclusivity
byChristian Robert
 
PDF
Adaptive Restore algorithm & importance Monte Carlo
byChristian Robert
 
PDF
discussion of ICML23.pdf
byChristian Robert
 
PDF
How many components in a mixture?
byChristian Robert
 
PDF
restore.pdf
byChristian Robert
 
PDF
Testing for mixtures at BNP 13
byChristian Robert
 
PDF
Inferring the number of components: dream or reality?
byChristian Robert
 
PDF
CDT 22 slides.pdf
byChristian Robert
 
PDF
Testing for mixtures by seeking components
byChristian Robert
 
PDF
discussion on Bayesian restricted likelihood
byChristian Robert
 
PDF
NCE, GANs & VAEs (and maybe BAC)
byChristian Robert
 
PDF
ABC-Gibbs
byChristian Robert
 
PDF
Coordinate sampler : A non-reversible Gibbs-like sampler
byChristian Robert
 
PDF
eugenics and statistics
byChristian Robert
 
PDF
asymptotics of ABC
byChristian Robert
 
PDF
Likelihood-free Design: a discussion
byChristian Robert
 
PDF
the ABC of ABC
byChristian Robert
 
PDF
CISEA 2019: ABC consistency and convergence
byChristian Robert
 
PDF
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment models
byChristian Robert
 
Insufficient Gibbs sampling (A. Luciano, C.P. Robert and R. Ryder)
byChristian Robert
 
The future of conferences towards sustainability and inclusivity
byChristian Robert
 
Adaptive Restore algorithm & importance Monte Carlo
byChristian Robert
 
discussion of ICML23.pdf
byChristian Robert
 
How many components in a mixture?
byChristian Robert
 
restore.pdf
byChristian Robert
 
Testing for mixtures at BNP 13
byChristian Robert
 
Inferring the number of components: dream or reality?
byChristian Robert
 
CDT 22 slides.pdf
byChristian Robert
 
Testing for mixtures by seeking components
byChristian Robert
 
discussion on Bayesian restricted likelihood
byChristian Robert
 
NCE, GANs & VAEs (and maybe BAC)
byChristian Robert
 
ABC-Gibbs
byChristian Robert
 
Coordinate sampler : A non-reversible Gibbs-like sampler
byChristian Robert
 
eugenics and statistics
byChristian Robert
 
asymptotics of ABC
byChristian Robert
 
Likelihood-free Design: a discussion
byChristian Robert
 
the ABC of ABC
byChristian Robert
 
CISEA 2019: ABC consistency and convergence
byChristian Robert
 
a discussion of Chib, Shin, and Simoni (2017-8) Bayesian moment models
byChristian Robert
 

Recently uploaded

PDF
CAM, DHT11, GAS, ESP32, FLAME, IR, TEMP, LCD, PIR, SOIL, SOUND, RELAY, TURBID...
byPravinDhanrao3
 
PPTX
Metabolism ( BIOCHEMISTRY & CLINICAL PATHOLOGY )
byBushraDeshpande
 
PPTX
West Hatch High School - GCSE Art Presentation
byWestHatch
 
PDF
West Hatch High School - GCSE Media Specification
byWestHatch
 
PPT
West Hatch High School - GCSE History Option
byWestHatch
 
PDF
Radio Ceylon Finals (An Indian Subcontinent Quiz).pdf
byConquiztadors- the Quiz Society of Sri Venkateswara College
 
PDF
Unit Plan and Unit Test-pdf-Dr. Rajashekhar Shirvalkar, Principal, SMRS B.Ed ...
byRajashekhar Shirvalkar
 
PDF
Fast Followers Project, Embedding Net Zero into Wakefield Metropolitan Distri...
byAssociation for Project Management
 
PDF
Schrodinger's Capital Finals (SciBiz Quiz).pdf
byConquiztadors- the Quiz Society of Sri Venkateswara College
 
PPTX
'Colonial Mentality and Social Identity in Karma by Khushwant Singh'
byKrishnarajsinh Parmar
 
PPTX
literary theory and criticism by Vivek p
byvivekrathodborda
 
PDF
Beak Modifications by Dr. Ramzan Virani pptx.pdf
byDr. Ramzan Virani
 
PPTX
How to Track Team Performance Analysis in Odoo 18 Recruitment
byCeline George
 
PPTX
Drugs modulating serotonergic system.pptx
byNorman Arockiaraj G
 
PDF
Information about the author Shashi Deshpande
bydarshanaparmar6522
 
PPTX
Appreciations - Jan 26 01.pptxkkkmmkmkmkmkm
bypreetheshparmar
 
PPTX
Biological source, chemical constituents, and therapeutic efficacy of the fol...
bySai Meer College of Pharmacy
 
PPTX
That long silence - Novel by Shashi Deshapande
bysanjanavaghela65
 
PPTX
COMMUNICATION ITS PROCESS ELEMENTS & BARRIER .pptx
byPoojaSen20
 
PDF
Sharad Bisen_Soil Sterilization_Objectives_ICAR Course, Sixth Dean Committee.pdf
bybisensharad
 
CAM, DHT11, GAS, ESP32, FLAME, IR, TEMP, LCD, PIR, SOIL, SOUND, RELAY, TURBID...
byPravinDhanrao3
 
Metabolism ( BIOCHEMISTRY & CLINICAL PATHOLOGY )
byBushraDeshpande
 
West Hatch High School - GCSE Art Presentation
byWestHatch
 
West Hatch High School - GCSE Media Specification
byWestHatch
 
West Hatch High School - GCSE History Option
byWestHatch
 
Radio Ceylon Finals (An Indian Subcontinent Quiz).pdf
byConquiztadors- the Quiz Society of Sri Venkateswara College
 
Unit Plan and Unit Test-pdf-Dr. Rajashekhar Shirvalkar, Principal, SMRS B.Ed ...
byRajashekhar Shirvalkar
 
Fast Followers Project, Embedding Net Zero into Wakefield Metropolitan Distri...
byAssociation for Project Management
 
Schrodinger's Capital Finals (SciBiz Quiz).pdf
byConquiztadors- the Quiz Society of Sri Venkateswara College
 
'Colonial Mentality and Social Identity in Karma by Khushwant Singh'
byKrishnarajsinh Parmar
 
literary theory and criticism by Vivek p
byvivekrathodborda
 
Beak Modifications by Dr. Ramzan Virani pptx.pdf
byDr. Ramzan Virani
 
How to Track Team Performance Analysis in Odoo 18 Recruitment
byCeline George
 
Drugs modulating serotonergic system.pptx
byNorman Arockiaraj G
 
Information about the author Shashi Deshpande
bydarshanaparmar6522
 
Appreciations - Jan 26 01.pptxkkkmmkmkmkmkm
bypreetheshparmar
 
Biological source, chemical constituents, and therapeutic efficacy of the fol...
bySai Meer College of Pharmacy
 
That long silence - Novel by Shashi Deshapande
bysanjanavaghela65
 
COMMUNICATION ITS PROCESS ELEMENTS & BARRIER .pptx
byPoojaSen20
 
Sharad Bisen_Soil Sterilization_Objectives_ICAR Course, Sixth Dean Committee.pdf
bybisensharad
 

ABC in London, May 5, 2011

  • 1.
    ABC Methods forBayesian Model Choice ABC Methods for Bayesian Model Choice Christian P. Robert Universit´ Paris-Dauphine, IuF, & CREST e http://www.ceremade.dauphine.fr/~xian May 4, 2011
  • 2.
    ABC Methods forBayesian Model Choice ABC for model choice Model choice Gibbs random fields Generic ABC model choice
  • 3.
    ABC Methods forBayesian Model Choice Model choice Bayesian model choice Several models M1 , M2 , . . . are considered simultaneously for a dataset y and the model index M is part of the inference. Use of a prior distribution. π(M = m), plus a prior distribution on the parameter conditional on the value m of the model index, πm (θ m ) Goal is to derive the posterior distribution of M , challenging computational target when models are complex.
  • 4.
    ABC Methods forBayesian Model Choice Model choice Generic ABC for model choice Algorithm 1 Likelihood-free model choice sampler (ABC-MC) for t = 1 to T do repeat Generate m from the prior π(M = m) Generate θ m from the prior πm (θ m ) Generate z from the model fm (z|θ m ) until ρ{η(z), η(y)} < Set m(t) = m and θ (t) = θ m end for [Toni, Welch, Strelkowa, Ipsen & Stumpf, 2009]
  • 5.
    ABC Methods forBayesian Model Choice Model choice ABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Early issues with implementation: should tolerances be the same for all models? should summary statistics vary across models? (incl. their dimension)? should the distance measure ρ vary as well?
  • 6.
    ABC Methods forBayesian Model Choice Model choice ABC estimates Posterior probability π(M = m|y) approximated by the frequency of acceptances from model m T 1 Im(t) =m . T t=1 Early issues with implementation: then needs to become part of the model this is incompatible with Bayesian model choice same difficulty Extension to a weighted polychotomous logistic regression estimate of π(M = m|y), with non-parametric kernel weights [Cornuet et al., DIYABC, 2009]
  • 7.
    ABC Methods forBayesian Model Choice Model choice The Great ABC controversy On-going controvery in phylogeographic genetics about the validity of using ABC for testing Against: Templeton, 2008, 2009, 2010a, 2010b, 2010c argues that nested hypotheses cannot have higher probabilities than nesting hypotheses (!)
  • 8.
    ABC Methods forBayesian Model Choice Model choice The Great ABC controversy On-going controvery in phylogeographic genetics about the validity of using ABC for testing Replies: Fagundes et al., 2008, Against: Templeton, 2008, Beaumont et al., 2010, Berger et 2009, 2010a, 2010b, 2010c al., 2010, Csill`ry et al., 2010 e argues that nested hypotheses point out that the criticisms are cannot have higher probabilities addressed at [Bayesian] than nesting hypotheses (!) model-based inference and have nothing to do with ABC...
  • 9.
    ABC Methods forBayesian Model Choice Gibbs random fields Gibbs random fields Gibbs distribution The rv y = (y1 , . . . , yn ) is a Gibbs random field associated with the graph G if 1 f (y) = exp − Vc (yc ) , Z c∈C where Z is the normalising constant, C is the set of cliques of G and Vc is any function also called potential U (y) = c∈C Vc (yc ) is the energy function
  • 10.
    ABC Methods forBayesian Model Choice Gibbs random fields Gibbs random fields Gibbs distribution The rv y = (y1 , . . . , yn ) is a Gibbs random field associated with the graph G if 1 f (y) = exp − Vc (yc ) , Z c∈C where Z is the normalising constant, C is the set of cliques of G and Vc is any function also called potential U (y) = c∈C Vc (yc ) is the energy function c Z is usually unavailable in closed form
  • 11.
    ABC Methods forBayesian Model Choice Gibbs random fields Potts model Potts model Vc (y) is of the form Vc (y) = θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure
  • 12.
    ABC Methods forBayesian Model Choice Gibbs random fields Potts model Potts model Vc (y) is of the form Vc (y) = θS(y) = θ δyl =yi l∼i where l∼i denotes a neighbourhood structure In most realistic settings, summation Zθ = exp{θ T S(x)} x∈X involves too many terms to be manageable and numerical approximations cannot always be trusted
  • 13.
    ABC Methods forBayesian Model Choice Gibbs random fields Bayesian Model Choice Comparing a model with energy S0 taking values in Rp0 versus a model with energy S1 taking values in Rp1 can be done through the Bayes factor corresponding to the priors π0 and π1 on each parameter space exp{θ T S0 (x)}/Zθ 0 ,0 π0 (dθ 0 ) 0 Bm0 /m1 (x) = exp{θ T S1 (x)}/Zθ 1 ,1 π1 (dθ 1 ) 1
  • 14.
    ABC Methods forBayesian Model Choice Gibbs random fields Neighbourhood relations Choice to be made between M neighbourhood relations m i∼i (0 ≤ m ≤ M − 1) with Sm (x) = I{xi =xi } m i∼i driven by the posterior probabilities of the models.
  • 15.
    ABC Methods forBayesian Model Choice Gibbs random fields Model index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) , Θm
  • 16.
    ABC Methods forBayesian Model Choice Gibbs random fields Model index Computational target: P(M = m|x) ∝ fm (x|θm )πm (θm ) dθm π(M = m) , Θm If S(x) sufficient statistic for the joint parameters (M, θ0 , . . . , θM −1 ), P(M = m|x) = P(M = m|S(x)) .
  • 17.
    ABC Methods forBayesian Model Choice Gibbs random fields Sufficient statistics in Gibbs random fields
  • 18.
    ABC Methods forBayesian Model Choice Gibbs random fields Sufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient.
  • 19.
    ABC Methods forBayesian Model Choice Gibbs random fields Sufficient statistics in Gibbs random fields Each model m has its own sufficient statistic Sm (·) and S(·) = (S0 (·), . . . , SM −1 (·)) is also (model-)sufficient. For Gibbs random fields, 1 2 x|M = m ∼ fm (x|θm ) = fm (x|S(x))fm (S(x)|θm ) 1 = f 2 (S(x)|θm ) n(S(x)) m where n(S(x)) = {˜ ∈ X : S(˜ ) = S(x)} x x c S(x) is therefore also sufficient for the joint parameters
  • 20.
    ABC Methods forBayesian Model Choice Gibbs random fields ABC model choice Algorithm ABC-MC Generate m∗ from the prior π(M = m). ∗ Generate θm∗ from the prior πm∗ (·). Generate x∗ from the model fm∗ (·|θm∗ ). ∗ Compute the distance ρ(S(x0 ), S(x∗ )). Accept (θm∗ , m∗ ) if ρ(S(x0 ), S(x∗ )) < . ∗ Note When = 0 the algorithm is exact
  • 21.
    ABC Methods forBayesian Model Choice Gibbs random fields Toy example iid Bernoulli model versus two-state first-order Markov chain, i.e. n f0 (x|θ0 ) = exp θ0 I{xi =1} {1 + exp(θ0 )}n , i=1 versus n 1 f1 (x|θ1 ) = exp θ1 I{xi =xi−1 } {1 + exp(θ1 )}n−1 , 2 i=2 with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phase transition” boundaries).
  • 22.
    ABC Methods forBayesian Model Choice Gibbs random fields Toy example (2) ^ BF01 −5 0 5 −40 −20 BF01 0 10 ^ BF01 −10 −5 0 5 10 −40 −20 BF01 0 10 (left) Comparison of the true BF m0 /m1 (x0 ) with BF m0 /m1 (x0 ) (in logs) over 2, 000 simulations and 4.106 proposals from the prior. (right) Same when using tolerance corresponding to the 1% quantile on the distances.
  • 23.
    ABC Methods forBayesian Model Choice Generic ABC model choice Back to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og]
  • 24.
    ABC Methods forBayesian Model Choice Generic ABC model choice Back to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm )
  • 25.
    ABC Methods forBayesian Model Choice Generic ABC model choice Back to sufficiency ‘Sufficient statistics for individual models are unlikely to be very informative for the model probability. This is already well known and understood by the ABC-user community.’ [Scott Sisson, Jan. 31, 2011, ’Og] If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and η2 (x) sufficient statistic for model m = 2 and parameter θ2 , (η1 (x), η2 (x)) is not always sufficient for (m, θm ) c Potential loss of information at the testing level
  • 26.
    ABC Methods forBayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior
  • 27.
    ABC Methods forBayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (T → ∞) ABC approximation T t=1 Imt =1 Iρ{η(zt ),η(y)}≤ B12 (y) = T , t=1 Imt =2 Iρ{η(zt ),η(y)}≤ where the (mt , z t )’s are simulated from the (joint) prior As T go to infinity, limit Iρ{η(z),η(y)}≤ π1 (θ 1 )f1 (z|θ 1 ) dz dθ 1 B12 (y) = Iρ{η(z),η(y)}≤ π2 (θ 2 )f2 (z|θ 2 ) dz dθ 2 η Iρ{η,η(y)}≤ π1 (θ 1 )f1 (η|θ 1 ) dη dθ 1 = η , Iρ{η,η(y)}≤ π2 (θ 2 )f2 (η|θ 2 ) dη dθ 2 η η where f1 (η|θ 1 ) and f2 (η|θ 2 ) distributions of η(z)
  • 28.
    ABC Methods forBayesian Model Choice Generic ABC model choice Limiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η , π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2
  • 29.
    ABC Methods forBayesian Model Choice Generic ABC model choice Limiting behaviour of B12 ( → 0) When goes to zero, η η π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η , π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 Bayes factor based on the sole observation of η(y)
  • 30.
    ABC Methods forBayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic for both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 = η g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g1 (y) η = B (y) . g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011]
  • 31.
    ABC Methods forBayesian Model Choice Generic ABC model choice Limiting behaviour of B12 (under sufficiency) If η(y) sufficient statistic for both models, fi (y|θ i ) = gi (y)fiη (η(y)|θ i ) Thus η Θ1 π(θ 1 )g1 (y)f1 (η(y)|θ 1 ) dθ 1 B12 (y) = η Θ2 π(θ 2 )g2 (y)f2 (η(y)|θ 2 ) dθ 2 η g1 (y) π1 (θ 1 )f1 (η(y)|θ 1 ) dθ 1 = η g2 (y) π2 (θ 2 )f2 (η(y)|θ 2 ) dθ 2 g1 (y) η = B (y) . g2 (y) 12 [Didelot, Everitt, Johansen & Lawson, 2011] No discrepancy only when cross-model sufficiency
  • 32.
    ABC Methods forBayesian Model Choice Generic ABC model choice Poisson/geometric example Sample x = (x1 , . . . , xn ) from either a Poisson P(λ) or from a geometric G(p) Sum n S= yi = η(x) i=1 sufficient statistic for either model but not simultaneously Discrepancy ratio g1 (x) S!n−S / i yi ! = g2 (x) 1 n+S−1 S
  • 33.
    ABC Methods forBayesian Model Choice Generic ABC model choice Poisson/geometric discrepancy η Range of B12 (x) versus B12 (x) B12 (x): The values produced have nothing in common.
  • 34.
    ABC Methods forBayesian Model Choice Generic ABC model choice Formal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011]
  • 35.
    ABC Methods forBayesian Model Choice Generic ABC model choice Formal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] In the Poisson/geometric case, if i xi ! is added to S, no discrepancy
  • 36.
    ABC Methods forBayesian Model Choice Generic ABC model choice Formal recovery Creating an encompassing exponential family T T f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θ1 η1 (x) + θ1 η1 (x) + α1 t1 (x) + α2 t2 (x)} leads to a sufficient statistic (η1 (x), η2 (x), t1 (x), t2 (x)) [Didelot, Everitt, Johansen & Lawson, 2011] Only applies in genuine sufficiency settings... c Inability to evaluate loss brought by summary statistics
  • 37.
    ABC Methods forBayesian Model Choice Generic ABC model choice The Pitman–Koopman lemma Lemma A necessary and sufficient condition for the existence of a sufficient statistic of a fixed dimension whatever the sample size is that the sampling distribution belongs to an exponential family. [Pitman, 1933; Koopman, 1933]
  • 38.
    ABC Methods forBayesian Model Choice Generic ABC model choice The Pitman–Koopman lemma Lemma A necessary and sufficient condition for the existence of a sufficient statistic of a fixed dimension whatever the sample size is that the sampling distribution belongs to an exponential family. [Pitman, 1933; Koopman, 1933] Provision of fixed support (consider U(0, θ) counterexample)
  • 39.
    ABC Methods forBayesian Model Choice Generic ABC model choice Meaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, ’Og]
  • 40.
    ABC Methods forBayesian Model Choice Generic ABC model choice Meaning of the ABC-Bayes factor ‘This is also why focus on model discrimination typically (...) proceeds by (...) accepting that the Bayes Factor that one obtains is only derived from the summary statistics and may in no way correspond to that of the full model.’ [Scott Sisson, Jan. 31, 2011, ’Og] In the Poisson/geometric case, if E[yi ] = θ0 > 0, η (θ0 + 1)2 −θ0 lim B12 (y) = e n→∞ θ0
  • 41.
    ABC Methods forBayesian Model Choice Generic ABC model choice MA example Consider the MA(q) model q xt = t+ ϑi t−i i=1 Simple prior: uniform prior over the identifiability zone, e.g. triangle for MA(2)
  • 42.
    ABC Methods forBayesian Model Choice Generic ABC model choice MA example (2) ABC algorithm thus made of 1. picking a new value (ϑ1 , ϑ2 ) in the triangle 2. generating an iid sequence ( t )−q<t≤T 3. producing a simulated series (xt )1≤t≤T
  • 43.
    ABC Methods forBayesian Model Choice Generic ABC model choice MA example (2) ABC algorithm thus made of 1. picking a new value (ϑ1 , ϑ2 ) in the triangle 2. generating an iid sequence ( t )−q<t≤T 3. producing a simulated series (xt )1≤t≤T Distance: basic distance between the series T ρ((xt )1≤t≤T , (xt )1≤t≤T ) = (xt − xt )2 t=1 or between summary statistics like the first q autocorrelations T τj = xt xt−j t=j+1
  • 44.
    ABC Methods forBayesian Model Choice Generic ABC model choice Comparison of distance impact Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
  • 45.
    ABC Methods forBayesian Model Choice Generic ABC model choice Comparison of distance impact 4 1.5 3 1.0 2 0.5 1 0.0 0 0.0 0.2 0.4 0.6 0.8 −2.0 −1.0 0.0 0.5 1.0 1.5 θ1 θ2 Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
  • 46.
    ABC Methods forBayesian Model Choice Generic ABC model choice Comparison of distance impact 4 1.5 3 1.0 2 0.5 1 0.0 0 0.0 0.2 0.4 0.6 0.8 −2.0 −1.0 0.0 0.5 1.0 1.5 θ1 θ2 Evaluation of the tolerance on the ABC sample against both distances ( = 100%, 10%, 1%, 0.1%) for an MA(2) model
  • 47.
    ABC Methods forBayesian Model Choice Generic ABC model choice MA(q) divergence 1.0 1.0 1.0 1.0 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factor equal to 17.71.
  • 48.
    ABC Methods forBayesian Model Choice Generic ABC model choice MA(q) divergence 1.0 1.0 1.0 1.0 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 1 2 1 2 1 2 1 2 Evolution [against ] of ABC Bayes factor, in terms of frequencies of visits to models MA(1) (left) and MA(2) (right) when equal to 10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sample of 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21 equal to .004.
  • 49.
    ABC Methods forBayesian Model Choice Generic ABC model choice Further comments ‘There should be the possibility that for the same model, but different (non-minimal) [summary] statistics (so ∗ different η’s: η1 and η1 ) the ratio of evidences may no longer be equal to one.’ [Michael Stumpf, Jan. 28, 2011, ’Og] Using different summary statistics [on different models] may indicate the loss of information brought by each set but agreement does not lead to trustworthy approximations.
  • 50.
    ABC Methods forBayesian Model Choice Generic ABC model choice A population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter
  • 51.
    ABC Methods forBayesian Model Choice Generic ABC model choice A population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter 24 summary statistics 2 million ABC proposal importance [tree] sampling alternative
  • 52.
    ABC Methods forBayesian Model Choice Generic ABC model choice A population genetics evaluation Population genetics example with 3 populations 2 scenari 15 individuals 5 loci single mutation parameter 24 summary statistics 2 million ABC proposal importance [tree] sampling alternative
  • 53.
    ABC Methods forBayesian Model Choice Generic ABC model choice Stability of importance sampling 1.0 1.0 1.0 1.0 1.0 q q 0.8 0.8 0.8 0.8 0.8 0.6 0.6 0.6 0.6 0.6 q 0.4 0.4 0.4 0.4 0.4 0.2 0.2 0.2 0.2 0.2 0.0 0.0 0.0 0.0 0.0
  • 54.
    ABC Methods forBayesian Model Choice Generic ABC model choice Comparison with ABC Use of 24 summary statistics and DIY-ABC logistic correction 1.0 qq qq qq qq qqq qq q q qq q q qq q q q q q q q q q q qq q q qq q q q qq qq qq q q q q qq q q q q q q q q q q q qq q q q qq q q 0.8 q q q q q q q q q q q q qq q q q q q q q qq q q q q q qqq q q qq q ABC direct and logistic q q qq q q q q q q q 0.6 q q q qq q q qq q q q q q q q q q q q q q q q q q q q q q q q q q qq qq q q qq 0.4 qq q q q q q q q q q q q q q q q q q 0.2 q qq q q q q q q q q q q 0.0 qq 0.0 0.2 0.4 0.6 0.8 1.0 importance sampling
  • 55.
    ABC Methods forBayesian Model Choice Generic ABC model choice Comparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction 6 4 q q q q q q q qq 2 q q q q qq ABC direct q q q qq q qq q q q q qq qq q qq q q q q q qq q q q q q qq q q qq q qq q q qq q q q qqq q q q q q q q q 0 qq qq q q q q qq qq q q qq q q q q q q q q −2 −4 −4 −2 0 2 4 6 importance sampling
  • 56.
    ABC Methods forBayesian Model Choice Generic ABC model choice Comparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction qq 6 q q q q q q q q q q q 4 q qq q qq qq q q q q q q q q q q qq q q q q q q q ABC direct and logistic q q q q q q q qq q q q q q q q qq q 2 q q q q q qq q qq q q q qq qq q qq qq q q q qq q q qq qq qq q q qqqq qq q q q qqqqq qq q q q qq q q q q qq q qq q q qqq q q qqqqq q q qq q q qq q q q qq 0 qq qqq q q qq q qqq q q q q q q q q q q q q q q q q q q q q qq q q −2 q q q q q −4 q q q −4 −2 0 2 4 6 importance sampling
  • 57.
    ABC Methods forBayesian Model Choice Generic ABC model choice Comparison with ABC Use of 15 summary statistics and DIY-ABC logistic correction q 2 q q 1 q q q q q q q q q q log−ratio q q q q q qq q q q qqq q q q q q q q qq q q q q q qq qq q qq q q qqqqqq q q qqqq qq qq q qq qq q qq q qq qq q q q qq q qq q q qqq q 0 q q q q qq q q q qq q q q q q q qq q qq q qq qq q qq qqq qq qq q q q q qq q qqqq q q q q q q q q qq qqq q qq qq q q qq q q q q qq q q qq q q q q q q q q qq qqqq q q q q q q q q q −1 q q q q −2 q 0 20 40 60 80 100 index
  • 58.
    ABC Methods forBayesian Model Choice Generic ABC model choice A second population genetics experiment three populations, two divergent 100 gen. ago two scenarios [third pop. recent admixture between first two pop. / diverging from pop. 1 5 gen. ago] In scenario 1, admixture rate 0.7 from pop. 1 100 datasets with 100 diploid individuals per population, 50 independent microsatellite loci. Effective population size of 1000 and mutation rates of 0.0005. 6 parameters: admixture rate (U [0.1, 0.9]), three effective population sizes (U [200, 2000]), the time of admixture/second divergence (U [1, 10]) and time of first divergence (U [50, 500]).
  • 59.
    ABC Methods forBayesian Model Choice Generic ABC model choice Results IS algorithm performed with 100 coalescent trees per particle and 50,000 particles [12 calendar days using 376 processors] Using ten times as many loci and seven times as many individuals degrades the confidence in the importance sampling approximation because of an increased variability in the likelihood. 0.9 0.8 q 0.7 0.6 q 0.5
  • 60.
    ABC Methods forBayesian Model Choice Generic ABC model choice Results Blurs potential divergence between ABC and genuine posterior probabilities because both are overwhelmingly close to one, due to the high information content of the data. 1.0 q qq q qq q q qqq qq q q qq qq q q q qq qq q qq q q q q q 0.8 q Importance Sampling estimates of q P(M=1|y) with 1,000 particles q q 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0 Importance Sampling estimates of
  • 61.
    ABC Methods forBayesian Model Choice Generic ABC model choice Results Blurs potential divergence between ABC and genuine posterior probabilities because both are overwhelmingly close to one, due to the high information content of the data. 1.0 q q qqq q qqqq q q qqqqq q qqq qq qq qq qq qq q q q qq qqq Importance Sampling estimates of P(M=1|y) with 50,000 particles q q qq q q q qq q q q q q 0.8 q q q q 0.6 0.4 0.2 0.0 0.0 0.2 0.4 0.6 0.8 1.0
  • 62.
    ABC Methods forBayesian Model Choice Generic ABC model choice The only safe cases Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010;Sousa et al., 2009]
  • 63.
    ABC Methods forBayesian Model Choice Generic ABC model choice The only safe cases Besides specific models like Gibbs random fields, using distances over the data itself escapes the discrepancy... [Toni & Stumpf, 2010;Sousa et al., 2009] ...and so does the use of more informal model fitting measures [Ratmann, Andrieu, Richardson and Wiujf, 2009]