Bayesian intro


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Bayesian intro

  1. 1. An Introduction to the Bayesian Approach J Guzmán, PhD 15 August 2011
  2. 2. Bayesian Evolution
  3. 3. Bayesian: one who asks youwhat you think before a study inorder to tell you what you thinkafterwards Adapted from: S Senn (1997). Statistical Issues in Drug Development. Wiley
  4. 4. Rev. Thomas BayesEnglish Theologian and Mathematician ca. 1700 – 1761
  5. 5. Bayesian Methods•  1763 – Bayes’ article on inverse probability•  Laplace extended Bayesian ideas in different scientific areas in Théorie Analytique des Probabilités [1812]•  Both Laplace & Gauss used the inverse method•  1st three quarters of 20th Century dominated by frequentist methods•  Last quarter of 20th Century – resurgence of Bayesian methods [computational advances]•  21st Century – Bayesian Century [Lindley]
  6. 6. Pierre-Simon LaplaceFrench Mathematician 1749 – 1827
  7. 7. Karl Friedrich Gauss“Prince of Mathematics” 1777 – 1855Used inverse probability
  8. 8. Bayesian Methods•  Key components: prior, likelihood function, posterior, and predictive distribution•  Suppose a study is carried out to compare new and standard teaching methods•  Ho: Methods are equally effective•  HA: New method increases grades by 20%•  A Bayesian presents the probability that new & standard methods are equally effective, given the results of the experiment at hand: P(Ho | data)
  9. 9. Bayesian Methods•  Data – observed data from experiment•  Find the probability that the new method is at least 20% more effective than the standard, given the results of the experiment [Posterior Probability]•  Another conclusion could be the probability distribution for the outcome of interest for the next student•  Predictive Probabilities – refer to future observations on individuals or on set of individuals
  10. 10. Bayes’ Theorem•  Basic tool of Bayesian analysis•  Provide the means by which we learn from data•  Given prior state of knowledge, it tells how to update belief based upon observations: ∝ P(H | data) = P(H) · P(data | H) / P(data) € α P(H) · P(data | H) ∝ α means “is proportonal to”•  Bayes’ theorem can be re-expressed in € odds terms: let data ≡ y
  11. 11. Bayes’ Theorem
  12. 12. Bayes’ Theorem•  Can also consider posterior probability of any measure θ: P(θ | data) α P(θ) · P( data | θ)•  Bayes’ theorem states that the posterior probability of any measure θ, is proportional to the information on θ external to the experiment times the likelihood function evaluated at θ: Prior · likelihood → posterior
  13. 13. Prior•  Prior information about θ assessed as a probability distribution on θ•  Distribution on θ depends on the assessor: it is subjective•  A subjective probability can be calculated any time a person has an opinion•  Diffuse prior - when a person’ s opinion on θ includes a broad range of possibilities & all values are thought to be roughly equally probable
  14. 14. Prior•  Conjugate prior – if the posterior distribution has same shape as the prior distribution, regardless of the observed sample values•  Examples: 1.  Beta prior & binomial likelihood yield a beta posterior 2.  Normal prior & normal likelihood yield a normal posterior 3.  Gamma prior & Poisson likelihood yield a gamma posterior
  15. 15. Community of Priors•  Expressing a range of reasonable opinions•  Reference – represents minimal prior information•  Expertise – formalizes opinion of well- informed experts•  Skeptical – downgrades superiority of new method•  Enthusiastic – counterbalance of skeptical
  16. 16. Likelihood Function P(data | θ)•  Represents the weighting of evidence from the experiment about θ•  It states what the experiment says about the measure of interest [Savage, 1962]•  It is the probability of getting certain result, conditioning on the model•  As the amount of data increases, Prior is dominated by the Likelihood : –  Two investigators with different prior opinions could reach a consensus after the results of an experiment
  17. 17. Likelihood Principle•  States that the likelihood function contains all relevant information from the data•  Two samples have equivalent information if their likelihoods are proportional•  Adherence to the Likelihood Principle means that inference are conditional on the observed data•  Bayesian analysts base all inferences about θ solely on its posterior distribution
  18. 18. Likelihood Principle•  Two experiments: one yields data y1 and the other yields data y2•  If the likelihoods: P(y1 | θ) & P(y2 | θ) are identical up to multiplication by arbitrary functions of y1 & y2 then they contain identical information about θ and lead to identical posterior distributions•  Therefore, to equivalent inferences
  19. 19. Example•  EXP 1: In a study of a •  EXP 2: Students are fixed sample of 20 entered into a study students, 12 of them until 12 of them respond positively to respond positively to the method [Binomial the method [Negative- distribution] binomial distribution]•  Likelihood is •  Likelihood at n = 20 is proportional to proportional to θ12 (1 – θ)8 θ12 (1 – θ)8
  20. 20. Exchangeability•  Key idea in statistical inference in general•  Two observations are exchangeable if they provide equivalent statistical information•  Two students randomly selected from a particular population of students can be considered exchangeable•  If the students in a study are exchangeable with the students in the population for which the method is intended, then the study can be used to make inferences about the entire population•  Exchangeability in terms of experiments: Two studies are exchangeable if they provide equivalent statistical information about some super-population of experiments
  21. 21. Laplace on ProbabilityIt is remarkable that a science, whichcommenced with the consideration ofgames of chance, should be elevated tothe rank of the most important subjectsof human knowledge.A Philosophical Essay on Probabilities.John Wiley & Sons, 1902, page 195.Original French edition 1814.
  22. 22. References•  Computation: OpenBUGS R packages: BRugs, bayesm, R2WinBUGS from CRAN: http://•  Gelman, A, Carlin, JB, Stern, HS, & Rubin, DB (2004). Bayesian Data Analysis. Second Ed.. Chapman and Hall•  Gilks, WR, Richardson, S, & Spiegelhalter, DJ (1996). Markov Chain Monte Carlo in Practice. Chapman & Hall•  More Advanced: Bernardo, J & Smith, AFM (1994). Bayesian Theory. Wiley OHagan, A & Forster, JJ (2004). Bayesian Inference, 2nd Edition. Vol. 2B of "Kendalls Advanced Theory of Statistics". Arnold