Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Classical inference in/for physics

432 views

Published on

A brief review of basics in Statistics and Classical Inference. This talk was given to a very specific public, interested in seeing how Statistics can be employed step-by-step. Especially, Maximum Likelihood estimators are discussed and applied to three simple data sets as a way to fit your probabilistic.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Classical inference in/for physics

  1. 1. Classical Inference in Physics Thiago Mosqueiro Institute of Physics of S˜ao Carlos University of S˜ao Paulo July 31 2012 Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 1 / 38
  2. 2. Equilibrium criticality – Ising model • Model of ferromagnetism in statistical mechanics • Contact with thermal reservoir • Lattice of N binary elements – spins • Each site of this lattice: or Energy of a configuragion S = (S1, S2, . . .), E(S) = −J p,j SjSp Probability of S = (S1, S2, . . .), P(S) = exp − J kT p,j SjSp ∀S exp − J kT p,j SjSp Monte Carlo step: each time step means N itera- tions of the algorithm. Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 2 / 38
  3. 3. The Market for Lemons: Quality Uncertainty and the Market Mechanism, G. Akerlof. The quarterly journal of economics, 1970 Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 3 / 38
  4. 4. Objectivies • What’s the difference between Probability and Statistics? • Inference • Statistics – models, hypotehsis and estimation • Main focus: Maximum likelyhood estimators • Glimpse of hypothesis testing “What! you have solved it already?” “Well, that would be too much to say. I have discovered a suggestive fact, that is all. It is, however, very suggestive.” Sign of Four (II), Sir Arthur Conan Doyle Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 4 / 38
  5. 5. Summary 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 5 / 38
  6. 6. Where are we? 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 6 / 38
  7. 7. Random variable • Let |ψ be the state of a particle • Suppose we know for every n the solutions H |n = n |n What is the probability the particle is in state |n ? • It can happen that |ψ = |1 . . . or |ψ = |2 . . . or . . . • In this sense, |ψ is a random variable • Suppose you have n dice Side j of each die have probability pj • The result of tossing the j-th die is Xj What is the probability of Xj > x? • Xj is another random variable Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 7 / 38
  8. 8. Random variables • Events: X = x, X ≤ x, X = x and Y < y • Probability of an event: P (X = x), P (X ≤ x), P (X = x, Y < y) • Moments: X := x P(X = x)x, X2 := x P(X = x)x2 , . . . • Surprisal: I (X = x) = − log [P (X = x)] • Entropy: I (X = x) = − x P (X = x) log [P (X = x)] Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 8 / 38
  9. 9. Example: gaussian distributed variable X ∼ N(µ, σ) P(x < X ≤ x + dx) = ρ(x)dx = A exp − (x − µ)2 σ2 dx - 3 - 2 - 1 φμ,σ2( 0.8 0.6 0.4 0.2 0.0 −5 −3 1 3 5 x 1.0 −1 0 2 4−2−4 x) 0,μ= 0,μ= 0,μ= −2,μ= 2 0.2,σ = 2 1.0,σ = 2 5.0,σ = 2 0.5,σ = Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 9 / 38
  10. 10. Example: gaussian distributed variable X ∼ N(µ, σ) P(x < X ≤ x + dx) = ρ(x)dx = A exp − (x − µ)2 σ2 dx 0 1 2 3 4 5 6 0 100 200 300 400 500 600 700 EventXj Realization j Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 9 / 38
  11. 11. Now the big problem... What if I don’t have a model...? How can one obtain information from observations? How can I know my model fits the reality? Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 10 / 38
  12. 12. Where are we? 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 11 / 38
  13. 13. Statistically infering... • Statistics validates and fits Probabilistic models • build a statistical model that should describe the process • interpret the data as realizations of your model • Inference gives you a statistical proposition • Models may be parametric, non-parametric or semi-parametric • Of course let’s focus on parametric models. Let’s guess our model: Xj ∼ N(µ, σ) What’s your best guess about µ and σ? Usual way of doing this estimate is by means of an estimator 0 1 2 3 4 5 6 0 100 200 300 400 500 600 700 EventXj Realization j Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 12 / 38
  14. 14. Estimator • Data: set of i.i.d. X1, X2, X3, . . . Xm • Given a data set and a statistical model, we have a probability with some parameter θ • An estimator is a function of the data set to some sample estimates Examples of estimators • X = 1 m m j=1 Xj • ˆσ = 1 m m j=1 Xj − X 2 • ˆxmin = min (X1, X2, X3, . . . Xm) Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 13 / 38
  15. 15. Mean Square Error • EQM ˆθ := ˆθ − θ 2 • EQM ˆθ = ˆθ2 − 2θˆθ + θ2 = ˆθ2 − 2θ ˆθ + θ2 = ˆθ2 − ˆθ 2 + ˆθ 2 − 2θ ˆθ + θ2 EQM ˆθ = Var ˆθ − B2 (ˆθ) • Bias: B(ˆθ) = ˆθ − θ • Non-biased estimator: B(ˆθ) = 0 ⇐⇒ ˆθ = θ Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 14 / 38
  16. 16. Example to gaussian mean and variance • Random sample: X1, X2, X3, . . . , Xm • Starting with the mean value: ˆµ = X := 1 m m j=1 Xj • ˆµ = 1 m m j=1 Xj = 1 m m j=1 Xj = 1 m m j=1 µ = µ Thus, X is a non-biased estimator for µ • Now let’s take a look at the • EQM [ˆµ] = Var X = Var 1 m m j=1 Xj = 1 m2 m j=1 Var [Xj] = 1 m2 m j=1 σ2 = σ2 m Moreover, X → µ when m → ∞ Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 15 / 38
  17. 17. Example to gaussian mean and variance • Let’s now do it with the variance: ˆσ2 b = 1 m m j=1 Xj − X 2 • ˆσ2 b = 1 m m j=1 Xj − X 2 = 1 m m j=1 Xj − X 2 = m − 1 m σ2 Thus, ˆσb is asymptotically non-biased estimator for σ • Conversely, let’s define ˆσ2 = 1 m − 1 m j=1 Xj − X 2 • EQM [ˆµ] = σ2 ˆσ is an unbiased estimator for σ Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 16 / 38
  18. 18. Back to Gaussian random variables 0 1 2 3 4 5 6 0 100 200 300 400 500 600 700 EventXj Realization j • Estimation: ˆσb = 0.49944735873 • Actual value used to generate the data: ˆσ = 0.49980448952 Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 17 / 38
  19. 19. Where are we? 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 18 / 38
  20. 20. Maximum Likelihood • Maximize the ”likelihood“: get an estimator for te parameter of a given statistical model • p(x1, x2, x3, . . . |θ) :: given the value θ for the parameter, this is the probability that X1 = x1, X2 = x2, . . . • p(x1, x2, x3, . . . |θ) = p(x1|θ)p(x2|θ)p(x3|θ) . . . = m j=1 p(xj|θ) • Let x = x1, x2, x3, . . . L(θ, x) = m j=1 p(xj|θ) • To maximize it, we can use ln (L(θ, x)) = m j=1 p(xj|θ) • ∂ ∂θ ln (L(θ, x)) = 0 – solve it for θ Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 19 / 38
  21. 21. Example for gaussian • Suppose Xj ∼ N(µ, σ) is our statistical model • L(ˆµ, x) = 1 √ 2πσ2 m exp 1 2σ2 m j=1 (xj − ˆµ)2 • By calculating ∂ ∂ˆµ ln (L(ˆµ, x)) = 0, we get to m j=1 (xj − ˆµ) = m j=1 xj − mˆµ = 0 • Finally, we get ˆµ = 1 m m j=1 xj • On the other hand, ∂ ∂ˆσ ln (L(ˆσ, x)) = 0 gives ........ • This is what is usually done to derive such an estimator Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 20 / 38
  22. 22. Example for gaussian • Suppose Xj ∼ N(µ, σ) is our statistical model • L(ˆµ, x) = 1 √ 2πσ2 m exp 1 2σ2 m j=1 (xj − ˆµ)2 • By calculating ∂ ∂ˆµ ln (L(ˆµ, x)) = 0, we get to m j=1 (xj − ˆµ) = m j=1 xj − mˆµ = 0 • Finally, we get ˆµ = 1 m m j=1 xj • On the other hand, ∂ ∂ˆσ ln (L(ˆσ, x)) = 0 gives ˆσ = 1 m m j=1 Xj − X 2 , which we have already discovered to be biased! • This is what is usually done to derive such an estimator Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 20 / 38
  23. 23. Deutsch Tank Problem • Suppose you have a box with (unkown) n tickets, labled from 1 to n. • You take one ticket, it’s label is x. What’s your best guess for n? Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 21 / 38
  24. 24. Estimation of the maximum • Statistical model: p(x|ˆn) = 1 ˆn , x ≤ ˆn – a uniform distribution • Remember p(x|ˆn) = 0, x > ˆn. This is important to this maximization. • L(ˆn, x) = 1 ˆn , which is maximized when ˆn is the largest! • Therefore, our best guess at this moment would be ˆn = x. • If we had conversely made several observations X1, X2, X3, . . ., then ˆn = max (X1, X2, X3, . . .) • ”However, this is awful“ – it is a poor estimation! • Other methods are far more accurate and, indeed, were succesfully used in WWII. Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 22 / 38
  25. 25. Where are we? 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 23 / 38
  26. 26. What is this? Large values significant? Is the mean informative? Probably not. 0 10 20 30 40 50 60 70 0 20 40 60 80 100 120 140 160 Xj Realization j Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 24 / 38
  27. 27. Empirical PDF • We don’t know p(x)∆x, but we have a lot data... • In this experiment, I will use 107 points. • We can then calculate the following estimator: ˆp(x) = 1 m m j=1 δ (Xj ∈ [x, x + ∆x]) 10-8 10-7 10-6 10-5 10-4 10-3 10-2 10-1 100 100 101 102 103 104 P(X=x) Event X = x In fact, we can show that ˆp(x) is an unbiased estimator for p(x). Let’s propose then a model: p(x) ∼ x−α But... α =? Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 25 / 38
  28. 28. Empirical CDF If you want to fit, use the CDF 10-7 10-6 10-5 10-4 10-3 10-2 10-1 100 100 101 102 103 104 105 P(X>=x) Event X = x However, how about a maximum likelihood estimator? Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 26 / 38
  29. 29. Exponent estimator • General power-law distribution: p(x)dx = α − 1 xmin x xmin −α dx • We have to maximize the likelihood: L (ˆα|X1, . . . Xm) = m j=1 ˆα − 1 xmin Xj xmin −ˆα • ln [L (α|X1, . . . Xm)] = m j=1 ln(α − 1) − ln(xmin) − α ln Xj xmin • Maximizing it... ˆα = 1 + m m j=1 Xj/xmin Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 27 / 38
  30. 30. PDF In our case, ˆα ∼ 2.65 10-8 10-7 10-6 10-5 10-4 10-3 10-2 10-1 100 100 101 102 103 104 P(X=x) Event X = x Testing data Fitting curve Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 28 / 38
  31. 31. Where are we? 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 29 / 38
  32. 32. What about this... -250 -200 -150 -100 -50 0 50 100 150 0 50 100 150 200 250 300 EventXj Realization j Let’s start by the empirical pdf! Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 30 / 38
  33. 33. Gaussian!!! :) Again, we can propose a model: A Gaussian! 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 -4 -2 0 2 4 P(X=x) Event X = x Estimating the variance: ˆσ ≈ 200 Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 31 / 38
  34. 34. Gaussian!!! :( Again, we can propose a model: A Gaussian! 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 -4 -2 0 2 4 P(X=x) Event X = x Empirical PDF Fitting Estimating the variance: ˆσ ≈ 200 – Something is not right! Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 31 / 38
  35. 35. Okay, not a gaussian 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 -10 -5 0 5 10 P(X=x) Event X = x Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 32 / 38
  36. 36. Okay, not a gaussian 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 -20 -15 -10 -5 0 5 10 15 20 P(X=x) Event X = x Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 32 / 38
  37. 37. Power-law decay – Cauchy distributions 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 1 10 100 1000 10000 100000 P(X=x) Event X = x Empirical CDF Very suggestive indeed. Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 33 / 38
  38. 38. Where are we? 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 34 / 38
  39. 39. Hypothesis testing • What we have been doing till now is ∼ exploratory analysis • We now want to confirm a prediction or hypothesis – confirmatory analysis • Is this last data set gaussian or cauchy distributed? • The general recipe: - An initial guess, possibly true - State an relevant null and its alternative hypothesis - Formulate an appropriate test – T and a significance level τ - Estimate the distribution of T under your null hypothesis - Compute the observed quantity t and verify your null hypothesis Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 35 / 38
  40. 40. Let’s try this shit! • H0: the data is normally distributed – N(0, σ) • We have the estimated pdf of the data sample – these are our m observations Oj • The test will be T = m j=1 (Oj − Ej)2 Ej • Ej = A exp − (x)2 σ2 are the expected frequencies! • It is easy to derive that T ∼ χ2 m−1 (Bolfarine) • In our case, the observed t ≈ 400 and P (T = t) → 0. • This rejects H0. Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 36 / 38
  41. 41. Suggestions • Other hypothesis testing techniques: * τ-Student test * minimax * Lagrange multiplier * Union-intersection * Fisher test * ... • Non-parametric testins, such as Kolmogorov-Smirnov Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 37 / 38
  42. 42. Where are we? 1 Probability and random variables 2 Estimators 3 Maximum likelihood 4 Non-trivial example 5 One last example 6 Hypothesis testing 7 Conclusions Thiago Mosqueiro (IFSC - USP) Classical inference in physics 31/07/2012 38 / 38

×