Bayes' Theorem for Programmers

2,003 views

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,003
On SlideShare
0
From Embeds
0
Number of Embeds
53
Actions
Shares
0
Downloads
34
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Bayes' Theorem for Programmers

  1. 1. Bayes TheoremFor programmers
  2. 2. Joke● Whats the difference between a mathematician, an engineer and a programmer?
  3. 3. Punchline● Mathematicians use natural log (base e)● Engineers use decibels (10 times log base 10)● Programmers use bits (log base 2)
  4. 4. Useful functions● odds(p)=p/(1-p) – Gambler talk: 1/3 → “1-to-2”● logit(p)=log(odds(p)) – Remember: logs are base 2, or bits● expit(p)=exp(p/(1+p)) – Inverse of logit
  5. 5. What is belief?● Belief(X) = logit(Probability you assign to X) – Measured in bits● Fun fact: Belief(not X)=-Belief(X)
  6. 6. Examples● Belief(X)=0: probability 0.5, zero knowledge● Belief(X)=1: probability is 2/3● Belief(X)=-1: probability is 1/3● Belief(X)=5: probability about 0.97
  7. 7. More examples● Belief(X)=10: “I’m 99.9% certain about this!”● Belief(X)=-10: “There’s a 0.001 chance of that!”● Belief(X)=infinity: probability 1, or “The religious belief”…
  8. 8. Accuracy of belief● Overconfidence: >>1/expit(B) of beliefs of strength >B are wrong (for some B>0)● Underconfidence: <<1/expit(B) of beliefs of 0<strength<B are wrong (for some B>0)● Well-calibrated: Neither overconfident nor underconfident
  9. 9. Evidence● Event E happened. Is X true?● E is helpful only when P(E given X) != P(E given not X). But how much?● Likelihood(E given X) = P(E given X)/P(E given not X)● Evidence(E about X) = log(Likelihood(E given X))● Evidence is measured in bits!
  10. 10. THE FORMULA Belief(X after seeing E) =Belief(X)+Evidence(E about X)
  11. 11. Bayes Theorem● “If you are well-calibrated, and update beliefs according to THE FORMULA, you remain well-calibrated”● Corrolary: If you sometimes count evidence twice, or sometimes only weakly, you FALL OUT OF CALIBRATION!
  12. 12. Remember, Kids!Bayes’ Theorem is math, not a suggestion. If you care about being right, you can’t afford to ignore it!

×