SlideShare a Scribd company logo
1 of 92
Download to read offline
Frontiers of
Computational Journalism
Columbia Journalism School
Week 7: Randomness and Spooky Significance
October 31, 2018
This class
• Randomness
• Significance testing in Journalism
• $%#$! P-Values
• Bayesian inference
• The Garden of Forking Paths
• Analysis of Competing Hypotheses
Randomness
Margin of Error
Which one is random?
One star per box – “less” random
Two principles of randomness
1. Random data has “patterns” in it way more often than you
think.
2. This problem gets much more extreme when you have
less data.
Is this die loaded?
How about this one?
Is this one loaded?
Two dice: non-uniform distribution
Is something causing cancer?
Cancer rate per county. Darker = greater incidence of cancer.
From Graphical Inference for Infovis, Wickham et. Al.
Which of these is real data?
Global temperature record
How likely is it that the temperature won't increase over next decade?
From The Signal and the Noise, Nate Silver
It is conceivable that the 14 elderly people who are reported to have died soon
after receiving the vaccination died of other causes. Government officials in
charge of the program claim that it is all a coincidence, and point out that old
people drop dead every day. The American people have even become familiar
with a new statistic: Among every 100,000 people 65 to 75 years old, there will
be nine or ten deaths in every 24-hour period under most normal
circumstances.
Even using the official statistic, it is disconcerting that three elderly people in
one clinic in Pittsburgh, all vaccinated within the same hour, should die within a
few hours thereafter. This tragedy could occur by chance, but the fact remains
that it is extremely improbable that such a group of deaths should take place in
such a peculiar cluster by pure coincidence.
- New York Times editorial, 14 October 1976
Assuming that about 40 percent of elderly Americans were vaccinated within
the first 11 days of the program, then about 9 million people aged 65 and
older would have received the vaccine in early October 1976. Assuming that
there were 5,000 clinics nationwide, this would have been 164 vaccinations
per clinic per day. A person aged 65 or older has about a 1-in-7,000 chance
of dying on any particular day; the odds of at least three such people dying
on the same day from among a group of 164 patients are indeed very long,
about 480,000 to one against. However, under our assumptions, there were
55,000 opportunities for this “extremely improbable” event to occur—5,000
clinics, multiplied by 11 days. The odds of this coincidence occurring
somewhere in America, therefore, were much shorter—only about 8 to 1
- Nate Silver, The Signal and the Noise, Ch. 7 footnote 20
The Howland Will Trial
Significance Testing in Journalism
Randomization to detect insider trading
Looking at executives' trading in the week before their companies made news,
the Journal found that one of every 33 who dipped in and out posted average
returns of more than 20% (or avoided 20% downturns) in the following week.
By contrast, only one in 117 executives who traded in an annual pattern did that
well.
Executives’ Good Luck in Trading Own Stock, Wall Street Journal, 2012
Randomization to detect tennis fixing
Why look at betting data? Well, the main point of fixing a match is to make
money off the betting. In a normal match, some people bet that one player will
win and some people bet on the other, based on the odds that bookmakers
have set. But if huge bets start pouring in on one side, that looks very much like
a sign that some gamblers think they know more than the bookmaker about
how that match is going to go. Perhaps they know one player is going to tank.
…
To estimate how often they should have been expected to lose, I ran 1 million
computer simulations per player.
How BuzzFeed News Used Betting Data To Investigate Match-Fixing In Tennis, John
Templon, Buzzfeed, 2016
Problems with statistical tests alone
“It’s very, very dangerous to make blasé assumptions about a match being
dubious because of prematch movements,” Dan Weston, a tennis analyst and
trader who writes for the website of the sports book Pinnacle, said in a
telephone interview. (Using only data on betting and results to demonstrate
fixing has proven problematic in other sports.)
“By itself, the analysis of betting data does not prove match-fixing,” Schoofs
said in his statement. “That’s why we did not name the players and are
declining to comment, and also why our investigation went much wider than the
algorithm and was based on a cache of leaked documents, interviews across
three continents, and much more.”
Why Betting Data Alone Can’t Identify Match Fixers In Tennis, FiveThirtyEight
Detecting campaign finance violations?
In late October 2016, Donald Trump’s personal attorney Michael Cohen paid
adult star Stormy Daniels $130,000 in order to purchase her silence about an
alleged affair a decade earlier. … Sharp-eyed observers have noted that, in late
October 2016, the Trump campaign made a series of five large payments to
Trump-affiliated entities, totaling $129,999.72.
Ultimately, our model suggests that the probability of a set of payments
coincidentally coming so close to $130,000 is approximately 0.1%, or one out
of one thousand. In other words, about 99.9% of the time, random chance
would not produce a set of payments this close to $130,000. Therefore, the
probability that the Trump campaign payments were related to the Daniels
payoff is very high.
Statistical Model Strongly Suggests the Stormy Daniels Payoff Came from the
Trump Campaign, Will Stancil
Statistical Model Strongly Suggests the Stormy Daniels Payoff Came from the
Trump Campaign, Will Stancil
“The simulation confirmed that it is extremely unlikely that, by random chance
alone, a set of payments near a specific date would almost equal $130,000.”
$%@*! P-Values
P-value
p(observed data > your data | null hypothesis)
What’s it good for? What’s it bad for?
From A dirty dozen: twelve p-value misconceptions, S.Goodman
T-test for two groups with different variance. Expected to have T-
distribution under under null hypothesis of equal scores
Is one classroom better than another?
Things that depend on which classroom a student is in
Things that don’t depend on which classroom they’re in
Reasons for possible differences
Things that depend on which classroom a student is in
Things that don’t depend on which classroom they’re in
Reasons for possible differences
Break the relationship
observed difference
between classes
observed difference
between classes
14% of all resamples have a class difference > observed, so p = 0.14
New samples from the data
Boostrapping: resample with repetition. This gives an excellent
approximation of the sampling distribution, even if non-normal.
Computing the sampling distribution
A dirty dozen: twelve p-value misconceptions, S. Goodman
A dirty dozen: twelve p-value misconceptions, S. Goodman
Bayesian Inference
Conditional Probability
Pr(B|A) = Pr(AB)/Pr(A)
Accident
No Accident
Blue Yellow
Accident
No Accident
Blue Yellow
P(Accident|Blue) = 0.1
Relative risk as conditional probability
N = a+b+c+d
N(disease) = a+c
N(no disease) = b+d
Pr(disease) = a+c / a+b+c+d
Pr(disease|smoker) = a / (a+b)
Pr(disease|non-smoker) = c / (c+d)
RR = Pr(disease|smoker)/Pr(disease|non-smoker) = (a/a+b) / (c/c+d)
Base Rates - Taxi Accidents
Imagine you live in a city where 15% of all rides end in an
accident, and last year there were
- 75 accidents involving yellow cabs
- 25 accidents involving blue cabs
Which taxi company is more dangerous?
Base rate
We know
P(accident) = 0.15
P(accident|blue) = 0.25
P(accident|yellow) = 0.75
We do not know the “base rate”:
P(yellow)
or equivalently
N(yellow)
Evidence and Conditional Probability
Hypothesis H = Alice has a cold
Evidence E = we just saw her cough
Alice is coughing. Does she have a cold?
Most people with colds cough
P(coughing|cold) = 0.9
P(A|B) ≠ P(B|A)
Most people with colds cough
P(coughing|cold) = 0.9
but we want
P(cold | coughing)
Bayes’ Theorem
Tells us how to go from Pr(A|B) to Pr(B|A)
Pr(B|A) = Pr(A|B)Pr(B) / Pr(A)
Alice is coughing. Does she have a cold?
Prior P(H) = 0.05 (5% of our friends have a cold)
Likelihood P(E|H) = 0.9 (most people with colds cough)
Base rate P(E) = 0.1 (10% of everyone coughs today)
P(H|E) = P(E|H)P(H)/P(E)
= 0.9 * 0.05 / 0.1
= 0.45
If you believe your initial probability estimates, you should now
believe there's a 45% chance she has a cold.
Bayes’ Theorem - Diagnostic tests
Suppose I tell you:
• 14 of 1000 women under 50 have breast cancer
• If a woman has cancer, a mammogram is positive 75%
of the time
• If a woman does not have cancer, a mammogram is
positive 10% of the time
If a woman has a positive mammogram, how likely is she
to have cancer?
The Signal and the Noise, Nate Silver
cancer
no cancer
positive negative
cancer
no cancer
positive negative
Pr(positive|cancer) = 0.75
= N(positive & cancer) / N(cancer)
N(cancer) = 4
N(positive & cancer) = 3
cancer
no cancer
positive negative
Pr(positive|no cancer) = 0.1
= N(positive & no cancer) / N(positive)
N(no cancer) = 1000
N(positive & no cancer) = 100
cancer
no cancer
positive negative
Pr(cancer) = 0.0014
= N(cancer) / N
Conditional probabilities
Pr(positive|cancer) = 75%
Pr(positive|no cancer) = 10%
What is Pr(cancer|positive)?
cancer
no cancer
positive negative
Pr(cancer|positive)
= 9.6%
Bayesian diagnostics
Pr(cancer|positive) =
Pr(positive|cancer) Pr(cancer) / Pr(positive)
Pr(positive|cancer) = 0.75
Pr(cancer) = 0.014
Pr(positive) = Pr(positive|no cancer)Pr(no cancer) +
Pr(positive|cancer)Pr(cancer)
= 0.10*0.986 + 0.75*0.014
= 0.1091
Bayesian diagnostics
Pr(cancer|positive) =
Pr(positive|cancer) Pr(cancer) / Pr(positive)
= (0.75 * 0.014) / (0.1091)
= 0.0962
= 9.6% chance she has cancer
if mammogram is positive
Evidence
Information that justifies a belief.
Presented with evidence E for X, we should believe X "more."
In terms of probability, P(X|E) > P(X)
Bayes “learns” from evidence
Pr(H|E) = Pr(E|H) Pr(H) / Pr(E)
or
P(H|E) = Pr(E|H)/Pr(E) * Pr(H)
Posterior
How likely is H
given evidence E?
Prior
How likely was
H to begin with?
Likelihood
Probability of
seeing E
if H is true
Base Rate
How commonly
do we see E at all?
A more complete theory
Compare probability of multiple alternatives.
Did the stoplight reduce accidents?
1
02468
2
02468
3
02468
4
02468
5
02468
6
02468
7
02468
8
02468
9
02468
Simulated without stoplight
1
02468
2
02468
3
02468
4
02468
5
02468
6
02468
7
02468
8
02468
9
02468
Simulated with a 50% effective stoplight
Probability distribution over hypotheses
Is the NYPD targeting mosques for stop-and-frisk?
1
0
H0 H1 H2
Never RoutinelyOnce or twice
*Tricky: you have to imagine a hypothesis before you can assign it a
probability.
Parameter Estimation
Computing probability for a continuum of hypotheses
P(𝛳|E) = Pr(E|𝛳)/Pr(E) * Pr(𝛳)
Relative strength of evidence
Can we find a p-value equivalent?
There is “Bayes factor”
Pr(H1|E)/Pr(H2|E)
= [Pr(E|H1)Pr(H1)/Pr(E)] / [Pr(E|H2)Pr(H2)/Pr(E)]
= Pr(E|H1)/Pr(E|H2) * Pr(H1)/Pr(H2)
Bayes Factor
Ok, but what’s a “significant” Bayes Factor?
From Bayes Factors, Kass and Raftery
There’s this, but the whole idea of “significance” is
probably flawed.
The Garden of Forking Paths
I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How.
John Bohannon
Science Isn’t Broken, FiveThirtyEight
“Statistical significance” is usually asking the wrong question.
Does the model reproduce the data?
Testing for Racial Discrimination in Police Searches of Motor Vehicles, Simoiu et al.
Analysis
of Competing Hypotheses
Cognitive biases
Availability heuristic: we use examples that come to mind, instead of
statistics.
Preference for earlier information: what we learn first has a much
greater effect on our judgment.
Memory formation: whatever seems important at the time is what gets
remembered.
Confirmation bias: we seek out and give greater importance to
information that confirms our expectations.
Confirmation bias
Comes in many forms.
...unconsciously filtering information that doesn't fit expectations.
...not looking for contrary information.
...not imagining the alternatives.
Method of competing hypotheses
Start with multiple hypotheses H0, H1, ... HN
(Remember, if you can't imagine it, you can't conclude it!)
Go looking for information that gives you the best ability to discriminate
between hypotheses.
Evidence which supports Hi is much less useful than evidence which supports
Hi much more than Hj, if the goal is to choose a hypothesis.
In practice: Triangulation
A good conclusion is one which is supported by multiple lines of evidence from
multiple methods.
“Philosophy ought to imitate the successful sciences in its methods, so far as to
proceed only from tangible premises which can be subjected to careful scrutiny,
and to trust rather to the multitude and variety of its arguments than to the
conclusiveness of any one. Its reasoning should not form a chain which is no
stronger than its weakest link, but a cable whose fibers may be ever so slender,
provided they are sufficiently numerous and intimately connected.”
- Charles Sanders Peirce
A difficult example
NYPD performs ~600,000 street stop and frisks per year.
What sorts of conclusions could we draw from this data?
How?
Stop and Frisk Causation
Suppose you take the address of every mosque in NYC, and
discover that there are 15% more stop-and-frisks within 100m of
mosques than the overall average.
Can we conclude that the police are targeting Muslims?

More Related Content

Similar to Frontiers of Computational Journalism week 7 - Randomness and Statistical Significance

Writing about P-values in science journalism, for National Association of Sci...
Writing about P-values in science journalism, for National Association of Sci...Writing about P-values in science journalism, for National Association of Sci...
Writing about P-values in science journalism, for National Association of Sci...Regina Nuzzo
 
Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)
Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)
Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)Brian Hughes
 
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Daniel Katz
 
Pre-Cal 40S Slides January 10, 2008
Pre-Cal 40S Slides January 10, 2008Pre-Cal 40S Slides January 10, 2008
Pre-Cal 40S Slides January 10, 2008Darren Kuropatwa
 
3Statistics in Criminal JusticeHomework 6 Each question is.docx
3Statistics in Criminal JusticeHomework 6 Each question is.docx3Statistics in Criminal JusticeHomework 6 Each question is.docx
3Statistics in Criminal JusticeHomework 6 Each question is.docxrhetttrevannion
 
Analyzing data in health care Dr.Majdi
Analyzing data in health care Dr.MajdiAnalyzing data in health care Dr.Majdi
Analyzing data in health care Dr.MajdiDr. Majdi Al Jasim
 
Introduction to probability
Introduction to probabilityIntroduction to probability
Introduction to probabilityGlobal Polis
 
Business stats assignment
Business stats assignmentBusiness stats assignment
Business stats assignmentInfosys
 
Slideshare cork skep3
Slideshare cork skep3Slideshare cork skep3
Slideshare cork skep3Brian Hughes
 
Covering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.orgCovering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.orgGary Schwitzer
 
introduction to biostatistics in clinical trials
introduction to biostatistics in clinical trialsintroduction to biostatistics in clinical trials
introduction to biostatistics in clinical trialsstopit2404
 
introduction to biostatistics in clinical trials
introduction to biostatistics in clinical trialsintroduction to biostatistics in clinical trials
introduction to biostatistics in clinical trialsstopit2404
 
Insights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibilityInsights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibilityDorothy Bishop
 
MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012GarySchwitzer
 
Introduction to Statistics - Part 2
Introduction to Statistics - Part 2Introduction to Statistics - Part 2
Introduction to Statistics - Part 2Damian T. Gordon
 
Politics Essay Writing. Online assignment writing service.
Politics Essay Writing. Online assignment writing service.Politics Essay Writing. Online assignment writing service.
Politics Essay Writing. Online assignment writing service.Michelle Brown
 
O Behave! Issue 11 - February 2015
O Behave! Issue 11 - February 2015O Behave! Issue 11 - February 2015
O Behave! Issue 11 - February 2015#ogilvychange
 

Similar to Frontiers of Computational Journalism week 7 - Randomness and Statistical Significance (20)

Writing about P-values in science journalism, for National Association of Sci...
Writing about P-values in science journalism, for National Association of Sci...Writing about P-values in science journalism, for National Association of Sci...
Writing about P-values in science journalism, for National Association of Sci...
 
Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)
Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)
Psychology, Science, and Pseudoscience: Class #06 (Probability Fails)
 
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
 
Pre-Cal 40S Slides January 10, 2008
Pre-Cal 40S Slides January 10, 2008Pre-Cal 40S Slides January 10, 2008
Pre-Cal 40S Slides January 10, 2008
 
3Statistics in Criminal JusticeHomework 6 Each question is.docx
3Statistics in Criminal JusticeHomework 6 Each question is.docx3Statistics in Criminal JusticeHomework 6 Each question is.docx
3Statistics in Criminal JusticeHomework 6 Each question is.docx
 
Analyzing data in health care Dr.Majdi
Analyzing data in health care Dr.MajdiAnalyzing data in health care Dr.Majdi
Analyzing data in health care Dr.Majdi
 
Introduction to probability
Introduction to probabilityIntroduction to probability
Introduction to probability
 
Business stats assignment
Business stats assignmentBusiness stats assignment
Business stats assignment
 
Slideshare cork skep3
Slideshare cork skep3Slideshare cork skep3
Slideshare cork skep3
 
P-values in crisis
P-values in crisisP-values in crisis
P-values in crisis
 
Covering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.orgCovering Cancer News - Lessons from HealthNewsReview.org
Covering Cancer News - Lessons from HealthNewsReview.org
 
introduction to biostatistics in clinical trials
introduction to biostatistics in clinical trialsintroduction to biostatistics in clinical trials
introduction to biostatistics in clinical trials
 
introduction to biostatistics in clinical trials
introduction to biostatistics in clinical trialsintroduction to biostatistics in clinical trials
introduction to biostatistics in clinical trials
 
Insights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibilityInsights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibility
 
O Behave! Issue 26
O Behave! Issue 26O Behave! Issue 26
O Behave! Issue 26
 
MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012
 
Introduction to Statistics - Part 2
Introduction to Statistics - Part 2Introduction to Statistics - Part 2
Introduction to Statistics - Part 2
 
Probability
ProbabilityProbability
Probability
 
Politics Essay Writing. Online assignment writing service.
Politics Essay Writing. Online assignment writing service.Politics Essay Writing. Online assignment writing service.
Politics Essay Writing. Online assignment writing service.
 
O Behave! Issue 11 - February 2015
O Behave! Issue 11 - February 2015O Behave! Issue 11 - February 2015
O Behave! Issue 11 - February 2015
 

More from Jonathan Stray

Frameworks for Algorithmic Bias
Frameworks for Algorithmic BiasFrameworks for Algorithmic Bias
Frameworks for Algorithmic BiasJonathan Stray
 
Frontiers of Computational Journalism week 11 - Privacy and Security
Frontiers of Computational Journalism week 11 - Privacy and SecurityFrontiers of Computational Journalism week 11 - Privacy and Security
Frontiers of Computational Journalism week 11 - Privacy and SecurityJonathan Stray
 
Frontiers of Computational Journalism week 10 - Truth and Trust
Frontiers of Computational Journalism week 10 - Truth and TrustFrontiers of Computational Journalism week 10 - Truth and Trust
Frontiers of Computational Journalism week 10 - Truth and TrustJonathan Stray
 
Frontiers of Computational Journalism week 9 - Knowledge representation
Frontiers of Computational Journalism week 9 - Knowledge representationFrontiers of Computational Journalism week 9 - Knowledge representation
Frontiers of Computational Journalism week 9 - Knowledge representationJonathan Stray
 
Frontiers of Computational Journalism week 8 - Visualization and Network Anal...
Frontiers of Computational Journalism week 8 - Visualization and Network Anal...Frontiers of Computational Journalism week 8 - Visualization and Network Anal...
Frontiers of Computational Journalism week 8 - Visualization and Network Anal...Jonathan Stray
 
Frontiers of Computational Journalism - Final project suggestions
Frontiers of Computational Journalism - Final project suggestionsFrontiers of Computational Journalism - Final project suggestions
Frontiers of Computational Journalism - Final project suggestionsJonathan Stray
 
Frontiers of Computational Journalism week 4 - Statistical Inference
Frontiers of Computational Journalism week 4 - Statistical InferenceFrontiers of Computational Journalism week 4 - Statistical Inference
Frontiers of Computational Journalism week 4 - Statistical InferenceJonathan Stray
 
Frontiers of Computational Journalism week 3 - Information Filter Design
Frontiers of Computational Journalism week 3 - Information Filter DesignFrontiers of Computational Journalism week 3 - Information Filter Design
Frontiers of Computational Journalism week 3 - Information Filter DesignJonathan Stray
 
Frontiers of Computational Journalism week 2 - Text Analysis
Frontiers of Computational Journalism week 2 - Text AnalysisFrontiers of Computational Journalism week 2 - Text Analysis
Frontiers of Computational Journalism week 2 - Text AnalysisJonathan Stray
 
Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...
Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...
Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...Jonathan Stray
 

More from Jonathan Stray (10)

Frameworks for Algorithmic Bias
Frameworks for Algorithmic BiasFrameworks for Algorithmic Bias
Frameworks for Algorithmic Bias
 
Frontiers of Computational Journalism week 11 - Privacy and Security
Frontiers of Computational Journalism week 11 - Privacy and SecurityFrontiers of Computational Journalism week 11 - Privacy and Security
Frontiers of Computational Journalism week 11 - Privacy and Security
 
Frontiers of Computational Journalism week 10 - Truth and Trust
Frontiers of Computational Journalism week 10 - Truth and TrustFrontiers of Computational Journalism week 10 - Truth and Trust
Frontiers of Computational Journalism week 10 - Truth and Trust
 
Frontiers of Computational Journalism week 9 - Knowledge representation
Frontiers of Computational Journalism week 9 - Knowledge representationFrontiers of Computational Journalism week 9 - Knowledge representation
Frontiers of Computational Journalism week 9 - Knowledge representation
 
Frontiers of Computational Journalism week 8 - Visualization and Network Anal...
Frontiers of Computational Journalism week 8 - Visualization and Network Anal...Frontiers of Computational Journalism week 8 - Visualization and Network Anal...
Frontiers of Computational Journalism week 8 - Visualization and Network Anal...
 
Frontiers of Computational Journalism - Final project suggestions
Frontiers of Computational Journalism - Final project suggestionsFrontiers of Computational Journalism - Final project suggestions
Frontiers of Computational Journalism - Final project suggestions
 
Frontiers of Computational Journalism week 4 - Statistical Inference
Frontiers of Computational Journalism week 4 - Statistical InferenceFrontiers of Computational Journalism week 4 - Statistical Inference
Frontiers of Computational Journalism week 4 - Statistical Inference
 
Frontiers of Computational Journalism week 3 - Information Filter Design
Frontiers of Computational Journalism week 3 - Information Filter DesignFrontiers of Computational Journalism week 3 - Information Filter Design
Frontiers of Computational Journalism week 3 - Information Filter Design
 
Frontiers of Computational Journalism week 2 - Text Analysis
Frontiers of Computational Journalism week 2 - Text AnalysisFrontiers of Computational Journalism week 2 - Text Analysis
Frontiers of Computational Journalism week 2 - Text Analysis
 
Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...
Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...
Frontiers of Computational Journalism week 1 - Introduction and High Dimensio...
 

Recently uploaded

LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF NAT...
LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF  NAT...LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF  NAT...
LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF NAT...pragatimahajan3
 
4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptxmary850239
 
Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...
Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...
Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...DrVipulVKapoor
 
4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptxmary850239
 
Self directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDLSelf directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDLspmdoc
 
DORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of Cybersecurity
DORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of CybersecurityDORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of Cybersecurity
DORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of CybersecurityPECB
 
CLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptxCLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptxAnupam32727
 
PART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFE
PART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFEPART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFE
PART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFEMISSRITIMABIOLOGYEXP
 
Supply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdfSupply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdftomeskell101
 
DiskStorage_BasicFileStructuresandHashing.pdf
DiskStorage_BasicFileStructuresandHashing.pdfDiskStorage_BasicFileStructuresandHashing.pdf
DiskStorage_BasicFileStructuresandHashing.pdfChristalin Nelson
 
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdfMJDuyan
 
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...Nguyen Thanh Tu Collection
 
How to Share Dashboard in the Odoo 17 ERP
How to Share Dashboard in the Odoo 17 ERPHow to Share Dashboard in the Odoo 17 ERP
How to Share Dashboard in the Odoo 17 ERPCeline George
 
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfijoemmbrillantes
 
18. Training and prunning of horicultural crops.pptx
18. Training and prunning of horicultural crops.pptx18. Training and prunning of horicultural crops.pptx
18. Training and prunning of horicultural crops.pptxUmeshTimilsina1
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...Nguyen Thanh Tu Collection
 
What is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP ModuleWhat is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP ModuleCeline George
 

Recently uploaded (20)

CARNAVAL COM MAGIA E EUFORIA _
CARNAVAL COM MAGIA E EUFORIA            _CARNAVAL COM MAGIA E EUFORIA            _
CARNAVAL COM MAGIA E EUFORIA _
 
LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF NAT...
LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF  NAT...LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF  NAT...
LEVERAGING SYNERGISM INDUSTRY-ACADEMIA PARTNERSHIP FOR IMPLEMENTATION OF NAT...
 
4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx4.9.24 School Desegregation in Boston.pptx
4.9.24 School Desegregation in Boston.pptx
 
Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...
Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...
Geoffrey Chaucer Works II UGC NET JRF TGT PGT MA PHD Entrance Exam II History...
 
4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx4.9.24 Social Capital and Social Exclusion.pptx
4.9.24 Social Capital and Social Exclusion.pptx
 
Self directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDLSelf directed Learning - SDL, introduction to SDL
Self directed Learning - SDL, introduction to SDL
 
DORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of Cybersecurity
DORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of CybersecurityDORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of Cybersecurity
DORA, ISO/IEC 27005, and the Rise of AI: Securing the Future of Cybersecurity
 
CLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptxCLASSIFICATION OF ANTI - CANCER DRUGS.pptx
CLASSIFICATION OF ANTI - CANCER DRUGS.pptx
 
PART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFE
PART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFEPART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFE
PART 1 - CHAPTER 1 - CELL THE FUNDAMENTAL UNIT OF LIFE
 
Supply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdfSupply agency market - aiming high 2.pdf
Supply agency market - aiming high 2.pdf
 
DiskStorage_BasicFileStructuresandHashing.pdf
DiskStorage_BasicFileStructuresandHashing.pdfDiskStorage_BasicFileStructuresandHashing.pdf
DiskStorage_BasicFileStructuresandHashing.pdf
 
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
(Part 3) CHILDREN'S DISABILITIES AND EXCEPTIONALITIES.pdf
 
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
CHUYÊN ĐỀ ÔN THEO CÂU CHO HỌC SINH LỚP 12 ĐỂ ĐẠT ĐIỂM 5+ THI TỐT NGHIỆP THPT ...
 
How to Share Dashboard in the Odoo 17 ERP
How to Share Dashboard in the Odoo 17 ERPHow to Share Dashboard in the Odoo 17 ERP
How to Share Dashboard in the Odoo 17 ERP
 
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
647291105-Ppt-Arts-10-4th-Quarter-1.pdfi
 
18. Training and prunning of horicultural crops.pptx
18. Training and prunning of horicultural crops.pptx18. Training and prunning of horicultural crops.pptx
18. Training and prunning of horicultural crops.pptx
 
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
BÀI TẬP BỔ TRỢ TIẾNG ANH 8 - I-LEARN SMART WORLD - CẢ NĂM - CÓ FILE NGHE (BẢN...
 
Israel Genealogy Research Assoc. April 2024 Database Release
Israel Genealogy Research Assoc. April 2024 Database ReleaseIsrael Genealogy Research Assoc. April 2024 Database Release
Israel Genealogy Research Assoc. April 2024 Database Release
 
Chi-Square Test Non Parametric Test Categorical Variable
Chi-Square Test Non Parametric Test Categorical VariableChi-Square Test Non Parametric Test Categorical Variable
Chi-Square Test Non Parametric Test Categorical Variable
 
What is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP ModuleWhat is Property Fields in Odoo 17 ERP Module
What is Property Fields in Odoo 17 ERP Module
 

Frontiers of Computational Journalism week 7 - Randomness and Statistical Significance

  • 1. Frontiers of Computational Journalism Columbia Journalism School Week 7: Randomness and Spooky Significance October 31, 2018
  • 2. This class • Randomness • Significance testing in Journalism • $%#$! P-Values • Bayesian inference • The Garden of Forking Paths • Analysis of Competing Hypotheses
  • 5. Which one is random?
  • 6. One star per box – “less” random
  • 7.
  • 8.
  • 9. Two principles of randomness 1. Random data has “patterns” in it way more often than you think. 2. This problem gets much more extreme when you have less data.
  • 10. Is this die loaded?
  • 12. Is this one loaded?
  • 13. Two dice: non-uniform distribution
  • 14.
  • 15. Is something causing cancer? Cancer rate per county. Darker = greater incidence of cancer. From Graphical Inference for Infovis, Wickham et. Al.
  • 16. Which of these is real data?
  • 17.
  • 18. Global temperature record How likely is it that the temperature won't increase over next decade?
  • 19. From The Signal and the Noise, Nate Silver
  • 20. It is conceivable that the 14 elderly people who are reported to have died soon after receiving the vaccination died of other causes. Government officials in charge of the program claim that it is all a coincidence, and point out that old people drop dead every day. The American people have even become familiar with a new statistic: Among every 100,000 people 65 to 75 years old, there will be nine or ten deaths in every 24-hour period under most normal circumstances. Even using the official statistic, it is disconcerting that three elderly people in one clinic in Pittsburgh, all vaccinated within the same hour, should die within a few hours thereafter. This tragedy could occur by chance, but the fact remains that it is extremely improbable that such a group of deaths should take place in such a peculiar cluster by pure coincidence. - New York Times editorial, 14 October 1976
  • 21. Assuming that about 40 percent of elderly Americans were vaccinated within the first 11 days of the program, then about 9 million people aged 65 and older would have received the vaccine in early October 1976. Assuming that there were 5,000 clinics nationwide, this would have been 164 vaccinations per clinic per day. A person aged 65 or older has about a 1-in-7,000 chance of dying on any particular day; the odds of at least three such people dying on the same day from among a group of 164 patients are indeed very long, about 480,000 to one against. However, under our assumptions, there were 55,000 opportunities for this “extremely improbable” event to occur—5,000 clinics, multiplied by 11 days. The odds of this coincidence occurring somewhere in America, therefore, were much shorter—only about 8 to 1 - Nate Silver, The Signal and the Noise, Ch. 7 footnote 20
  • 24. Randomization to detect insider trading
  • 25. Looking at executives' trading in the week before their companies made news, the Journal found that one of every 33 who dipped in and out posted average returns of more than 20% (or avoided 20% downturns) in the following week. By contrast, only one in 117 executives who traded in an annual pattern did that well. Executives’ Good Luck in Trading Own Stock, Wall Street Journal, 2012
  • 26.
  • 27. Randomization to detect tennis fixing Why look at betting data? Well, the main point of fixing a match is to make money off the betting. In a normal match, some people bet that one player will win and some people bet on the other, based on the odds that bookmakers have set. But if huge bets start pouring in on one side, that looks very much like a sign that some gamblers think they know more than the bookmaker about how that match is going to go. Perhaps they know one player is going to tank. … To estimate how often they should have been expected to lose, I ran 1 million computer simulations per player. How BuzzFeed News Used Betting Data To Investigate Match-Fixing In Tennis, John Templon, Buzzfeed, 2016
  • 28. Problems with statistical tests alone “It’s very, very dangerous to make blasé assumptions about a match being dubious because of prematch movements,” Dan Weston, a tennis analyst and trader who writes for the website of the sports book Pinnacle, said in a telephone interview. (Using only data on betting and results to demonstrate fixing has proven problematic in other sports.) “By itself, the analysis of betting data does not prove match-fixing,” Schoofs said in his statement. “That’s why we did not name the players and are declining to comment, and also why our investigation went much wider than the algorithm and was based on a cache of leaked documents, interviews across three continents, and much more.” Why Betting Data Alone Can’t Identify Match Fixers In Tennis, FiveThirtyEight
  • 29. Detecting campaign finance violations? In late October 2016, Donald Trump’s personal attorney Michael Cohen paid adult star Stormy Daniels $130,000 in order to purchase her silence about an alleged affair a decade earlier. … Sharp-eyed observers have noted that, in late October 2016, the Trump campaign made a series of five large payments to Trump-affiliated entities, totaling $129,999.72. Ultimately, our model suggests that the probability of a set of payments coincidentally coming so close to $130,000 is approximately 0.1%, or one out of one thousand. In other words, about 99.9% of the time, random chance would not produce a set of payments this close to $130,000. Therefore, the probability that the Trump campaign payments were related to the Daniels payoff is very high. Statistical Model Strongly Suggests the Stormy Daniels Payoff Came from the Trump Campaign, Will Stancil
  • 30. Statistical Model Strongly Suggests the Stormy Daniels Payoff Came from the Trump Campaign, Will Stancil “The simulation confirmed that it is extremely unlikely that, by random chance alone, a set of payments near a specific date would almost equal $130,000.”
  • 32. P-value p(observed data > your data | null hypothesis) What’s it good for? What’s it bad for? From A dirty dozen: twelve p-value misconceptions, S.Goodman
  • 33. T-test for two groups with different variance. Expected to have T- distribution under under null hypothesis of equal scores Is one classroom better than another?
  • 34. Things that depend on which classroom a student is in Things that don’t depend on which classroom they’re in Reasons for possible differences
  • 35. Things that depend on which classroom a student is in Things that don’t depend on which classroom they’re in Reasons for possible differences
  • 37.
  • 39. observed difference between classes 14% of all resamples have a class difference > observed, so p = 0.14
  • 40. New samples from the data
  • 41. Boostrapping: resample with repetition. This gives an excellent approximation of the sampling distribution, even if non-normal. Computing the sampling distribution
  • 42.
  • 43.
  • 44. A dirty dozen: twelve p-value misconceptions, S. Goodman
  • 45. A dirty dozen: twelve p-value misconceptions, S. Goodman
  • 50. Relative risk as conditional probability N = a+b+c+d N(disease) = a+c N(no disease) = b+d Pr(disease) = a+c / a+b+c+d Pr(disease|smoker) = a / (a+b) Pr(disease|non-smoker) = c / (c+d) RR = Pr(disease|smoker)/Pr(disease|non-smoker) = (a/a+b) / (c/c+d)
  • 51. Base Rates - Taxi Accidents Imagine you live in a city where 15% of all rides end in an accident, and last year there were - 75 accidents involving yellow cabs - 25 accidents involving blue cabs Which taxi company is more dangerous?
  • 52. Base rate We know P(accident) = 0.15 P(accident|blue) = 0.25 P(accident|yellow) = 0.75 We do not know the “base rate”: P(yellow) or equivalently N(yellow)
  • 53. Evidence and Conditional Probability Hypothesis H = Alice has a cold Evidence E = we just saw her cough
  • 54. Alice is coughing. Does she have a cold? Most people with colds cough P(coughing|cold) = 0.9
  • 55. P(A|B) ≠ P(B|A) Most people with colds cough P(coughing|cold) = 0.9 but we want P(cold | coughing)
  • 56. Bayes’ Theorem Tells us how to go from Pr(A|B) to Pr(B|A) Pr(B|A) = Pr(A|B)Pr(B) / Pr(A)
  • 57. Alice is coughing. Does she have a cold? Prior P(H) = 0.05 (5% of our friends have a cold) Likelihood P(E|H) = 0.9 (most people with colds cough) Base rate P(E) = 0.1 (10% of everyone coughs today) P(H|E) = P(E|H)P(H)/P(E) = 0.9 * 0.05 / 0.1 = 0.45 If you believe your initial probability estimates, you should now believe there's a 45% chance she has a cold.
  • 58. Bayes’ Theorem - Diagnostic tests Suppose I tell you: • 14 of 1000 women under 50 have breast cancer • If a woman has cancer, a mammogram is positive 75% of the time • If a woman does not have cancer, a mammogram is positive 10% of the time If a woman has a positive mammogram, how likely is she to have cancer?
  • 59. The Signal and the Noise, Nate Silver
  • 61. cancer no cancer positive negative Pr(positive|cancer) = 0.75 = N(positive & cancer) / N(cancer) N(cancer) = 4 N(positive & cancer) = 3
  • 62. cancer no cancer positive negative Pr(positive|no cancer) = 0.1 = N(positive & no cancer) / N(positive) N(no cancer) = 1000 N(positive & no cancer) = 100
  • 64. Conditional probabilities Pr(positive|cancer) = 75% Pr(positive|no cancer) = 10% What is Pr(cancer|positive)?
  • 66. Bayesian diagnostics Pr(cancer|positive) = Pr(positive|cancer) Pr(cancer) / Pr(positive) Pr(positive|cancer) = 0.75 Pr(cancer) = 0.014 Pr(positive) = Pr(positive|no cancer)Pr(no cancer) + Pr(positive|cancer)Pr(cancer) = 0.10*0.986 + 0.75*0.014 = 0.1091
  • 67. Bayesian diagnostics Pr(cancer|positive) = Pr(positive|cancer) Pr(cancer) / Pr(positive) = (0.75 * 0.014) / (0.1091) = 0.0962 = 9.6% chance she has cancer if mammogram is positive
  • 68. Evidence Information that justifies a belief. Presented with evidence E for X, we should believe X "more." In terms of probability, P(X|E) > P(X)
  • 69. Bayes “learns” from evidence Pr(H|E) = Pr(E|H) Pr(H) / Pr(E) or P(H|E) = Pr(E|H)/Pr(E) * Pr(H) Posterior How likely is H given evidence E? Prior How likely was H to begin with? Likelihood Probability of seeing E if H is true Base Rate How commonly do we see E at all?
  • 70. A more complete theory Compare probability of multiple alternatives.
  • 71. Did the stoplight reduce accidents?
  • 74. Probability distribution over hypotheses Is the NYPD targeting mosques for stop-and-frisk? 1 0 H0 H1 H2 Never RoutinelyOnce or twice *Tricky: you have to imagine a hypothesis before you can assign it a probability.
  • 75. Parameter Estimation Computing probability for a continuum of hypotheses P(𝛳|E) = Pr(E|𝛳)/Pr(E) * Pr(𝛳)
  • 76. Relative strength of evidence Can we find a p-value equivalent? There is “Bayes factor” Pr(H1|E)/Pr(H2|E) = [Pr(E|H1)Pr(H1)/Pr(E)] / [Pr(E|H2)Pr(H2)/Pr(E)] = Pr(E|H1)/Pr(E|H2) * Pr(H1)/Pr(H2) Bayes Factor
  • 77. Ok, but what’s a “significant” Bayes Factor? From Bayes Factors, Kass and Raftery There’s this, but the whole idea of “significance” is probably flawed.
  • 78. The Garden of Forking Paths
  • 79. I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How. John Bohannon
  • 80.
  • 81. Science Isn’t Broken, FiveThirtyEight
  • 82. “Statistical significance” is usually asking the wrong question.
  • 83. Does the model reproduce the data? Testing for Racial Discrimination in Police Searches of Motor Vehicles, Simoiu et al.
  • 85. Cognitive biases Availability heuristic: we use examples that come to mind, instead of statistics. Preference for earlier information: what we learn first has a much greater effect on our judgment. Memory formation: whatever seems important at the time is what gets remembered. Confirmation bias: we seek out and give greater importance to information that confirms our expectations.
  • 86. Confirmation bias Comes in many forms. ...unconsciously filtering information that doesn't fit expectations. ...not looking for contrary information. ...not imagining the alternatives.
  • 87. Method of competing hypotheses Start with multiple hypotheses H0, H1, ... HN (Remember, if you can't imagine it, you can't conclude it!) Go looking for information that gives you the best ability to discriminate between hypotheses. Evidence which supports Hi is much less useful than evidence which supports Hi much more than Hj, if the goal is to choose a hypothesis.
  • 88. In practice: Triangulation A good conclusion is one which is supported by multiple lines of evidence from multiple methods. “Philosophy ought to imitate the successful sciences in its methods, so far as to proceed only from tangible premises which can be subjected to careful scrutiny, and to trust rather to the multitude and variety of its arguments than to the conclusiveness of any one. Its reasoning should not form a chain which is no stronger than its weakest link, but a cable whose fibers may be ever so slender, provided they are sufficiently numerous and intimately connected.” - Charles Sanders Peirce
  • 89. A difficult example NYPD performs ~600,000 street stop and frisks per year. What sorts of conclusions could we draw from this data? How?
  • 90.
  • 91.
  • 92. Stop and Frisk Causation Suppose you take the address of every mosque in NYC, and discover that there are 15% more stop-and-frisks within 100m of mosques than the overall average. Can we conclude that the police are targeting Muslims?