I use this presentation to teach students to evaluate 6 aspects of popular press articles: 1. What are the credentials & motivation of the person making the claim? 2. Is evidence based on correlation or causation? 3. What is the inference space? 4. How large is the effect? 5. How large of a population was tested? 6. Is the effect statistically significant?

1. Evaluating Scientific Data
BIO 101L
Fall 2014

2. Why be cautious?
• Author’s motivations
are not the same as
your own
– Sell you on a drug,
therapy, hospital, etc
– Sell you a magazine
– ‘Publish or perish’
• Applies more broadly
than health claims Eric
Poehlman

3. What is pattern here?
Turner & Spillich 1997

4. Questions to ask:
1. What is the motivation of the person making
the claim?
2. Is evidence based on correlation or
causation?
3. What is the inference space?
4. How large is the effect?
5. How large of a population was tested?
6. Is the effect statistically significant?

5. Correlation or causation?
Correlation
• shows a pattern
• might suggest a cause
• pattern may instead result from unmeasured
3rd variable

6. Correlation or causation?

7. Do nightlights cause nearsightedness
(myopia)?
• What is
pattern?
• What is
explanation?

8. Do nightlights cause nearsightedness
(myopia)?
• What is
pattern?
• What is
explanation?

9. Inference space?
• What is the population being studied?

10. Inference space
• s

11. Inference space
• What is the range of treatments being
studied?

12. How large is the effect?
• ‘Increase’ or ‘Decrease’ alone does not
indicate whether it is enough to matter.
• What is measurement without treatment?
• How much does measurement change with
treatment?

13. Cell phones and brain cancer
•

14. How large of a population was tested?
•

15. How large of a population was tested?
•

16. Is effect statistically significant?
• Did results arise by chance?
• Or is the effect itself actually important?
• Often indicated by P-values
• Lower P-values means that chance is less likely
to explain pattern
• P < 0.05 is a common cut-off point
• Does article specify “statistically significant?”