Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
Abstract
Claims coming from human medical observational studies, when tested rigorously,
most often fail to replicate. W...
22
Contact
Information
Stan Young
National Institute of Statistical Sciences
www.niss.org
young@niss.org
919 685 9328
NISS
NISS 3
Hayek, 1974, Nobel Lecture
It is often difficult enough for the expert,
and certainly in many instances impossible
...
NISS 4
Hayek (2)
...much effort will have to be directed toward debunking
such arrogations*, some of which have by now bec...
5
Reliability of Literature Claims
S. Stanley Young
National Institute of Statistical Sciences
Young@niss.org, 919 685 932...
66
Science point of view
What is the meaning of life?
What is real?
What is reproducible?
Fooled (fooling) by randomness?
...
77
The Players
1. The workers – scientists
2. The communicators –
a. PR people
b. Bloggers
c. Reporters
d. Science writers...
88
The Worker is not the Problem.
W. Edwards Deming,
the most visionary innovator ever on quality control, said
The worker...
9
Problems with observational studies
“Everything is dangerous”
1. Data staging
2. No written analysis protocol
3. Multipl...
Assertion : Every study is positive
Data Staging
 Bias
Multiple testing
Multiple model searching
Any or all will lead ...
11
First, data staging
Stan:
Why do you think data staging is a big issue?
Because it can be done in myriad ways, is
rarel...
12
Multiple Testing: P-value, t-test
Population,
real or theoretical
Two samples,
random
NISS
10-sided dice experiment
12/25/12
NISS
14
How do you get a “p < 0.05”?
Answer: Ask lots of questions.
61 questions
95% chance of
a positive study!
NISS
1515
Let’s run an epidemiology study!
10-sided dice simulation:
Coffee causes X.
NISS
1616
P-value plot – 60 p-values.
NISS
17
Cereal determines human gender
Really?????
17NISS
NISS
19
2 Cancer types, 48 pesticies, 96 questions
Three claims made, only one appears valid.
NISS
Multiple Modeling/Bias
Take a simple difference
Paper, data, claim
American Cancer Society Cancer Prevention Study II
No association with CV deaths, corrected for PM2.5.
...
Jerrett et al. Large Search Space
22
Covariate Adjustment
27
= 128
23
Large search space
32 x 128 = 4,096
The data used in this paper is not available.
We are asked to trust that analysis deci...
2525
Crisis in science? 2011, 2012
Nature, 2012
Significance, 2011
NISS
26
Claims from observational studies tested in RCTs
27
What can funding agencies do?
Fund data generation and analysis
separately.
Fund replication studies.
Require data used...
28
What can journal editors do?
Quality by inspection, p-value < 0.05, is not working.
(Many workers are gaming the system...
29
What can you, the consumer, do?
(not much)
1. Be skeptical of observational study claims.
2. Read the actual paper.
3. ...
30
“New” p-value plot, - log10
(p-value), Dmitri Zaykin
31
Conclusions
Most science claims do not replicate.
Deming: Don't blame the worker
(or expect them to adopt different met...
32
One irate study evaluator, 2012
Mens Sana Monograph, 2012
3333
Contact
Information
Stan Young
National Institute of Statistical Sciences
www.niss.org
young@niss.org
919 685 9328
NI...
Ozone/PM2.5 Acute Deaths LA
34
3535
Suggestions for effective management
of observational studies
No funding / publication without:
1. Public posting pro...
3636
Congressional Management:
True Science Transparency Act
Any federal agency proposing rule-making or legislation
shall...
Upcoming SlideShare
Loading in …5
×

02 young vpi lecture 2014

Stan Young, PhD,. slides from Phil 6334 guest presentation.

  • Be the first to comment

  • Be the first to like this

02 young vpi lecture 2014

  1. 1. 1 Abstract Claims coming from human medical observational studies, when tested rigorously, most often fail to replicate. Whereas randomized clinical trials replicate over 80% of the time, medical observational studies replicate only 10 to 20% of the time. Multiple re-test studies reported JAMA failed to replicate. For example in the early 1990s, Vitamin E was reported to protect against heart attacks. Large, well-conducted randomized clinical trials did not replicate this claim. The claim that Type A Personality leads to heart attacks failed to replicate in two separate studies, yet the myth still lives. Clearly, there are systematic problems with how observational studies are conducted and analyzed that need to be identified and fixed. Edwards Deming, the most famous quality expert ever, says that any problem with a failed process is not the fault of the workers, scientists conducting observational studies, but of management. Funding agencies and journal editors need to fix a clearly broken process. Technical problems are identified. Tough management solution are proposed. A simple statistical analysis strategy is presented. Many human health problems can only be examined using observational data. Our proposals, technical and managerial, should lead to more reliable claims along with fair ways to judge their reliability. NISS
  2. 2. 22 Contact Information Stan Young National Institute of Statistical Sciences www.niss.org young@niss.org 919 685 9328 NISS
  3. 3. NISS 3 Hayek, 1974, Nobel Lecture It is often difficult enough for the expert, and certainly in many instances impossible for the layman, to distinguish between legitimate and illegitimate claims advanced in the name of science. It is often difficult enough for the expert, and certainly in many instances impossible for the layman, to distinguish between legitimate and illegitimate claims advanced in the name of science.
  4. 4. NISS 4 Hayek (2) ...much effort will have to be directed toward debunking such arrogations*, some of which have by now become the vested interests of established university departments. *Claims without proper foundation
  5. 5. 5 Reliability of Literature Claims S. Stanley Young National Institute of Statistical Sciences Young@niss.org, 919 685 9328 VIP Lecture NISS
  6. 6. 66 Science point of view What is the meaning of life? What is real? What is reproducible? Fooled (fooling) by randomness? NISS
  7. 7. 77 The Players 1. The workers – scientists 2. The communicators – a. PR people b. Bloggers c. Reporters d. Science writers 3. The consumers – public, regulatory agencies, trial lawyers 4. The management – funding agencies, journal editors NISS
  8. 8. 88 The Worker is not the Problem. W. Edwards Deming, the most visionary innovator ever on quality control, said The worker is not the problem. The problem is at the top! Management! To Deming, blaming the workers—individual researchers— is as incorrect as it is useless. Bringing the system under control is the responsibility of those managing it. NISS
  9. 9. 9 Problems with observational studies “Everything is dangerous” 1. Data staging 2. No written analysis protocol 3. Multiple testing 4. Multiple modeling 5. Uncorrected bias 6. Self-serving paper writing 7. Self-serving press release 8. Actually believe the claims 9NISS
  10. 10. Assertion : Every study is positive Data Staging  Bias Multiple testing Multiple model searching Any or all will lead to essentially all observational studies being positive! 10NISS
  11. 11. 11 First, data staging Stan: Why do you think data staging is a big issue? Because it can be done in myriad ways, is rarely documented, and is usually not reproducible? David Madigan 11NISS
  12. 12. 12 Multiple Testing: P-value, t-test Population, real or theoretical Two samples, random NISS
  13. 13. 10-sided dice experiment 12/25/12 NISS
  14. 14. 14 How do you get a “p < 0.05”? Answer: Ask lots of questions. 61 questions 95% chance of a positive study! NISS
  15. 15. 1515 Let’s run an epidemiology study! 10-sided dice simulation: Coffee causes X. NISS
  16. 16. 1616 P-value plot – 60 p-values. NISS
  17. 17. 17 Cereal determines human gender Really????? 17NISS
  18. 18. NISS
  19. 19. 19 2 Cancer types, 48 pesticies, 96 questions Three claims made, only one appears valid.
  20. 20. NISS Multiple Modeling/Bias Take a simple difference
  21. 21. Paper, data, claim American Cancer Society Cancer Prevention Study II No association with CV deaths, corrected for PM2.5. Ozone associated with respiratory deaths. 21
  22. 22. Jerrett et al. Large Search Space 22
  23. 23. Covariate Adjustment 27 = 128 23
  24. 24. Large search space 32 x 128 = 4,096 The data used in this paper is not available. We are asked to trust that analysis decisions were good and claims are robust. Any adjustment for multiple testing and/or multiple modeling renders p-values NS. 24
  25. 25. 2525 Crisis in science? 2011, 2012 Nature, 2012 Significance, 2011 NISS
  26. 26. 26 Claims from observational studies tested in RCTs
  27. 27. 27 What can funding agencies do? Fund data generation and analysis separately. Fund replication studies. Require data used in publication be posted on publication.
  28. 28. 28 What can journal editors do? Quality by inspection, p-value < 0.05, is not working. (Many workers are gaming the system.) Management needs to re-design the system to build quality into the product. Papers following good manufacturing procedures and addressing important questions, should be accepted without regard to statistical significance. Require data used in publication be posted on publication.
  29. 29. 29 What can you, the consumer, do? (not much) 1. Be skeptical of observational study claims. 2. Read the actual paper. 3. Count the claims under consideration. 4. Ask for the data set. 5. Letter to editor : voodoo stats and trust me science. (Educate editors.) 6. Write to funding agency. 7. Write to congressman. 29NISS
  30. 30. 30 “New” p-value plot, - log10 (p-value), Dmitri Zaykin
  31. 31. 31 Conclusions Most science claims do not replicate. Deming: Don't blame the worker (or expect them to adopt different methods). Funding agencies and journal editors have been AWOL. Require data to be placed in depository on publication.
  32. 32. 32 One irate study evaluator, 2012 Mens Sana Monograph, 2012
  33. 33. 3333 Contact Information Stan Young National Institute of Statistical Sciences www.niss.org young@niss.org 919 685 9328 NISS
  34. 34. Ozone/PM2.5 Acute Deaths LA 34
  35. 35. 3535 Suggestions for effective management of observational studies No funding / publication without: 1. Public posting protocol before study initiation. 2. Public posting of data set on publication. 3. Clear statement of questions under consideration. 4. Conform to “Reproducible Research” guidelines. 5. Any claims must be independently replicated. NISS
  36. 36. 3636 Congressional Management: True Science Transparency Act Any federal agency proposing rule-making or legislation shall specifically name each document used to support the proposed rule-making or legislation and provide all data used in said document for viewing by the public. See also OSTP memorandum, 22Feb2013. NISS

    Be the first to comment

    Login to see the comments

Stan Young, PhD,. slides from Phil 6334 guest presentation.

Views

Total views

9,934

On Slideshare

0

From embeds

0

Number of embeds

8,984

Actions

Downloads

21

Shares

0

Comments

0

Likes

0

×