“Why most published
research is wrong.”
Louise Cullen
(Clinician researcher)
Disclosure Information
“It is everyone’s responsibility to
find out how to
ask questions systematically,
find answers from searching the
literatu...
“It is everyone’s responsibility to
find out how to
ask questions systematically,
find answers from searching the
literatu...
40 ingredients
associate with
cancer
Most single studies showed implausibly
large effects.
The p value
The p value
Observed size of Effect
p=0.01
p=0.01
There is a 1% chance of results as extreme as these would occur
when there is really no difference occurring in the...
1000 hypotheses
Replication of studies
Replication of studies
Problems with the study itself.
Wrong question
Wrong Theory
Wrong population studied
2
ACS: Trial and community populations
Circulation. 115(19):2549-69, 2007 May 15.
n=2
Wrong design
• Greater the flexibility in
– designs
– definitions
– outcomes
– analytical modes
• Greater the flexibility in
– designs
– definitions
– outcomes
– analytical modes
• Hotter a scientific field with more t...
Wrong Endpoints
Ad and high dose Ad
Ca++ in cardiac arrest
COX-2 inhibitors
Milrinone
Methodology
Statistical hypothesis inference testing
Problems with reporting
Interpretation
• “a little significance”
• “a definite trend is
evident”
• “a clear tendency”
• “almost achieved
significance”
• “a little significance”
• “a definite trend is
evident”
• “a clear tendency”
• “almost achieved
significance”
The data i...
• “In my experience”
• “In case after case”
• “In a series of cases”
• “It is generally believed
that..”
• “A highly signi...
Omitting facts deliberately
….!
Why?
Incentives
Pharma
Pharma
Why?
Ethical practice of researchers
Problems with publishing
Don’t believe in the review process
Journal publishing practices
• 2004 “original articles” in NEJM
– 363 tested an established therapy
– 146 (40%) reversed that practice
– 138 (38%) reaf...
What can you do about it?
Read more than the title!
Reporting Framework
CONSORT (http://bit.ly/14qUNEF)
– Standards for reporting of trials
STARD
– Standards for the Reportin...
Biases
Be sceptical!
Thank you
@louiseacullen
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Cullen: Why most published research is wrong
Upcoming SlideShare
Loading in...5
×

Cullen: Why most published research is wrong

709

Published on

Question everything! Louise cullen examines the minefield of published research and importance of reading around topics, not articles.

Published in: Health & Medicine, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
709
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
7
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • When Chris asked me to do this I thought – are you for real Chris? Why all published research is wrong….Do you really think I can cover all of this in 20mins?
  • Sounds easy? Right? Great – but how…
  • Sounds easy? Right? Great – but how…
  • Where this all come from.
    Open access article by John in 2005.
    In here John – began the discussion about issues relating to problems with studies, analyses, their publication and reporting
  • Ioannidis randomly selected 50 ingredients
    PubMed queries for recent studies that evaluated relationship with cancer.
  • Effect size shrunk with meta-analyses.
  • Ronald Fisher (a UK statistician) introduced the P value in the 1920s.
    He did not mean it to be a definitive test
    He intended it simply as an informal way to judge whether evidence was significant in the old-fashioned sense: i.e. that it was worth a second look.
    The idea was to run an experiment, then see if the results were consistent with what random chance may produce.
    Researchers would first set up a ‘null hypothesis” that they wanted to disprove – such as there being no correlation or no difference between 2 groups.
    Next hey would play devils advocate and, assuming that this null hypothesis was in fact true, calculate the chances of getting results at least as extreme as what was actually observed.
    This probability s the P value
  • Most look at this and say that there is a 1% chance of the findings/results being wrong.
    But this is wrong.
    The P value cannot say this – you also need to know the odds that the real effect was there in the first place.
    The P value actually means that there is a 1% chance of results as extreme as these would occur when there is really no difference occurring in the experiment – e.g. that a drug has no effect.
  • Most look at this and say that there is a 1% chance of the findings/results being wrong.
    But this is wrong.
    The P value cannot say this – you also need to know the odds that the real effect was there in the first place.
    The P value actually means that there is a 1% chance of results as extreme as these would occur when there is really no difference occurring in the experiment – e.g. that a drug has no effect.
  • Consider 1000 hypotheses, of which only 10% are true.
  • Random error makes a hypothesis that is really false – look true.
    These = FP.
    Medicine accepts that this happens in the order of 1 in 20 times, so in 1000 hypotheses (where 100 are TP) this means there are 45 FPs
  • If there are 100 TP and 45 FP then almost a third of the results that look positive would be wrong.
  • But it is worse than that.
    There is another type of error – and that is False negatives. Where there is a true effect, but it is misinterpreted as a false one.
    Say 20% of the true finding fail to be detected (and this figure is difficult to quantity). That is in this case - 20 cases.
    Now researches see 125 hypotheses as true (where 45 are not true)
  • A growing number cannot be replicated, because many studies may have not found a real result in the first place.
    Perhaps we should only be looking at P values <0.005
  • A growing number cannot be replicated, because many studies may have not found a real result in the first place.
    Perhaps we should only be looking at P values <0.005
  • Start at the beginning:
    Studies themselves
  • Interesting but not really relevant?
  • Steroids for traumatic brain injury
    Others Medical example – Investigations into H2 blockers in PUD
    Before realised that PUD related to H bactor pylori
  • Selective
    ACS – pts over 75 excluded from most studies
    Underpowered
  • The National Registry of Myocardial Infarction (NRMI)
    Large, US observational registry
    Collects baseline data, procedural, therapeutic and outcome data on discharge
    >1million NSTEACs
  • More lilkey that the findigns are false with Small size
  • More lilkey that the findigns are false
  • More lilkey that the findigns are false
  • Not clinically relevant endpoints
    (Ad/Vasoproessin/ high dose Ad)
  • Others – Milronone – Congestive heart failure – inc cardiac contractility – inc mortaility.
  • Since publication in 1990, results from the National Acute Spinal Cord Injury Study II (NASCIS II) trial have changed the way patients suffering an acute spinal cord injury (SCI) are treated.
    though well-designed and well-executed, both NASCIS II and III failed to demonstrate improvement in primary outcome measures as a result of the administration of methylprednisolone. Post-hoc comparisons, although interesting, did not provide compelling data to establish a new standard of care in the treatment of patients with acute SCI.
    Evidence of the drug's efficacy and impact is weak and may only represent random events.
  • Renamed
  • In the late 1940s before there was a polio vaccine Health authorities noted that polio cases increased with ice cream and soft drink consumption.
    Eliminating such treats were part of the advice given to combat the spread of the diseases.
    Polio was more common in summer, when people eat more icecream
    Hence a Correlation vs causation
  • Spin
  • Incentives
    Egos, cudos,
    Academics – KPIs, output, promotions, grants
    Tied to remuneration – QH contracts - KPI
  • Pharma – All bad
    75% of all US research funded by Pharma
  • Pharma – All bad
    75% of all US research funded by Pharma
    Surgeon – cannot recommend surgery
    Interventional cardiologist - stenting
  • UQ RORT:
    One of Australia's leading universities is investigating new concerns of possible academic misconduct by two former academics.
    The University of Queensland's Dr Caroline Barwood and Professor Bruce Murdoch published a peer-reviewed paper in the prestigious European Journal of Neurology, heralding a major breakthrough in the treatment of Parkinson's disease.
    theuniversity made the unusual admission that it could find no data or evidence that the research was ever conducted.
    Before the article was retracted, the study's apparent success led to a number of grants.
    Ten months after the allegations of academic misconduct were first raised and one month after the investigation was referred to the CMC, the university accepted part of a $300,000, five-year research fellowship on behalf of Dr Barwood.
  • Incentives
    Egos, cudos,
    Academics – KPIs, output, promotions, grants
    Tied to remuneration – QH contracts - KPI
  • Don’t believe open access – where if I pay the $ to many journal they will simply accept my paper!
  • Neg trials hard to publish
    High impact journals – ‘1st’ of something – often exaggerated.
  • Where we end up at is this>>>
    What are we doing now that is harmful to our patients?
  • Don’t just skim article
    Not just abstract and conclusions
    Learn a little about stats
    Don’t be fooled by high quality journal
    Know/review the literature/topic. Not just article
  • I don’t recommend that you go home tonight and try a “booty call” with your partner
    Perhaps read the article and find out the details.
  • Consistent and transparent
  • Be aware of your own biases – especially confirmation biases.
    Stop searching for information to confirm your own views. Read broadly.
    Don’t believe a single articles findings – look for bodies of work around a topic.
  • Who do you believe and what do you believe?
    Do you believe who speaks loudest?
  • Sit and contemplate your position…………Put the patient (not the patients leg) at the forefront of your focus.
    Be sceptical!
  • Transcript of "Cullen: Why most published research is wrong"

    1. 1. “Why most published research is wrong.” Louise Cullen (Clinician researcher)
    2. 2. Disclosure Information
    3. 3. “It is everyone’s responsibility to find out how to ask questions systematically, find answers from searching the literature, critically appraise the literature and apply the results to practice.” Rinaldo Bellomo
    4. 4. “It is everyone’s responsibility to find out how to ask questions systematically, find answers from searching the literature, critically appraise the literature and apply the results to practice.” Rinaldo Bellomo
    5. 5. 40 ingredients associate with cancer Most single studies showed implausibly large effects.
    6. 6. The p value
    7. 7. The p value Observed size of Effect
    8. 8. p=0.01
    9. 9. p=0.01 There is a 1% chance of results as extreme as these would occur when there is really no difference occurring in the experiment.
    10. 10. 1000 hypotheses
    11. 11. Replication of studies
    12. 12. Replication of studies
    13. 13. Problems with the study itself.
    14. 14. Wrong question
    15. 15. Wrong Theory
    16. 16. Wrong population studied
    17. 17. 2 ACS: Trial and community populations Circulation. 115(19):2549-69, 2007 May 15.
    18. 18. n=2
    19. 19. Wrong design
    20. 20. • Greater the flexibility in – designs – definitions – outcomes – analytical modes
    21. 21. • Greater the flexibility in – designs – definitions – outcomes – analytical modes • Hotter a scientific field with more teams involved.
    22. 22. Wrong Endpoints
    23. 23. Ad and high dose Ad Ca++ in cardiac arrest COX-2 inhibitors Milrinone
    24. 24. Methodology Statistical hypothesis inference testing
    25. 25. Problems with reporting
    26. 26. Interpretation
    27. 27. • “a little significance” • “a definite trend is evident” • “a clear tendency” • “almost achieved significance”
    28. 28. • “a little significance” • “a definite trend is evident” • “a clear tendency” • “almost achieved significance” The data is practically meaningless
    29. 29. • “In my experience” • “In case after case” • “In a series of cases” • “It is generally believed that..” • “A highly significant area for exploratory study” • Once • Twice • Three times • A couple of others think so too • A totally useless topic in my underpowered study…….
    30. 30. Omitting facts deliberately ….!
    31. 31. Why? Incentives
    32. 32. Pharma
    33. 33. Pharma
    34. 34. Why? Ethical practice of researchers
    35. 35. Problems with publishing
    36. 36. Don’t believe in the review process
    37. 37. Journal publishing practices
    38. 38. • 2004 “original articles” in NEJM – 363 tested an established therapy – 146 (40%) reversed that practice – 138 (38%) reaffirmed it
    39. 39. What can you do about it?
    40. 40. Read more than the title!
    41. 41. Reporting Framework CONSORT (http://bit.ly/14qUNEF) – Standards for reporting of trials STARD – Standards for the Reporting of diagnostic accuracy studies
    42. 42. Biases
    43. 43. Be sceptical!
    44. 44. Thank you @louiseacullen
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×