Primum non nocere “First do no harm”Hippocrates, Epidemics
Sources of Evidence in Medicine• Traditional Teaching• Textbooks• Basic sciences• Observational studies• Computer simulation• Decision Analysis• Case-Control Studies• Randomised Controlled Trials (RCT)• Meta-analyses
RCOG Classification of Evidence Levels1++ High quality meta-analyses of RCTs1+ Meta-a. Or RCTs at low risk of bias1- Meta-a. Or RCTs at high risk of bias2++ High quality meta-analyses of CCs2+ Well-conducted cc or cohort studies2- Case-control or cohort studies with ? bias3 Case reports4 Expert opinion
Archie Cochrane (1909-88) “Effectiveness and Efficiency: Random Reflections of Health Services “, 1971
Randomized Controlled Trials “Gold standard” in evaluating new therapies or surgical techniques May also be applied to new diagnostic tests
Randomized Controlled TrialsObjectives of RCT: Minimize bias by randomisation Achieve statistical power through adequate sample size “Blinding “ - single or double Analysis by intention to treat
Randomized Controlled TrialsRandomisation Several techniques available Computer software linked to central monitoring station “Block “ randomisation Sealed envelope method
RCT’s and Observational Studies• Two studies published in the NEJM in 2000 suggested thatRCTs and observational studies overall produced similarresults• JAMA 2001: “discrepancies beyond chance do occur anddifferences in estimated magnitude of treatment effect arevery common”• RCTs may be unnecessary for treatments that havedramatic and rapid effects relative to the expected
RCT’s and Industry Funding•RCT’s funded by industry are significantly more likely toreport positive results•Possibly due to publication bias•RCTs may be unnecessary for treatments that havedramatic and rapid effects relative to the expected
RCT’s and Statistical Error• Type I error – “false positive”• Type II error – “false negative”• Sample size calculations often inaccurate
"It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials." Archie Cochrane, 1972
Role of systematic reviewsBefore commencing a new project: to determinewhether further studies are really indicated: ‘state-of-the-art’ literature review.Gain in statistical power for average estimates.Cumulative meta-analysis can determine whenfurther studies are no longer indicated.Design of subsequent studies.Setting policy for treatment and health care –making the best use of the data available.
Can Studies be Combined? Identification of optimal inclusion criteria can bedifficult. The most critical step is choosing the appropriateresearch question. A fairly general question is more preferable to a veryspecific one. Tukey : "...far better an approximate answer to theright question, than an exact answer to the wrongquestion.."
Publication BiasEntire research studies may fail to reach publicationbecause of the nature of the results.Identification of unpublished trials can be verydifficult - in one study it accounted for 22% of thepapers included in the meta-analysis. Failure to publish rests with the investigators ratherthan editors.
PREDICTIVE ABILITY OF META-ANALYSES Villar et al, Lancet 1995Comparison of the meta-analyses of smaller studieswith the corresponding result of the largest study. 30 meta-analyses including a total of 185 randomisedcontrolled studies (RCT) obtained from the Cochranepregnancy and childbirth database. The meta-analyses were only included if they had at least onetrial with a total sample size of over 1000. Calculations differ from the Cochrane database inthat the largest trial was excluded, this being used asthe gold standard for outcome
The Cochrane Collaboration • Established in 1993 by Sir Iain Chalmers • International: 100 countries • Independent • Not-for-profit • Over 27000 contributors
RANZCOG and EBM“RANZCOG endorses the principles of Evidence-based Medicine and recognizes the NHMRC levels ofevidence and grades of recommendations”College Statement C-Gen 15, Nov. 2009
…a scientific idea can never be proven true,because no matter how many observations seem toagree with it, it may still be wrong. On the otherhand, a single contrary experiment can prove atheory forever false Sir Karl Popper