040910 EBM Course, Oxford6“As it is no easy matter to root outprejudices, …. it became requisite toexhibit a full and impartial view of whathad hitherto been published on thescurvy, and that in a chronologicalorder, by which the sources of thesemistakes may be detected. Indeed,before the subject could be set in aclear and proper light, it was necessaryto remove a great deal of rubbish.”James Lind, 1753
040910 EBM Course, Oxford7Joseph Goldberger, 1907• identified 44 studies and provided comprehensivereferences in a bibliography• used a newly developed serum agglutination test toseparate reliable studies from those he consideredunreliable• tabulated the raw data from 26 selected studies• calculated the mean rate of bacteriuria from thepooled data.Winkelstein, 1998
040910 EBM Course, Oxford8More recently,physicists and social scientistsled the way
040910 EBM Course, Oxford10Light RJ, ed. (1983).Evaluation Studies Review Annual.Beverly Hills, CA: SAGE Publications.Light RJ, Pillemer DB (1984).Summing up: the science of reviewingresearch.Cambridge, Mass.: Harvard UniversityPress.
040910 EBM Course, Oxford11Mulrow C (1987).The medical review article: stateof the science. Ann Int Med106:485-488."Current medical reviews do notroutinely use scientific methodsto identify, assess, andsynthesize information."
040910 EBM Course, Oxford12The evolution ofstatistical methods forreducing the play of chancein research synthesis -meta-analysis
040910 EBM Course, Oxford15Ronald Fisher (1932)“... although few or [no statisticaltests] can be claimed individually assignificant, yet the aggregate givesan impression that the probabilitiesare lower than would have beenobtained by chance.”
040910 EBM Course, Oxford16In 1976, Gene Glass, anAmerican social scientist coinedthe term ‘meta-analysis’ torefer to:“the statistical analysis of alarge collection of analysisresults from individual studiesfor purposes of integrating thefindings”
040910 EBM Course, Oxford17Rosenthal and Rubin (1978); Smith and Glass (1977);Glass and Smith (1979); Hunter et al. (1979).Rosenthal R (1984). Meta-Analytic Procedures forSocial Research. Newbury Park, CA: SagePublications.Hedges L, Olkin I (1985). Statistical Procedures forMeta-Analysis. Orlando: Academic Press.Yusuf S, Peto R, Lewis J, Collins R, Sleight P (1985).Beta blockade during and after myocardial infarction:An overview of the randomised trials. Prog CardiovascDis XXVII (5), 336-371.
040910 EBM Course, Oxford18By the end of the 20thcentury, research synthesiswas beginning to be takenmore seriously
040910 EBM Course, Oxford19“There are research syntheses in suchdiverse topics as advertising,agriculture, archaeology, astronomy,biology, chemistry, criminology,ecology, education, entomology, law,manufacturing, parapsychology,psychology, public policy, zoology andeven of eyewitness accounts of theIndian rope trick.”Mark Petticrew, BMJ 2001.
040910 EBM Course, Oxford20As science is cumulative,scientists should cumulatescientifically!
040910 EBM Course, Oxford21Luc de Clapiers Vauvenarques, 1715-47 (Réflexions et Maximes)."Il est plus aisé de dire des chosesnouvelles que de concilier celles quiont étés dites“.‘It is easier to say something new than toreconcile things that have already been said.’or‘Synthesis of ideas is harder than merelyproposing a new idea.’
040910 EBM Course, Oxford22Lord Rayleigh, 1884“If, as is sometimes supposed, scienceconsisted in nothing but the laboriousaccumulation of facts, it would soon come toa standstill, crushed, as it were, under itsown weight…The work which deserves, but Iam afraid does not always receive, the mostcredit is that in which discovery andexplanation go hand in hand, in which notonly are new facts presented, but theirrelation to old ones is pointed out.”
040910 EBM Course, Oxford24Hill AB. Reasons for writing.BMJ 1965;4:870.People reading research reports wantinvestigators to answer four questions:Why did you start?What did you do?What answer did you get?What does it mean anyway?”
040910 EBM Course, Oxford25PremiseThe results of a particularresearch study cannot beinterpreted with any confidenceunless they have beenconsidered, systematically,together with the results ofother studies addressing thesame or similar questions.
040910 EBM Course, Oxford26How well is this premise reflectedin papers published in majorgeneral medical journals?
040910 EBM Course, Oxford27Classification of Discussion sections inRCT reports published in Ann Int Med,BMJ, JAMA, Lancet, and N Eng J Med19No apparent systematic attempt to setnew results in context of other trials4Discussed a previous review but did notattempt to integrate new results2Contained an updated systematicreview integrating the new results1First trial addressing the questionMay ’01n=33May ’97n=26
040910 EBM Course, Oxford28Classification of Discussion sections inRCT reports published in Ann Int Med,BMJ, JAMA, Lancet, and N Eng J Med2719No apparent systematic attempt to setnew results in context of other trials34Discussed a previous review but did notattempt to integrate new results02Contained an updated systematicreview integrating the new results31First trial addressing the questionMay ’01n=33May ’97n=26
040910 EBM Course, Oxford29A contemporary example of theconsequences of not takingresearch synthesis seriouslyDrugs to prevent heart rhythmabnormalities (arrhythmias)after heart attacks
040910 EBM Course, Oxford30The theory:Patients with arrhythmias are atincreased risk of early death afterheart attackAnti-arrhythmic drugs reducearrhythmias after heart attackThese drugs should reduce earlydeath after heart attack
040910 EBM Course, Oxford31The evidence:A 1983 systematic review of 14randomized controlled trials of anti-arrhythmic drugs in heart attack“The theoretical potential for a preventiveor prophylactic effect of antiarrhythmicdrugs …..in the treatment of coronarypatients with ventricular arrhythmias hasnot been realized.”Furberg, 1983
040910 EBM Course, Oxford32A 1993 systematic review of 51randomized trials of anti-arrhythmicdrugs in heart attack660 deaths among 11,712 patientsallocated drugs571 deaths among 11,517 patientsallocated to controlTeo et al. JAMA 1993.
040910 EBM Course, Oxford33The consequence:At the peak of their use in the late1980s, it has been estimated thatanti-arrhythmic drugs were causingevery year comparable numbers ofdeaths to the total number ofAmericans who died in the Vietnamwar (Moore 1995).
040910 EBM Course, Oxford34The discovery that these drugsare lethal could have been madea decade earlier if the Discussionsections in each report of a new trialhad set the new results in the contextof a systematic review of the resultsof all previous trials – in other words,if scientists had cumulatedevidence scientifically.
040910 EBM Course, Oxford35The human costs of failing torecognise the importance ofcumulating evidence scientifically“Advice on some life-saving therapieshas been delayed for more than adecade, while other treatments havebeen recommended long aftercontrolled research has shown themto be harmful.”Antman et al. 1992Antman et al. JAMA 1992;268:240-8.
040910 EBM Course, Oxford36Setting the results of new resultsin the context of a systematicreview of the results of all otherrelevant studies would becomestraightforward if systematicreviews were always done beforeembarking on new research.
040910 EBM Course, Oxford37Would any of you have agreedto participate in a placebocontrolled trial of prophylacticantibiotics for colorectalsurgery after 1975?
040910 EBM Course, Oxford38Reduction of perioperative deaths byantibiotic prophylaxis for colorectal surgery
040910 EBM Course, Oxford39Would you ever have putbabies to sleep on theirtummies?
040910 EBM Course, Oxford42Would a healthy young womanvolunteer at Johns Hopkins havedied after inhaling hexamethoniumif the investigators had done asystematic review of relevantevidence?Clark O, Clark L, Djulbegovic B. Is clinicalresearch still too haphazard?Lancet 2001;358:1648.
040910 EBM Course, Oxford44Would 7000 stroke patients have beenpersuaded to participate in controlledtrials of nimodipine if researchers haddone systematic reviews of the relevantanimal experiments?Johanna Horn et al (2001).Nimodipine in animal model experiments offocal cerebral ischaemia.Stroke 32:2433-2438.
040910 EBM Course, Oxford45Pound P, Ebrahim S, Sandercock P,Bracken MB, Roberts I. The ReviewingAnimal Trials Systematically (RATS)Group.Where is the evidence that animalresearch benefits humans?BMJ 2004;328:514-17 (28 February)
040910 EBM Course, Oxford46Principles of good practiceAll new research studies should be:• designed in the light ofscientifically-defensible syntheses ofrelevant existing research evidence• reported using the new evidence toupdate these research syntheses(thus making clear what contributionthe new study has made to the totalevidence)
040910 EBM Course, Oxford47An important innovation by the BMJ
040910 EBM Course, Oxford48The main obstacles to applyingthese principles in practice arepsychological and social, notpractical.The practical challenge is beingmet by very rapid evolution andapplication of the science ofresearch synthesis andelectronic publishing.
040910 EBM Course, Oxford49Can the public rely on scientiststo use research synthesisto ensure that limited resourcesfor research are used moreefficiently and ethically?
040910 EBM Course, Oxford51“….. The work which deserves, but Iam afraid does not always receive,the most credit is that in whichdiscovery and explanation go handin hand, in which not only are newfacts presented, but theirrelation to old ones is pointedout.”Lord Rayleigh 1884
040910 EBM Course, Oxford52If academia cannot be reliedon to ensure that newresearch projects begin andend with syntheses of theresults of other relevantstudies, who will protect thepublic from the adverseconsequences of currentscientific indiscipline?
040910 EBM Course, Oxford53Danish Research Ethics CommitteeSystem, 1997“For a research ethical evaluation ofscientific experiments involving man it iscrucial that all relevant literature has beenreviewed by the research group beforesubmission……....This will be a precondition when theevaluating committee is judging theoriginality of the project and, for example,the permissibility of using placebo and notan already known treatment in a controlgroup.”
040910 EBM Course, Oxford54UK Medical Research Councilrequirements of people applying forsupport for new controlled trials, 1999.“Give references to any relevant systematicreview(s)* and discuss the need for your trialin the light of the(se) review(s).”“If you believe that no relevant previous trialshave been done, give details of your searchstrategy for existing trials.”“* For definition of a systematic review, seeOxman, AD. Checklists for review articles, BMJ, 1994;309;648-51.”
040910 EBM Course, Oxford55“The European Science Foundation (2001)endorses the view that embarking on new researchwithout first preparing systematic reviews of relevantexisting evidence is indefensible on scientific andethical grounds, and recommends that its MemberOrganisations should:• Require applicants for support of new trials to referto scientifically defensible reviews of relevant existingtrials, or demonstrate that no other relevant trialsexist.• Require data monitoring committees to take accountof new evidence accumulating in updated systematicreviews.”
040910 EBM Course, Oxford56In conclusionScience is cumulative, andscientists must cumulatescientifically!As the main funder of science, the public has aright to expect that this will be reflected moreeffectively in the way that science is conductedand reported.It remains unclear how effectively academiawill respond to this fundamental challenge toits traditional ways of working.
040910 EBM Course, Oxford57This talk has been based in part onmaterial published in:Chalmers I, Hedges L, Cooper H.A brief history of research synthesisEvaluation and the Health Professions2002;25:12-37.
040910 EBM Course, Oxford58Francis Galton, 1901"I have begun to think that no oneought to publish biometric results,without lodging a well-arranged andwell-bound copy of his data in someplace where it should be accessible,under reasonable restrictions, tothose who desire to verify his work.”