• Save
Critical appraisal of meta-analysis
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Critical appraisal of meta-analysis

on

  • 2,742 views

 

Statistics

Views

Total Views
2,742
Views on SlideShare
2,742
Embed Views
0

Actions

Likes
5
Downloads
93
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • First published in 1997Second impression 1997Third impression 1998Fourth impression 1998Fifth impression 1999Sixth impression 2000Seventh impression 2000Second Edition 2001
  • The initial report of Ranson’s criteria was based on 100 patients (21 of whom underwent early surgery as part of a randomized trial or for uncertainty of diagnosis). The study identified 11 factors that predicted severe diseases (defined as death or an ICU stay beyond 7 days). The 11-point scoring system is measured in 2 stages: 5 initial data points on admission and a further 6 data points within the subsequent 48 hours. The initial report demonstrated a linear relationship between the number of criteria and the likelihood of mortality. Subsequently, modifications were made on the 11-point system for those with gallstone pancreatitis (the original studies were a mixture of alcoholic and biliary pancreatitis).These modifications reduced the number of criteria to 10 for those with gallstone pancreatitis.References:Ranson JHC, Rifkind KM, Roses DF, et al: Prognostic signs and the role of operative management in acute pancreatitis.  SurgGynecolObstet  1974; 139:69. Ranson JHC: Etiological and prognostic factors in human acute pancreatitis: A review.  Am J Gastroenterol  1982; 77:633.
  • Concealment: إخفاء - كتمان
  • However, after randomisation, it is almost inevitable that some participants would not complete the study for whateverreason. Participants may deviate from the intended protocol because of misdiagnosis, non-compliance, or withdrawal.When such patients are excluded from the analysis, we can no longer be sure that important baseline prognostic factorsin the two groups are similar. Thus the main rationale for random allocation is defeated, leading to potential bias.To reduce this bias, results should be analyzed on an ‘intention to treat’ basis.
  • Patients: Patients who are aware that they are receiving what they believe to be an expensive new treatment may report being better than they really are. Doctors: The judgment of a doctor who expects a particular treatment to be more effective than another may be clouded in favor of what he perceives to be the more effective treatment. Analysts: When people analyzing data know which treatment group was which, there can be the tendency to ‘‘overanalyze’’ the data for any minor differences that would support one treatment. It is important for authors of papers describing RCTs to state clearly whether participants, researchers, or data evaluators were or were not aware of assigned treatment.
  • Egger et al. (2003) have pointed out that the completeness of the literature search is an important feature of the meta-analysis to avoid publication bias or selection bias.
  • Publication & reporting biases:Positive results bias Grey Literature bias Time-lag bias Language & country bias Multiple publication bias Selective citation bias Database indexing bias Selective outcome reporting biasHealth Technology Assessment, 2000; 4(10):1-115
  • Near-patient testing Near-patient testing is an area of emerging technology, and a larger proportion than usual of papers were possibly unpublished, published in less common sources or presented at conferences.
  • CCTR (Cochrane Controlled Trials Register):Largest electronic compilation of CT in existence 527 885 citations as of the 2008, Issue 1
  • Scales: مقياس
  • CI is important because it gives an idea about how precise an estimate is. The width of the interval indicates the precision of the estimate. The wider the interval, the less the precision.A very wide interval may indicate that more data should be collected before anything definite can be said about the estimate.
  • NNT:Number of people who need to receive a treatment in order to achieve the required outcome in one of them.
  • We considered that it would be more appropriate to calculate the RR instead of the OR (odds ratio) with RCTs, which would be the more conservative approach. We also generally use only the random effects models. This would be the more conservative approach, and takes into account the natural heterogeneity, including the patient population, the critical care experience of the physicians who took care of these patients, the type and duration of antibiotics, and the standard of care across the medical centers, among these types of studies independent of statistical heterogeneity testing. Actually, even when the statistical method was switched to fixed effects models, the main conclusion of this meta-analysis did not change at all.
  • X-squared:X-squared: Has, on average, a value equal to its degrees of freedomDegree of freedom (df): Number of trials in the MA minus 1 (in this case: 7 – 1 = 6) Interpretation of x2: x-squared = number of trial in MA: no evidence of statistical heterogeneityx-squared much greater than number of trials in MA: serious heterogeneityReference of x-squared:Thompson SG. Why sources of heterogeneity in meta-analysis should be investigated. In: Chalmers I, Altman DG, eds. Systematic reviews. London: BMJ Publications, 1995: 48–63.I-squared:< 25% No or little heterogeneity50 – 75 % Serious heterogeneity
  • X-squared:X-squared: Has, on average, a value equal to its degrees of freedomDegree of freedom (df): Number of trials in the MA minus 1 (in this case: 7 – 1 = 6) Interpretation of x2: x-squared = number of trial in MA: no evidence of statistical heterogeneityx-squared much greater than number of trials in MA: serious heterogeneityReference of x-squared:Thompson SG. Why sources of heterogeneity in meta-analysis should be investigated. In: Chalmers I, Altman DG, eds. Systematic reviews. London: BMJ Publications, 1995: 48–63.I-squared:< 25% No or little heterogeneity50 – 75 % Serious heterogeneity
  • We considered that it would be more appropriate to calculate the RR instead of the OR (odds ratio) with RCTs, which would be the more conservative approach. We also generally use only the random effects models. This would be the more conservative approach, and takes into account the natural heterogeneity, including the patient population, the critical care experience of the physicians who took care of these patients, the type and duration of antibiotics, and the standard of care across the medical centers, among these types of studies independent of statistical heterogeneity testing. Actually, even when the statistical method was switched to fixed effects models, the main conclusion of this meta-analysis did not change at all.
  • Focus should be placed on patients with a high risk for infected pancreatic necrosis, such as those with the presence of organ failure.
  • Statement: كشف - بيان - إفادةDiagram: مخططChecklist:
  • Meta-analysis: the core of the truth?Kristiansen IS & Mooneyّ G. Evidence-Based Medicine In its place.Taylor & Francis e-Library, 1sted, 2004.

Critical appraisal of meta-analysis Presentation Transcript

  • 1. Critical appraisal of a meta-analysis studySamir Haffar M.D.Associate Professor of GastroenterologyAl-Mouassat University Hospital – Damascus – Syria
  • 2. Hierarchy of evidence in quantitative studiesMcGovern D, Summerskill W, Valori R, Levi M. Key topics in EBM.BIOS Scientific Publishers, 1st Edition, Oxford, 2001.
  • 3. Gene GlassAmerican statistician – University of ColoradoInvolved in social science researchHe coined the term meta-analysis in 1976
  • 4. Logo of Cochrane collaborationhttp://www.cochrane.orgDatabase available free online in many countries
  • 5. Number of publications about MA (1986 - 1999)Results from Medline search using MeSH“meta-analysis” & text word “systematic review”Egger M et all. Systematic reviews in health care: Meta-analysis in context.BMJ Publishing Group, London, 2nd edition, 2001.
  • 6. Greenhalgh T. How to read a paper - The basics of evidence based medicine.BMJ Publishing Group -2nd Edition - London - 2001.Trisha Greenhalgh“ If I had to pick one word which exemplifiesthe fear felt by so many students, clinicians, &consumers towards evidence-based medicine,that word would be meta-analysis”
  • 7. 1- Ask2- Acquire4- Apply5- AssessPatientdilemmaPrinciplesof EBPEvidence alone does notdecide – combine with otherknowledge & values3- AppraiseHierarchyof evidence5 A
  • 8. Steps of EBM Ask
  • 9. Clinical history• 60-year-old man with acute biliary pancreatitis• Ranson‟s score: 4 – No fever – Normal WBCs• CECT* on day 7: CT grading system of Balthazar 3Necrosis score 2CT severity index 5• You wonder if prophylactic antibiotics prevents infectionof non-infected pancreatic necrosis & decreases mortality*CECT: Contrast-Enhanced Computed Tomography
  • 10. Ranson’s score for gallstone pancreatitisAge > 70 yrBlood glucose >220 mg/dlWBC >18,000/mm3LDH > 400 IU/LASAT > 250 IU/LAt presentation1 point for each positive factorSevere acute pancreatitis: ≥ 3Ranson JHC. Am J Gastroenterol 1982;77:633.During initial 48 hrHt >10% decreaseSerum calcium < 8 mg/dlBase deficit > 5 mEq/LBUN > 2 mg/dl increaseFluid sequestration > 4 L
  • 11. CT grading system of BalthazarGrade Description PointsA Normal pancreas 0Balthazar EJ et al. Radiology 1990 ; 174 : 331 – 6.B Pancreatic enlargement 1C Inflammation of pancreas or peripancreatic fat 2D Single peripancreatic fluid collection 3E ≥ 2 fluid collections or retroperitoneal air 4
  • 12. Necrosis scoreNecrosis PointsNo pancreatic necrosis 0 pointsOne third of pancreas 2 pointsOne half of pancreas 4 points> one half of pancreas 6 points
  • 13. CT severity indexThe index ranges from 0 to 10Severe acute pancreatitis ≥ 3CT grading of Balthazar(0 – 4 points)Necrosis score(0 – 6 points)+Morgan DE. Clin Gastroenterol Hepatol 2008 ; 6 : 1077 – 1085.
  • 14. CT Severity Index (CTSI)Localized fluid collection adjacent to tail: CT grading (3 points)Lack of enhancement of pancreatic tail: Necrosis <30 % (2 points)Absence of retroperitoneal air
  • 15. Key components of your clinical questionPICOP Patient Severe AP with CT-proven necrosisI Intervention Prophylactic antibioticsC Comparaison Placebo or no treatmentO Outcome Infected pancreatic necrosis – MortalityProphylactic antibiotics in pancreatic necrosis
  • 16. Steps of EBMAcquire
  • 17. PubMed translation of query into search termsPICO Element Search terms for PubMedP Acute necrotizing pancreatitis “acute necrotizing pancreatitis” [MeSH]* MeSH: Medical Subject Headings in PubMedI Prophylactic antibiotics “antibiotic prophylaxis” [MeSH term]C PlaceboNo treatment“placebo” [MeSH term]O Infected necrosisMortality“infection” [MeSH term]“necrosis” [MeSH term]“mortality” [MeSH term]Other Meta-analysis SR in PubMed Clinical Queries
  • 18. PubMed Clinical Queries
  • 19. Search onDec 16, 2009
  • 20. Steps of EBM Appraise
  • 21. Systematic review & meta-analysisSystematic reviews(SR)Meta-analyses(MA)MA may, or may not, include a SREgger M et all. Systematic reviews in health care: Meta-analysis in context.BMJ Publishing Group, London, 2nd edition, 2001.
  • 22. Definition of meta-analysis“Statistical analysis that combines or integratesthe results of several independent clinical trialsconsidered by the analyst to be combinable”Proceedings of biopharmaceutical section of American statistical association.1988 ; 2 : 28 – 33.
  • 23. Rationale for a meta-analysisBy combining the samples of individual studies,the overall sample size is increased, therebyimproving the statistical power of the analysis aswell as precision of estimates of treatment effects
  • 24. Steps of meta-analysis Formulation of the problem to be addressed Data collection Data recording Data analysis Reporting the results (Forest plot)Researchers should write in advance a detailed protocol
  • 25.  Formulation of the addressed problemPICOP Patient Severe AP with CT-proven necrosisI Intervention Prophylactic antibioticsC Comparaison Placebo or no treatmentO Outcome Infected pancreatic necrosis -MortalityStudy design: RCTs
  • 26.  Formulation of the addressed problem• Controlled trials• Randomization of patients• Intention to treat principle (ITT)• Preferably blinded• Outcome assessment: p – RR – OR – CIs – NNTGuyatt G, et al. User’s guide to the medical literature.Essentials of evidence based clinical practice. Mc Graw Hill, 2nd ed, 2008.Specify inclusion & exclusion criteria
  • 27. Basic structure of a RCT / Parallel trialPetrie A, Sabin C. Medical statistics at a glance. Blackwell Publishing, 2nd edition, 2005.Most frequently used design
  • 28. Randomization• Simple randomization• Random table• Block randomization• Stratified randomization• Minimization method• Unequal randomization• Allocation concealmentInacceptablePreferred
  • 29. Intention to treat analysisQuality control rather than analytic tool• Strategy in conduct & analysis of RCT ensuring that allpatients allocated to treatment or control groups areanalyzed together as representing that treatment armwhether or not they received the prescribed treatment orcompleted the studyMcGovern D, Summerskill W, Valori R, Levi M. Key topics in EBM.BIOS Scientific Publishers, 1st ed, Oxford, 2001.Randomized participants = Analyzed participants
  • 30. Blinding or Masking• Participants• Investigators who administer interventions• Investigators taking care of the participants• Investigators assessing the outcomes• Data analyst• Investigators who write results of the trialBlinding can be implemented in at least 6 levels in RCTsUsuallythe same
  • 31.  Data collectionFinding all studies (Is there an existing SR?)• Electronic searchInitial search PubMed – Cochrane ReviewOthers databases: EMBASE, CINAHLFurther search References of relevant reviewsFind terms you didn’t use (MeSH*)Search again Snowballing• Supplementary searchHand searchWrite to researchers* MeSH: Medical Subject Headings in MEDLINE
  • 32. Studies included in meta-analysisStudies reviewedGrayliteratureAll studies publishedAll studies conducted
  • 33. Why using multiple sources?Papers identified in a SR of near patient testingUniqueNot uniqueGlasziou P et al. Systematic reviews in health care: a practical guide.Cambridge University Press, 1st edition, 2001.
  • 34.  Data collectionProphylactic antibiotics in pancreatic necrosis• Electronic databases – MEDLINE– EMBASE– CCTR– Cochrane Library– Science Citation Index• Hand search – References from published trials– Major conference abstractsCCTR: Cochrane Controlled Trials RegisterBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.
  • 35.  Data recording• 2 independent observers extract the data• Quality of the studies may be rated with speciallydesigned checklist or scales• Blinding observers to names of authors, institutions,names of journals, funding & acknowledgments
  • 36. Existing tools to assess trial quality• Several components grouped inScales Each item scored numericallyOverall quality score is generatedChecklists Components evaluated separatelyNo numerical scores• Systematic search of literature in 1995 identified25 scales & 9 checklists for assessing trial quality** Moher D et all. Controlled clinical trials 1995 ; 16 : 62 – 73.
  • 37. • Quality assessment performed independently by 2authors using empirical evidence 1-2• Disagreement resolved by discussion between 2 reviewers• Low risk of bias Generation of allocation sequenceAllocation concealmentBlinding• High risk of bias 1 or more component inadequate Data recordingProphylactic antibiotics in pancreatic necrosis1 Schulz KF et al. JAMA 1995 ; 273 : 408 – 12.2 Moher D et al. Lancet 1998 ; 352 : 609 – 13.
  • 38. Bai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Antibiotic prophylactic in pancreatic necrosisFlow diagram
  • 39. Characteristics of RCTs included in MABai Yu & al. Am J Gastroenterol 2008 ; 103 : 104 – 110.467 patients included in 7 trials
  • 40.  Data analysis2 stage statistical process of MAStatistical power of MA is often very high• Treatment effect for each studyp value (p)Relative Risk (RR) or Odds Ratio (OR)Confidence Intervals (CIs)Number Needed to Treat (NNT)• Overall treatment effectCalculated as weighted average of individual statistics
  • 41. • p > 0.05 Statistically insignificant• p < 0.05 Statistically significantProbability value (p value)StatisticallysignificantClinicallysignificantDoesntmean
  • 42. Risk & OddsRiskaa + bRisk =RR: risk patients/risk controlsOddsabOdds =OR: odds patients/odds controls
  • 43. Interpretation of RR & OROR or RR should be accompanied by CIRR or OR > 1Increased likelihood of outcome in treatment groupRR or OR < 1Decreased likelihood of outcome in treatment groupRR or OR = 1No difference of outcome between tt & control group
  • 44. Odds ratio or relative risk?OR will be close to RR if endpoint occurs infrequently (<15%)If outcome is more common, OR will differ increasingly from RREgger M et all. Systematic reviews in health care: Meta-analysis in context.BMJ Publishing Group, London, 2nd edition, 2001.
  • 45. Confidence intervalsValue 95 % CI are commonly used90 or 99% CI are sometimes usedWidth of CI Indicates precision of the estimateWider the interval, less the precisionCI includes 1 No statistically significant differenceCI doesn‟t include 1 Statistically significant difference
  • 46. Statistical significance & CI(a) Statistically significant , low precision(b) Statistically significant, high precision(c) Not statistically significant, low precision(d) Not statistically significant, high precisionGlasziou P et al. Evidence based practice workbook. Blackwell, 2nd edition, 2007.
  • 47. Number Needed to Treat (NNT)• Relative risk (RR)Risk in treatment group / risk in control group• Absolute risk reduction (ARR)Risk in control group – risk in treatment group• NNT (expressed in clinically relevant way)1 /ARR
  • 48. Statistical methods/overall treatment effectLarger trials have more influence than smaller onesFixed effects model 1Random effects model 2Bayesian models3 ControversialFixed & random effects1 Prog Cardiovasc Dis 1985 ; 17 : 335 – 71.2 Stat Med 1992 ; 11 : 141 – 58.3 BMJ 1996 ; 313 : 603 – 7.No singlecorrect method
  • 49.  Data analysisProphylactic antibiotics in pancreatic necrosisBai Yu & al. Am J Gastroenterol 2008 ; 103 : 104 – 110.• Treatment effect for each study• Overall treatment effectRandom effects model onlyInherited heterogeneity between the studiesMore conservative estimate of effect by using wider CIsp value (p)Relative risk (RR)95% confidence intervals (CIs)
  • 50.  Reporting the resultsThe typical graph for displaying resultsof a meta-analysis is called a „„forest plot‟‟
  • 51. Antibiotic prophylaxis & pancreatic necrosisBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Forest plot
  • 52. Bai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Antibiotic prophylaxis & pancreatic necrosisHorizontal lineScale measuring the treatment effect
  • 53. Bai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Antibiotic prophylaxis & pancreatic necrosisVertical line or line of no effectTreatment & control groups have the same effect
  • 54. Antibiotic prophylaxis & pancreatic necrosisBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Point estimate & CIs for each study
  • 55. Point estimate (RR or OR) & CIGallin JI, Ognibene FP. Principles & practice of clinical research.A Press, 2nd ed, 2005.
  • 56. Antibiotic prophylaxis & pancreatic necrosisBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Diamond
  • 57. The diamondPerera R, Heneghan C, Badenoch D. Statistics Toolkit.Blackwell Publishing Ltd, Oxford, 1st edition, 2008.Shows combined point estimate (OR or RR)& CI for the meta-analysis
  • 58. Diamond in meta-analysisDiamond on Left of the line of no effectLess episodes of outcome of interest in treatment groupDiamond on Right of the line of no effectMoRe episodes of outcome in treatment groupDiamond touches the line of no effectNo statistically significant difference between groupsDiamond does not touch the line of no effectDifference between two groups statistically significant
  • 59. Bai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Antibiotic prophylaxis & pancreatic necrosisThe diamondShows the overall result of MA
  • 60. Antibiotic prophylactic effect on mortalityBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.The diamond
  • 61. Interpretation of forest plotNames on left First authors of primary studiesBlack squares RR or OR of individual studiesBlack square size Weight of each trial in MAHorizontal lines 95% confidence intervalsVertical line Line of no effect (OR or RR = 1)Diamond Overall treatment effectDiamond Center Combined treatment effectTips of diamond 95% CI
  • 62. Meta-analytic analyses are prone to bias& need to be interpreted with cautionBias: difference between study results & truth
  • 63. Bias in meta-analysis (1)• Publication bias: studies never publishedStudies with no beneficial effect of treatmentStudies sponsored by pharmaceutical industryStudies from a single centre versus multiple centers• English language bias:Positive findings published in a international journalNegative findings published in a local journal• Database bias:Journals not indexed in major databases
  • 64. Language bias40 pairs of trials published by the same authorControlled trials with statistically significant results washigher among reports published in EnglishEgger M et all. Lancet 1997 ; 350 : 326 – 9.
  • 65. Bias in meta-analysis (2)• Multiple publication biasStudies with significant results lead to multiple publications• Bias in provision of dataAdditional data not reported in print needed for MA• Biased inclusion criteriaSelective inclusion of studies with positive findingsExclusion of studies with negative findings
  • 66. Explaining heterogeneityIn language of meta-analysis- Homogeneity means results of each individual trialare compatible with the results of any of the others- Heterogeneity means results of each individual trialare incompatible with results of any of the others
  • 67. Do the pieces fit together?Simon SD. Statistical evidence in medical trials: What do the data really tell us?Oxford University Press, Oxford, 1st edition, 2006
  • 68. How to measure heterogeneity in MA?• QualitativeForest plot Visual evidence of heterogeneityFunnel plot Visual evidence of heterogeneity• QuantitativeX-squaredI-squared Based on Cochran‟s QSimon SD. Statistical evidence in medical trials: What do the data really tell us?Oxford University Press, Oxford, 1st edition, 2006
  • 69. Heterogeneity & forest plotHypothetical MASome trials with lower C.I. above upper C.I. of other trialsSome lines do not overlapMcGovern D, Summerskill W, Valori R, Levi M. Key topics in EBM.BIOS Scientific Publishers, 1st Edition, Oxford, 2001.
  • 70. Funnel plotsBias detected by simple graphical test• Plot for each trial RR or OR on x axisSample size on y axis• Absence of biasPlot should resemble inverted funnel or Christmas tree• Presence of biasPlot shows asymmetrical & skewed shape
  • 71. Ideal funnel plotThe smaller the trial, the larger the distribution of resultsCleophas TJ et all. Statistics applied to clinical trials.Springer, The Netherlands , 3rd edition, 2006.
  • 72. Cut Christmas treeCleophas TJ et all. Statistics applied to clinical trials.Springer, The Netherlands , 3rd edition, 2006.Negative trials not published (missing)Suspicion of considerable publication bias in this MA
  • 73. Funnel plotPublication bias of antibiotics for infected necrosisBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.
  • 74. Funnel plotPublication bias of trials of antibiotics for mortalityBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.
  • 75. Quantitative measure of heterogeneityMany prefer not to use quantitative measure• X-squared:Degree of freedom (df) Number of trials in MA – 1X2 ≈ df No heterogeneityX2 much greater than df Serious heterogeneity• I-squared (0 – 100%)< 25% No heterogeneity50% – 75% Serious heterogeneitySimon SD. Statistical evidence in medical trials: What do the data really tell us?Oxford University Press, Oxford, 1st edition, 2006
  • 76. Bai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Antibiotic prophylaxis & pancreatic necrosisHeterogeneityX2 = 7.82 (df 6 – No heterogeneity)I2 = 23.2% (No or little heterogeneity)
  • 77. Antibiotic prophylactic effect on mortalityHeterogeneityBai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.X2 = 4.66 (df 6 – No heterogeneity)I2 = 0 % (No or little heterogeneity)
  • 78. Appraising & applying meta-analysisHeneghan C, Badenoch D. EBM toolkit. BMJ Books, London, 1st edition 2002.
  • 79. Questions for appraising MA – 1Critical Appraisal Skills Programme. Appraisal Tools. Oxford, UK.http://www.phru.nhs.uk/casp/appraisa.htm (accessed 10 Dec 2004). Clearly focused question Focused question Identification of all relevant studies Good search Inclusion the right type of study Yes (RCTs) Assessment quality of all studies Yes but no blinding Reasonable to combine study results Yes (good X2 & I2)
  • 80. Questions for appraising MA – 2Critical Appraisal Skills Programme. Appraisal Tools. Oxford, UK.http://www.phru.nhs.uk/casp/appraisa.htm (accessed 10 Dec 2004). Precision of the results No (wide 95% CI) Result presentation & main result RR (95% CI)No difference Results applied to local population Mainly alcoholic Change practice as result of MA No?All important outcomes considered Antibiotic SE?
  • 81. Steps of EBM Assess
  • 82.  AssessProphylactic antibiotics in pancreatic necrosisLimitations of this MA• Timing of initiation of antibiotics• Subgroup analysis AgeEtiology of pancreatitisPresence of organ failure• Wide 95% CI Infected necrosis 0.81 (0.54 –1.22)Mortality 0.70 (0.42 – 1.17)Bai Y et al. Am J Gastroenterol 2008 ; 103 : 104 – 110.Further large scale better design RCTs are needed
  • 83. * Altman DG et al. Ann Intern Med 2001 ; 134 : 663 - 94.Improving quality of reportsConsolidatedStandards ofReporting TrialsRCTsCONSORT*Diagnosticaccuracy studySTARD**Standards forReporting ofDiagnostic Accuracy*** Bossuyt PM et all. BMJ 2003; 326 : 41 – 44.Quality ofReporting ofMeta-analysesMeta-analysisQUOROM**** Moher D et al. Lancet 1999 ; 354 : 1896 - 900.
  • 84. QUOROM* statementTargeted authors of MA rather than readers• Experts 30 experts (epidemiologists, clinicians, editors,statisticians, researchers)• Date Oct 2–3, 1996 (Chicago – USA)• Aim Improve quality of reporting MA & may be SR• Results Flow diagram: progress through stages of MAChecklist: 21 headings & subheadings* Quorom: Quality of Reporting of Meta-analysesMoher D et al. Lancet 1999 ; 354 : 1896 - 900.
  • 85. The QUOROM checklistHeading Subheading Descriptor ReportedPage NoTitle Identify report as MA or SR of RCTsAbstractObjectivesData sourcesReview methodsResultsConclusionUse a structured formatClinical question explicitlyDatabases (list) & other information sourcesSelection criteria, validity assessment, data synthesisCharacteristics of RCTs, point estimates, CIMain resultsIntroduction Clinical problem, rationales for intervention & reviewMethods SearchingSelectionValidity assessmentData abstractionStudy characteristicsData synthesisInformation sources in detail, precise restrictionsInclusion & exclusion criteriaCriteria & process used (masked conditions, ..)Process used (completed independently, in duplicate)Study design, intervention, outcome & heterogeneityMeasures of effect (RR), method of combining results(statistical testing & CI), missing data; statisticalheterogeneity, assessment of publication biasResults Trial flowStudy characteristicsData synthesisProfile summarizing trial flowData for each trial (age, sample size, dose, follow-up)Agreement, summary results, effect sizes & CI in ITTDiscussion Key findings, internal & external validity, biases, …
  • 86. How much work is a meta-analysis?• Analysis of 37 MA by Allen & Olkin of MetaWorks*• Hours Average 1139 (216 – 2518)• Breakdown 588 Protocol, searching, & retrieval44 Statistical analysis206 Report writing201 Administration• Total time depends on number of citations* Company based in Massachusetts (USA) specializes in doing SRAllen, I.E. Olkin, I. JAMA 1999; 282 : 634 – 5.
  • 87. “Doing a meta-analysis is easy,doing one well is hard”Ingram Olkin
  • 88. Importance of meta-analysis• For some cliniciansMA is seen as exercises in "mega-silliness"• For other cliniciansMA left no place for narrative review article• The truthIs likely to lie somewhere between these 2 extremes
  • 89. ReferencesCambridge Press2001BMJ Publishing Group2001John Wiley & Sons2009
  • 90. Thank You