The purpose of this chapter is to define the project, the purpose and scope of the review, the key research questions, the analytic framework, et cetera.
Chapter 2: Methods
The purpose of this chapter is to explain the methods used in the review, including the experts involved, the literature search strategy used, the inclusion and exclusion criteria applied, how the evidence tables were developed, the approach used to assess the quality of studies, and data abstraction and data synthesis methods.
This chapter should not present any “results” (e.g., tables), but rather should serve as a guide for how the study information was collected and the evidence tables were created so that the research can be replicated.
The purpose of this chapter is to report the results of the data analyses, which should be broken down according to the key research questions.
Subsections should be used when applicable.
Chapter 4: Discussion
The purpose of this chapter is to discuss the strength of the literature and evidence, the principal findings (broken down by key questions) of the review, areas of future research, and any conclusions that can be drawn.
Describe all information sources used in the literature search:
Databases with dates of coverage
Contacts with study authors to identify additional studies
Date of the last search
We searched the following databases for primary studies for the periods in parentheses: MEDLINE ® (1966 to January 2006), EMBASE ® (1974 to January 2006), and the Cochrane Central Register of Controlled Trials (1966 to January 2006). We also searched for systematic reviews until November 2005.
Methods: Data Sources Bolen S, et al. AHRQ Comparative Effectiveness Review No. 5. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/6/39/OralFullReport.pdf.
State the processes used to select studies for review:
Inclusion/exclusion criteria applied for the systematic review and, if applicable, the meta-analysis
We included trials if the randomization scheme included groups that assigned patients to treatment guided by the PAC [pulmonary artery catheter] or treatment without the PAC. We only included trials if they reported death and number of days hospitalized or the number of days in the ICU as outcome measures. Studies were excluded if the randomization scheme did not specify groups as PAC or no PAC, if patients were not randomized to a conventional PAC, if investigators combined randomized and nonrandomized groups when reporting outcomes, or if there were no outcome data on death or hospitalizations.
Methods: Eligibility Criteria (l) Shah MR, et al. JAMA 2005;294:1664-70.
Randomised clinical trials studying the administration of hepatitis B vaccine to CRF [chronic renal failure] patients, with or without dialysis. No language, publication date, or publication status restrictions were imposed.
Types of participants:
Participants of any age with CRF or receiving dialysis (haemodialysis or peritoneal dialysis) were considered.… Renal transplant patients were excluded from this review.
Methods: Eligibility Criteria (II) Schroth RJ, et al. Cochrane Database Syst Rev 2004;(3):CD003775.
Trials comparing the beneficial and harmful effects of hepatitis B vaccines with adjuvant or cytokine co-interventions [and] trials comparing the beneficial and harmful effects of immunoglobulin prophylaxis.… Hepatitis B vaccines (plasma or recombinant [yeast] derived) of all types, dose, and regimens versus placebo, control vaccine, or no vaccine.
Types of outcomes:
Primary outcome measures: Seroconversion , [that is], proportion of patients with adequate anti-HBs response (10 IU/L or Sample Ratio Units).…Secondary outcome measures: Adverse events of hepatitis B vaccinations…[and] mortality .
We developed a data extraction sheet[,]…pilot-tested it on ten randomly-selected included studies, and refined it accordingly. One review author extracted the…data…and the second author checked the extracted data. …Disagreements were resolved by discussion between the two review authors.
Describe any processes used to obtain and confirm data from other investigators.
Methods: Data Extraction Mistiaen P, Poot E. Cochrane Database Syst Rev 2006;(4):CD004510.
List and define all variables for which data were sought, using the PICOTS framework as a guide.
List any assumptions and simplifications that were made in defining the variables.
Information was extracted from each included trial on: (1) characteristics of trial participants…and the trial’s inclusion and exclusion criteria; (2) type of intervention…(versus placebo or versus the type, dose, duration and frequency of another NSAID [nonsteroidal antiinflammatory drug]; or versus another pain management drug; or versus no treatment); (3) type of outcome measure.
Methods: Data Items Allen C, et al. Cochrane Database Syst Rev 2005;(4):CD004753. PICOT(S) = population, intervention, comparator, outcome, timing, and study design or setting
Describe the methods and criteria used to assess the quality (risk of bias) of individual studies.
Specify whether or not the assessment was carried out at the study or outcome level, or both.
Describe how this information is to be used in any data synthesis.
Examples from two separate studies:
Pairs of reviewers…determined the adequacy of randomization and concealment of allocation, blinding of patients, health care providers, data collectors, and outcome assessors; and extent of loss to follow-up .
To explore variability in study results (heterogeneity) we specified the following hypotheses…
Methods: Quality Assessment Tracz MJ, et al . J Clin Endocrinol Metab 2006;91:2011-6; Bucher HC, et al. BMJ 2000;321:73-7.
Describe the methods used to handle the data and to combine the results of studies.
Describe measures of consistency (e.g., I-squared).
Examples from two separate studies:
In very few instances, estimates of baseline mean or mean QOL [quality of life] responses were obtained without corresponding estimates of variance (standard deviation [SD] or standard error). In these instances, an SD was imputed from the mean of the known SDs. In a number of cases, the response data available were the mean and variance in a prestudy condition and after therapy.…
We tested for heterogeneity with the Breslow-Day test, and used the method proposed by Higgins et al. to measure inconsistency.
Methods: Synthesis of Results Jones M, et al. Cancer 2004;101:1720-32; Briel M, et al. Am J Med 2004;117:596-606.
Detail any assessment of risk of bias that may affect the cumulative evidence.
Selective reporting within studies
For each trial we plotted the effect by the inverse of its standard error. The symmetry of such “funnel plots” was assessed both visually, and formally with Egger’s test, to see if the effect decreased with increasing sample size.
Methods: Grading Strength of Evidence Hróbjartsson A, Gøtzsche PC. Cochrane Database Syst Rev 2004;(1):CD003974.
Describe the methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression).
Indicate which of these analyses were prespecified.
Sensitivity analyses were pre-specified. The treatment effects were examined according to quality components (concealed treatment allocation, blinding of patients and caregivers, blinded outcome assessment), time to initiation of statins, and the type of statin. One post-hoc sensitivity analysis was conducted including unpublished data from a trial using cerivastatin.
Methods: Additional Analyses Briel M, et al. JAMA 2006;295:2046-56.
Place at the beginning of the Results section, not in the Methods section.
Give numbers of studies screened, assessed for eligibility, and included in the review.
Give the reasons for exclusions at each stage of the assessment, ideally illustrated with a flow diagram.
The search of Medline, PsycInfo and Cinahl [sic] databases provided a total of 584 citations. After adjusting for duplicates 509 remained. Of these, 479 studies were discarded because…. Three additional studies…were discarded because…. The full text of the remaining 27 citations was examined in more detail. It appeared that 22 studies did not meet the inclusion criteria as described. Five studies…met the inclusion criteria and were included in the systematic review.
Results: Study Selection Uitterhoeve RJ, et al. Br J Cancer. 2004;91:1050-62.
Examples of Flow Diagrams Fuccio L, et al. Ann Intern Med 2007;147:53-62. Reprinted with permission from the American College of Physicians. Sharma M, et al. Ann Intern Med 2009:151:622-30. Reprinted with permission from the American College of Physicians.
All four studies finally selected for the review were randomised controlled trials published in English. The duration of the intervention was 24 months for the RIO-North America and 12 months for the RIO-Diabetes RIO-Lipids and RIO-Europe study. Although the last two described a period of 24 months during which they were conducted, only the first 12-months results are provided. All trials had a run-in, as a single blind period before the randomisation.
Results: Study Characteristics Curioni C, André C. Cochrane Database Syst Rev 2006;(4):CD006162.
Present synthesized results in text, summary tables, or evidence maps.
Mortality data were available for all six trials, randomizing 311 patients and reporting data for 305 patients. There were no deaths reported in the three respiratory syncytial virus/severe bronchiolitis trials; thus our estimate is based on three trials randomizing 232 patients, 64 of whom died. In the pooled analysis, surfactant was associated with significantly lower mortality (relative risk = 0.7, 95% confidence interval = 0.4 – 0.97, P = 0.04). There was no evidence of heterogeneity (I 2 = 0%).
Summary tables: refer to the Presentation of Findings module
Evidence maps: refer to the Presentation of Findings module
Results: Synthesis of Results Duffett M, et al. Crit Care 2007;11:R66.
Include risk of bias, directness, consistency, and precision in reporting how evidence was graded.
There is a low level of evidence…that RFA [radiofrequency catheter ablation] improves quality of life more than medical treatment. Three RCTs [randomized controlled trials] and one observational study reported more improvement in the general or physical functioning…in patients who underwent RFA.…However, these studies assessed the results at nonuniform time points and therefore the findings may be difficult to interpret.
Results: Grading Strength of Evidence Ip S, et al. AHRQ Comparative Effectiveness Review No. 15. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/51/114/2009_0623RadiofrequencyFinal.pdf.
Give the results of additional analyses, such as sensitivity or subgroup analyses and meta-regressions.
Include the results of additional analyses to facilitate a better understanding of heterogeneity.
[B]enefits of chondroitin were smaller in trials with adequate concealment of allocation compared with trials with unclear concealment (P for interaction = 0.050), in trials with an intention-to-treat analysis compared with those that had excluded patients from the analysis (P for interaction = 0.017), and in large compared with small trials (P for interaction = 0.022).
Results: Additional Sensitivity Analyses Reichenbach S, et al. Ann Intern Med 2007;146:580-90.
Subgroup analyses according to antibody status, antiviral medications, organ transplanted, treatment duration, use of antilymphocyte therapy, time to outcome assessment, study quality and other aspects of study design did not demonstrate any differences in treatment effects. Multivariate meta-regression showed no significant difference in CMV [cytomegalovirus] disease after allowing for potential confounding or effect-modification by prophylactic drug used, organ transplanted or recipient serostatus in CMV positive recipients and CMV negative recipients of CMV positive donors.
Compared with men who used watchful waiting , men with clinically localized prostate cancer detected by methods other than PSA [prostate-specific antigen] testing and treated with radical prostatectomy experienced fewer deaths from prostate cancer, marginally fewer deaths from any cause, and fewer distant metastases. The greater benefit of RP on cancer-specific and overall mortality appears to be limited to men under 65 years of age but is not dependent on baseline PSA level or histologic grade.
Discussion: Summary of Evidence Wilt TJ, et al. AHRQ Comparative Effectiveness Review No. 13. Available at: . http://www.effectivehealthcare.ahrq.gov/ehc/products/9/80/2008_0204ProstateCancerFinal.pdf.
Discuss the limitations of the review at different levels:
Study level (e.g., risk of bias)
Outcome level (e.g., benefits or harms)
Review level (e.g., incomplete retrieval of identified research; reporting bias)
The meta-analysis reported here combines data across studies in order to estimate treatment effects with more precision than is possible in a single study. The main limitation of this meta-analysis, as with any overview, is that the patient population, the antibiotic regimen and the outcome definitions are not the same across studies.
Discussion: Limitations Liberati A, et al. Cochrane Database Syst Rev 2004;(1):CD000022.
Provide a general interpretation of the results in the context of other evidence and the implications for future research.
[T]he available clinical trial evidence supporting the use of combination therapies over higher dose statin therapy is insufficient to guide clinical decisions. The long term clinical benefits and risks of combination therapies have yet to be demonstrated. There are some instances, such as failure to reach targets in spite of maximal statin therapy, and populations with elevated triglycerides who need to achieve secondary goals, in which clinicians may choose combinations pending definitive evidence.
Discussion: Conclusions Sharma M, et al. AHRQ Comparative Effectiveness Review No. 16. Available at: http://www.effectivehealthcare.ahrq.gov/ehc/products/11/171/reptbodyfin-typofixed4-12-2010.pdf.
Alejandria MM, Lansang MA, Dans LF, et al. Intravenous immunoglobulin for treating sepsis and septic shock. Cochrane Database Syst Rev 2002;(1):CD001090.
Allen C, et al. Non-steroidal anti-inflammatory drugs for pain in women with endometriosis. Cochrane Database Syst Rev 2005;(4):CD004753.
Bolen S, Wilson L, Vassy J, et al. Comparative Effectiveness and Safety of Oral Diabetes Medications for Adults With Type 2 Diabetes . Comparative Effectiveness Review No. 8 (Prepared by The Johns Hopkins University Evidence-based Practice Center under Contract No. 290-02-0018). Rockville, MD: Agency for Health Care Research and Quality, July 2007. AHRQ Publication No. 07-EHC010-EF.
Briel M, Studer M, Glass T, et al. Effects of statins on stroke prevention in patients with and without coronary heart disease: a meta-analysis of randomized controlled trials. Am J Med 2004;117:596-606.
Briel M, Schwartz GG, Thompson PL, et al. Effects of early treatment with statins on short-term clinical outcomes in acute coronary syndromes: a meta-analysis of randomized controlled trials. JAMA 2006;295:2046-56.
Bucher HC, Hengstler P, Schindler C, et al. Percutaneous transluminal coronary angioplasty versus medical treatment for non-acute coronary heart disease: meta-analysis of randomised controlled trials. BMJ 2000;321:73-7.
Curioni C, André C. Rimonabant for overweight or obesity. Cochrane Database Syst Rev 2006;(4):CD006162.
Devereaux PJ, Beattie WS, Choi PT, et al. How strong is the evidence for the use of perioperative beta blockers in non-cardiac surgery? Systematic review and meta-analysis of randomised controlled trials. BMJ 2005;331:313-21.
Ip S, Terasawa T, Balk EM, et al. Comparative Effectiveness of Radiofrequency Catheter Ablation for Atrial Fibrillation . Comparative Effectiveness Review No. 15 (Prepared by Tufts –Mew England Medical Center Evidence-based Practice Center under Contract No. 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality, July 2009. AHRQ Publication No. 09-EHC015-EF.
Jones M, Schenkel B, Just J, et al. Epoetin alfa improves quality of life in patients with cancer: results of metaanalysis. Cancer 2004;101:1720-32.
Lakhdar R, Al-Mallah MH, Lanfear DE. Safety and tolerability of angiotensin-converting enzyme inhibitor versus the combination of angiotensin-converting enzyme inhibitor and angiotensin receptor blocker in patients with left ventricular dysfunction: a systematic review and meta-analysis of randomized controlled trials. J Card Fail 2008;14:181-8.
Liberati A, Altman DF, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 2009;151:W65-94.
Liberati A, D'Amico R, Pifferi, et al. Antibiotic prophylaxis to reduce respiratory tract infections and mortality in adults receiving intensive care. Cochrane Database Syst Rev 2004;(1):CD000022.
Mistiaen P, Poot E. Telephone follow-up, initiated by a hospital-based health professional, for postdischarge problems in patients discharged from hospital to home. Cochrane Database Syst Rev 2006;(4):CD004510.
PRISMA Statement Web site. Home page. Available at: http://www.prisma-statement.org/index.htm. Reichenbach S, Sterchi R, Scherer M, et al. Meta-analysis: chondroitin for osteoarthritis of the knee or hip. Ann Intern Med 2007;146:580-90.
Schroth RJ, Hitchon CA, Uhanova J, et al. Hepatitis B vaccination for patients with chromic renal failure. Cochrane Database Syst Rev 2004;(3):CD003775.
Shah MR, Hasselvlad V, Stevenson LW, et al. Impact of the pulmonary artery catheter in critically ill patients: meta-analysis of randomzied clinical trials. JAMA 2005;294:1664-70.
Sharma M, Ansari MT, Soares-Weiser K, et al. Comparative Effectiveness of Lipid-Modifying Agents . Comparative Effectiveness Review No. 16 (Prepared by the University of Ottawa Evidence-based Practice Center under contract No. 290-02-0021). Rockville, MD: Agency for Healthcare Research and Quality, September 2009. AHRQ Publication No. 09-EHC024-EF.
Sharma M, Ansari MT, Abou-Setta AM, et al. Systematic review: comparative effectiveness and harms of combination therapy and monotherapy for dyslipidemia. Ann Intern Med 2009;151:622-30.
Tracz MJ, Sideras K, Boloña ER, et al. Testosterone use in men and its effects on bone health. A systematic review and meta-analysis of randomized placebo-controlled trials. J Clin Endocrinol Metab 2006;91:2011-6.
Uitterhoeve RJ, Vernooy M, Litjens M, et al. Psychosocial interventions for patients with advanced cancer — a systematic review of the literature. Br J Cancer 2004;91:1050-62.
Wilt TJ, Shamliyan T, Taylor B, et al. Comparative Effectiveness of Therapies for Clinically Localized Prostate Cancer . Comparative Effectiveness Review No. 13 (Prepared by Minnesota Evidence-based Practice Center under Contract No. 290-02-0009). Rockville, MD: Agency for Healthcare Research and Quality, February 2008. AHRQ Publication No. 08-EHC010-EF.
This presentation was prepared by David Moher, Ph.D., director, University of Ottawa Evidence-based Practice Center.
Many of the examples in the presentations are taken from the PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions (Liberati A, et al. PLoS Med 2009;6(7): e1000100).