CADTH 2015 Symposium

1. Does it Matter —
Timeliness or Accuracy of Results?
Results of a Research Program on Rapid Reviews
Andrea C. Tricco PhD MSc
CADTH Symposium 2015

2.  I have no actual or potential conflict of
interest in relation to this presentation.
Conflict of interest
2

3.  Background
 Scoping Review of Rapid Reviews
 International Survey of Rapid Review Producers
 International Consensus-building Exercise with Key
Stakeholders
 Ultimate Goal of Our Research Program
 Questions
Outline
2

4. Background
 Rapid reviews are a form of knowledge synthesis in which
components of the systematic review process are simplified or
omitted to produce information in a timely manner [Khangura 2012]
 Evidence suggests that decision-makers are currently using rapid
reviews to inform their decision-making processes
 Few studies have examined the methodological characteristics of
rapid reviews
 We aimed to conduct a research program on rapid reviews to clarify
the methods and perceptions of rapid review approaches
3

5. A scoping review
of rapid review methods
Submitted to BMC Medicine
Methods project 1:

6. Objective & methods
 Objective
– To examine rapid review approaches, guidance, impact, and comparisons
through a scoping review
 Methods
– Used methodologically rigorous scoping review methods proposed by
Arksey and O’Malley (2005)
– MEDLINE, EMBASE, the Cochrane Library, internet websites of rapid review
producers, and reference lists were searched to identify articles for inclusion
– Two reviewers independently screened citations and full-text articles, and
abstracted data
5

7. Results
Study flow figure
N=3397 citations from MEDLINE, EMBASE, Cochrane Library,
previous systematic reviews, and grey literature
N=3135 excluded titles and abstracts
N=100 rapid reviews
(plus 1 companion report)
N=262 potentially relevant full-text articles
N=161 excluded full-text
reports
6

8. Results (continued)
Word cloud figure for the frequency of terms
7

9. Study Characteristics
No. of Rapid Reviews
(n=100)
Year of Publication
1997-2000 2
2001-2004 10
2005-2008 30
2009-2012 51
2013 5
Not Reported 4
Corresponding
author’s
continent
Europe (including UK) 58
North America (Canada & United States) 20
Australia 15
Multiple continents 3
Asia 1
South America 1
Not Reported 2
Study characteristics
Results (continued)
8

10. Results (continued)
Study Characteristics No. of Rapid Reviews (n=100)
Article Type
Application (82 with methods) 84
Development 7
Impact 6
Comparison 4
Topic of Review
Intervention 62 (74%)
Frequency 10 (12%)
Causal association 4 (5%)
Diagnosis 4 (5%)
Patient experience 2 (2%)
Screening 2 (2%)
Not applicable 16
Study characteristics (continued)
9

11. Results (continued)
Methods characteristics
4%
2%
83%
70%
51%
22%
12%
17%
35%
25%
10%
17%
23%
98%
12%
24%
10%
11%
68%
49%
54%
46%
68%
57%
73%
6%
6%
40%
67%
20%
34%
17%
29%
22%
26%
0% 20% 40% 60% 80% 100%
Duration of review
Protocol mentioned
Databases Searched
Grey literature search
Scanning references
Contacting authors
Date limit
Language limit
Titles and abstracts screening
Full-texts screening
Data abstraction
Quality appraisal
SR method Streamlined method Not reported
10

12. Results (continued)
Approach Literature search Search limit Screening Data abstraction
Risk of bias
appraisal
1
>1 database,
published only
Both date and
language
One reviewer
One person
abstracts, other
verifies
One person
assesses, other
verifies
2
Updating the literature
search of a previous
review, published only
None One reviewer One reviewer Not performed
3
>1 database, grey
literature
Both date and
language
One reviewer One reviewer Not performed
4
>1 database, grey
literature
Either date or
language
One reviewer One reviewer Not performed
5
>1 database, grey
literature
Date One reviewer One reviewer One reviewer
5 most frequent rapid review approaches
11

13. Conclusions
 Numerous rapid review approaches were identified
 Little consistency exists in the field
 Poor quality of reporting was observed
 Prospective study comparing the results from rapid reviews to
those obtained through systematic reviews is warranted.
12

14. An international survey
of rapid review producers
Submitted to the J of Clin Epi
Methods project 2:

15. Objective and methods
 Objective
– To determine different rapid review approaches used by rapid review
producers
 Methods
– International survey of 63 organizations administered via FluidSurvey
– Survey pilot-tested prior to administration
– Reminders to non-respondents sent every 2 weeks
– Contacted through postal mail if no response
– $10 financial incentive from Amazon
14

16. Results
Study flow figure of participants
63 organizations contacted
41 responses (65%)
22 did not respond
15

17. Results (continued)
Summary of rapid review characteristics
Review Characteristics Count (%)
Duration of Review
(weeks)
1-12 62 (70)
12-26 18 (20)
26-36 6 (7)
≥ 52 2 (2)
Commissioning Agency
Government Agencies & Health Ministries 69 (78)
Healthcare Organizations, Hospitals & Community Health 51 (58)
Healthcare Professionals 13 (15)
Industry 4 (5)
Target Audience
Government Agencies & Health Ministries 73 (83)
Healthcare Professionals 46 (52)
Patients 19 (22)
Researchers 21 (24)
16

18. Results (continued)
Rationale provided Count (%)
Decision-maker timeline 57 (66)
Focused or brief question 8 (9)
Lack of resources 5 (6)
Increase efficiency (including timeliness) 4 (5)
Broad understanding of an area 4 (5)
Identify topics requiring a systematic review 2 (2)
Update a systematic review 2 (2)
Well-established intervention 1 (1)
Evidence is unclear 1 (1)
Rationale for conducting rapid reviews
17

19. Results (continued)
Review Stage Most frequent streamlined approach Count (%)
Identifying relevant studies Used previous review(s) as a starting point 79 (92)
Limitations on search strategy Limited review by date of publication 75 (88)
Study selection Screening conducted by ONE reviewer only 68 (85)
Data Abstraction Data abstraction performed by ONE reviewer only 67 (84)
Quality (risk of bias) appraisal
process
Risk of bias assessed by ONE reviewer only 68 (86)
Synthesis Narrative summary 75 (90)
Summary results of most frequently streamlined approaches
18

20. Conclusions
 Results are consistent with scoping review of rapid reviews
 Rapid reviews usually conducted in 1-12 weeks
 Government agencies and health ministries are primary
commissioners
 Many different streamlined methods are being used.
19

21. International consensus-building
exercise regarding rapid reviews
Submitted to the J of Clin Epi
Methods project 3:

22. Objective & methods
 Objective
– To conduct a consensus-building exercise to select a rapid review approach
that will be prospectively tested in a diagnostic study
 Methods
– Editors, healthcare providers, researchers, policy-makers, and industry
stakeholders (including participants of the CADTH rapid review summit)
– Asked to rank the 5 most frequent rapid review approaches identified in our
scoping review and survey using FluidSurvey
– Results presented to participants, followed by a facilitated discussion (online
and in-person) and re-ranking exercise using FluidSurvey
21

23. Results
Study flow figure of participants
26 individuals
contacted
113 responses (72%)
3 did not respond
130 individuals
contacted
40 did not respond
22
Online delphi In-person delphi

24. Summary of ranking results by approach
Results (2)
Rapid review
approach
Feasibility Timeliness Comprehensiveness Risk of Bias
Approach 1 1st 2nd 4th 1st
Approach 2 2nd 1st 5th 5th
Approach 3 3rd 3rd 3rd 2nd
Approach 4 4th 4th 2nd 4th
Approach 5 5th 5th 1st 3rd
*Ranked based on the distribution of "very" and "extremely" on the 7-point Likert scale, except Risk of Bias was ranked on
distribution of “not at all” and “very”
23

25.  The highest ranked method was: Approach 1
‒ Most feasible (72%, n=81 out of 113 responses)
‒ Lowest perceived risk of bias (12%, n=12 out of 103)
‒ 2nd in timeliness (37%, n=38 out of 102)
‒ 5th in comprehensiveness (5%, n=5 out of 100)
 We will use the information from the consensus-building exercise
to select the rapid review approach for a prospective study.
Conclusion
24

26. Ultimate goal of this research
“We can give you results within 4 months, but the meta-
analysis estimates will be inaccurate by 35%”
Rapid review definition (Shannon Kelly)
Identify 5 frequently used methods
Diagnostic study to test a rapid review approach
Identify and characterize rapid review methods
25

27.  Will use these results from our research program to inform a
diagnostic study:
− Index test: Rapid Review Approach
− Reference standard: Systematic Review
 Collaboration between 3 Canadian Knowledge Synthesis Centers
 Targeting CIHR and PCORI
Proposed diagnostic study
Diagnostic Accuracy of Rapid reviews compared To
Systematic reviews (DARTS)
26

28. Summary
 This research program provides up-to-date information on rapid
review methods reported in the literature, as well as stakeholder
experiences and perceptions regarding rapid reviews
 Poor quality of reporting was observed in the literature
 Rapid reviews have many names and approaches and some
methods might be more desirable than others
 A prospective study comparing the results of rapid reviews to those
obtained through systematic reviews is necessary.
27

29. Acknowledgements
 Funding: Canadian Institutes of Health Research/Drug Safety
and Effectiveness Network
− Operating grant to update 2 systematic reviews,
international survey, Delphi
− New investigator award
 Research team: Jesmin Antony, Wasifa Zarin
 Co-investigators: Drs. Straus, Moher, Hutton, Sherifali
28

30. Questions?
triccoa@smh.ca
30