CADTH 2015 Symposium

1. uOttawa.cauOttawa.ca
Deconstructing Rapid Reviews
• An Exploration of Knowledge, Traits and Attitudes
Presented by: Shannon Kelly, CADTH Symposium 2015 – Saskatoon, SK
uOttawa.ca
Faculty of Medicine
School of Epidemiology, Public Health and Preventive Medicine

2. uOttawa.ca
Objectives
• To present results from 3 projects on
rapid reviews conducted as part of
MSc. Epidemiology at uOttawa
I. Defining rapid reviews – a consensus approach
II. Attitudes and perceptions towards rapid reviews
III. Defining rapid reviews – a pragmatic approach
2

3. uOttawa.ca
RAPID REVIEWS?
Why do we care about

4. uOttawa.ca
What is a rapid review?
“…a streamlined approach to synthesizing evidence in a
timely manner - typically for the purpose of informing
emergent decisions faced by decision makers in health
care settings.”
~ Khangura, 2012
Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the
evolution of a rapid review approach. Systematic Reviews. 2012;1:10.

5. uOttawa.ca
Rapid reviews influence decision-
making in Canada
• Knowledge users in Canada (everywhere) need support
to make timely, evidence-informed health care decisions
• Systematic reviews and HTA often take too long!
• Delay important policy decisions
• Use less-than-best evidence
• Rapid reviews help fill this void
• Balance rigour, relevance, timelines
…We still need to know so much more…

6. uOttawa.ca
What do we know?
Ultra-rapid reviews, first test results

7. uOttawa.ca
What don’t we know?
“While rapid review approaches are being
used by HTA producers across the world in
a variety of ways an agreed-upon
definition describing its constitution and
methods is lacking” (Khangura, 2014)“While the concept
of rapid evidence
synthesis, or rapid
review, is not
novel, it remains
a poorly
understood and
yet ill-defined set
of disparate
methodologies
supported by a
paucity of
published,
available scientific
literature”
(Khangura, 2014)
“..evidence is only
slowly emerging as to
which steps in the
systematic review process
may be altered by
increased speed”
(Schünemann, 2015)
“In order to consider offering rapid reviews, one first has to be able to
define what they are. Previous reviews on the topic of rapid reviews
describe them as ‘ill-defined,’ ‘not well-defined,’ lacking a single
definition, or ‘varying widely in terms of the language used to
describe them.’” (AHRQ, Hartling, 2015)
“…a
literature
search by the
authors does
not elucidate
any clear or
final
definition of
what a RR is
and how
exactly the
methodology
of a RR
differs from
a full SR.”
(Harker,
2012)
“There is little empirical evidence comparing the
continuum of products among rapid reviews
and full systematic reviews, or analysing the
diverse methods used in rapid reviews. It is
important not only to establish transparent
methodologies for rapid reviews, but also to
understand the implications of what is lost in terms
of rigour, bias, and results when methods
associated with full systematic review are
streamlined.” (Ganaan, 2010)

8. uOttawa.ca
Needed a research program to
improve knowledge on rapid reviews
An improved understanding of rapid reviews will help us
better meet the needs of decision-makers

9. uOttawa.ca
DEFINING RAPID REVIEWS
A consensus approach

10. uOttawa.ca
Objective
• The study will employ a modified Delphi method to
survey of experts in the field of evidence synthesis and
rapid reviews to discover what they consider the key
defining characteristics of rapid reviews to be.
Note: For our purposes, defining characteristics are:
‘one of a number of essential features by which a rapid
reviews can be recognized’.

11. uOttawa.ca
Method: The Delphi Technique
Definition: a procedure that structures a communication
process among a group of experts.(Linstone & Turoff, 1975)
Characteristics of the Delphi technique:
 Staged process, assessment of agreement or consensus
 Assessment of the group’s view;
 An opportunity for informants to revise their views;
 Opportunity to react to and assess differing view points;
 Anonymity of informants.
Panelists are selected based on their expertise in the
subject matter; not random.

12. uOttawa.ca
Participants
Experts targeted for participation were:
1. Subject-matter experts or experienced researchers
with knowledge of a variety of evidence synthesis
methods and practical experience with rapid reviews;
2. Authors with publications relevant to rapid reviews;
and,
3. Delegates presenting pertinent work at recent
meetings, workshops, conferences or symposia.
• International Cochrane Colloquium, Health Technology
Assessment international (HTAi), CADTH Symposium,
Cochrane Canada Symposium).

13. uOttawa.ca
The Modified Delphi Process
STUDY SCOPING AND
EXPERT PANEL DEFINITION
ROUND 1:
WEB-BASED SURVEY ON KEY
RR THEMES
(FluidSurveys)
ROUND 2:
DRAFT CHARACTERISTIC
STATEMENTS
(email)
ROUND 3:
REVISED STATEMENTS,
EXPERT REVISION OF
AGREEMENT
(email)
FINAL 11 CHARACTERISTIC
STATEMENTS
PUBLICATION
PENDING
Review agreement,
refine statements
120 POTENTIAL EXPERTS
Pilot Review,
summarize,
analyze, draft
statements66 EXPERTS 44 EXPERTS
26 EXPERTSCONSENSUS ON 10 OF 11
STATEMENTS
Consensus level = 70%, Approved by TOHREB
Review
agreement
Summarize
results

14. uOttawa.ca
Results: Key Characteristics (1)
• Conducted in less time than a systematic review;
• Have a protocol (at least describing objectives, scope,
PICO, approach);
• Tailor the explicit, reproducible methods conventionally
used in a systematic review in some manner to
expedite the process to meet the needs of a
decision-maker;
• Use a spectrum of approaches to complete an
evidence synthesis related to a defined research
question(s) using the most systematic or rigourous
methods the limited time frame will allow.
Rapid reviews:

15. uOttawa.ca
RR time to completion
Delphi Expert Opinion
Previous RR Samples
GANAAN
2010
HARKER
2012
INAHTA scan
2012
POLISENA
2015
AHRQ
2015
TRICCO
(RR SYMP.
CADTH)
Minimum 3 wk >3 mo 1 wk 1 wk 5 min <1mo
Maximum 6 mo < 24 mo 12 mo 12 mo 8 mo 12 mo
0 20 40 60 80 100 120
Rapid Review
Systematic Review
Time (in weeks)
Khangura et al. (2012): SRs take 6 months to 2 years, and RRs ≤ 5 weeks.
Delphi panel: Median time for SR is 36 weeks, while median for RR is 9 weeks.
Min = minute; d = days, wk = week; mo = month; yr = year

16. uOttawa.ca
Results: Key Characteristics (2)
• Balance the risk of the yet undetermined impact of
workflow adaptations to the validity of the review and
its results, with the needs of the knowledge user;
• Report methods and findings transparently, with
appropriate detail required to
– answer the research question;
– meet the requirements of the knowledge user and
allow assessment of confidence in the results;
– inform the audience for which the review is intended;
– while meeting delivery time line agreed upon in
advance.
Consensus (72 - 100%)

17. uOttawa.ca
Still up for debate…
• Tension: risk of bias/critical appraisal;
– individual study or review level
– distinct and opposing camps
• Achieved consensus on no consensus – RRs as a unique
method;
• Time is still relative…;
• Protocol format, publication/registration;
• Peer review;
• Unpublished (grey) literature;
• Not all rapid reviews meet this definition? Overlap with
some key features of SR, HTA.
Statements based only on the experience of the expert
panel…must be balanced with published literature.

18. uOttawa.ca
RAPID REVIEWS
Exploring attitudes and perceptions towards

19. uOttawa.ca
Method: Q methodology
A research
technique that
permits the
systematic study of
subjectivity
(opinions, beliefs
and preferences).
INTERPRETATION
FACTOR ANALYSIS
RANK ORDER OF THE STATEMENTS: Q SORT
DEVELOPMENT OF REPRESENTATIVE STATEMENTS: Q SET
DEFINING STUDY PARTICIPANTS: P SET
COLLECTION OF OPINION STATEMENTS: CONCOURSE
RESEARCH QUESTION
*Adapted from Amin, 2000

20. uOttawa.ca
Conceptual Q sort layout
We replicated this process using online Q methodology tools:

21. uOttawa.ca
• When time allows, a comprehensive systematic review of all available evidence should always be conducted.
• Further research comparing the methods and results of rapid reviews and systematic reviews is required before I
decide how I feel about rapid reviews.
• Rapid reviews mean different things to different people.
• Rapid reviews should only precede a more comprehensive and rigorous systematic review.
• The opportunity cost of a comprehensive SR or HTA is too high and it is more advantageous to conduct rapid reviews
when timeliness is a factor.
• All evidence synthesis products, including rapid reviews, SRs, or HTAs, can be conducted very well or very poorly.
• Rapid reviews are comparable to SRs except they are done in a more timely fashion.
• Rapid reviews are ‘quick and dirty' systematic reviews.
• Rapid reviews need to be tailored to the specific needs of the knowledge user.
• Rapid reviews meet the needs of knowledge users.
• It is always appropriate to conduct a rapid review.
• It is important to have minimum standards for the reporting of rapid reviews (e.g., a PRISMA-RR).
• Standardization of rapid review methods may conflict with the needs of knowledge users
• A good quality review of evidence is determined by the methods used, not by the speed at which it is completed.
• It is difficult to tell a rapid review from a systematic review unless very specific nomenclature is used in the title or
description of methods.
• A rapid review cannot be a systematic review.
Sample statements

22. uOttawa.ca
Summary
• 11 participants performed
the online Q sort
• 50 statements representing
a range of viewpoints and
and subjective opinion on RRs
• Following factor analysis
3 factors extracted
Geographic Location n (%)
Canada 11 (100)
Age: n (%)
18 to 35 2 (18.2)
36 to 50 4 (36.4)
50 or above 5 (45.5)
Sex: n (%)
Female 8 (72.7)
Education: n (%)
Doctorate 7 (63.6)
Masters 3 (27.3)
Undergraduate 1 (9.1)
Consider Themselves: n (%)
Researcher/Producer 8 (72.7)
Knowledge User 2 (18.2)
Neither 1 (9.1)
Have ever been the
author of a rapid review
n (%)
Yes 7 (63.6)
No 4 (36.4)
Unsure 0 (0)
Have used a rapid
review to aid in a policy
or decision-making?
n (%)
Yes 7 (63.6)
No 3 (27.3)
Unsure 1 (0.9)

23. uOttawa.ca
Title

24. uOttawa.ca
Title

25. uOttawa.ca
Factor 3. “Pragmatism over principle”

26. uOttawa.ca
Consensus Statements
Agree
6. Rapid reviews mean different things to different people.
Disagree
7. Rapid reviews should only precede a more
comprehensive and rigorous systematic review.
34. It is appropriate to endeavor to define a single, unique
methodology for rapid reviews.
42. Any review of evidence that takes longer than 3 months
to produce is not a rapid review.
43. Any review of evidence that takes longer than 1 month to
produce is not a rapid review.

27. uOttawa.ca
DEFINING RAPID REVIEWS
A pragmatic approach

28. uOttawa.ca
Summary of results
• Review of RRs published in journals or as reports by
various agencies and organizations worldwide;
– Supplemented with INAHTA scan to query HTA agencies.
• Screened 1,800+ database records and grey literature
results - Single reviewer, checking by a second;
• Included RR samples were examined for completion
time, nomenclature, study characteristics,
approach/tailoring and other relevant features, including
whether the results are contextualized or framed with
policy implications.

29. uOttawa.ca
In general
• Findings matched previous work on RR methods:
– “ One species, many breeds” Heterogeneity of approaches,
nomenclature – supports the evidence continuum concept;
– May find more than one approach within a single
organization (e.g., CADTH);
– Methods not well-reported; Self-identifying RRs with no
way to assess what was tailored.
• In addition:
– RRs are getting published;
– Need to focus on contextualization of results, policy
implications for end-user;
– Starting to see RR protocols published (n=4) or registered
in PROSPERO (n=3).

30. uOttawa.ca
Summary
• Addressed RR knowledge gaps with 3
separate but interrelated studies;
• Experts have identified defining
characteristics that may help
differentiate RRs from other evidence
synthesis approaches;
• Agree to disagree in some areas;
• Three distinct viewpoints on RRs were
ascertained during a Q methodology
study of subjective opinion;
• A selection of RR samples show that
there is room for improvement (without
compromising on the needs of end-
users).

31. uOttawa.ca
Questions? Thoughts?
A special thank you to Dr. Tammy Clifford and Dr. David Moher,
to CADTH for the initial funding for this student project, and
everyone who has participated in surveys, piloted questionnaires,
shared data and supported this research with their own time