This document provides an overview of systematic reviews, emphasizing the importance of developing clear review questions and utilizing explicit methodologies to ensure high-quality evidence. Key steps in the systematic review process include planning, searching, screening, and appraising research, alongside the application of various question frameworks such as PICO and SPICE. It encourages researchers to address quality issues in evidence by using structured approaches to formulate and answer relevant research questions.

1. Getting started with a
systematic review:
developing your
review question
Ruaraidh Hill PhD
Institute of Population Health

2. This session
ΠIntroduction Рissues with research evidence
 Reviews – overview of systematic reviews |
types of review in the evidence ecosystem
Ž Developing review questions
 Reviews – planning next steps
5

4. WeMap
LRIG – Health Data Science, IPH
Liverpool Reviews and Implementation Group (LRiG) – 20 years at the
core of evidence-based guidance development at NICE.
LRiG delivers a range of high-quality Health Technology Assessments and
capacity development initiatives
Capacity development – support for individuals and teams to use evidence
to develop practice, services and research.
@LivUniLRiG | liverpool.ac.uk/LRiG

5. 9
#&❤’d

6. Updated – new edition & resources
TinyURL.com/Doingareview2

7. Book – chapter format, samples
TinyURL.com/Doingareview2

8. Research – quantity at varying quality
12

9. Quality
What quality issues are relevant to your
discipline or area of practice or research?
13
think

10. Research – bad science?
14

11. Research – bad science?
15

13. Quality – selecting evidence
Instead of just mooching through
the research literature, consciously
or unconsciously picking out papers
here and there that support your
pre-existing beliefs, you take a
scientific, systematic approach to
the very process of looking for
scientific evidence, ensuring that
your evidence is as complete and
representative as possible of all the
research that has ever been done
Bad Pharma. Goldacre (2013)
Fourth Estate, London.
18
https://www.youtube.com/
watch?v=RKmxL8VYy0M

14. Quality – bringing evidence together
20
Predefined
method in
stages

15. Quality – key review activities
Identifying
relevant
research
(finding)
Describing
relevant
research
(describing)
Critically
appraising
research
reports in a
systematic
manner
(appraisal)
Bringing
together the
findings into
a coherent
statement
(synthesis)
Reviewing research systematically involves
key activities:
22

16. Quality – explicit method
1-2 Development – review question(s), protocol
3 Searching
4 Screening titles and abstracts
5 Obtaining full text papers
6 Selecting full text papers
7 Quality assessment
8 Data extraction
9 Analysis
10 Reporting
23

17. Question
Explicit
method
Findings
24
Quality –
Essentials
of SRs

18. Systematic review – in 3 numbers
3 | 4 | 10essentials activities steps

19. Review approaches – for different questions
• To present a conceptDescriptive
• Explore preferences about what should
happen
Normative
• Investigate a relationship between two
or more variables
Observational/relational
• Investigate the effect of one or more
independent variables on one or more
outcome variables
Causal
• Explore factors that cause a condition,
event or process
Theoretical

20. Review approaches – examples (1)
• What is the nature (volume, topic coverage, study
designs) of the literature on online postgraduate
training in English law (2000-to present)?
Descriptive
• What are the views and experiences of people
using a internet banking services?Normative
• Are adults with disabilities at an increased risk of
violence compared to those without disabilities?Observational/relational
• What are the effects of fluoridation of drinking
water supplies on the incidence of caries (tooth
decay)?
Causal
• What are the links between crime, fear of crime,
the environment and health and wellbeing? How
might these be explained?
Theoretical

22. This session
ΠIntroduction Рissues with research evidence
 Reviews – overview of systematic reviews |
types of review in the evidence ecosystem
Ž Developing review questions
 Reviews – planning next steps
31

23. Essentials of systematic review
Defined question -
Explicit method* -
Statement of findings -
*which selects relevant evidence while minimising bias
Question
Explicit
method
Findings

24. Question development
§ Essential first step in evidence
development
33

25. Asking the right question
https://www.azquotes.com/quote/526112

26. Questions about questions – for example:
Who Who is asking the question?
Whom About whom? Who is the question about?
What What is the choice to be made?
Where In what setting is the choice to made?
When
When is the choice to be made? Is the
question about a particular level of service
or stage?
Why Why is the question being asked?
36
DeborahCohenbit.ly/2EqULGF

27. Question perspectives – who’s asking?
Policy
What should commissioners of services
fund?
Practice
What should providers of services do
(assess, measure, offer, follow-up)?
Research
What is the (difference in) level, rate,
preferences, inequalities, effectiveness,
cost-effectiveness (between alternatives)?
Person
What are my options? What are the risks
and benefits? What should I accept?
37
Ulysses

28. Question framework – who, what, how
Who What How
38

29. Q framework – Who, what, how
Who
Who is the focus?
Specific health condition or age,
sex/gender, occupation
socioeconomic, ethic groups?
What
What is the intervention?
What is idea is being ‘tested’?
An exposure, hypothesis or
description of a phenomenon?
How
How does the ‘what’ affect ‘who’?
Effects, outcomes…
39

30. Q framework – WWH & MindMap
See also Table 2.3/ Doing a Review

31. Q framework – Who, what, how…
Who Doctors in acute settings
What
Face-to-face communication
methods/ techniques
How
Quality of doctor – patient
communication (knowledge,
attitudes, behaviours, health)
41

32. P I C O
Question framework - PICOSs
42

33. Question framework – PICOSs
Population -
Intervention -
Comparator -
Outcome -
Study design -
setting -
43

34. Formulated question – PICO
P
I
C
O
Children and adolescents
before dental treatment…
Pain relief medication
Pain…
satisfaction, cost, completion,
adverse effects
Placebo or no medication

35. PICOSs – example
Review objective
To assess the effects of preoperative analgesics for intraoperative or postoperative
pain relief (or both) in children and adolescents undergoing dental treatment without
general anaesthesia or sedation.
Participants/ population (setting)
Children and adolescents up to 17 years of age before dental treatment without
GA/sedation (international)
Intervention(s)
Analgesics [pain relief medications] given before dental treatment
Comparator
Placebo or
No analgesic
Outcomes
Postoperative pain
Intraoperative pain, Preoperative and postoperative anxiety measures, Patient
satisfaction, Parental satisfaction, Cost, Completion of treatment, Adverse events
Study design
Randomised controlled clinical trials
http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD008392.pub3/full

36. Developing research
Let’s have a go…!
Design a study to assess the effects of a
technology
For the following technologies think about:
§ Population
§ Intervention and comparator
§ Outcomes
§ Study design
46

37. Developing research – PICO exercise
Let’s have a go…!
Design a study to assess the effects of a
technology
For the following technologies think about:
§ Population
§ Intervention and comparator
§ Outcomes
§ Study design
47

38. Technology 1
§ Dramatically Different™
Moisturizing Lotion+
§ £30.00
§ Skin Types: 1, 2
§ The big plus: Our new formula
now helps strengthen skin's
own moisture barrier. More
moisture stays in. Skin feels
soft, springy. Has a healthy-
looking glow.
48

39. Technology 2
§ Clinique For Men Dark Spot
Corrector
§ £42.00
§ Benefits: Uneven Skin Tone
§ Skin Types: Works for all
skin types.
§ Clinique For Men Dark Spot
Corrector helps reduce the
appearance of
discolourations caused by
sun damage, blemish scars
and shaving to deliver
improvements in radiance
and smoothness.
49

40. BRIEF – Evidence Evaluation Unit
• Your are part of the prestigious, internationally respected University of
Liverpool Evidence Evaluation Unit (U-EEU)
• The Unit is publically funded to provide frank, expert advice to
government and to conduct internationally respected research
• Design a study to robustly explore the technology…
• Think
• P
• I/C
• O
• Sss – study design, source of evidence, setting
51
Even
numbered
breakout
rooms

41. BRIEF – Innovation Support Unit
• Your are part of an expert group of research methodologists;
technology developers and policy, communication and implementation
specialists – the Innovation Support Unit (ISU).
• The Unit is funded by an industry consortium to provide expert advice
to consortium members to ensure the public understand the benefits of
and have timely access to the consortium members’ innovative
technologies
• Design a study to help the public access the technology…
• Think
• P
• I/C
• O
• Sss – study design, source of evidence, setting
52
ODD
numbered
rooms

42. Developing research
How did that go…?
§ Who was asking the question?
§ What types of effects did you consider?
§ What types of designs or evidence?
§ Was PICO useful? What was challenging?
53

43. Quality – user perspectives
Skin Concern: Pores/ Skin Type: 1 - Very Dry/ Using Clinique
For: Less than 1 year
*****
I'm a recent convert to Clinique products, and I'm so glad I took the
plunge.
This moisturiser has really helped my very very dry skin, making
my make up look better once it's applied. A little really does go a
long way so I predict the bottle lasting me a while. My skin looks
so much more brighter and I've been having a lot more make up
free days as I feel much more confident about how my skin looks.
Yes, I would recommend this product to a friend.
[Emphasis added]
54

44. Formulated question - PICo
A clearly formulated question…
Co
P
I
Services that test for
Hepatitis C
People who inject drugs
Views, experiences
and attitudes of
hepatitis C testing and
diagnosis

45. Formulated question - SPICE
62
A clearly formulated question…
S
P
I
C
E
Services that test for
Hepatitis C
People who inject drugs
None
Hepatitis C testing and
diagnosis
Views, experiences
and attitudes

46. Question frameworks – overview
WWH:
Who | What | How
For early thinking
PICO:
Population | Intervention
Comparator | Outcome
For choices between/
comparing alternatives
(A versus B)
PICo:
Population
Interest | Context
For any question, including
experiences, descriptive
reviews or scoping
Others: SPICE,
PROGRESS+
For reviews of qualitative
studies or equalities (e.g.)
63

47. Question frameworks – overview
WWH:
Who | What | How
For early thinking
PICO:
Population | Intervention
Comparator | Outcome
For choices between/
comparing alternatives
(A versus B)
PICo:
Population
Interest | Context
For any question, including
experiences, descriptive
reviews or scoping
Others: SPICE,
PROGRESS+
For reviews of qualitative
studies or equalities (e.g.)
64

48. Question frameworks – overview
65
Watch from around 40 minutes for
PICo and SPICE frameworks
https://stream.liv.ac.uk/w93ffxxu

49. Research questions - key points (1)
A good research question is:
ü Clear
ü Well defined
ü Appropriate
ü Manageable
ü Relevant

50. Research questions - key points (2)
Question frameworks can help develop
a good question
Some notes:
ü Interventions can be a range of choices
ü Interest can be explored in a range of ways
ü Different frameworks can be used, to think about
§ the same research question
§ different research questions (or aspects of related Q)
§ different review approaches
§ different evidence
ü Use of frameworks supports transparency and
contestability à better quality

51. Evidence – consider…
What evidence informs your discipline or area
of practice or research?
What evidence quality issues are relevant to
your discipline or area of practice or research?
think

52. Question
Explicit
method
Findings
70
Essentials
of SRs

53. This session
ΠIntroduction Рissues with research evidence
 Reviews – overview of systematic reviews |
types of review in the evidence ecosystem
Ž Developing review questions
 Reviews – planning next steps
71

55. Systematic review – planning
3 | 4 | 10essentials activities steps

56. Systematic review – getting started
§ Plan your ‘10 steps’ – as required
§ Develop your question(s) and review approach(es)
§ Scope your questions; develop questions further
§ Meet the needs of your research question or
‘decision problem’
§ Refine and record in your research protocol
§ Consider registering on PROSPERO or other register

57. Systematic review – protocol standards

58. Systematic review – protocol processes
Harris et al 2017

59. Systematic review – question frameworks
§ Cochrane Qualitative and Implementation Methods Group
guidance series paper 2: methods for question formulation,
searching, and protocol development for qualitative evidence
synthesis (Harris et al 2017)
https://pubmed.ncbi.nlm.nih.gov/29248725/
§ Formulating the Evidence Based Practice Question: A Review of
the Frameworks (Davies 2011)
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3140151/
§ Developing NICE guidelines: the manual.(NICE 2014)
https://www.nice.org.uk/process/pmg20/chapter/developing-
review-questions-and-planning-the-evidence-review

60. Support, next steps
How will you use systematic review methods in
your discipline, area of practice or research?

61. Doing a review?
https://study.sagepub.com/doingasystematicreview2e

62. Doing a review?

63. Doing evidence?
https://training.cochrane.org/essentials

64. This session
ΠIntroduction Рissues with research evidence
 Reviews – overview of systematic reviews |
types of review in the evidence ecosystem
Ž Developing review questions
 Reviews – planning next steps
85

65. Additional links from the Q&A
Screen capture of a similar session (for Liverpool Doctoral College) on
developing review questions
https://stream.liv.ac.uk/w93ffxxu
Open science/ protocol registration options – please check suitability/ remit
of each before choosing to register:
https://www.crd.york.ac.uk/prospero/
https://f1000research.com/
https://osf.io/
Researcher Know How (2020):
https://libcal.liverpool.ac.uk/event/3595466
https://libcal.liverpool.ac.uk/event/3595489
https://libcal.liverpool.ac.uk/event/3595494

66. Ruaraidh Hill
Lecturer in
evidence synthesis
Institute of Population
Health
§ TinyURL.com /IPHSrahill
§ LinkedIn: /RuaraidhHill
§ Twitter: @RuaraidhHill
§ Skype/Snap: RuaraidhlHill