This document outlines a session on developing review questions for systematic reviews, emphasizing the importance of a structured question formulation process. It introduces various frameworks (such as PICO and SPICE) to assist researchers in crafting effective research questions and discusses the essential steps for conducting a systematic review. Additionally, it addresses issues related to evidence quality and provides resources for further learning and support.

1. Getting started with a
systematic review:
developing your
review question
Session 1/3
#systematicreview
@LivUniKnowHow | @LivUniLRiG

2. Ruaraidh Hill PhD MSc FHEA
Lecturer in evidence synthesis
Michelle Maden PhD MA FHEA
Postdoc research associate in evidence synthesis
linktr.ee/RuaraidhHill

3. Your comfort & safety
 Slides ✓
 Recording ✓
 Chat ✓
 Practical activity on screen
 Evaluation see link in chat & follow-up e-mail
 References see final slides

4. This session
 Introduction – issues with research evidence
 Reviews – overview of systematic reviews | types of
review in the evidence ecosystem
 Developing review questions
 Reviews – planning next steps
5

5. issues with evidence
◉◎◎◎

6. WeMap
LRiG – Health Data Science, IPH
Liverpool Reviews and Implementation Group (LRiG) – 20+ years
at the core of evidence-based guidance development at NICE.
LRiG delivers a range of high-quality Health Technology
Assessments and capacity development initiatives.
Capacity development – support for individuals and teams to use
evidence to develop practice, services and research.
@LivUniLRiG | liverpool.ac.uk/LRiG

7. 9
#&❤’d

8. Book – updated 2nd edition & resources
TinyURL.com/DoingaReview2

9. Book – chapter format
TinyURL.com/DoingaReview2

10. Research – quantity at varying quality
12

11. Quality
What quality issues are relevant to your discipline
or area of practice or research?
13
think

12. Research – bad science?
14

13. Research – bad science?
15

14. Research – current challenges
Getty Images | https://www.bbc.co.uk/news/uk-politics-55106567

15. Research – current challenges
pandemic
syndemic
infodemic

16. overview
◎◎◎◎

17. Quality – selecting evidence
Instead of just mooching through the
research literature, consciously or
unconsciously picking out papers here and
there that support your pre-existing beliefs,
you take a scientific, systematic approach
to the very process of looking for
scientific evidence, ensuring that your
evidence is as complete and representative
as possible of all the research that has ever
been done
Bad Pharma. Goldacre (2013) Fourth
Estate, London.
19
https://www.youtube.com/
watch?v=RKmxL8VYy0M

18. Quality – bringing evidence together
21
Predefined
method in
stages

19. Identifying
relevant
research
(finding)
Describing
relevant
research
(describing)
Critically
appraising the
research
(appraisal)
Bringing
together the
findings into a
coherent
statement
(synthesis)
Quality – key review activities
Reviewing research systematically involves
key activities:
23

20. Quality – explicit method in 10 steps
24
1-2 Development – review question(s), protocol
3 Searching
4 Screening titles and abstracts
5 Obtaining full text papers
6 Selecting full text papers
7 Quality assessment
8 Data extraction
9 Analysis
10 Reporting
Predefined
method in
stages

21. Systematic review – our 10-step road
map

22. 26
Quality – essentials of SRs
Question
Explicit
method
Findings

23. Systematic review – in 3 numbers
3 | 4 | 10
essentials activities steps

24. Review approaches – different questions
• To present a concept
Descriptive
• Explore preferences about what should
happen
Normative
• Investigate a relationship between two or
more variables
Observational/relational
• Investigate the effect of one or more independent
variables on one or more outcome variables
Causal
• Explore factors that cause a condition,
event or process
Theoretical

25. Review approaches – examples (1)
• What is the nature (volume, topic coverage, study
designs) of the literature on online postgraduate training
in English law (2000-to present)?
Descriptive
• What are the views and experiences of people
using a internet banking services?
Normative
• Are adults with disabilities at an increased risk of
violence compared to those without disabilities?
Observational/relational
• What are the effects of fluoridation of drinking water
supplies on the incidence of caries (tooth decay)?
Causal
• What are the links between crime, fear of crime, the
environment and health and wellbeing? How might these
be explained?
Theoretical

26. Stella Talic et al. BMJ 2021;375:bmj-2021-068302
©2021 by British Medical Journal Publishing Group

27. review questions
◎◎◉◎

28. This session
 Introduction – issues with research evidence
 Reviews – overview of systematic reviews | types of
review in the evidence ecosystem
 Developing review questions
 Reviews – planning next steps
33

29. Question
Explicit
method
Findings
Essentials of systematic review
Defined question -
Explicit method* -
Statement of findings -
*which selects relevant evidence while minimising bias

30. Question development
 Essential first step in evidence development
35

31. Asking the right question
“To ask the right question is already half the solution of a problem” Carl
Jung, 1959
https://www.azquotes.com/quote/879797

32. Asking the right question
https://www.azquotes.com/quote/526112

33. Questions about questions – for example:
Who Who is asking the question?
Whom About whom? Who is the question about?
What What is the choice to be made?
Where In what setting is the choice to made?
When
When is the choice to be made? Is the question
about a particular level of service or stage?
Why Why is the question being asked?
38
Deborah
Cohen
bit.ly/2EqULGF

34. Question perspectives – who’s asking?
Policy What should commissioners of services fund?
Practice
What should providers of services do (assess,
measure, offer, follow-up)?
Research
What is the (difference in) level, rate,
preferences, inequalities, effectiveness, cost-
effectiveness (between alternatives)?
Person
What are my options? What are the risks and
benefits? What should I accept?
39
Ulysses

35. Question framework – who, what, how
40
Who What How

36. Q framework – Who, what, how
Who
Who is the focus?
Specific health condition or age,
sex/gender, occupation socioeconomic,
ethic groups?
What
What is the intervention?
What is idea is being ‘tested’?
An exposure, hypothesis or description of a
phenomenon?
How
How does the ‘what’ affect ‘who’?
Effects, outcomes…
41

37. Q framework – WWH & MindMap
See also Table 2.3/ Doing a Review 2nd Ed

38. Q framework – Who, what, how…
Who Doctors in acute settings
What Face-to-face communication methods/ techniques
How
Quality of doctor – patient communication
(knowledge, attitudes, behaviours, health)
43

39. P I C O
Question framework - PICOSs
44

40. Question framework – PICO-Sst…
Population -
Intervention -
Comparator -
Outcome -
Study design -
Setting, timing -
45

41. Formulated question – PICO
P
I
C
O
Children and adolescents before
dental treatment…
Pain relief medication
Pain…
satisfaction, cost, completion,
adverse effects
Placebo or no medication

42. PICOSs – example
Review objective
To assess the effects of preoperative analgesics for intraoperative or postoperative pain relief (or both) in children
and adolescents undergoing dental treatment without general anaesthesia or sedation.
Participants/ population (setting)
Children and adolescents up to 17 years of age before dental treatment without GA/sedation (international)
Intervention(s)
Analgesics [pain relief medications] given before dental treatment
Comparator
Placebo or
No analgesic
Outcomes
Postoperative pain
•Intraoperative pain, Preoperative and postoperative anxiety measures, Patient satisfaction, Parental satisfaction,
Cost, Completion of treatment, Adverse events
Study design
Randomised controlled clinical trials http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD008392.pub3/full

43. Question framework – PEO
P ee O
48

44. Question framework – PEO
Population
Exposure
The exposure relates to a defined event, activity,
experience or process that groups are exposed
to (and others are not)
Outcome
49

45. Question frameworks – check our learning (1)
 For the following systematic review description:
 Choose a framework
 Identify
 Population, intervention , comparator (if applicable), outcome(s)*
 Think about
 Settings, timing or study design to include
 Review approach

46. Question frameworks – check our learning
Effectiveness of public
health measures in
reducing the incidence of
covid-19, SARS-CoV-2
transmission, and covid-19
mortality: systematic
review and meta-analysis
https://doi.org/10.1136/bmj-2021-068302

47. Question frameworks – check our learning
Effectiveness of public health
measures in reducing the incidence of
covid-19, SARS-CoV-2 transmission,
and covid-19 mortality
 Population?
 Comparator?
 Study designs?
https://doi.org/10.1136/bmj-2021-068302

48. Stella Talic et al. BMJ 2021;375:bmj-2021-068302
©2021 by British Medical Journal Publishing Group

49. Question frameworks – check our learning (2)
 For the following systematic review description:
 Choose a framework
 Identify
 Population, intervention or exposure, comparator (if applicable), outcome(s)*
 Think about
 Settings, timing or study design to include
 Review approach

50. Question framework – PICo/PIC
P I Co
59

51. Question framework – PICo
Population
What are the characteristics of the person,
patient, group or population?
Interest
The phenomena of interest relates to a defined
event, activity, experience or process
Context
Context is the setting or other distinct
characteristics
60

52. Formulated question - PICo
Co
P
I
Services that test for
Hepatitis C
People who inject drugs
Views, experiences
and attitudes of
hepatitis C testing and
diagnosis

53. Question framework - SPICE
sP I C E
62

54. Question framework – SPICE
Setting -
Perspective -
Intervention -
Comparison -
Evaluation -
-
63

55. Formulated question - SPICE
64
S
P
I
C
E
Services that test for
Hepatitis C
People who inject drugs
None
Hepatitis C testing and
diagnosis
Views, experiences
and attitudes

56. Question frameworks – check our learning (4)
 For the following systematic review description:
 Choose a framework
 Identify
 Population; interest, intervention or exposure (as applicable); outcome(s)*
 Think about
 Settings, timing or study design to include
 Review approach

57. Question frameworks – check our learning
Barriers and facilitators to the uptake
of tuberculosis diagnostic and
treatment services by hard-to-reach
populations in countries of low and
medium tuberculosis incidence: a
systematic review of qualitative
literature

58. Question frameworks – check our learning
Barriers and facilitators to the uptake
of tuberculosis diagnostic and
treatment services by hard-to-reach
populations in countries of low and
medium tuberculosis incidence
 SPICE
 Study designs?

59. Question frameworks – overview
WWH:
Who | What | How
For early thinking
PICO:
Population | Intervention
Comparator | Outcome
For choices between/ comparing
alternatives
(A versus B)
PICo:
Population
Interest | Context
For any question, including
experiences, descriptive reviews
or scoping
Others: SPICE, PROGRESS+
For reviews of qualitative studies
or equalities (e.g.)
69

60. Question frameworks – overview
WWH:
Who | What | How
For early thinking
PICO:
Population | Intervention
Comparator | Outcome
For choices between/ comparing
alternatives
(A versus B)
PICo:
Population
Interest | Context
For any question, including
experiences, descriptive reviews
or scoping
Others: SPICE, PROGRESS+
For reviews of qualitative studies
or equalities (e.g.)
70

61. Question frameworks – overview
71
• Watch from around 40 minutes for
• PICo and SPICE frameworks
• https://stream.liv.ac.uk/w93ffxxu

62. Research questions - key points (1)
• A good research question is:
 Clear
 Well defined
 Appropriate
 Manageable
 Relevant

63. Research questions - key points (2)
Question frameworks can help develop a good review question
Some notes:
 Interventions can be a range of choices
 Interest can be explored in a range of ways
 Different frameworks can be used, to think about
 the same research question
 different research questions (or aspects of related Q)
 different review approaches
 different evidence
 Use of frameworks supports transparency and contestability
 better quality

64. Evidence – consider…
What evidence informs your discipline or area of practice or
research?
What evidence quality issues are relevant to your discipline or
area of practice or research?
think

65. Question
Explicit
method
Findings
76
Essentials
of SRs

66. This session
 Introduction – issues with research evidence
 Reviews – overview of systematic reviews | types of
review in the evidence ecosystem
 Developing review questions
 Reviews – planning next steps
77

67. next steps
◎◎◎◉

68. Systematic review – in 3 numbers
3 | 4 | 10
essentials activities steps

69. Systematic review – getting started
 Plan your ‘10 steps’ – as required
 Develop your question(s) and review approach(es)
 Scope your questions  develop questions further
 Check your review meets the needs of your research
question or ‘decision problem’
 Refine and record in your review protocol
 Consider registering on PROSPERO or other register

70. Systematic review – protocol standards

71. Systematic review – protocol processes
Harris et al 2017

72. Systematic review – question frameworks
 Cochrane Qualitative and Implementation Methods Group guidance series paper 2: methods
for question formulation, searching, and protocol development for qualitative evidence
synthesis (Harris et al 2017)
https://pubmed.ncbi.nlm.nih.gov/29248725/
 Formulating the Evidence Based Practice Question: A Review of the Frameworks (Davies
2011)
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3140151/
 Developing NICE guidelines: the manual.(NICE 2014)
https://www.nice.org.uk/process/pmg20/chapter/developing-review-questions-and-planning-
the-evidence-review
 Our Researcher Know How series resources/tools
TinyURL.com/KnowHowSRtools

73. Support, next steps
How will you use systematic review methods in your discipline,
area of practice or research?

74. UoL library class no:
R853.S94.B68
tinyURL.com/DoingaReview2
Sample chapters, FAQ, videos available online
86
 Watch
 Explore
 Read
Doing a review?

75. Doing a review?
https://study.sagepub.com/doingasystematicreview2e

76. Doing a review?

77. Doing evidence?
https://training.cochrane.org/essentials

78. This session
 Introduction – issues with research evidence
 Reviews – overview of systematic reviews | types of
review in the evidence ecosystem
 Developing review questions
 Reviews – planning next steps
91

79. Additional links from previous Q&A
Open science/ protocol registration options – please check suitability/
remit of each before choosing to register:
• https://www.crd.york.ac.uk/prospero/
• https://f1000research.com/
• https://osf.io/
Researcher Know How HomePage:
• https://libguides.liverpool.ac.uk/researcherknowhow

80. Getting started with a
systematic review:
developing your
review question
Session 1/3
#systematicreview
@LivUniKnowHow | @LivUniLRiG
www.liverpool.ac.uk/LRiG