SLR

1. Systematic reviews of
health promotion and
public health interventions
Rebecca Armstrong
Elizabeth Waters
Cochrane Health Promotion & Public Health Field

2. Overview
 Overview of systematic reviews
 Outline of The Cochrane Collaboration
 Role of the HPPH Field
 Function of systematic reviews in informing
policy and practice
 Key elements of systematic reviews
Asking answerable questions
Searching for evidence
Assessing quality
Synthesising results
Applicability and transferability

3. Meta-analysis
Systematic reviews
Reviews
(narrative/literature/
traditional)
Types of reviews

4. Narrative reviews
Usually written by experts in the field
Use informal and subjective methods to
collect and interpret information
Usually narrative summaries of the
evidence
Read: Klassen et al. Guides for Reading and Interpreting Systematic
Reviews. Arch Pediatr Adolesc Med 1998;152:700-704.

5. What is a systematic review?
A review of the evidence on a clearly
formulated question that uses systematic
and explicit methods to identify, select
and critically appraise relevant primary
research, and to extract and analyse data
from the studies that are included in the
review*
*Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or
Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination,
University of York. March 2001.

6. Key elements of a systematic review
Structured, systematic process involving
several steps :
1. Formulate the question
2. Plan the review
3. Comprehensive search
4. Unbiased selection and abstraction process
5. Critical appraisal of data
6. Synthesis of data (may include meta-analysis)
7. Interpretation of results
All steps described explicitly in the review

7. Systematic vs. Narrative reviews
 Scientific approach to a
review article
 Criteria determined at
outset
 Comprehensive search
for relevant articles
 Explicit methods of
appraisal and synthesis
 Meta-analysis may be
used to combine data
 Depend on authors’
inclination (bias)
 Author gets to pick any
criteria
 Search any databases
 Methods not usually
specified
 Vote count or narrative
summary
 Can’t replicate review

8. Advantages of systematic reviews
Reduce bias
Replicability
Resolve controversy between conflicting
studies
Identify gaps in current research
Provide reliable basis for decision making

9. Limitations of systematic reviews
specific to health promotion
Results may still be inconclusive
There may be no trials/evidence
The trials may be of poor quality
The intervention may be too complex to be
tested by a trial
Practice does not change just because
you have the evidence of
effect/effectiveness

10. Consider these interventions…
Interventions to promote smoke alarm ownership
and function
School-based driver education for the prevention
of traffic crashes
Helmets for preventing head and facial injuries in
bicyclists
Do you think the results identified in SRs will be
good, promising or absent (and potentially
harmful)?

11. Results from systematic reviews
 Helmets reduce bicycle-related head and facial
injuries for bicyclists of all ages involved in all types of
crashes including those involving motor vehicles.
 The results provide no evidence that drive education
reduces road crash involvement, and suggest that it
may lead to a modest but potentially important increase
in the proportion of teenagers involved in traffic crashes.
 Results from this review suggest that area-wide traffic
calming in towns and cities may be a promising
intervention for reducing the number of road traffic
injuries and deaths. However, further rigorous
evaluations of this intervention are needed.

12. The Cochrane Collaboration
International non-profit
organisation that
prepares, maintains, and
disseminates systematic
up-to-date reviews of
health care interventions

13. Cochrane Collaboration
Named in honour of Archie Cochrane, a British
researcher
In 1979:
“It is surely a great criticism
of our profession that we
have not organised a critical
summary, by specialty or
subspecialty, adapted
periodically, of all relevant
randomised controlled trials”

14. The Cochrane Library
 Cochrane Systematic reviews : Cochrane reviews and
protocols
 Database of Reviews of Effects: Other systematic reviews
appraised by the Centre for Reviews and Dissemination.
 Cochrane Central Register of Controlled Trials:
Bibliography of controlled trials (some not indexed in MEDLINE).
 Health Technology Assessment Database: HTA reports
 NHS Economic evaluation database:
Economic evaluations of health care interventions.

15. The Cochrane Library
www.thecochranelibrary.com

17. Cochrane HPPH Field
Represent the needs and interests of
those in health promotion and public
health in Cochrane matters
Represent Cochrane in health promotion
and public health forums
In transition from Field to Review Group
Will edit PH reviews for the Cochrane Library

18. Cochrane Collaboration Structure
Steering Group
Review Groups
Centres Fields
Methods Groups
Consumer Network

19. Collaborative Review Groups (50)
 Produce systematic reviews relevant to a
particular disease or health issue for inclusion in
the Cochrane Library
 Examples
Airways Group
Drug and Alcohol Group
Heart Group
Injuries Group
Skin Group
Pregnancy and Childbirth Group
Stroke Group
Breast Cancer Group

20. Methods Groups (12)
Provide advice and support in the
development of the methods of systematic
reviews
Examples
Non-Randomised Studies
Screening and Diagnostic Tests
Empirical Methodological Studies
Qualitative Methods

21. Cochrane Centres (14)
Work to assist all Cochrane entities within
a specific geographical area
Examples
Australasian Cochrane Centre (at Monash)
South African Cochrane Centre
Italian Cochrane Centre
Chinese Cochrane Centre

22. Cochrane Fields/Networks (9)
Represent an area of interest which spans
a number of health problems - and hence
a number of Review Groups
Examples
Health Promotion and Public Health Field
Primary Health Care Field
Cancer Network
Child Health Field

23. Cochrane HPPH Field
Cochrane Fields represent a population
group, or type of care that overlaps
multiple Review Group areas
HPPH Field
Registered in 1996
Administered from Melbourne
Funded by VicHealth
Over 400 members on contact database across
>30 countries

24. Staff
Elizabeth Waters (Director)
Jodie Doyle (Coordinator)
Rebecca Armstrong (Senior Research Fellow)
Naomi Priest (Research Fellow)

25. Asking an answerable
question

26. Questions of interest
Effectiveness:
 Does the intervention work/not work?
 Who does it work/not work for?
Other important questions:
 How does the intervention work?
 Is the intervention appropriate?
 Is the intervention feasible?
 Is the intervention and comparison relevant?

27. Answerable questions
EFFECTIVENESS
A description of the populations P
An identified intervention I
An explicit comparison C
Relevant outcomes O

28. A PICO question
Time-consuming question:
What is the best strategy to prevent smoking
in young people?

29. An answerable question
Q. Are mass media (or school-based or
community-based) interventions effective in
preventing smoking in young people?

30. Problem,
population
Intervention Comparison Outcome Types of
studies
Young people
under 25 years
of age
a) Television
b) Radio
c) Newspapers
d) Bill boards
e) Posters
f) Leaflets
g) Booklets
a) School-based
interventions
b) No
intervention
a) objective
measures of
smoking (saliva
thiocyanate
levels, alveolar
CO)
b) self-reported
smoking
behaviour
c) Intermediate
measures
(intentions,
attitude,
knowledge,
skills)
d) Media reach
a) RCT
b) Controlled
before and after
studies
c) Time series
designs
The PICO(T) chart

32. Finding the evidence

33. Systematic review process
1. Well formulated question
2. Comprehensive data search
3. Unbiased selection and abstraction process
4. Critical appraisal of data
5. Synthesis of data
6. Interpretation of results

34. A good search
 Clear research question
 Comprehensive search
All domains, no language restriction,
unpublished and published literature, up-to-date
 Document the search (replicability)

35. Components of electronic
searching
1. Describe each PICO component
2. Start with primary concept
3. Find synonyms
a) Identify MeSH / descriptors / subject headings
b) Add textwords
4. Add other components of PICO question to
narrow citations (may use study filter)
5. Examine abstracts
6. Use search strategy in other databases
(may need adapting)

36. So you want to do a ‘quick & dirty’?
 DARE
 CENTRAL
 PubMed (clinical queries, related records)
 CDC
 NICE
 Organisations who do work in your area
 …google

37. The Cochrane Library:
www.thecochranelibrary.com

38. Cochrane HPPH Field

39. Health-evidence.ca

40. The Guide to Community Preventive
Services
http://www.thecommunityguide.org/

41. National Institute for Health and
Clinical Excellence
http://www.publichealth.nice.org.uk/page.aspx?o=home

42. Evidence for Policy and Practice Information and Co-
ordinating Centre (EPPI-Centre)
http://eppi.ioe.ac.uk

43. Effective Public Health Practice
Project (EPHPP)
http://www.myhamilton.ca/myhamilton/CityandGo
vernment/HealthandSocialServices/Research/E
PHPP/

44. Centre for Reviews and
Dissemination
http://www.york.ac.uk/inst/crd

45. Other sources of primary research

46. Searching
 www.google.com.au
 The order of terms will effect the results so
start with the obvious or key concept
 No need for ‘and’
 Google will ignore common words
If they are important use + (e.g. policy + 3)
 Phrase searching is useful eg “suicide
prevention”
 Google searches for variations on words eg
diet, dietary

47. Searching
 Where terms have multiple meanings you
can direct google to remove sites you want to
avoid (e.g. bass –music)
 Keep your search strings brief
Mental health promotion initiatives to prevent suicide
in young people
 Compartmentalise your search strings
“mental health promotion” suicide
“Suicide prevention” “young people”
Prevention and suicide and youth

48. Searching
 http://www.ncbi.nlm.nih.gov/entrez/query.fcgi
 Use the same principles for google – keep it
short and sweet.
 Key features
Journals Database
MeSH Database
Single Citation Matcher
Clinical Queries

49. Searching

50. Searching

51. Searching
Select the
Limits tab –
just under the
search string

52. Searching

53. These next few slides show you how to
search MeSH terms in PubMed. Useful if
you don’t have access to electronic
databases. You combine this method with
the one for text words outlined above. The
process for combining text words and
MeSH terms is outlines below.

57. Example
Mass media interventions to prevent
smoking in young people
P= Young people
STEP ONE:
Find MeSH and textwords to
describe young people

58. Example
Mass media interventions to prevent
smoking in young people
P= Young people
MeSH: Adolescent
Child
Minors

59. Example
Mass media interventions to prevent smoking in
young people
P= Young people
Textwords:
Adolescent Girl
Child Boy
Juvenile Teenager
Young people Young adult
Student Youth

60. Textwords
Truncation $:
To pick up various forms of a word
Teen$.tw Smok$.tw
Teenage Smoke
Teenager Smoking
Teenagers Smokes
Teens Smoker
Teen Smokers

61. Textwords
Wild cards ? and #:
To pick up different spellings
Colo?r.tw (? Can be substituted for one or
no characters)
Colour
Color
Wom#n.tw (# Substitutes for one
character)
Woman
Women

62. Textwords
Adjacent ADJn:
 retrieves two or more query terms within n
words of each other, and in any order
 Great when you are not sure of phraseology
Eg sport adj1 policy
Sport policy
Policy for sport
Eg mental adj2 health
Mental health
Mental and physical health

63. Example continued
Mass media interventions to prevent
smoking in young people
I = Mass media interventions
STEP TWO:
Find MeSH and textwords to
describe mass media interventions

64. Example continued
MeSH
Mass media
Audiovisual aids
Television
Motion pictures
Radio
Telecommunications
Newspapers
Videotape recording
Advertising

65. Example continued
Mass media interventions to prevent
smoking in young people
O = Prevention of smoking
STEP THREE:
Find MeSH and textwords to
describe prevention of smoking

66. Example of search
P = YOUNG PEOPLE
MeSH Textwords
………………………. ……………………….
………………………. ……………………….
………………………. ……………………….
I = MASS MEDIA
MeSH Textwords
………………………. ……………………….
………………………. ……………………….
………………………. ……………………….
C = (if required)
O = PREVENTION OF SMOKING
MeSH Textwords
………………………. ……………………….
………………………. ……………………….
………………………. ……………………….
OR
OR
OR
P
AND
I
AND
C
AND
O

67. Different bibliographic databases
Databases use different types of
controlled vocabulary
Same citations indexed differently on
different databases
Medline and EMBASE use a different
indexing system for study type
PsycINFO and ERIC do not have specific
terms to identify study types
Need to develop search strategy for each
database

68. Study design filters
 RCTs
 See Cochrane Reviewer’s Handbook
 Non-RCTs
 Not yet developed, research in progress
 Qualitative research
 Specific subject headings used in CINAHL, ‘qualitative research’
used in Medline
 CINAHL Filter: Edward Miner Library
http://www.urmc.rochester.edu/hslt/miner/digital_library/tip_sheets/
Cinahl_eb_filters.pdf
 Systematic reviews/meta-analyses
 CINAHL: as above
 Medline
http://www.urmc.rochester.edu/hslt/miner/digital_library/tip_sheets/
OVID_eb_filters.pdf
 Medline and Embase
http://www.sign.ac.uk/methodology/filters.html
 PubMed

69. 2. Unpublished literature
 Not all known published trials are identifiable
in Medline (depending on topic)
 Only 25% of all medical journals in Medline
 Non-English language articles are under-
represented in Medline (and developing
countries)
 Publication bias – tendency for investigators
to submit manuscripts and of editors to
accept them, based on strength and direction
of results (Olsen 2001)

70. 2. Unpublished literature
Hand searching of key journals and
conference proceedings
Scanning bibliographies/reference lists
of primary studies and reviews
Contacting individuals/agencies/
academic institutions
Neglecting certain sources may result in
reviews being biased

71. Librarians are your friends!

72. Principles of critical
appraisal

73. Critical appraisal
The process of systematically
examining research evidence to
assess its validity, results and
relevance before using it to
inform a decision.
Alison Hill, Critical Appraisal Skills Programme, Institute of Health
Sciences, Oxford http://www.evidence-based-medicine.co.uk

74. Critical appraisal I:
Quantitative studies

75. Why appraise validity?
 Not all published and unpublished
literature is of satisfactory methodological
rigour
Just because it is in a journal does not mean it is
sound!
Onus is on you to assess validity!
 Quality may be used as an explanation for
differences in study results
 Guide the interpretation of findings and
aid in determining the strength of
inferences

76. Bias – quality assessment tool
1. Selection bias
2. Allocation bias
3. Confounding
4. Blinding (detection bias)
5. Data collection methods
6. Withdrawals and drop-outs
7. Statistical analysis
8. Intervention integrity

77. Recruit participants
Allocation
Exposed to
intervention
Not exposed to
intervention
Follow-up
Follow-up
Outcome Outcome
Allocation of
concealment
Blinding of outcome
assessment
Intention-to-treat
Withdrawals
Data collection
methods
Selection bias
Integrity of
intervention
Intervention Control
Analysis Analysis
Statistical analysis
Confounding

78. Critical appraisal tools
RCTs
The Quality Assessment Tool for Quantitative
Studies
(http://www.city.hamilton.on.ca/PHCS/EPHPP/).
Non-RCTs
Cochrane Effective Practice and Organisation
of Care Group (http://www.epoc.uottawa.ca/).
The Berkeley Systematic Reviews Group
(http://www.medepi.net/meta/)

79. Critical appraisal II:
Qualitative studies

80. Qualitative research
… explores the subjective world. It
attempts to understand why people
behave the way they do and what
meaning experiences have for people.
Qualitative studies of experience
Process evaluation
Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying
Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and
Dissemination, University of York. March 2001.

81. CASP appraisal checklist
1. Clear aims of research (goals, why it is
important, relevance)
2. Appropriate methodology (what, how, why)
3. Sampling strategy
4. Data collection
5. Relationship between researcher and
participants
6. Ethical issues
7. Data analysis
8. Findings
9. Value of research (context dependent)

82. Other qualitative checklist
Quality framework
Government Chief Social Researcher’s Office,
UK
http://www.strategy.gov.uk/files/pdf/Quality_framewo
rk.pdf
19 question checklist for process
evaluations (EPPI-Centre)

83. Appraisal of a systematic review
 10 questions
1. Clearly-focused question
2. The right type of study included
3. Identifying all relevant studies
4. Assessment of quality of studies
5. Reasonable to combine studies
6. What were the results
7. Preciseness of results
8. Application of results to local population
9. Consideration of all outcomes
10. Policy or practice change as a result of evidence
CASP

84. Interpretation of results

85. Factors influencing effectiveness
Type III error (integrity of intervention)
Theoretical framework of intervention
Context
For whom did the intervention work,
why, in what circumstances, at what
cost

86. Difficulties addressing inequalities
Studies rarely present information on
differential effects of interventions
Cannot locate studies addressing
inequalities
May need original data from authors
Low power to detect subgroup differences

87. Integration of evidence into
practice

88. Assessing the applicability and
transferability of interventions
Applicability – whether the intervention
process could be implemented in the local
setting, no matter what the outcome is.
Is it possible to run this intervention in this local
setting?
Eg. provision of condoms in area where they
are not acceptable for religious reasons
Wang et al 2005

89. Assessing the applicability and
transferability of interventions
Transferability – if the intervention were to
be implemented in the local setting, would
the effectiveness of the program be similar
to the level detected in the study setting?
E.g. if the interventionists lack experience and
have few skills in delivering the intervention
then its effectiveness in the local setting may be
lower than that demonstrated in the study
setting
Wang et al 2005

90. Review of questions proposed by Wang
et al

91. Contact details
Rebecca Armstrong
rarmstrong@vichealth.vic.gov.au
03 9667 1336
If I can’t help you I might be able to point you in
the right direction.
If you are interested in training and support for
conducting SRs or increasing uptake within
your organisation, region, state please let me
know