Presentation by Nanor Minoyan and Myriam Cielo (Université de Montréal). Global Health Workshop: Methods For Implementation Science in Global Health. http://www.equitesante.org/implementation-science-methods-in-global-health/

1. Fidelity assessment in cluster randomised trials of public
health interventions in low- and middle-income
countries: systematic review
METHODS FOR IMPLEMENTATION SCIENCE
IN GLOBAL HEALTH
April, 20th. 2017
McGill University
Workshop co-organised by REALISME Chair and McGill of Global Health Programs
Nanor Minoyan & Cielo Perez

2. Context
• Cluster-randomised trials (CRTs) :
• Key tools to assess evidence of intervention effectiveness
• Problem: CRTs of complex public health interventions conducted in low- and
middle-income countries (LMICs) are prone to deficiencies in implementation1
• Multiple components, resource/logistical challenges
• Internal & external validity threatened
 Intervention fidelity (IF) assessment may be particularly useful
• = component of process evaluation that aims to understand and measure to what
extent interventions are implemented as planned
• CRT reporting guidelines: no recommendations for IF

3. Objective
• To study current practices concerning the assessment of
intervention fidelity in CRTs of public health interventions
conducted in LMICs
• Specifically:
– To quantify the proportion of studies that report IF assessment
– To compare reporting of IF assessment across trial registries / study
protocols and peer-reviewed publications of primary trial results
– To examine reported dimensions of IF and methods

4. Method: Systematic review
Participants + Settings Interventions Comparators Outcomes
LMICs Public health
interventions
As defined by CRT
design
Protocol and trial reporting
of IF assessment (agreement)
Databases: PubMed / MEDLINE, EMBASE, CINAHL, EMB Reviews
Filters: Jan 2012-May 2016; Eng-Fr-Spa; regist* in abstract
Inclusion criteria
• Method to evaluate ≥ 1 key fidelity component2: content, coverage, frequency, or duration, in:
a) Main trial report
OR
a) Protocol or trial registry
Data extraction:
• Whether / where IF assessment was reported + methods  mapped onto framework
• 1o outcomes
• Risk of bias data (Cochrane)
(key components + moderating factors3)

5. Results • 72.5% (66/91) report method for ≥1 fidelity dimension
 ~10% report all 4 key dimensions
 6% report all moderating factors
 Multiple factors often reported, variability in what/whether methods used
 Overlap in dimensions
• Best reporting: peer-reviewed protocol/process evaluation
• 56% : no mention of IF in control group
• Different reporting “intensity” = 1
6879
3501 Ti/Ab screen
227 full text review
91 IF assessment
25 66
IF IF ✓✗
Content Coverage Frequency Duration
~97% 48 – 62% 32-45% 19-23%
“All classes were delivered and teachers reported that 95% of content was
explained as scheduled.”4
“QA team reviewed intervention activities (…) according to standardised tools
that included direct observation of community mobilisation activities, direct
observations of household counselling sessions, register reviews, activity plan
reviews and knowledge testing of intervention team members.5

6. Lessons learned for
implementation
science in global
health
- Well-reported IF provides
rich information regardless of
study "result"
- IF most useful to reader
when clearly stated
- vs. inferring
- Robust design (CRT) does
not guarantee robustness
- IF ≈ Confounding
Strengths
• Systematic review best manner of reviewing current
practices + learning about method
• Strategy captured broad range of studies + practices
• Used an iterative process to reach understanding of IF
Limits
• Remaining subjectivity / ambiguity in what consitutes
IF, what constitutes a method
• e.g. modelling impact of intervention over time,
having recruitment criteria
• Unclear if adapted to new practices, better suited if
framework is well-established
• Agreement between protocol + main report TBD

7. To go further
1. Eldridge S, Ashby D, Bennett C, Wakelin M, Feder G. Internal and external validity of
cluster randomised trials: systematic review of recent trials. BMJ. 2008;336(7649):876–80.
2. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for
implementation fidelity. Implement Sci. 2007;2(1):40
3. Hasson H. Systematic evaluation of implementation fidelity of complex interventions in
health and social care. Implement Sci. 2010;5(1):67.
4. Andrade S, Lachat C, Ochoa-aviles A, et al. A school-based intervention improves physical
fitness in Ecuadorian adolescents: a cluster-randomized controlled trial. Int J Behav Nutr
Phys Act. 2014;11:153.
5. Ayles H, Muyoyeta M, Du toit E, et al. Effect of household and community interventions on
the burden of tuberculosis in southern Africa: the ZAMSTAR community-randomised trial.
Lancet. 2013;382(9899):1183-94.
Pérez MC, Minoyan N, Ridde V, Sylvestre MP, Johri M. Comparison of registered and published
intervention fidelity assessment in cluster randomised trials of public health interventions in low- and
middle-income countries: systematic review protocol. Syst Rev. 2016;5(1):177.

8. Let’s discuss together!
Meet the experts
Thank you for your attention

9. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40.
Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci. 2010;5(1):67.
Additional slides

11. • a-b) Intervention developed + implemented in close collaboration with natl. policy makers, experts and district health
management teams. Design workshop and pilot study done // detailed intervention, combining health facility
strengthening and CHW home visits (conceptual framework developed in collaboration with many actors, multiple
publications detailing ) // CHW trained for 5 days e) CHWs recruited by community w/ aim of identifying individuals
with attributes favouring success (e.g. empathy, respect by community
• b) In response to formative research, efforts made to design intervention to ensure that all health facilities in/around study
area were strenghthened
• Formative research: In-depth interviews w health managers/workers and community to understand perceptions of
quality/acceptability of MNH services, review of local verbal autopsy survey data to understand causes of newborn deaths,
results of formative research used in intervention design workshop with Ministry representatives, health teams, NGOs, UN
agencies, experts in neonatal health/BCC, + working with community volunteers - results presented to stakeholders
• d) Results of in-depth interviews with local council leaders, health workers, families identified that CHWs were highly
appreciated in the community, seen as important contributors to maternal and newborn health. CHWs reported being
highly motivated
• f) engagement of policymakers and decision leaders from village to national level. Several forms of engaging were used,
including meetings, brochures, news- paper publications, television interviews, documentary videos, publications,
conferences, field visits
Comparison was made between self-reported users of the intervention group with those of the control. While
householders exaggerated reported use, confirmed users nevertheless accounted for most (60%) of reported
users. Although reported users of the intervention group had significantly less contaminated water than those
of the placebo group, they did not have a lower prevalence of diarrhoea. This result speaks against low
compliance being the only explanation for the lack of impact.
Boisson et al. 2013
Boisson et al. 2015