This document summarizes a research study on using systems-based evaluation (SBE) to assess Zambia's antiretroviral therapy (ART) scale-up strategy. The study used system dynamics modeling to evaluate relationships among ART program elements. Key findings include that the model triggered discussion of system boundaries and interrelationships, but did not lead to transformative action. For SBE to be most effective, stakeholders need opportunities to engage in boundary-setting discussions during both model development and after completion. Determining the system's boundaries is a methodological requirement that can have transformative impact through facilitated discussion.

1. I M P A C T , L E A R N I N G A N D I N N O V A T I O N : T O W A R D S A
R E S E A R C H A N D P R A C T I C E A G E N D A F O R T H E F U T U R E
I N S T I T U T E F O R D E V E L O P M E N T S T U D I E S , B R I G H T O N ( U K )
M A R C H 2 6 , 2 0 1 3
JOHN T. GROVE, M.A., PH.D .
B I L L A N D M E L I N D A G A T E S F D N . *
* C U R R E N T A F F I L I A T I O N . T H I S W O R K W A S C O N D U C T E D B E T W E E N 2 0 0 6 A N D 2 0 1 1 W H E N J G W A S
P A R T O F T H E C E N T E R S F O R D I S E A S E C O N T R O L & P R E V E N T I O N A N D A S A N I N D E P E N D E N T
R E S E A R C H D E G R E E
M O D E L I N G W A S C O N D U C T E D B Y D R . J A C K H O M E R
Aiming for utility in ‘systems-based
evaluation’:
A research-based framework for practitioners

2. Outline
 Defining SBE
 Research Objectives, Methods, Stance
 Key Features and Utility of SBE
 The Zambia Case: Strategic Evaluation with SD
 Preparatory (Pre-Model) Phase Findings
 SD Model Highlights & Reflections
 Post-Model Findings
 Recommendations for Practice

3. What Is Systems-Based Evaluation?
 Systems-Based Evaluation (SBE) is the application
of specific methods that are influenced by systems
concepts and methodologies to the task of
evaluation.
 These methods articulate and analyze the
interrelationships and interdependencies
of real-life factors, including perspectives and
boundaries, in a process of intervention toward
action.

4. Research Scope & Objectives
 The broad objectives of the research were:
1. To investigate the utility of system dynamics
(SD) in terms of how it does or does not add
value for evaluation of the ART strategy in
Zambia.
2. To identify useful principles and methods for
the broader application of SBE to other health-
services research.

5. Methods & Philosophical Stance
 The research questions and utility framework were
informed by relevant literature, in-depth interviews,
extensive field notes, and a 6-year personal research
journal.
 Data collection and analysis for the research was
guided by narrative-based inquiry and critical-
hermeneutics philosophy:
 Gadamer’s (1998) role of language;
 Habermas’s (1976) communicative action;
 and Ricoeur’s (1985) threefold mimesis.

6. Literature: Key Features of SBE Approaches
 Three unique core concepts in application together
are essential for an SBE:
Interrelationships
Perspectives
Boundaries
• (MIDGLEY 2000: 45; WILLIAMS & HUMMELBRUNNER 2010: 3).

7. Utility Framework: Transformative vs Representative
 Tensions of SBE utility:
 ‘Now that I understand that the problem is a system, I can see
or consider (evaluate) what’s needed to possibly change it. I
may be the one to make the change or I can help advocate for
someone else to do so.’ (Representative)
 ‘I was part of describing the problem and the system, I am part
of the system, I care about the system, and I value certain
factors and connections. I will now go create and be part of the
new system.’ (Transformative)

8. Utility Framework:
Transformative & Representative

9. Utility Framework: Spectrum of Utility—
From Alignment to Action
 For Utility, we mediate Representative and Transformative
on a spectrum:
 aligning pre-understandings
 developing a shared comprehension of the dynamic
situation and components
 providing space for argument on the validity of opinions and
data and reinterpretation of inter-relationships,
perspectives, and boundaries.
 Move to action
Trigger, Comprehend, Validate, Re-Interpret, Act

10. Case Study: Zambian ART Scale-Up Strategy & Context
 2005–2009 :
 Zambian Ministry of Health’s ART scale-up strategy aimed
to rapidly expand HIV/AIDS services
 Overarching goal to achieve dramatic change in the
epidemiological profile of HIV/AIDS in the country.
 Complexity of the Zambian ART scale-up plan
raises need for dynamic and systems-oriented view
in the midst of linear target-setting.
 Opportunities:
1) Apply a systems lens, and 2) Study the process

11. Preparatory Research Data and Findings
 Data revealed three common themes:
1. Critique of current M&E practice for ART
2. Necessity for boundary-setting for the system of
interest
3. Interest in systems-oriented approaches but
cautious about new methods.

12. Preparatory Research Data and Findings
So we’re good at getting the clinical stuff, and we’re good
at counting things and following patients, but we’re bad at
the more psycho/social stuff … ‘Psycho/social’ is a term of
art, which, I don’t really know what it means exactly.
When I say it I’m talking about taking care of the patient’s
non-medical needs, so: counselling and making sure that
the systems and structures are in place in the family, in
order to take care of the family unit. In the point of view of
things we measure, we’d like to know how well those other
services are doing, because they affect the clinical
outcomes that we’re trying to deliver.
- Rep from lead NGO partner

13. Preparatory Research Data and Findings

14. Case Study: Strategic Evaluation
 Objectives:
 to assess the typical point at which the ART patient load in any given
clinic has reached beyond its capacity to provide sustainable, long-
term care
 to determine how increasing the number and/or type of supportive
services networked with an ART clinic reduces the patient load of the
clinical site
 to ascertain what level of access to and participation in supportive
services is necessary to sustain ART for clients into the future
Strategic evaluation is for: ‘evaluating relationships among elements of
strategy’, including ‘coherence of different strategy elements, complexity of
interrelationships, understanding of interdependence and interrelationships,
and evidence of systems thinking and complexity understandings’
(Patrizi: 2010: 99)

15. SD Model Highlights

16. SD Model Highlights
800,000
600,000
400,000
200,000
0
1990 2000 2010 2020 2030 2040
Year
Treating OI’s
Screening
Enrolling
Monitoring
Hoursperyear
Non-HIV tasks
The Monitoring “Bulge”

17. Strategic Evaluation Conclusions
1. Zambia’s strategic goal to ‘prevent, halt and begin to
reverse the spread and impact of HIV and AIDS by 2015’
might be plausible for Lusaka if the huge need for
additional clinical staff is met.
2. Unless supportive service workers can perform
particular tasks (e.g. screening & monitoring) tasks, their
presence may actually increase patient load.

18. Post-Model Data and Findings
SD model:
1. readily triggered stakeholders to reflect on their
existing boundaries (i.e., their beliefs and values)
2. stimulated reinterpretation of interrelationships
3. generated new possible boundaries of the ART care
system
Stakeholders:
1. had comfort with and an understanding of the model’s
structure talked about concerns with the perceived
complicatedness of the approach
2. expressed concern about the potential for false
assumptions to be reified in the model calculations
3. placed confidence/trust in investigators to be rigorous

19. Post-Model Data and Findings
 Benefits
 SD model triggered discussion and elaboration of
system boundaries among participants, trending
towards unified understandings.
 SD model generated new and unfolding
understandings through dialogue.
 Constraints
 Process did not lead to transformative action
 could have been more useful had stakeholders been
engaged in intensive boundary-setting dialogue
both during the development and after the
completion of the model

20. Discussion: Utility of SBE
 Participants asked for more time and opportunities to
understand modeling and analysis.
 Need to collect a broad variety of perspectives at very
outset of modeling process.
 SBE demands planner/evaluator to emphasize
representative and/or transformative process aims.
 Determining system’s boundaries is methodological
requirement/asset for/of systems-based approaches:
transformative power of this discussion in itself, where
it can be an evaluative intervention.
 SBE can be facilitated with an aim to move
stakeholders to consensual coordinated action.
 Sometimes things fall apart…

21. Conclusions for SBE Practice

22.  Acknowledgments
 Government of the Republic of Zambia, Ministry of Health
 CDC Zambia
 Bill and Melinda Gates Foundation
 Patricia Rogers
 Bob Williams
 John Barton
 Jack Homer
 Countless others…..
 Dissertation is available at:
http://researchbank.rmit.edu.au/eserv/rmit:160050/Grove.pdf
Thank You