IDS Impact Innovation and Learning Workshop March 2013: Day 1, Paper session 1 John Grove


Published on

Published in: Education, Health & Medicine
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

IDS Impact Innovation and Learning Workshop March 2013: Day 1, Paper session 1 John Grove

  1. 1. I M P A C T , L E A R N I N G A N D I N N O V A T I O N : T O W A R D S AR E S E A R C H A N D P R A C T I C E A G E N D A F O R T H E F U T U R EI N S T I T U T E F O R D E V E L O P M E N T S T U D I E S , B R I G H T O N ( U K )M A R C H 2 6 , 2 0 1 3JOHN T. GROVE, M.A., PH.D .B I L L A N D M E L I N D A G A T E S F D N . ** C U R R E N T A F F I L I A T I O N . T H I S W O R K W A S C O N D U C T E D B E T W E E N 2 0 0 6 A N D 2 0 1 1 W H E N J G W A SP A R T O F T H E C E N T E R S F O R D I S E A S E C O N T R O L & P R E V E N T I O N A N D A S A N I N D E P E N D E N TR E S E A R C H D E G R E EM O D E L I N G W A S C O N D U C T E D B Y D R . J A C K H O M E RAiming for utility in ‘systems-basedevaluation’:A research-based framework for practitioners
  2. 2. Outline Defining SBE Research Objectives, Methods, Stance Key Features and Utility of SBE The Zambia Case: Strategic Evaluation with SD Preparatory (Pre-Model) Phase Findings SD Model Highlights & Reflections Post-Model Findings Recommendations for Practice
  3. 3. What Is Systems-Based Evaluation? Systems-Based Evaluation (SBE) is the applicationof specific methods that are influenced by systemsconcepts and methodologies to the task ofevaluation. These methods articulate and analyze theinterrelationships and interdependenciesof real-life factors, including perspectives andboundaries, in a process of intervention towardaction.
  4. 4. Research Scope & Objectives The broad objectives of the research were:1. To investigate the utility of system dynamics(SD) in terms of how it does or does not addvalue for evaluation of the ART strategy inZambia.2. To identify useful principles and methods forthe broader application of SBE to other health-services research.
  5. 5. Methods & Philosophical Stance The research questions and utility framework wereinformed by relevant literature, in-depth interviews,extensive field notes, and a 6-year personal researchjournal. Data collection and analysis for the research wasguided by narrative-based inquiry and critical-hermeneutics philosophy: Gadamer’s (1998) role of language; Habermas’s (1976) communicative action; and Ricoeur’s (1985) threefold mimesis.
  6. 6. Literature: Key Features of SBE Approaches Three unique core concepts in application togetherare essential for an SBE:InterrelationshipsPerspectivesBoundaries• (MIDGLEY 2000: 45; WILLIAMS & HUMMELBRUNNER 2010: 3).
  7. 7. Utility Framework: Transformative vs Representative Tensions of SBE utility: ‘Now that I understand that the problem is a system, I can seeor consider (evaluate) what’s needed to possibly change it. Imay be the one to make the change or I can help advocate forsomeone else to do so.’ (Representative) ‘I was part of describing the problem and the system, I am partof the system, I care about the system, and I value certainfactors and connections. I will now go create and be part of thenew system.’ (Transformative)
  8. 8. Utility Framework:Transformative & Representative
  9. 9. Utility Framework: Spectrum of Utility—From Alignment to Action For Utility, we mediate Representative and Transformativeon a spectrum: aligning pre-understandings developing a shared comprehension of the dynamicsituation and components providing space for argument on the validity of opinions anddata and reinterpretation of inter-relationships,perspectives, and boundaries. Move to actionTrigger, Comprehend, Validate, Re-Interpret, Act
  10. 10. Case Study: Zambian ART Scale-Up Strategy & Context 2005–2009 : Zambian Ministry of Health’s ART scale-up strategy aimedto rapidly expand HIV/AIDS services Overarching goal to achieve dramatic change in theepidemiological profile of HIV/AIDS in the country. Complexity of the Zambian ART scale-up planraises need for dynamic and systems-oriented viewin the midst of linear target-setting. Opportunities:1) Apply a systems lens, and 2) Study the process
  11. 11. Preparatory Research Data and Findings Data revealed three common themes:1. Critique of current M&E practice for ART2. Necessity for boundary-setting for the system ofinterest3. Interest in systems-oriented approaches butcautious about new methods.
  12. 12. Preparatory Research Data and FindingsSo we’re good at getting the clinical stuff, and we’re goodat counting things and following patients, but we’re bad atthe more psycho/social stuff … ‘Psycho/social’ is a term ofart, which, I don’t really know what it means exactly.When I say it I’m talking about taking care of the patient’snon-medical needs, so: counselling and making sure thatthe systems and structures are in place in the family, inorder to take care of the family unit. In the point of view ofthings we measure, we’d like to know how well those otherservices are doing, because they affect the clinicaloutcomes that we’re trying to deliver.- Rep from lead NGO partner
  13. 13. Preparatory Research Data and Findings
  14. 14. Case Study: Strategic Evaluation Objectives: to assess the typical point at which the ART patient load in any givenclinic has reached beyond its capacity to provide sustainable, long-term care to determine how increasing the number and/or type of supportiveservices networked with an ART clinic reduces the patient load of theclinical site to ascertain what level of access to and participation in supportiveservices is necessary to sustain ART for clients into the futureStrategic evaluation is for: ‘evaluating relationships among elements ofstrategy’, including ‘coherence of different strategy elements, complexity ofinterrelationships, understanding of interdependence and interrelationships,and evidence of systems thinking and complexity understandings’(Patrizi: 2010: 99)
  15. 15. SD Model Highlights
  16. 16. SD Model Highlights800,000600,000400,000200,00001990 2000 2010 2020 2030 2040YearTreating OI’sScreeningEnrollingMonitoringHoursperyearNon-HIV tasksThe Monitoring “Bulge”
  17. 17. Strategic Evaluation Conclusions1. Zambia’s strategic goal to ‘prevent, halt and begin toreverse the spread and impact of HIV and AIDS by 2015’might be plausible for Lusaka if the huge need foradditional clinical staff is met.2. Unless supportive service workers can performparticular tasks (e.g. screening & monitoring) tasks, theirpresence may actually increase patient load.
  18. 18. Post-Model Data and FindingsSD model:1. readily triggered stakeholders to reflect on theirexisting boundaries (i.e., their beliefs and values)2. stimulated reinterpretation of interrelationships3. generated new possible boundaries of the ART caresystemStakeholders:1. had comfort with and an understanding of the model’sstructure talked about concerns with the perceivedcomplicatedness of the approach2. expressed concern about the potential for falseassumptions to be reified in the model calculations3. placed confidence/trust in investigators to be rigorous
  19. 19. Post-Model Data and Findings Benefits SD model triggered discussion and elaboration ofsystem boundaries among participants, trendingtowards unified understandings. SD model generated new and unfoldingunderstandings through dialogue. Constraints Process did not lead to transformative action could have been more useful had stakeholders beenengaged in intensive boundary-setting dialogueboth during the development and after thecompletion of the model
  20. 20. Discussion: Utility of SBE Participants asked for more time and opportunities tounderstand modeling and analysis. Need to collect a broad variety of perspectives at veryoutset of modeling process. SBE demands planner/evaluator to emphasizerepresentative and/or transformative process aims. Determining system’s boundaries is methodologicalrequirement/asset for/of systems-based approaches:transformative power of this discussion in itself, whereit can be an evaluative intervention. SBE can be facilitated with an aim to movestakeholders to consensual coordinated action. Sometimes things fall apart…
  21. 21. Conclusions for SBE Practice
  22. 22.  Acknowledgments Government of the Republic of Zambia, Ministry of Health CDC Zambia Bill and Melinda Gates Foundation Patricia Rogers Bob Williams John Barton Jack Homer Countless others….. Dissertation is available at: You