Presented by Xavier Alterescu as part of the Brown Bag Series given at USAID on MEASURE Evaluation's contribution to the Global Health Initiative Principles

1. Sustainability andHealth Systems StrengtheningWhat Have We Learned? Xavier Alterescu, LLM MIB PhD (ABD) Team Leader, Organizational Development Eric Sarriot, MD PhD Activity Leader, Mali MEASURE Evaluation November 10, 2010. Washington, DC

2. Sustainability: A High Concern for the Global Health Initiative GHI— [Implementation of the Global Health Initiative: Consultation Document] “The challenge of the next decade and beyond is to take these impressive accomplishments to the next level by helping countries achieve long-term sustainability in their health services.” “Building on a long tradition of U.S. government global health leadership and the unprecedented level of commitment manifested in recent years, the Obama Administration’s Global Health Initiative has the opportunity to move global health to a new level of effectiveness, with a vision of long-term sustainability led by partner countries.”

3. Growing Emphasis Or Unsolved Riddle? Russell Theory and practice in sustainability and sustainable development. USAID center for Development Information and Evaluation: USAID’s concern about sustainability referenced as early as 1979 Bossert TJ.: Can they get along without us? 1990 Lafond. Case Studies. 1992-1993 Office of Sustainable Development—Bureau for Africa: Health and Family Planning Indicators; Volume II: Measuring Sustainability. 2000 USAID/CSHGP– CSTS+-CORE Group Sustainability Initiative. 2000-2002 Taking the Long View (Manual) – 2010 USAID/OPRH – Leadership, Management and Sustainability Project. 2005-2010

4. Definitions of Sustainability for Health Systems : “The capacity of a local Health System to deliver needed, quality health services and programs according to sound technical and management and leadership practices and in a way that guarantees continuity and the achievement of its and mandate today and in the future.” “The emerging property within a local Health System which allows to maintain improved population health status within the locale.”

7. Part 2 – A Local Systems Approach toPlanning for and Evaluating Sustainability

9. HSS: Putting people at the center of all levels of the health system One of the biggest challenges of a health care system is how to turn a demoralized or overworked and stressed staff into a proactive, motivated team focused on delivering quality health services every day.

10. A Framework for People- Centered Health Systems Strengthening

11. Leadership Development Program (LDP) Process: A process delivered over 4-6 months that focuses managers and their teams on achieving measurable results using proven public health practices. 3 workshops (2-3 days), Team meetings, Leadership Projects and Implementation, Coaching, M&E, Results Presentations Outcomes: -Measurable health results -Committed and inspired work teams -An improvement process that is owned and sustained

13. Infant mortality declined by 13%

14. Maternal mortality decreased to from over 86 to 47.5/ 100,000

16. Individual ministries benefited from the program and internalized the process to create improved results in their own work groups.

19. Country ownership can thrive; and,

21. Part 2 A Local System Approach to Sustainability Planning and Evaluation

22. Why we need a system perspective?E.g. Urban Health System, Bangladesh ADB, DfID, USAID, etc. MOLGRD MOHFW City Government Chairman MOHFW District & Sub-District Health Platform (MESPCC) Health Inspector (in absence of Med Off.) Health Department NGO Health Providers Govt & Private Health Facilities Social & religious leaders Commissioner Private pharmacists Community organizations Ward Health Committee Teachers Youth volunteers Traditional health providers Civil Society

23. Sustainability in HSS: a Complex Equation? “Health -programme sustainability is the ultimate manifestation of a complex web of inter-relations between health concerns, stakeholders, resources, and actions analogous to an ecosystem.” Gruen et al. The Lancet. 2008 Illustration inspired by SamirRihani: “Complex Systems: Theory and Development Practice. Understanding Non-Linear Realities.”

25. Are you at least improving expectations for continued progress in the context?

26. Are you thinking “within a system?” [a.k.a. is the “Sustainability Scenario” coherent from an internal development logic?]

27. Do you have information about where your efforts are heading?

28. Are your partners mastering the processes which produce the results?

29. Do the M&E systems inform the local actors about their progress?Questions I don’t understand: Is it sustainable? Can you ensure it will be sustained? Who will take over after you leave?

30. The Sustainability Framework –Time and Perspective in Evaluation Constraints & Black Swans Improvements continue and/or Achievements are maintained Implement HSS Intervention Stakeholders Adapt & Organize Intervention Ends SF Retrospective: Were gains maintained? Did progress continue? How did system adapt? Prospective: What are the long-term odds based on the results, processes, and systems being built? Source: Taking the Long View: A Practical Guide to Sustainability Planning and Measurement in Community-Oriented Health Programming. 2008. http://www.mchipngo.net/controllers/link.cfc?method=tools_sustain. And: Black Swan, Grey Swan, Sustainability. Or, the difference between Planning and Predicting. Available at: http://cedarscenter.com/resdetail.cfm?resid=125

31. Three Evaluation Examples Prospective(USAID/NEPAL) Point in time evaluation of progress toward sustainability by supported districts Prospective & Retrospective(USAID/CSHGP/Concern Bangladesh) Planning for sustainability What has been sustained? How much has been sustained? Modeling Investments(USAID/CSHGP/Save the Children Guinea) What could we achieve if…?

32. Measuring sustainability as a programming tool for health sector investments – USAID NepalComposite / Dashboard Measures Compo 1 index Poor Nascent Compo 6 index 2 Intermediate Promising Strong 3 Compo 5 index Compo 4 index Source: Measuring sustainability as a programming tool for health sector investments—report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Manage. 2009 Oct-Dec;24(4):326-50 http://www3.interscience.wiley.com/journal/118903030/abstract

33. Can districts sustain the health of their mothers & children? Kanchanpur and Rasuwa Districts, Nepal - 2006

34. Sustained Results 5 Years Post-Project. Urban Health / CWI Bangladesh: 1999-2004↨-2009 From Urban Child Survival Project to Urban Health System Strengthening Defining a Sustainability Scenario consistent with national policy Defining ±consistent measures (incl. outcomes) Repeated assessments & collective learning 18 months post-project coaching (minimal) -> 3-year and 5-year post-project sustainability assessments Source: Sustainability of the Saidpur and Parbatipur Urban Health Model (Bangladesh) Five Years After the End of Concern’s Child Survival Project . Final Evaluation Report – January 10, 2010. USAID, Concern Worldwide. ICF Macro/. Available at: http://cedarscenter.com/resdetail.cfm?resid=106

35. CWI Bangladesh: 1999-2004↨-2009How Much Sustainability in Health Outcomes? Project Ends

36. HSS at District Level: Sustainability Investments and Child Deaths Averted Source: Pro-sustainability choices and child deaths averted: from project experience to investment strategy Health Policy and Planning 2010;1–12. ; doi: 10.1093/heapol/czq042

37. A Simplified Traditional Scenario for Investment Total Investment: $7.5 millions U5 Lives Saved: 2,530 [1,569 – 6,167]

38. Investment Scheme for Maximizing Sustainability – Phase 1

39. Investment Scheme for Maximizing Sustainability – Phase 2

40. Investment Scheme for Maximizing Sustainability – Phase 3

41. Investment Scheme for Maximizing Sustainability – Phase 4

42. Investment Scheme for Maximizing Sustainability – Phase 5 Total Investment: $7.5 millions U5 Lives Saved: 8,485 [4,169 – 8,909]

43. Testable Hypothesis: three-time the impact for the same investment

44. Part 3 Conclusions and Suggestions to the GHI

45. Food for Thought Time is inaccessible to human dictates. It is fiercely independent from our desires, and denying this reality does not lead to sustainable results. Health System achievements can be attributed to one group or they can be owned by another. It cannot be both. “We will align and harmonize when they make us!” “Per diems are the indicator of anti-alignment.” * Source: Field Notes – Country Partnership Assessment. Unpublished. And discussions with authors.

46. Put people first– The right info to the right people Respect Time as a Factor – Demonstrate Consistency of Purpose Keep it simple…but not simplistic: build the internal logic of the local system Build-in self-learning steps (heuristics) & metrics for internal accountability of the system Examine all components of sustainability systematically A&H: primum non nocere Build evidence through research What CAN we do?

47. Toward an Evaluation-Research Agenda Evaluation ownership: internal or external ? Processes and Metrics for Learning Strengthen empiricism in HSSRobust studies, which will yield lessons five years from now need to be designed now. Learn from Complexity & Systems Sciences

49. MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at http://www.cpc.unc.edu/measure.

50. Does Strengthening Leadership and Management Lead to Improved Health Outcomes?Results from a Quasi-experimental Study in Kenya

52. Baseline, endline, and sustainability measures taken of a key indicator addressed by each of 67 LDP teams;

53. Same measures taken of the same key HMIS indicator in intervention and comparison groups with no intervention;

54. Matching using Coarsened Exact Matching (Stata, v. 11), with equivalence of intervention and comparison groups increased by matching on selected criteria;

56. Delivery by a skilled birth attendant (23 teams)

57. Four antenatal care visits (10 teams)

61. Results (percentages) were averaged for each time period for LDP teams and district-level comparisons

64. Figure 1: Results from LDP Teams at District and Facility Levels: Average Coverage Rates for Selected Health Indicators– Kenya 2008-2010 n=67 teams Source: LMS Kenya Project, MSH

65. Figure 2: Results from LDP Teams and Comparison Areas at District Level: Average Coverage Rates for Selected Health Indicators – Kenya 2008-2010 n=18 teams Source: LMS Kenya Project, MSH; Kenya HMIS MOH

67. Different indicators with different service volumes selected by the LDP teams

69. Shortage of medicines or supplies, especially vaccines; and,

71. Increased access by providing more outreach sites or more service hours or days; and,

73. Sustainability Components and Illustrative Indicators Programmatic: the organization delivers products and services that respond to client needs and anticipates new areas of need Service delivery indicators: CYP, VCT counseling and treatment, TB detection rate Quality indicators: client satisfaction, drop-outs, clients receive preferred method

74. Sustainability Components and Illustrative Indicators Financial: the organization draws on various sources of revenue and has effective financial management practices Cost recovery - Rate of revenue vs. annual operating costs Percentage of annual revenue generated from diverse sources Percentage of annual operating budget covered by income generated through service delivery Organization has submitted at least one proposal for external funding within the last 12 months

75. Sustainability Components and Illustrative Indicators Institutional : Systems Planning Human resource management Monitoring and evaluation Information management: data collection Information management: use of information Quality assurance Financial management Revenue generation Supply management