Gates fdn measuring impact presentation nov 30 2010


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Strategies must be developed, tracked and measured, reviewed, and adjusted based on data. This shows the strategy business processes (measurement is implied): create the strategy, review it annually, “refresh” it periodically when a major course change and fresh look is needed. **********Development – New area of work. After period of learning. Not very common. Coming up – Urban Poverty (increase income of slumdwellers by helping them organize and develop income opportunities e.g, wastepicking).Review – annual. Every strategy every year, with the CEO, presidents, co-chairs and team. Purpose – for the team to shareWhat’s working/what’s notChallengesLessonsCourse corrections (areas of adjustment)Critical issues they’d like feedback onRefresh. Would like to drill in here. Development and refresh are closely linked – nearly the same process. How does the foundation develop a new strategy, or refresh one that already exists? What process and set of analyses do teams follow? Note this is done every 3-5 years.
  • These are analytical tools to facilitate reasoning, analysis, consideration of alternative solutions, help us understand the critical role of other players and clearly articulate our role. Ultimate goal - create realistic, but ambitious strategies that can be implemented and measured. Opportunity mapTool to open the thinking, find creative ideas, challenge assumptionsProblem may be segmented by geography, end beneficiary, disease state, etc.Typically a precursor to developing a TOC2. TOCIncludes an explicit goal statementShows all the critical “levers” and causal links to achieve the goal, not just what we will doHelps ground our actions in the big picture (what others must do)Should stand on its own (i.e., be self explanatory)Right level of granularity important 3. TOAMust overlay on the TOCMade up of initiatives and sub-initiativesAs a companion to or part of the TOA, important to also state what we won’t do, and whyWhen done well, reveals risksAlso – solution leverage -- How and to what extent will the choices we make position us to achieve scale?
  • Start with an anecdote – real donor terms of reference for an evaluation in Northern Uganda. Community economic development program Few years, $10 million TOR read: evaluate impact program has had on economic development in the region, conflict prevention and welfare in the target communities. Time: 18 days. Shows how measurement can be difficult for many reasons – not just the sheer act of doing it, but understanding what it is, what it can and cant do, the kind of knowledge that can be produced by collecting data, the limits to that knowledge and reality on the ground Example of no common understanding of language, no purpose in mind probably – donor in this case – were they holding the organization accountable for conflict prevention in No U?
  • We use these categories to help teams within the foundation to think about what results (output, outcomes: coverage), and how (with/without attribution; shorter term feedback) they should measure above the grant level – trying to identify their strategic intent for a cluster of grants, where you can and can’t add up….
  • - Putting it all back together – the theory to the practice – we still have more questions than answers. On Develop/refresh strategy: How do we identify, articulate and evaluate alternative paths to impact? On Track Progress: What results do we aggregate? What are the most promising interventions to evaluate? On Review strategy: How do we adjust our strategies while maintaining continuity in our partnerships? On Learn, Adapt, Improve:  How do we keep the bar high and hold people accountable for results, while leaving room for failure and learning? 
  • Gates fdn measuring impact presentation nov 30 2010

    1. 1. Prove It: How the Gates Foundation Ties Strategy to Results<br />Schwab Charitable Philanthropy Speaker Series<br />Center for Nonprofit and Public Leadership<br />Haas School of Business<br />Jodi Nelson, Interim Director<br />Emily Parker, Sr. Strategy Officer<br />Impact Planning and Improvement<br />September 30, 2010<br />
    2. 2. What are your questions about strategy and measurement at the Bill & Melinda Gates Foundation?<br />© 2010 Bill & Melinda Gates Foundation | <br />2<br />
    3. 3. Today we would like to:<br />Collect your questions<br />Share our aspiration: is it about “proving” it? <br />Share what we do in practice <br />Share our big questions <br />Ask for your ideas from other experiences and sectors <br />© 2010 Bill & Melinda Gates Foundation | <br />3<br />
    4. 4. Fast Facts about the Bill & Melinda Gates Foundation<br />(1) Number of countries in which the foundation has grantees<br />Source: Bill & Melinda Gates Foundation Fact Sheet, September 2010,<br />© 2010 Bill & Melinda Gates Foundation | <br />4<br />
    5. 5. Changes over the past decade set the context for strategy and measurement<br />To<br />From<br /><ul><li>Foundation at a stable size
    6. 6. Focus on use of limited resources
    7. 7. 25 strategies across 3 broad sectors: global health, development, US education
    8. 8. Approach to strategy maturing
    9. 9. Actionable Measurement beginning to take shape
    10. 10. in learning mode—from inside and outside
    11. 11. Foundation growing rapidly
    12. 12. Pay out requirement hard to meet
    13. 13. A few strategies (i.e., high school education, US libraries)
    14. 14. No consistent approach to strategy development or measurement
    15. 15. little guidance
    16. 16. let a thousand flowers bloom</li></ul>© 2010 Bill & Melinda Gates Foundation | <br />5<br />
    17. 17. Imagine… <br />You lead the Agricultural Development program at the Bill & Melinda Gates Foundation. <br />You have to decide how the foundation should invest in agriculture to help alleviate poverty in Sub-Saharan Africa and parts of Asia. <br />What will the foundation’s goals be? How will we achieve them? How will we know if they are achieved? <br />You have a meeting with Bill & Melinda in six months and need to tell them your plan. <br />What do you do?<br />© 2010 Bill & Melinda Gates Foundation | <br />6<br />
    18. 18. Why is it so tough?<br /><ul><li>The world is complex — change is caused by many different things.
    19. 19. We are just one actor — governments, NGOs, other donors, private citizens are also trying to improve people’s lives
    20. 20. We are not on the “ground” — we are strategic donors, but still just donors.
    21. 21. Data can be hard to come by — we may know we need it, but collecting meaningful data is hard to do.</li></ul>© 2010 Bill & Melinda Gates Foundation | <br />7<br />
    22. 22. So what do we do?<br />Use tools of strategy and measurement to plan our work<br />despite the complexity <br />© 2010 Bill & Melinda Gates Foundation | <br />8<br />
    23. 23. Our aspiration is to create feedback loops so we can plan, execute, measure, learn, adjust, plan …<br />data & experience<br />data & experience<br />data & experience<br />data & experience<br />© 2010 Bill & Melinda Gates Foundation | <br />9<br />
    24. 24. Let’s break it down….<br />© 2010 Bill & Melinda Gates Foundation | <br />10<br />
    25. 25. What is a strategy at the foundation?<br />Causal pathway to impact. Outlines the investments and programmatic activities aligned with that pathway. Defines the results of these investments and activities over time. <br />Bill & Melinda Gates Foundation Approved Strategies<br />Global Development Program<br /><ul><li>Agricultural Development
    26. 26. Financial Services for the Poor
    27. 27. Global Libraries
    28. 28. Water, Sanitation, and Hygiene
    29. 29. Policy and Advocacy</li></ul>Global Health Program<br /><ul><li>Discovery
    30. 30. Enteric Diseases and Diarrhea
    31. 31. Family Planning
    32. 32. HIV
    33. 33. Malaria
    34. 34. Maternal, Newborn, and Child Health
    35. 35. Neglected and Other Infectious Diseases
    36. 36. Nutrition
    37. 37. Pneumonia
    38. 38. Policy and Advocacy
    39. 39. Tobacco
    40. 40. Tuberculosis
    41. 41. Vaccine Preventable Diseases/Delivery</li></ul>United States Program<br /><ul><li>College Ready
    42. 42. Early Learning
    43. 43. Pacific Northwest
    44. 44. Postsecondary Success
    45. 45. U.S. Advocacy
    46. 46. U.S. Libraries</li></ul>Policy and Government Affairs<br /><ul><li>Charitable Sector Support</li></ul>© 2010 Bill & Melinda Gates Foundation | <br />11<br />
    47. 47. Our vision of successStrategies are constantly improving, based on measurement and learning<br /><ul><li>“No strategy survives first contact with the enemy.”
    48. 48. Measurement and learning—coupled with systematic review, reflection, and intellectual dialogue—drives improvement.</li></ul>unrealized strategy<br />intended strategy<br />deliberate strategy<br />realized strategy<br />emergent strategy<br />Figure created by Henry Mintzberg, strategy theorist.<br />© 2010 Bill & Melinda Gates Foundation | <br />12<br />
    49. 49. Our business process reflects this philosophy <br />When<br />What<br /><ul><li>Strategy development: creating a strategy“from scratch”
    50. 50. Strategy review: annual update for the CEO and co-chairs on progress against an approved strategy
    51. 51. Strategy refresh: re-examining an approved strategy with the expectation there will be a significant change in strategic direction
    52. 52. Once: after a period of learning
    53. 53. Every strategy, every year
    54. 54. Every three to five years</li></ul>© 2010 Bill & Melinda Gates Foundation | <br />13<br />
    55. 55. Strategy development and refresh is an iterative, phased processwith progressive analyses from broad to narrow<br />© 2010 Bill & Melinda Gates Foundation | <br />14<br />
    56. 56. Three concepts underpin strategy development and refresh<br />Theory of action<br /><ul><li>solution leverage
    57. 57. partner leverage</li></ul>Theory of change<br />Opportunity map<br /><ul><li>hypothesis about what needs to happen for the stated goal to be achieved
    58. 58. What must the world do?
    59. 59. what we will do (directly or through others) to help achieve the stated goal
    60. 60. What must we do?
    61. 61. How will our actions maximize leverage?
    62. 62. segmentation(s) of the problem/ space and identification of opportunities for change</li></ul>What it is<br />Key question<br /><ul><li>Where is the leverage?</li></ul>15<br />© 2010 Bill & Melinda Gates Foundation | <br />
    63. 63. Limited grantmaking begins<br />Strategy developed & approved--the sky’s the limit<br /> “Learning initiative” established<br />Endless possibilities; limited resources<br />???<br />Strategy review—strategy is adjusted <br />© 2010 Bill & Melinda Gates Foundation | <br />16<br />
    64. 64. What about measurement? <br />What is it anyway? <br />Why is it so difficult to do well? <br />What is our philosophy and approach? <br />17<br />© 2010 Bill & Melinda Gates Foundation | <br />
    65. 65. What is measurement? <br />By definition: Obtaining the magnitude of a quantity <br />Performance measurement <br /><ul><li>Assessing the achievement of pre-determined goals and objectives
    66. 66. Produce objective, relevant information on program or organization performance
    67. 67. Developing measurable indicators that can be tracked to assess progress
    68. 68. What is unfortunate about the word “measurement”
    69. 69. Not everything is about counting or “tracking”
    70. 70. Under emphasizes evaluation, mixed methods, asking and answering questions about how things happen and why — not just what happens </li></ul>© 2010 Bill & Melinda Gates Foundation | <br />18<br />
    71. 71. What's our philosophy? Actionable Measurement <br />Direction from the co-chairs in 2008<br />Measurement should be strategic and actionable<br />We should not measure everything:<br /><ul><li>Rely on grantee reporting as much as possible
    72. 72. Align our data requests with those of other funders
    73. 73. Learn from others
    74. 74. Leverage technology for data collection and viewing results</li></ul>© 2010 Bill & Melinda Gates Foundation | <br />19<br />
    75. 75. Actionable Measurement<br />Measure to inform decisions and actions. How you do it (the approach, methodology, evaluation design) is determined by the purpose. There is no one measurement approach that works all the time, for all purposes. <br /><ul><li>Why is it unique to be pragmatic?
    76. 76. Where do we weigh in on the philosophical debates?
    77. 77. The “gold standard” is use </li></ul>© 2010 Bill & Melinda Gates Foundation | <br />20<br />
    78. 78. <ul><li>Tracking results across time/space
    79. 79. Tracking execution is good for accountability
    80. 80. But tough to do well but can describe change over time
    81. 81. Evaluation designs
    82. 82. Impact evaluation purpose: demonstrating effectiveness of a model/approach, good for replication, informing the field, testing assumptions to check strategy
    83. 83. Process evaluation purpose: to improve implementation/execution at key times
    84. 84. Participatory evaluation purpose: to empower participants with information to act
    85. 85. Developmental evaluation: to identify components of a model or approach in real time </li></ul>Getting useful data requires choosing the right methods or <br />approaches to measurement given what you need to do with it<br />© 2010 Bill & Melinda Gates Foundation | <br />21<br />Example of purpose driven measurement <br />
    86. 86. Inputs<br />Activities<br />Outputs<br />Outcomes<br />Impacts<br />Strategy<br />Initiative<br />Sub<br />-<br />Initiative<br />Grant<br />Sub<br />-<br />Grant<br />It is an approach to measure results of our strategies<br /><ul><li>How is this different from the measurement challenge facing implementing</li></ul> organizations (grantees)? <br />© 2010 Bill & Melinda Gates Foundation | <br />22<br />
    87. 87. © 2010 Bill & Melinda Gates Foundation | <br />What’s the essence of the approach?<br />Measure impact (change in people’s lives) to track collective progress toward targets, not to attribute impact to the foundation’s actions or investments. <br />Strategy<br />Measure key output and outcome results to track progress of the foundation’s work; use evaluation to inform decision making by testing assumptions, learning what works, how and why. <br />Initiatives<br />Selectively track results of grants at critical points for accountability; evaluate to learn about implementation of key investments; use impact evaluation when it seeks to answer a strategic question for an initiative not as an accountability tool. <br />Grants<br />23<br />
    88. 88. Measuring investment types <br />Measurement Guidelines Specific to Investment Type<br />Types of Investment<br />Priority for evaluation resources<br />Research<br />Product Development<br />Policy and Advocacy<br />Model Development<br />and Demonstration of<br />Effectiveness<br />Systems Change<br />Delivery at Scale<br />Measure desired outcomes, <br />track progress, and focus <br />on shorter-term feedback<br />Monitor outputs and track process<br />Track execution and coverage/reach<br />Measure success or failure and extent of fit with target product profile<br />Measure attribution where it is technically feasible and ethical<br />Use multiple measurement <br />methods to draw conclusions<br />© 2010 Bill & Melinda Gates Foundation | <br />24<br />
    89. 89. Our questions… <br />How do we identify, articulate, and evaluate alternative paths to impact?<br />How do we develop theories of change where evidence of what works is slim?<br />How do we keep the bar high and hold people accountable for results, while leaving room for failure and learning?<br />data & experience<br />data & experience<br />What results do we aggregate? What are the most promising interventions to evaluate?<br />data & experience<br />data & experience<br />How do we adjust our strategies while maintaining continuity in our partnerships?<br />© 2010 Bill & Melinda Gates Foundation | <br />25<br />
    90. 90. Thank You<br />© 2010 Bill & Melinda Gates Foundation. All Rights Reserved. Bill & Melinda Gates Foundation is a registered trademark in the United States and other countries.<br />