Gates fdn measuring impact presentation nov 30 2010

  • 2,492 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
2,492
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
97
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Strategies must be developed, tracked and measured, reviewed, and adjusted based on data. This shows the strategy business processes (measurement is implied): create the strategy, review it annually, “refresh” it periodically when a major course change and fresh look is needed. **********Development – New area of work. After period of learning. Not very common. Coming up – Urban Poverty (increase income of slumdwellers by helping them organize and develop income opportunities e.g, wastepicking).Review – annual. Every strategy every year, with the CEO, presidents, co-chairs and team. Purpose – for the team to shareWhat’s working/what’s notChallengesLessonsCourse corrections (areas of adjustment)Critical issues they’d like feedback onRefresh. Would like to drill in here. Development and refresh are closely linked – nearly the same process. How does the foundation develop a new strategy, or refresh one that already exists? What process and set of analyses do teams follow? Note this is done every 3-5 years.
  • These are analytical tools to facilitate reasoning, analysis, consideration of alternative solutions, help us understand the critical role of other players and clearly articulate our role. Ultimate goal - create realistic, but ambitious strategies that can be implemented and measured. Opportunity mapTool to open the thinking, find creative ideas, challenge assumptionsProblem may be segmented by geography, end beneficiary, disease state, etc.Typically a precursor to developing a TOC2. TOCIncludes an explicit goal statementShows all the critical “levers” and causal links to achieve the goal, not just what we will doHelps ground our actions in the big picture (what others must do)Should stand on its own (i.e., be self explanatory)Right level of granularity important 3. TOAMust overlay on the TOCMade up of initiatives and sub-initiativesAs a companion to or part of the TOA, important to also state what we won’t do, and whyWhen done well, reveals risksAlso – solution leverage -- How and to what extent will the choices we make position us to achieve scale?
  • Start with an anecdote – real donor terms of reference for an evaluation in Northern Uganda. Community economic development program Few years, $10 million TOR read: evaluate impact program has had on economic development in the region, conflict prevention and welfare in the target communities. Time: 18 days. Shows how measurement can be difficult for many reasons – not just the sheer act of doing it, but understanding what it is, what it can and cant do, the kind of knowledge that can be produced by collecting data, the limits to that knowledge and reality on the ground Example of no common understanding of language, no purpose in mind probably – donor in this case – were they holding the organization accountable for conflict prevention in No U?
  • We use these categories to help teams within the foundation to think about what results (output, outcomes: coverage), and how (with/without attribution; shorter term feedback) they should measure above the grant level – trying to identify their strategic intent for a cluster of grants, where you can and can’t add up….
  • - Putting it all back together – the theory to the practice – we still have more questions than answers. On Develop/refresh strategy: How do we identify, articulate and evaluate alternative paths to impact? On Track Progress: What results do we aggregate? What are the most promising interventions to evaluate? On Review strategy: How do we adjust our strategies while maintaining continuity in our partnerships? On Learn, Adapt, Improve:  How do we keep the bar high and hold people accountable for results, while leaving room for failure and learning? 

Transcript

  • 1. Prove It: How the Gates Foundation Ties Strategy to Results
    Schwab Charitable Philanthropy Speaker Series
    Center for Nonprofit and Public Leadership
    Haas School of Business
    Jodi Nelson, Interim Director
    Emily Parker, Sr. Strategy Officer
    Impact Planning and Improvement
    September 30, 2010
  • 2. What are your questions about strategy and measurement at the Bill & Melinda Gates Foundation?
    © 2010 Bill & Melinda Gates Foundation |
    2
  • 3. Today we would like to:
    Collect your questions
    Share our aspiration: is it about “proving” it?
    Share what we do in practice
    Share our big questions
    Ask for your ideas from other experiences and sectors
    © 2010 Bill & Melinda Gates Foundation |
    3
  • 4. Fast Facts about the Bill & Melinda Gates Foundation
    (1) Number of countries in which the foundation has grantees
    Source: Bill & Melinda Gates Foundation Fact Sheet, September 2010, gatesfoundation.org
    © 2010 Bill & Melinda Gates Foundation |
    4
  • 5. Changes over the past decade set the context for strategy and measurement
    To
    From
    • Foundation at a stable size
    • 6. Focus on use of limited resources
    • 7. 25 strategies across 3 broad sectors: global health, development, US education
    • 8. Approach to strategy maturing
    • 9. Actionable Measurement beginning to take shape
    • 10. in learning mode—from inside and outside
    • 11. Foundation growing rapidly
    • 12. Pay out requirement hard to meet
    • 13. A few strategies (i.e., high school education, US libraries)
    • 14. No consistent approach to strategy development or measurement
    • 15. little guidance
    • 16. let a thousand flowers bloom
    © 2010 Bill & Melinda Gates Foundation |
    5
  • 17. Imagine…
    You lead the Agricultural Development program at the Bill & Melinda Gates Foundation.
    You have to decide how the foundation should invest in agriculture to help alleviate poverty in Sub-Saharan Africa and parts of Asia.
    What will the foundation’s goals be? How will we achieve them? How will we know if they are achieved?
    You have a meeting with Bill & Melinda in six months and need to tell them your plan.
    What do you do?
    © 2010 Bill & Melinda Gates Foundation |
    6
  • 18. Why is it so tough?
    • The world is complex — change is caused by many different things.
    • 19. We are just one actor — governments, NGOs, other donors, private citizens are also trying to improve people’s lives
    • 20. We are not on the “ground” — we are strategic donors, but still just donors.
    • 21. Data can be hard to come by — we may know we need it, but collecting meaningful data is hard to do.
    © 2010 Bill & Melinda Gates Foundation |
    7
  • 22. So what do we do?
    Use tools of strategy and measurement to plan our work
    despite the complexity
    © 2010 Bill & Melinda Gates Foundation |
    8
  • 23. Our aspiration is to create feedback loops so we can plan, execute, measure, learn, adjust, plan …
    data & experience
    data & experience
    data & experience
    data & experience
    © 2010 Bill & Melinda Gates Foundation |
    9
  • 24. Let’s break it down….
    © 2010 Bill & Melinda Gates Foundation |
    10
  • 25. What is a strategy at the foundation?
    Causal pathway to impact. Outlines the investments and programmatic activities aligned with that pathway. Defines the results of these investments and activities over time.
    Bill & Melinda Gates Foundation Approved Strategies
    Global Development Program
    • Agricultural Development
    • 26. Financial Services for the Poor
    • 27. Global Libraries
    • 28. Water, Sanitation, and Hygiene
    • 29. Policy and Advocacy
    Global Health Program
    United States Program
    Policy and Government Affairs
    • Charitable Sector Support
    © 2010 Bill & Melinda Gates Foundation |
    11
  • 47. Our vision of successStrategies are constantly improving, based on measurement and learning
    • “No strategy survives first contact with the enemy.”
    • 48. Measurement and learning—coupled with systematic review, reflection, and intellectual dialogue—drives improvement.
    unrealized strategy
    intended strategy
    deliberate strategy
    realized strategy
    emergent strategy
    Figure created by Henry Mintzberg, strategy theorist.
    © 2010 Bill & Melinda Gates Foundation |
    12
  • 49. Our business process reflects this philosophy
    When
    What
    • Strategy development: creating a strategy“from scratch”
    • 50. Strategy review: annual update for the CEO and co-chairs on progress against an approved strategy
    • 51. Strategy refresh: re-examining an approved strategy with the expectation there will be a significant change in strategic direction
    • 52. Once: after a period of learning
    • 53. Every strategy, every year
    • 54. Every three to five years
    © 2010 Bill & Melinda Gates Foundation |
    13
  • 55. Strategy development and refresh is an iterative, phased processwith progressive analyses from broad to narrow
    © 2010 Bill & Melinda Gates Foundation |
    14
  • 56. Three concepts underpin strategy development and refresh
    Theory of action
    • solution leverage
    • 57. partner leverage
    Theory of change
    Opportunity map
    • hypothesis about what needs to happen for the stated goal to be achieved
    • 58. What must the world do?
    • 59. what we will do (directly or through others) to help achieve the stated goal
    • 60. What must we do?
    • 61. How will our actions maximize leverage?
    • 62. segmentation(s) of the problem/ space and identification of opportunities for change
    What it is
    Key question
    • Where is the leverage?
    15
    © 2010 Bill & Melinda Gates Foundation |
  • 63. Limited grantmaking begins
    Strategy developed & approved--the sky’s the limit
    “Learning initiative” established
    Endless possibilities; limited resources
    ???
    Strategy review—strategy is adjusted
    © 2010 Bill & Melinda Gates Foundation |
    16
  • 64. What about measurement?
    What is it anyway?
    Why is it so difficult to do well?
    What is our philosophy and approach?
    17
    © 2010 Bill & Melinda Gates Foundation |
  • 65. What is measurement?
    By definition: Obtaining the magnitude of a quantity
    Performance measurement
    • Assessing the achievement of pre-determined goals and objectives
    • 66. Produce objective, relevant information on program or organization performance
    • 67. Developing measurable indicators that can be tracked to assess progress
    • 68. What is unfortunate about the word “measurement”
    • 69. Not everything is about counting or “tracking”
    • 70. Under emphasizes evaluation, mixed methods, asking and answering questions about how things happen and why — not just what happens
    © 2010 Bill & Melinda Gates Foundation |
    18
  • 71. What's our philosophy? Actionable Measurement
    Direction from the co-chairs in 2008
    Measurement should be strategic and actionable
    We should not measure everything:
    • Rely on grantee reporting as much as possible
    • 72. Align our data requests with those of other funders
    • 73. Learn from others
    • 74. Leverage technology for data collection and viewing results
    © 2010 Bill & Melinda Gates Foundation |
    19
  • 75. Actionable Measurement
    Measure to inform decisions and actions. How you do it (the approach, methodology, evaluation design) is determined by the purpose. There is no one measurement approach that works all the time, for all purposes.
    • Why is it unique to be pragmatic?
    • 76. Where do we weigh in on the philosophical debates?
    • 77. The “gold standard” is use
    © 2010 Bill & Melinda Gates Foundation |
    20
  • 78.
    • Tracking results across time/space
    • 79. Tracking execution is good for accountability
    • 80. But tough to do well but can describe change over time
    • 81. Evaluation designs
    • 82. Impact evaluation purpose: demonstrating effectiveness of a model/approach, good for replication, informing the field, testing assumptions to check strategy
    • 83. Process evaluation purpose: to improve implementation/execution at key times
    • 84. Participatory evaluation purpose: to empower participants with information to act
    • 85. Developmental evaluation: to identify components of a model or approach in real time
    Getting useful data requires choosing the right methods or
    approaches to measurement given what you need to do with it
    © 2010 Bill & Melinda Gates Foundation |
    21
    Example of purpose driven measurement
  • 86. Inputs
    Activities
    Outputs
    Outcomes
    Impacts
    Strategy
    Initiative
    Sub
    -
    Initiative
    Grant
    Sub
    -
    Grant
    It is an approach to measure results of our strategies
    • How is this different from the measurement challenge facing implementing
    organizations (grantees)?
    © 2010 Bill & Melinda Gates Foundation |
    22
  • 87. © 2010 Bill & Melinda Gates Foundation |
    What’s the essence of the approach?
    Measure impact (change in people’s lives) to track collective progress toward targets, not to attribute impact to the foundation’s actions or investments.
    Strategy
    Measure key output and outcome results to track progress of the foundation’s work; use evaluation to inform decision making by testing assumptions, learning what works, how and why.
    Initiatives
    Selectively track results of grants at critical points for accountability; evaluate to learn about implementation of key investments; use impact evaluation when it seeks to answer a strategic question for an initiative not as an accountability tool.
    Grants
    23
  • 88. Measuring investment types
    Measurement Guidelines Specific to Investment Type
    Types of Investment
    Priority for evaluation resources
    Research
    Product Development
    Policy and Advocacy
    Model Development
    and Demonstration of
    Effectiveness
    Systems Change
    Delivery at Scale
    Measure desired outcomes,
    track progress, and focus
    on shorter-term feedback
    Monitor outputs and track process
    Track execution and coverage/reach
    Measure success or failure and extent of fit with target product profile
    Measure attribution where it is technically feasible and ethical
    Use multiple measurement
    methods to draw conclusions
    © 2010 Bill & Melinda Gates Foundation |
    24
  • 89. Our questions…
    How do we identify, articulate, and evaluate alternative paths to impact?
    How do we develop theories of change where evidence of what works is slim?
    How do we keep the bar high and hold people accountable for results, while leaving room for failure and learning?
    data & experience
    data & experience
    What results do we aggregate? What are the most promising interventions to evaluate?
    data & experience
    data & experience
    How do we adjust our strategies while maintaining continuity in our partnerships?
    © 2010 Bill & Melinda Gates Foundation |
    25
  • 90. Thank You
    © 2010 Bill & Melinda Gates Foundation. All Rights Reserved. Bill & Melinda Gates Foundation is a registered trademark in the United States and other countries.