Impact Workshop Slides - Taylor Newberry Consulting

512 views

Published on

2013 OVCN INNOVATION & ACTION! Conference
'If Demonstrating Impact Seems Boring, You're Doing it Wrong' facilitated by Andrew Taylor of Taylor Newberry Consulting Inc.
http://taylornewberry.ca/
#OVCNaction

Published in: Business, Technology
1 Comment
0 Likes
Statistics
Notes
  • Thanks OVCN for posting the slides! It was a great discussion.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total views
512
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Impact Workshop Slides - Taylor Newberry Consulting

  1. 1. If Demonstrating Impact Seems Boring, You’re Doing it Wrong  A PRESENTATION FOR THE OVCN INNOVATION & ACTION! CONFERENCE
  2. 2. Pre - Test !  When someone says “lets talk about demonstrating impact,” I: Get excited and energized. !  Get moderately enthusiastic. !  Picture my long to do list getting even longer, sigh, and say “Ok, sure. I guess that’d probably be a good thing for me to think about.” !  Make a mental note to bring my phone to the meeting so I can get caught up on some emails. !  Think “just kill me now.” ! 
  3. 3. Andrew Taylor Contact info ANDREW@TAYLORNEWBERRY.CA 519 546 4789 @TAYLORNEWBERRYC Evidence Insight Action
  4. 4. A Basic Overview of Outcomes Measurement
  5. 5. Consultation Partnership Building Achieving Buy-in Connecting Visioning and Reflecting Program Planning; Strategic Planning; Program Design and Implementation Acting Listening Needs Assessment Outcome Measurement Program Evaluation
  6. 6. Setting the stage for good program monitoring 1.  2.  3.  Get buy-in first. Make sure you and your partners understand your program and its outcomes clearly and consistently. Remember that the key is using the data, not gathering the data. You don’t necessarily need to gather more data.
  7. 7. 1. Get Buy-in First ENGAGING WITH STAKEHOLDERS
  8. 8. Thinking About Use before You Start ! Who do you want to act on the basis of your outcome measurement findings? ! How do you want them to act?
  9. 9. Why are people not interested in outcome measurement? We tend to assume: But the real reasons may be: They don’t have the training to understand They know that the big decisions aren’t really based on outcome measurement findings (despite the rhetoric). They aren’t interested in learning They see measurement as complicated, dry and boring. They don’t see how it will help them to learn. They’ve got something to hide They worry that people won’t understand the context. They are afraid of being unfairly judged. They are too busy They see no tangible “payoff” for the significant time investment.
  10. 10. Stakeholder How might evaluation How might evaluation help them? make their lives difficult? Exercise: What’s in it for them? Board Member Manager Front-line Staff Member Participant in a Program Funder Partner Organization Volunteer
  11. 11. Make Sure You Understand the Program Clearly and Consistently CLARIFYING THEORIES OF CHANGE
  12. 12. Theories of Change !  The beliefs or assumptions that inform the design of a program or an intervention !  In particular, those beliefs and assumptions about how change happens, and how the intervention will lead to change. !  Sometimes theories of change are very explicit and structured, but sometimes they are implicit and emergent.
  13. 13. The importance of short-term outcomes Program Activities Train volunteer coordinators Long-Term Outcome or Goal Publicize volunteer positions Create a more engaged, connected community
  14. 14. The importance of short-term outcomes Program Activities Members manage volunteers better Publicize volunteer positions Train volunteer coordinators Members use other best practices Short-Term Outcomes Volunteers have more meaningful experiences Members are Stronger Long-Term Outcome or Goal More people volunteer Volunteers are more engaged Create a more engaged, connected community More diversity in volunteers
  15. 15. Breaking Down the Complexity What were we trying to change? (Long-Term Outcome Objectives) Did our program have an impact??? What particular contribution were we going to make to that change? (Short-Term Outcome Objectives) How were we planning to make that contribution? (Activities) What sorts of things would we see if the expected change was happening? (Indicators) How can we document those observations in a systematic way? (Methods)
  16. 16. Problems with Logic Models !  They are dense !  They are abstract !  They talk about change in a very linear, mechanical way !  They are good for getting your own thinking organized, but not so good for communicating with outsiders, especially if those outsiders ALSO have a complex program logic model that they want to communicate with YOU.
  17. 17. Hello. My life goal is to getI am new to town. I Hello. married and have kids. I am many people, or don’t know currently pursuing many places to go, and I’m several short term outcomes related toa bit lonely. My feeling this goal, including meeting a objectives are to outcome nice person, getting in better make some friends and get physical shape, moving out of My goal is to feel out more. my parents’ house, and connected to my new more choosing names for the kids. community, although I recognize that this goal may need to be made more measurable.
  18. 18. Yes! There’s a great place downtown. Want to go after work? Me too, actually! I’d love to live downtown someday. Know anywhere good to go for coffee around here? Sounds great! I’ve been wanting to get to know that neighbourhood.
  19. 19. What are the most important shared short-term outcomes for volunteer centres? What are the short-term outcomes we are well positioned to achieve that “plug in” to the shared “collective theory of change?” What are the shared outcomes that the community has identified as important? What is the shape of the “collective theory of change?”
  20. 20. The Really Useful Idea Identify short-term outcome objectives that are within your control, measurable and achievable, and use a theory of change to explain how these short-term achievements contribute to the achievements of collective outcomes over time.
  21. 21. What do we do? What do we do? What’s the big, shared change that matters to the community? What do we do? What do we do?
  22. 22. 3. Remember that the key is using the data, not gathering the data. MAKING SMART USE OF SIMPLE DATA
  23. 23. What are Indicators? !  Bits of information that provide part of the answer to one of your questions !  Things you can see or touch or hear – things that are observable in the world and don’t involve ‘interpretation’ !  Are often numbers but can also be (e.g.) stories, quotations, examples, pictures
  24. 24. Indicators are a Useful Idea Because... !  They help to break down the complex task of “outcome measurement” into manageable chunks !  They help others to understand what you mean in practical terms when you talk about a particular outcome !  They help you build up a strong meaurement plan by combining different kinds of information from several sources
  25. 25. The Notion of Triangulation
  26. 26. A Quick Way to See if You’ve got the Right Kind of Indicators (from Friedman, 2005) Quantity Quality How much work did we do? # of people who attended the workshop # of handouts created # of workshops run # How well did we do it? % of invited people who attended % % % Is anybody better off? # # # # % of participants who are better prepared to complete outcome measurement plans % of measurement plans that are useful % of plans that are manageable and practical
  27. 27. Different kinds of research on social programs
  28. 28. Measuring outcomes through ongoing program monitoring ! Carried out by program staff and volunteers on a routine basis ! Simple, easy to use methods ! Focused on short-term outcomes ! Usually measures a few basic things well, rather than many different things
  29. 29. Measuring outcomes through ongoing program monitoring ! Ideally, becomes a seamless element of program delivery ! Periodic analysis of data ! Focuses on ongoing improvement of programs
  30. 30. Outcome measurement planning sheet What are the questions you need answered? What indicators What additional How can you get can you use information do it? from you need? information you’ve already got? These may be Be as specific as questions about you can! outcomes, but they may also be questions about process, lessons learned, etc. Think stories and examples and also quantitative data What is the simplest, least intrusive method?
  31. 31. Making Your Evaluation “Worth It:” SHARING FINDINGS
  32. 32. Its all about “messaging” !  The key to powerful communication about evaluation findings is to know: Who your audience is !  What that audience is already passionate about, and how that passion connects to your work !  How you want them to act with the knowledge you give them !  How you can present your information in the way that maximizes the chances of them acting ! 
  33. 33. Consultation Partnership Building Achieving Buy-in Connecting Visioning and Reflecting Program Planning; Strategic Planning; Program Design and Implementation Acting Listening Needs Assessment Outcome Measurement Program Evaluation
  34. 34. Outcomes Measurement: Why do it?
  35. 35. The challenges !  Steep learning curve !  Time consuming !  Different expectations from different funders !  Lack of control over what you measure and how you measure it !  Funders that demand certain indicators but don’t explain what they think those indicators mean !  Differing jargon across funders !  Knowing how much measurement is “enough” and what type is expected !  Poor quality of data !  Lack of obvious “payoff”
  36. 36. The (potential) benefits !  More mindful practice !  more engaged, satisfied staff and volunteers !  A better sense of “where you fit” in the system of services and supports !  Improved programming; more benefits for the people you serve !  Improved capacity to explain the value of your work to donors, volunteers, board members, and the general public
  37. 37. Yeah, but …. QUESTIONS? COMMENTS?
  38. 38. Where do I start? 1.  2.  3.  Get buy-in first. Make sure you and your partners understand your program and its outcomes clearly and consistently. Remember that the key is using the data, not gathering the data. You don’t necessarily need to gather more data.
  39. 39. Post - Test !  When someone says “lets talk about demonstrating impact,” I: Get excited and energized. !  Get moderately enthusiastic. !  Picture my long to do list getting even longer, sigh, and say “Ok, sure. I guess that’d probably be a good thing for me to think about.” !  Make a mental note to bring my phone to the meeting so I can get caught up on some emails. !  Think “just kill me now.” ! 

×