2013 OVCN INNOVATION & ACTION! Conference 'If Demonstrating Impact Seems Boring, You're Doing it Wrong' facilitated by Andrew Taylor of Taylor Newberry Consulting Inc. http://taylornewberry.ca/ #OVCNaction

1. If Demonstrating Impact Seems
Boring, You’re Doing it Wrong
A PRESENTATION FOR THE
OVCN INNOVATION & ACTION!
CONFERENCE

2. Pre - Test
! When someone says “lets talk about demonstrating
impact,” I:
Get excited and energized.
! Get moderately enthusiastic.
! Picture my long to do list getting even longer, sigh, and say
“Ok, sure. I guess that’d probably be a good thing for me to
think about.”
! Make a mental note to bring my phone to the meeting so I can
get caught up on some emails.
! Think “just kill me now.”
!

3. Andrew Taylor
Contact info
ANDREW@TAYLORNEWBERRY.CA
519 546 4789
@TAYLORNEWBERRYC
Evidence
Insight
Action

4. A Basic Overview of Outcomes
Measurement

5. Consultation
Partnership Building
Achieving Buy-in
Connecting
Visioning and
Reflecting
Program Planning;
Strategic Planning;
Program Design
and
Implementation
Acting
Listening
Needs Assessment
Outcome Measurement
Program Evaluation

6. Setting the stage for good program monitoring
1.
2.
3.
Get buy-in first.
Make sure you and your partners
understand your program and its
outcomes clearly and consistently.
Remember that the key is using the
data, not gathering the data. You
don’t necessarily need to gather more
data.

7. 1. Get Buy-in First
ENGAGING WITH STAKEHOLDERS

8. Thinking About Use before You Start
! Who do you want to act on the
basis of your outcome
measurement findings?
! How do you want them to act?

9. Why are people not interested
in outcome measurement?
We tend to
assume:
But the real reasons may be:
They don’t have
the training to
understand
They know that the big decisions aren’t really based
on outcome measurement findings (despite the
rhetoric).
They aren’t
interested in
learning
They see measurement as complicated, dry and
boring. They don’t see how it will help them to
learn.
They’ve got
something to
hide
They worry that people won’t understand the
context. They are afraid of being unfairly judged.
They are too
busy
They see no tangible “payoff” for the significant time
investment.

10. Stakeholder
How might evaluation How might evaluation
help them?
make their lives
difficult?
Exercise: What’s in it for them?
Board Member
Manager
Front-line Staff
Member
Participant in a
Program
Funder
Partner
Organization
Volunteer

11. Make Sure You Understand the
Program Clearly and Consistently
CLARIFYING THEORIES OF CHANGE

12. Theories of Change
! The beliefs or assumptions that inform the
design of a program or an intervention
! In particular, those beliefs and assumptions
about how change happens, and how the
intervention will lead to change.
! Sometimes theories of change are very
explicit and structured, but sometimes they
are implicit and emergent.

13. The importance of short-term outcomes
Program Activities
Train
volunteer
coordinators
Long-Term Outcome or Goal
Publicize
volunteer
positions
Create a more engaged,
connected community

14. The importance of short-term outcomes
Program Activities
Members
manage
volunteers
better
Publicize
volunteer
positions
Train
volunteer
coordinators
Members
use other
best
practices
Short-Term Outcomes
Volunteers
have more
meaningful
experiences
Members
are Stronger
Long-Term Outcome or Goal
More
people
volunteer
Volunteers
are more
engaged
Create a more engaged,
connected community
More
diversity in
volunteers

15. Breaking Down the Complexity
What were we trying to change?
(Long-Term Outcome Objectives)
Did our program have
an impact???
What particular contribution were we
going to make to that change?
(Short-Term Outcome Objectives)
How were we planning to make that
contribution? (Activities)
What sorts of things would we see if
the expected change was
happening? (Indicators)
How can we document those
observations in a systematic way?
(Methods)

16. Problems with Logic Models
! They are dense
! They are abstract
! They talk about change in a very linear, mechanical
way
! They are good for getting your own thinking
organized, but not so good for communicating with
outsiders, especially if those outsiders ALSO have a
complex program logic model that they want to
communicate with YOU.

17. Hello. My life goal is to getI am new to town. I
Hello.
married and have kids. I am many people, or
don’t know
currently pursuing many places to go, and I’m
several
short term outcomes related toa bit lonely. My
feeling
this goal, including meeting a objectives are to
outcome
nice person, getting in better
make some friends and get
physical shape, moving out of My goal is to feel
out more.
my parents’ house, and connected to my new
more
choosing names for the kids.
community, although I
recognize that this goal may
need to be made more
measurable.

18. Yes! There’s a great
place downtown.
Want to go after
work?
Me too, actually!
I’d love to live
downtown someday.
Know anywhere
good to go for coffee
around here?
Sounds great! I’ve
been wanting to get
to know that
neighbourhood.

19. What are the most important shared short-term
outcomes for volunteer centres?
What are the short-term outcomes we are well
positioned to achieve that “plug in” to the shared
“collective theory of change?”
What are the shared outcomes that the community has
identified as important? What is the shape of the
“collective theory of change?”

20. The Really Useful Idea
Identify short-term outcome objectives that are
within your control, measurable and achievable,
and use a theory of change to explain how these
short-term achievements contribute to the
achievements of collective outcomes over time.

21. What do we do?
What do we do?
What’s the big,
shared change
that matters to
the community?
What do we do?
What do we do?

22. 3. Remember that the key is
using the data, not gathering the
data.
MAKING SMART USE OF SIMPLE DATA

23. What are Indicators?
! Bits of information that provide part of
the answer to one of your questions
! Things you can see or touch or hear –
things that are observable in the world
and don’t involve ‘interpretation’
! Are often numbers but can also be (e.g.)
stories, quotations, examples, pictures

24. Indicators are a Useful Idea
Because...
! They help to break down the complex task of
“outcome measurement” into manageable
chunks
! They help others to understand what you
mean in practical terms when you talk about
a particular outcome
! They help you build up a strong meaurement
plan by combining different kinds of
information from several sources

25. The Notion of
Triangulation

27. A Quick Way to See if You’ve got the Right Kind
of Indicators (from Friedman, 2005)
Quantity
Quality
How much work did we do?
# of people who attended the workshop
# of handouts created
# of workshops run
#
How well did we do it?
% of invited people who attended
%
%
%
Is anybody better off?
#
#
#
#
% of participants who are better
prepared to complete outcome
measurement plans
% of measurement plans that are useful
% of plans that are manageable and
practical

28. Different kinds of research on social programs

29. Measuring outcomes through ongoing program
monitoring
! Carried out by program staff and
volunteers on a routine basis
! Simple, easy to use methods
! Focused on short-term outcomes
! Usually measures a few basic things
well, rather than many different things

30. Measuring outcomes through ongoing program
monitoring
! Ideally, becomes a seamless element of
program delivery
! Periodic analysis of data
! Focuses on ongoing improvement of
programs

31. Outcome measurement planning sheet
What are the
questions you
need answered?
What indicators What additional How can you get
can you use
information do it?
from
you need?
information
you’ve already
got?
These may be
Be as specific as
questions about
you can!
outcomes, but they
may also be
questions about
process, lessons
learned, etc.
Think stories and
examples and also
quantitative data
What is the
simplest, least
intrusive method?

32. Making Your Evaluation
“Worth It:”
SHARING FINDINGS

33. Its all about “messaging”
! The key to powerful communication about
evaluation findings is to know:
Who your audience is
! What that audience is already passionate about, and how that
passion connects to your work
! How you want them to act with the knowledge you give them
! How you can present your information in the way that
maximizes the chances of them acting
!

34. Consultation
Partnership Building
Achieving Buy-in
Connecting
Visioning and
Reflecting
Program Planning;
Strategic Planning;
Program Design
and
Implementation
Acting
Listening
Needs Assessment
Outcome Measurement
Program Evaluation

35. Outcomes Measurement:
Why do it?

36. The challenges
! Steep learning curve
! Time consuming
! Different expectations from different funders
! Lack of control over what you measure and how you measure it
! Funders that demand certain indicators but don’t explain what
they think those indicators mean
! Differing jargon across funders
! Knowing how much measurement is “enough” and what type is
expected
! Poor quality of data
! Lack of obvious “payoff”

37. The (potential) benefits
! More mindful practice
! more engaged, satisfied staff and volunteers
! A better sense of “where you fit” in the system of services and
supports
! Improved programming; more benefits for the
people you serve
! Improved capacity to explain the value of your work
to donors, volunteers, board members, and the
general public

38. Yeah, but ….
QUESTIONS? COMMENTS?

39. Where do I start?
1.
2.
3.
Get buy-in first.
Make sure you and your partners
understand your program and its
outcomes clearly and consistently.
Remember that the key is using the
data, not gathering the data. You
don’t necessarily need to gather more
data.

40. Post - Test
! When someone says “lets talk about demonstrating
impact,” I:
Get excited and energized.
! Get moderately enthusiastic.
! Picture my long to do list getting even longer, sigh, and say
“Ok, sure. I guess that’d probably be a good thing for me to
think about.”
! Make a mental note to bring my phone to the meeting so I can
get caught up on some emails.
! Think “just kill me now.”
!