Identifying Collective Impact Outcomes

Identifying
Collective Impact Outcomes
Isaac Castillo - @Isaac_outcomes 1
Isaac D. Castillo
Deputy Director
DC Promise Neighborhood Initiative
@Isaac_outcomes
Isaac.Castillo@dcpni.org
December 4, 2015

Learning Objectives
• What is Collective Impact?
• What type of Collective Impact are you talking
about?
• What does it take to measure outcomes in a
collective impact framework?
December 2015 Isaac Castillo - @isaac_outcomes 2

What is Collective Impact?
• As defined by FSG (the nonprofit consulting
firm that invented the term)…..
“Collective Impact is the commitment of a group
of actors from different sectors to a common
agenda for solving a complex social problem.”
December 2015 Isaac Castillo - @Isaac_outcomes 3

Five Conditions of Collective Impact
• Collective impact is more rigorous and specific than collaboration among
organizations. There are five conditions that, together, lead to meaningful
results from collective impact:
– Common Agenda: All participants share a vision for change that includes a
common understanding of the problem and a joint approach to solving the
problem through agreed-upon actions.
– Shared Measurement: All participating organizations agree on the ways success
will be measured and reported, with a short list of common indicators identified
and used for learning and improvement.
– Mutually Reinforcing Activities: A diverse set of stakeholders, typically across
sectors, coordinate a set of differentiated activities through a mutually reinforcing
plan of action.
– Continuous Communication: All players engage in frequent and structured open
communication to build trust, assure mutual objectives, and create common
motivation.
– Backbone Support: An independent, funded staff dedicated to the initiative
provides ongoing support by guiding the initiative’s vision and strategy, supporting
aligned activities, establishing shared measurement practices, building public will,
advancing policy, and mobilizing resources.
December 2015 4
Source: FSG Consulting website (group that developed collective impact concept)
Isaac Castillo - @Isaac_outcomes

But is your Collective Impact
the same as mine?
• Different people / groups interpret the
concept of ‘collective impact’ in many
different ways.
• I have found the three most common to be:
– One problem / one geographic area
– ‘Franchise model’ – multiple locations
– Focus on core group of individuals

One Problem / One Geographic Area
• Most commonly discussed form of collective
impact.
• Single large problem is identified for solution
in one area.
• Multiple agencies and organizations
(government, schools, community, nonprofits)
‘collectively’ try to solve problem.
• Attempt to measure ‘impact’ of their efforts in
solving the problem.
3/17/2014 DCPNI – Isaac Castillo - @Isaac_outcomes 6

Example
Reduce Gang
Crime in City
Mayor’s
Office
Government
Agencies
Residents
Funding
Community
Nonprofit
Organizations
Law
Enforcement
Schools Evaluators

Characteristics
• Commonly defined goal (with indicators) that
everyone agrees with.
• Multiple agencies/groups all working towards
this common goal.
• Some agency/group takes lead or serves as
coordinator.
• Not every agency/group contributes the same
– some do more than others.

Franchise Model – Multiple Locations
• Most common model (not talked about as
much).
• Same program/intervention replicated by
single organization in multiple locations.
• Larger organization is trying to determine the
‘impact’ across all of their sites / locations.
• The aggregated outcomes of each location is
then considered ‘collective’.

Example
Collective Impact=+++++

Characteristics
• Parent organization sets goal(s) and local
organizations agree to goal(s).
• Same program / approach in lots of locations.
• Parent organization responsible for quality
control / fidelity.

Core Group of Individuals
• Rare – put present in some locations.
• Specified group of service recipients that are
served my multiple organizations.
• Service recipients are pre-identified at the
individual level.
• Attempt to measure the ‘impact’ of all service
providers on the same participants.

Example
After-school Tutoring
Pre-Identified Youth Service Recipients
Recreation
Exercise
Programming
Health Education College Readiness
Nutrition
Programming

Characteristics
• Service providers need to agree upon
common set of outcomes/indicators.
• Some group / organization needs to
coordinate.
• Intent is to provide extensive services to a
limited set of participants.

Common Concerns for All
Collective Impact Approaches
• What kind of data will you collect?
– Demographic and/or socioeconomic
– Service amounts or types
– Outcomes or changes in conditions
• Who is responsible for reporting data (and what is their
motivation)?
• Who collects the data and in what format?
• Who trains people on proper data submissions?
• Who analyzes and then shares/reports data and
findings?
• Are there consequences for not sharing data (or not
sharing data in correct format or on time)?

How the Simple
Can Create a Challenge
• Define each data point along with acceptable
answer categories
– Gender – what will you ask for and what will you
report?
– Race / ethnicity – are these two different concepts
to be measured, or will they be grouped? And
what are you answer categories?
– Neighborhood, census tract, census block group,
or some other geographic area?

Agreeing on an Outcome
• Should come to consensus on what the
outcome is that you want to make progress
towards.
• Three at a maximum – the fewer the better.
• This will be the hardest part – but important
to gain consensus.
• Shouldn’t get too detailed or worry too much
on how to measure it. First just agree on what
you want to change.

Next Agree on Indicators
• Indicators are what you will measure to
determine if you are making progress towards
your outcomes.
• They are more detailed than outcomes and
can exist at multiple levels and time periods.

Examples of Indicators and Outcomes
• Outcome: Youth are college and career ready
– Indicator: Youth graduate from college
– Indicator: Youth graduate from high school
– Indicator: Youth are proficient in math and/or reading
on state assessments
• Outcome: Reduce poverty in defined area
– Indicator: % of adults at or below poverty line
– Indicator: % of children at or below poverty line
– Indicator: % of people living at 150% or less of poverty
level

How Will You Measure?
• Once you have your outcomes and indicators,
you can think about how to measure change.
• Will you….
– Use publicly available data? (And how regularly
available is it)?
– Collect data from multiple sources (and how will
you aggregate it)?
– Ask multiple people to submit data (and what is in
it for them)?

Kenilworth-Parkside Community
DCPNI’s Collective Impact Approach
Youth Parents
Residents

DCPNI Collective Impact
Characteristics
• DCPNI serves as coordinating agency.
• Some outcomes/indicators set by DCPNI
(Dept. of Ed.), others developed with partners.
• Will determine which combinations of
partners are most effective.

Measuring at the Community and
Individual Levels
• How do these indicators help DCPNI evaluate
its work in the community? That occurs at
two levels:
– At the community level – we look at change that
occurs over time for community level data.
– At the individual level – we look at how each child
and parent in the community improves their
outcomes over time.

Community Level Evaluation
• Examine community level change that occurs.
• Use existing data sets (Census, ACS, Vital
Statistics, Dept. of Health, Crime, etc.)
• Conduct primary data collection:
– Neighborhood survey
– School climate survey
• Community interviews and focus groups.
• Is there a clear link to changes observed and
our interventions?

Individual Level Evaluation
• Each partner / program defines success in
alignment with one (or more) of DCPNI’s
indicators.
• Each partner articulates outcomes and methods
to measure outcomes at individual (child,
student, parent) level.
• Each partner shares individual level demographic,
attendance, and outcome data with DCPNI.
• DCPNI combines with other data sets (school
data) to measure individual level change over
time.

Individual Level Evaluation Example
Math
Tutoring
Effective Education – Students Proficient in Math
Report Card Grades
Chess
Instruction
Critical Thinking
Skills
School
Standardized Test

Individual Level Evaluation Example
Tutoring
Effective Education – Students Proficient in Math
Report Card Grades
Chess
Critical Thinking
School
Standardized Test
How Does DCPNI Fit In?
Child 1
Child 3
Child 2



 

Individual Level Evaluation Insights
• Critical part is having student / resident names
(or unique identifiers).
• Need to have common outcomes / indicators.
• Each partner must have adequate data
collection and measurement capacity.
• Can be used to identify effective combinations
of programs.

Isaac’s Contact Information
30December 2015 Isaac Castillo - @Isaac_outcomes
Isaac D. Castillo
Deputy Director
DC Promise Neighborhood Initiative
On Twitter: @Isaac_outcomes
Email: Isaac.Castillo@dcpni.org

Identifying Collective Impact Outcomes

More Related Content

What's hot

Viewers also liked

Similar to Identifying Collective Impact Outcomes

More from Isaac Castillo

Recently uploaded

Identifying Collective Impact Outcomes