Interim Outcomes to Measure Progress toward Significant Policy Change

1. Presentation
Notes
“The
Six
Conditions:
Interim
Outcomes
to
Measure
Progress
toward
Significant
Policy
Change”
Kristi
Kimball
Hewlett
Foundation
Contact:
kristi
DOT
kimball
AT
gmail
DOT
com
Presented
at
Meeting
of
Education
Funders
Strategy
Group
March
2011
Also
previously
presented
at:
Hewlett
Foundation
Conference
on
Evaluating
Advocacy
July
2010
American
Evaluator’s
Association
Annual
Conference
Session
on
“Advocacy
Evaluation:
Identifying
and
Using
Interim
Outcomes
to
Tell
the
Whole
Story”
November
2009
1. A
philanthropic
perspective
-­‐-­‐
thoughts
on
assessing
progress
towards
policy
change
• My
background
–
7
years
at
Hewlett
Foundation
investing
in
organizations
working
on
state
level
policy
change
to
improve
education
outcomes
for
California
students.
(Past
work
in
Congress,
the
Federal
Dept
of
Education,
policy
research/evaluation,
and
state/local
level
policy
advocacy).
• Disclaimer
–
I’m
going
to
discuss
the
whole
policy
change
process,
with
recognition
that
foundations
are
limited
by
state
and
federal
laws
in
terms
of
the
activities
they
can
support
in
the
policy
arena.
Foundations
can
support
a
wide
range
of
policy
research,
analysis,
communications
and
advocacy
work,
but
foundations
cannot
earmark
grant
funds
for
lobbying
or
lobby
themselves.
My
comments
today
are
intended
to
benefit
all
actors
engaged
in
policy
change
work,
recognizing
we
all
have
different
roles
in
the
policy
process.
• Mismatch
between
timeframe
of
policy
change
process
and
timeframe
in
which
funders
want
impact
data.

It
can
take
a
long
time
to
achieve
a
significant
policy
change
(10-­‐20
yrs)

But
foundations
increasingly
want
concrete
evidence
of
impact
in
the
short
term
(1-­‐3
yrs);

So
there
is
danger
that
foundations
will
lose
patience
and
abandon
1
Copyright
©
2012
Kristi
Kimball

2.
promising
policy
change
efforts
before
they
have
played
out.
•How
would
intermediate
progress
metrics
help?
If
we
had
better,
more
concrete
measures
of
the
intermediate
steps
before
a
policy
change
happens,
then…

Advocacy
organizations
would
be
better
able
to
demonstrate
measurable
progress
to
funders
annually
and
potentially
attract
more
funding;

Foundations
might
have
more
patience
to
continue
investing
in
grantees
working
on
long-­‐term
campaigns;
and

It
would
also
be
easier
to
recognize
when
policy
change
efforts
have
stalled
or
lost
ground
over
time
and
need
a
new
approach.
2. Which
intermediate
outcomes
matter
most?
• Some
good,
comprehensive
lists
of
intermediate
outcomes
for
advocacy
work
and
measurement
tools
have
been
developed
by
Julia
Coffman
at
Harvard
and
by
the
presenters
on
this
panel.
These
are
great
steps
forward.
• A
remaining
challenge
is
that
we
don’t
know
which
intermediate
outcomes
really
matter
and
which
are
less
important.
So
as
a
result…
 We
don’t
know
if
achieving
a
greater
number
of
intermediate
outcomes
(from
a
menu
of
many
possible
intermediate
outcomes)
is
more
likely
to
lead
to
policy
change;
 Or
if
spending
a
lot
of
time
and
energy
on
a
few
key
intermediate
outcomes
is
more
likely
to
lead
to
policy
change.
• We
need
a
theory
about
the
intermediate
steps
necessary
for
significant
policy
changes
-­‐-­‐
so
we
can
have
confidence
that
the
intermediate
outcomes
we
track
are
meaningful
and
actually
increase
the
likelihood
of
a
successful
policy
change.
• I
have
been
working
with
two
consultants
to
develop
a
set
of
intermediate
progress
metrics
along
these
lines.
 Started
from
academic
theory
(e.g.
Kingdon’s
theory
of
policy
windows);
 Also
considered
our
experience
from
recent
policy
grantmaking
in
CA;
 Developed
generic
set
of
intermediate
steps
towards
policy
change
that
might
hold
across
different
campaigns
and
different
issues;
2
Copyright
©
2012
Kristi
Kimball

3.  Examined
lots
of
historical
cases
to
test
whether
these
intermediate
steps
hold
generally
and
how
they
could
be
measured
concretely.
We
studied
21
policy
state
reform
campaigns
that
have
taken
place
in
California
over
the
past
32
years.
(Wide
range
including:
education,
environment,
transportation,
energy,
healthcare,
property
tax
reform,
governance
reforms,
etc).
Most
of
these
efforts
(16)
were
ultimately
successful
but
included
some
failures
along
the
way
–
some
spanning
more
than
30
years
time.
Five
were
not
ultimately
successful
(but
are
still
“works
in
progress”).
 Focused
on
significant
/
“heavy
lift”
policy
change
cases
that
either:
1)
shift
lots
of
government
funding;
or
2)
substantially
threaten
powerful
interests.
 Used
qualitative
methods
-­‐-­‐
interviews
with
key
actors,
review
of
news
articles,
polling,
reports
and
other
documentation
of
the
cases.
• Looking
across
the
historical
cases,
we
refined
the
list
of
intermediate
outcome
measures
to
produce
the
following
six,
which
we
think
could
serve
as
generic
measures
of
progress
toward
a
significant
policy
change
in
many
different
cases.
 Define
the
problem
and
create
a
sense
of
urgency
• Achieving
this
may
require
-­‐-­‐
research,
media
work,
communications
and
outreach
• Possible
measures
-­‐-­‐
public
opinion
polling,
policymaker
polling,
media
coverage
 Develop
a
specific
solution
and
build
consensus
• Achieving
this
may
require
-­‐-­‐
research,
media
work,
communications
and
outreach.
Also
policy
analysis,
technical
policy
development,
and
negotiation
with
key
interests
• Possible
measures
-­‐-­‐
public
opinion
polling,
policymaker
polling,
special
interest
positions,
media
coverage
 Strong
“outside
champion”
launches
a
campaign
(aka
“policy
entrepreneur”
in
Kingdon’s
terms)
• Usually
an
ngo
org,
not
just
an
individual
/could
be
multiple
orgs.
•
This
role
often
requires
-­‐-­‐
advocacy,
media
work,
coalition
building,
playing
a
leadership/
hub
role
for
a
campaign,
etc.
• Possible
measures
–
recognition
in
policy
arena
of
champion
org
as
the
“go-­‐to”
org
on
this
issue.
Champion
org
has
24/7
focus
and
proactive
approach
to
the
campaign.
Champion
org
is
credible
and
effective
with
media,
policymakers,
stakeholders.
3
Copyright
©
2012
Kristi
Kimball

4.
 Strong
support
coalition
is
created
and
mobilized
• Possible
measures
-­‐-­‐
size,
diversity,
bipartisanship,
credibility,
and
$
 “Inside
champion(s)”
(policymaker
leadership)
commit
to
action
• Ex
-­‐-­‐
governor,
party
leaders
in
legislature,
other
top
decision-­‐makers
commit
publicly
to
address
problem/
support
solution
(can
be
measured
through
speeches,
floor
statements,
press
releases,
media
interviews,
etc).
• Role
may
also
include
-­‐-­‐
hosting
meetings
or
hearings,
sponsoring
policy
proposals,
and
advocating
with
other
decision-­‐makers.
 Major
opposition
is
addressed
/
minimized.
• Possible
measures
–
no
powerful,
credible
opponents
(declared
or
undeclared)
working
actively
against
the
policy
solution.
• No
big
spending
to
oppose
the
change.
“Big
spending”
here
is
relative
to
the
amount
of
spending
on
the
support
side.
• We
find
that
these
conditions
have
developed
in
all
of
the
successful
cases
that
we
have
studied,
with
a
few
exceptions:
1)
some
measures
are
able
to
succeed
even
in
the
face
of
strong
opposition
(e.g.
Prop
13
had
strong
policymaker
opposition
rather
than
support
and
was
ultimately
successful
anyway;
Pavley
measure
–
had
very
strong
and
well-­‐funded
auto
industry
opposition
and
was
ultimately
successful
anyway,
etc);
and
2)
the
support
of
policymaker
leadership/
“inside
champions”
was
not
a
necessary
condition
for
many
of
the
successful
ballot
initiatives
we
studied,
however
it
did
show
up
as
a
critical
factor
for
all
of
the
legislative
victories.
• We
also
find
that
one
or
more
conditions
are
missing
or
not
robustly
developed
in
most
of
the
major
failure
cases
that
we
studied.
Further,
the
failure
cases
often
preceded
eventual
policy
change
successes,
and
we
also
see
evidence
of
the
strengthening
of
the
6
conditions
in
these
cases
over
time
between
failures
and
successes.
• The
presence
of
these
6
conditions
certainly
does
not
guarantee
that
a
major
policy
change
will
happen,
but
our
research
suggests
that
these
conditions
(when
robustly
developed)
do
probably
increase
the
likelihood
of
a
successful
change.
So
I
think
of
these
as
“necessary
enabling
conditions”
for
significant
policy
change
to
happen.
They
are
necessary,
but
not
sufficient.
• I
believe
these
six
conditions
can
be
measured
concretely,
with
a
little
creativity
in
terms
of
evaluation
methods.
Collecting
data
on
the
6
conditions
could
be
an
impt
4
Copyright
©
2012
Kristi
Kimball

5. way
to
help
funders
understand
the
state
of
the
field
in
which
they’re
investing,
and
to
understand
where
investments
are
needed
to
make
success
more
likely.
• These
conditions
do
not
necessarily
develop
in
a
linear
way.
They
can
develop
simultaneously.
They
can
interact
with
each
other.
They
can
regress.
There
is
enormous
complexity
in
the
policy
change
process
(bc
of
the
large
number
of
actors
and
factors
at
work)
that
often
makes
the
process
non-­‐linear.
• There
are
many
outside
factors
and
influences
at
work
in
establishing
these
enabling
conditions.
It
wouldn’t
be
reasonable
to
hold
any
one
organization
or
collection
of
organizations
accountable
for
controlling
these
conditions.
The
best
approach
is
to
measure
an
organization’s
“contribution”
to
creating
these
conditions,
rather
than
expecting
proof
of
causality
and
“attribution.”
• We
are
just
in
the
beginning
stages
of
discussing
this
research
and
the
enabling
conditions
publicly,
so
this
is
very
much
a
work
in
progress,
and
I
am
very
much
looking
forward
to
feedback
from
folks
in
the
room
today.
The
metrics
described
above
relate
to
measuring
progress
at
the
field
level
–
in
a
way
that
captures
the
impact
of
many
organizations
collectively.
This
kind
of
information
is
critical
to
assess
the
overall
impact
of
a
foundation’s
investment
portfolio.
We
have
also
funded
some
individual
grantee
evaluations
to
measure
outcomes
that
are
meaningful
for
their
specific
projects,
which
might
focus
on
developing
only
one
or
two
of
the
enabling
conditions.
5
Copyright
©
2012
Kristi
Kimball