This presentation, shared at the 2015 AEA conference in November 2015, provides a high-level overview of key issues facing evaluators as they do multi-stakeholder advocacy evaluation. This includes:
• Parsing out the nuance of advocacy strategy in multi-stakeholder efforts
• Dividing up credit or contribution of the multiple partners
• Understanding the right evaluation parameters given multiple stakeholder perspectives and values
• Balancing interpersonal dynamics as the evaluator
The presentation also shares insights gained from evaluating a number of large multi-stakeholder advocacy initiatives.
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Advocacy as a Team Game -- Evaluating Multi-Stakeholder Advocacy: Overview of Key Issues of Multi-Stakeholder Advocacy
1. Advocacy as a Team Game—
Evaluating Multi-Stakeholder Advocacy:
Overview of Key Issues of Multi-Stakeholder Advocacy
Evaluation 2015: Exemplary Evaluations in a Multicultural World
November 13, 2015
Presenter:
Jared Raynor,
TCC Group
@jraynor1
2. Strategies: Outcomes: Impact:Project Inputs/
Resources:
Logic Model: Advocacy Initiatives
Issue analysis/
research
Media advocacy/
Public awareness
raising
Grass roots
organizing
Coalition building/
networking
Policy analysis/
research
Legal action
Assure good policy
implementation
(Administrative/
Regulatory oversight,
TA, monitoring, etc.)
Lobbying and direct
policy-maker
influence
ImprovementintheQualityofLivingfortheCommunity
Better defined and
framed problems
Policy decisions
effectively implemented
Policy makers adopt new
policies based on the
agenda / do not adopt
harmful policies
Issues more clearly set
on the public agenda and
gain prominence
(momentum, interest,
awareness, etc.)
Impact of new policies is
evaluated
Advocacy Staff:
•Time, experience and
expertise (administrative,
legislative, election-related
and legal)
•Core skills (analytical,
communication, research,
etc.)
Funding:
•Monetary resources
•Non-monetary/ non-staff
resources (e.g. volunteers,
in-kind donations)
Organizational Reputation:
•Advocacy related
•General perception as a
quality organization
•Representative for base
constituency
Network:
•Policy makers
•Partner organizations
•Media
•Mobilized base
(C) TCC Group 2010
ChangeinSocialStructureReflectingPositiveSocialJusticeShift
Coalition/Network
(asanActoras
comparedtoa
strategy)
NonprofitorAdvocacy
Organization(usescoalition
asastrategy)Acting
Organization:
4. What Are We Evaluating?
Organization
Coalition/
Network
External
Environment
Individual
5. What you see depends on where you sit:
5 Evaluation Models in Campaigns
1) Retrospective
2) Formative
3) Developmental
4) Technical Assistance
5) Program Officer
6. What Challenges Does That Leave?
• Understanding Advocacy Strategy
• Parsing Impact
• Defining the Evaluation Parameters
• Balancing Interpersonal Dynamics
11. Solutions We Will Hear About
Leveraging
the
evaluator
The players
and how they
position
themselves
Contribution
analysis
Field
framing
Audience
Discussion
12. Leveraging the Evaluator
• Navigate power differentials
• Use to:
– Build
– Sustain
– Reflect
• Understand how to position
organizations in a network
• Help articulate goal
destination and value
proposition
Editor's Notes
This is for an advocacy initiative. Implementing program work is not significantly different.
Understanding advocacy strategy: There remain several challenges in considering advocacy strategy, some of which overlaps with the parsing impact discussion below. One participant asked about the limits of what advocacy can accomplish while another asked about the implications of perceived polarization in the political climate and whether such polarization had significant impact beyond Washington, DC. Other participants raised ongoing questions about more sacrosanct practices: To what extent is planning an important part of effective advocacy? What is the real strategic value of policy briefs? And, What is the difference between making good bets on individual activists versus engaging in movement building more deliberately?
Parsing Impact: At the heart of evaluation is the evaluative judgment and some participants identified several ongoing challenges. These include getting beyond the platitude of “contribution not attribution” common in much of advocacy to the more gritty “how much contribution is required for attribution?” Similarly, how do you know when you have saturation in a particular area—where you would start to see diminishing returns from increased efforts? Another participant asked about considering the opportunity cost of doing/not doing certain things, raising the complex issue of value comparison in multi-stakeholder advocacy.
Defining the evaluation parameters: Understanding what one is evaluating was another area raised in regard to multi-stakeholder advocacy. One participant asked the question about why, if at all, is it important to name things like field-building (as opposed to just doing the work without naming it) while a colleague questioned the strategic difference between developing a “deep advocacy bench” versus creating a movement. Another asked about boundary setting—who gets counted in or out when doing multi-stakeholder assessment?
Balancing interpersonal dynamics: Finally, as might be imagined when there are multiple personalities at play, some participants described a need for an ongoing focus on how to work effectively with stakeholder groups. One participant asked the question in relation to managing funder power dynamics while another asked about the dynamics involved in a small place with large amounts of interconnectedness.
Understanding advocacy strategy: There remain several challenges in considering advocacy strategy, some of which overlaps with the parsing impact discussion below. One participant asked about the limits of what advocacy can accomplish while another asked about the implications of perceived polarization in the political climate and whether such polarization had significant impact beyond Washington, DC. Other participants raised ongoing questions about more sacrosanct practices: To what extent is planning an important part of effective advocacy? What is the real strategic value of policy briefs? And, What is the difference between making good bets on individual activists versus engaging in movement building more deliberately?
Parsing Impact: At the heart of evaluation is the evaluative judgment and some participants identified several ongoing challenges. These include getting beyond the platitude of “contribution not attribution” common in much of advocacy to the more gritty “how much contribution is required for attribution?” Similarly, how do you know when you have saturation in a particular area—where you would start to see diminishing returns from increased efforts? Another participant asked about considering the opportunity cost of doing/not doing certain things, raising the complex issue of value comparison in multi-stakeholder advocacy.
Defining the evaluation parameters: Understanding what one is evaluating was another area raised in regard to multi-stakeholder advocacy. One participant asked the question about why, if at all, is it important to name things like field-building (as opposed to just doing the work without naming it) while a colleague questioned the strategic difference between developing a “deep advocacy bench” versus creating a movement. Another asked about boundary setting—who gets counted in or out when doing multi-stakeholder assessment?
Balancing interpersonal dynamics: Finally, as might be imagined when there are multiple personalities at play, some participants described a need for an ongoing focus on how to work effectively with stakeholder groups. One participant asked the question in relation to managing funder power dynamics while another asked about the dynamics involved in a small place with large amounts of interconnectedness.
Leveraging the evaluator (Jared)
The players and how they position themselves (Sue)
Contribution analysis (Carlyn)
Field framing (Jewlya)
Audience (moderated by David Devlin-Foltz)
Navigate power differentials
Use to:
Build (inform development)
Sustain (strengthen existing performance; identify inefficiencies)
Reflect (accountability for resources and prep for future)
Understand how to position yourself in a network
Help articulate goal destination and value proposition