3ie funds high-quality impact evaluations that are policy-relevant and useful for decision-making. 3ie has learned that researchers must engage stakeholders early and throughout the process to increase the likelihood of findings being taken up and used. 3ie now requires researchers to develop policy influence plans and engage in ongoing discussions with implementing agencies to ensure studies answer relevant questions and produce feasible recommendations. While impact evaluations can provide compelling evidence, uptake is a political process and single studies rarely drive major policy changes.

1. Impact Evaluation for policy-making
Promoting uptake of
impact evaluation
findings: the importance
of relevance, utility and
engagement
What 3ie is learning
Beryl Leach, deputy director, 3ie
18 February 2015
Istanbul, Turkey

2. Overview
• Who we are and how we promote high-quality,
relevant and useful studies that are taken up
and used
• What we are learning about impact evaluation
and policy influence
• How think tanks can produce, engage and use
impact evaluation in policy advocacy

3. What is 3ie?
• International grant-making NGO founded in 2008
• Filling evidence gaps in what we know works,
why, how and the costs in development
• We fund particular types of evidence production
– high-quality experimental and quasi-experimental,
theory-based, mixed-method impact evaluations
– systematic reviews
– evidence syntheses
Why are IEs important for policy-making?

4. Policy relevance and usefulness
• Evidence that can improve the
effectiveness of development
policies and programmes
• Must be policy relevant
• We expect policy influence and
impact on policy and programming
because of how we approach
design and implementation

5. What do we mean by useful?
 RELEVANT: Helps answer a
specific policy question
 CONTEXTUAL: Appropriate
political context—it makes
sense
 CLEAR: message gives
options
 FEASIBLE: Affordable and
possible
 TIMELY: interest exists

6. 3ie funds high-quality
policy-relevant studies
3ie encourages
researchers to engage
with key stakeholders
Uptake of study
findings and improved
policies and practice
Applications
- Ask policy
relevant
question(s)
- Have potential
for policy impact
- Study team has
experience in
policy influence
ASSUMPTIONS
Researchers
-Committed
- Understand how
policy influence
happens
- Have the tools
and resources to
invest in policy
engagement
- Decision-makers
are interested
Study
-Makes policy-
relevant
recommendations
– Answers main
IE questions
- Proposes
feasible
solutions
- Had ongoing
engagement
ASSUMPTIONS ASSUMPTIONS

7. What we learned from analysing early
influencing in our grants
• 3ie needed to be directly engaging with
researchers and much earlier
• Researchers needed to be engaging with
implementers much earlier and differently
• Paper-driven processes were not effective
• 3ie approach needed to be based on evidence
about policy change and research uptake

8. Main ways we are ensuring policy
relevance and usefulness
• Early and direct engagement with implementing agencies
during the development of funding windows
• Require country nationals on the research teams in
substantive roles; more funding is for country-based
teams – role and opportunity for think tanks
• Preparation phase
• Direct engagement with researchers about PIPs from the
start and ongoing dialogue with the team

9. Communicating impact evaluation
evidence: ongoing, integrated, multi-level
• Engage from the start
• Explain the study and why it will be useful--build interest
• Report preliminary findings-get feedback-promotes
ownership
• Engage a range of key actors: beneficiaries and other
local actors, civil society, media from early phases
• Translate into plain language and produce in a variety of
formats and disseminate through multiple channels:
– Meetings, conferences, social media, multiple briefs, papers, reports
Think
Tanks

10. Impact evaluation and
think tank policy
influencing: opportunities
and constraints

11. Bridging researchers & policymakers
Translating, relationship building, networking

12. Using impact evaluation evidence for
policy advocacy: continued
• Constraints
• Two-sided coin
– Rigor as the basis for being high quality
and for sound decision-making
– IEs seek to measure causal
relationships, we need to be able to
assess validity
• Two-part problem
– Poor or weak designs mean you can’t
really know which cause
– Users have limited critical appraisal
capacity

13. Using impact evaluations for advocacy
• How much can you generalise based on one IE study?
– Not much, even though we stress generalisability a lot
– More than IE evidence needed to know about potential for scaling up
– Researchers should always bring knowledge of existing evidence to the
analysis, so that the new evidence is situated in the existing body of
evidence, doesn’t always happen
• Evidence for programme change can be more
immediately useful than for policy change
– Especially the case with theory-based evaluation, which is what we
require
– Modification to existing programme theory of change and operations

14. Benefits of increased production and use
of impact evaluations by think tanks
• IEs particularly need a strong emphasis on policy relevance and
utility that think tanks can provide
• More think tanks doing quality IEs and being effective
intermediaries in policymaking will help increase uptake
• Participating in IE teams and commissioning can strengthen
relevance and utility
• Big potential for improving think tank understanding of IE evidence
and critical appraisal to be knowledge translators in advocacy
process

15. Thank you
www.3ieimpact.org
#3ieNews #TTIX2015

16. END OF TTIX PRESENTATION

17. What are we looking at next?
• Developing our approaches to working with implementers
• Doing case studies on the cumulative impact of IEs done
over time: direct and indirect contributions to what types of
change
• Evaluating evidence uptake from 3ie studies
– Finding an appropriate method and framework
– Exploring QCA
– Why we probably will not be doing IEs of policy influence
– Why we will look at synthesising evidence

18. Preparation phase-why is it so important
• Implementing agencies do not necessarily understand
impact evaluations
• Researchers do not necessarily speak the implementers’
language or understand their evidence needs well enough
• Fund working together
• Monitor through direct contact
• Have inception workshops

19. Policy influencing using impact
evaluations
• Evidence does not even play a major role in most
decision-making
• Research uptake is a political process, not a technical one
• Researchers are vital to translating and building trust and
credibility
• Single study evidence most often does not result in major
policy change, nor should it
• 3ie does not advocate for wider change based on single
studies, nor should you

20. Limitations
• Expensive
• Work best with large
programmes
• Highly specialised
methods
• Demands of
counterfactuals
• Need to translate for policy
and programming uptake

21. Prevailing view of policy engagement in
the IE community
Get evidence
Disseminate
as reports
and papers
Change
happens
Assumption: You need the evidence
before you engage and promote uptake

22. 3ie funds high-
quality policy
relevant studies
Applications
- Ask policy relevant
question(s)
- Have potential for
policy impact
- Study team has
demonstrated
experience in
policy influence
ASSUMPTIONS
Increased
scoring for
policy
aspects
National
researchers
on team
Preparation
phase
How
we
support

23. 3ie requires
stakeholder
engagement
Researchers
-Committed
- Understand how
policy influence
happens
- Have the tools
and resources to
invest in policy
engagement
- Decision-makers
interested
ASSUMPTIONS
Inception
workshop includes
implementing
agencies
Identify useful
evaluable
questions
Develop a policy
influence plan
Earmark study
budget
How
we
support

24. Uptake of study
findings and
improved policies
and practice
Study
-Makes policy-
relevant
recommendations
- Answers what
works and why
- Proposes
feasible solutions
- Has had ongoing
engagement
ASSUMPTIONS Ongoing
engagement with
3ie as part of grant
monitoring– graduating to
a more dynamic model of
interaction
Intensive review
and feedback of study
reports by internal and
external reviewers
How
we
support

25. Using impact evaluation evidence for
policy advocacy: 3ie perspective
• Benefits
– Policymakers can understand numbers
– Talking in effect size and costs can be
compelling
– Policymakers often are looking for cost
effectiveness evidence
– Programme managers want the IE and
are invested in the questions and want
to be able to act on findings
– They have credibility in today’s climate