This document provides guidance on evaluating the impact of research projects, particularly those funded by the Global Challenges Research Fund (GCRF). It discusses defining impact goals, selecting evaluation designs, determining the reach of impacts, and communicating evaluation findings as evidence of impact. Key evaluation concepts covered include theories of change, formative and summative evaluation, attribution versus contribution, and qualitative and quantitative methods. The document also provides templates and examples to help researchers plan for impact evaluation from the start of a project.

1. Fast Track Impact
Training by researchers for researchers
Research for Official Development Assistance with
lasting impact

2. Question:Question
 What is impact?

3. www.fasttrackimpact.com/what-is-impact

4.  Significance is the degree to which the
impact has enriched, influence, informed or
changed policies, practices, products,
opportunities or perceptions of individuals,
communities or organisations
 Reach is the extent and diversity of the
communities, environments, individuals,
organisations or any other beneficiaries that
may have been impacted by the research
Evaluating ImpactEvaluating Impact

5. Question:Question
 What is Official Development
Assistance?

6.  Research that delivers significant, far-
reaching, measurable benefits for
economic growth and welfare that are
urgently needed in a DAC list country
Evaluating ImpactODA is impact with a purpose

7. “…aims to promote the welfare and economic development
of a country or countries on the Development Assistance
Country (DAC) list of ODA recipients”
Evaluating ImpactODA

8. 5 WAYS
to Fast Track your
Research Impact
How to write a winning
proposal

9. £1.5B for ODA compliant research in
DAC list countries delivered by RCUK,
Academies, HEFCE & Space Agency
Evaluating ImpactGCRF

10. Equitable access to sustainable development
 secure and resilient food systems supported by
sustainable marine resources and agriculture
 sustainable health and well being
 inclusive and equitable quality education
 clean air, water and sanitation
 affordable, reliable, sustainable energy
Evaluating ImpactChallenge areas (1 of 3)

11. Sustainable economies and societies
 Sustainable livelihoods supported by strong
foundations for inclusive economic growth and
innovation
 Resilience and action on short-term environmental
shocks and long-term environmental change
 Sustainable cities and communities
 Sustainable production and consumption of materials
and other resources
Evaluating ImpactChallenge areas (2 of 3)

12. Human rights, good governance & social justice
 Understand and respond effectively to forced
displacement and multiple refugee crises
 Reduce conflict and promote peace, justice and
humanitarian action
 Reduce poverty and inequality, including gender
inequalities
Evaluating ImpactChallenge areas (3 of 3)

13. Geographical focus:
 Development challenges are more clear-cut in Least
Developed Countries and Other Low Income Countries
 GCRF work in Lower and Upper Middle Income Countries
should strengthen Newton Fund partnerships where possible
 Current focus on all DAC list countries may come into line
with UK Aid policy in future
GCRF success will be measured by the government in terms
of:
 Maximising impact on the wellbeing of people in developing
countries, in line with the UK aid strategy objectives
 Maximising impact on UK research [not UK economy]
 Operating in the most cost-effective way possible
Evaluating ImpactGCRF strategy (June 2017)

14. Evaluating ImpactGCRF strategy (June 2017)
Risks to successful implementation include:
 Lack of buy-in from the communities that need to be
mobilised to realise its ambitions, including researchers
and non-academic partners in UK and in Global South
 Poor coordination between delivery partners, leading to
duplication of efforts rather than complementarity
 Failures of governance, monitoring, evaluation,
reporting and/or communications, meaning that
confidence in the ODA compliance, value for money
and/or effectiveness of the GCRF as a whole is eroded
 Failure to create real impact in developing countries
 (Lack of active portfolio management by GCRF)

15.  Short timescales:
 Strategic team and concept building around key
themes
 Poised pre-review teams
 Build capacity/confidence with colleagues
whose work could be relevant in DAC countries
 In-country credibility
 Map and build strategic collaborations with
global and DAC list country NGOs and
research institutions
Evaluating ImpactStrategic challenges

16. “Cutting-edge research that addresses the
challenges faced by developing countries”
1. Deliver internationally excellent research
2. Effectively address a specific, evidenced
development need in a DAC country that will
deliver economic growth and wellbeing (note:
revisited every 3 years)
Evaluating Impact
2 key elements of a GCRF
proposal

17.  No different to any other RCUK call
 Demonstrate why your research is exciting, cutting-
edge, novel and significant
 Balance between disciplines
 If demanded in call for proposals
 Otherwise can be flexible
Evaluating ImpactResearch excellence

18. Evaluating Impact7 pathways to ODA compliance

19. 1. Benefit disadvantaged, impoverished populations
and major development issues in DAC list
countries
2. Explain how many people are affected by the
issues the project will address (or how you will
scale the benefits) but don’t over-extrapolate
3. Use plain English and link explicitly to call in title,
summary and objectives to make it through sift
stage. Make impact summary/pathway & ODA
relevance stand alone and in plain English for
non-academic development panelists
Evaluating Impact7 pathways to ODA compliance

20. 4. Build capacity for research/development with in-
country partners e.g. via collaboration and/or
training
5. Demonstrate in-country partnerships based on
prior/preparatory work
6. Build on existing research and development work
in country and add value to global efforts
7. Co-create solutions rather than imposing (get a
second opinion in case your choice of language
implicitly contradicts your approach)
Evaluating Impact7 pathways to ODA compliance

21. Evaluating ImpactDiscussion

22. Tools for Impact

23. Who has a stake in my research?
1. Stakeholder/publics analysis template
2. Partnership building template
3. Impact planning template
Two Tools

24. Who has a stake in my research?
Who are my beneficiaries?
Who may be disadvantaged or harmed?
Who is able to powerfully facilitate my
(research and) impact?
Who has the power to block my (research and)
impact?
Three un-asked questions

25. High
Low
Benefit
Hard-to-reach publics who
are disinterested but could
benefit significantly from
engagement
Find out what would
motivate them to engage
Easy-to-reach target publics who
benefit significantly from
engagement
Reach out systematically in
priority order – contact the first
one on your list now
Other publics that have little
interest and are unlikely to
benefit much if they were to
engage
Keep a watching brief as
their needs and interests
may change over time
Easy-to-reach non-target publics
may engage more than hard-to-
reach publics but benefit less
Be careful not to focus on these
groups at the expense of those
who who have greater need
Level of Interest High
Which publics to engage?

26. High
Influence
Hard-to-reach influential
stakeholders who could
block or facilitate impact but
are not interested enough to
prioritise engagement
Find out what would
motivate them to engage
Easy-to-reach influential
stakeholders who could block
or facilitate impact and engage
easily/regularly
Reach out systematically in
priority order – contact the first
one on your list now
Other stakeholders with
limited interest or influence,
whose interest or influence
may change over time
Keep a watching brief as
their needs and interests
may change over time
Easy-to-reach marginalised
stakeholders who may want to block
or facilitate impact but have limited
influence or voice
Identify strategic alliances with more
powerful stakeholders who share their
interests so you can all work together
Which stakeholders to engage?
Low Level of Interest High

27. ...adapt to your own needs
Stakeholder/publics analysis

28. Partnership building template
For more information about identifying partners, see:
http://www.ukcds.org.uk/resources/finding-and-building-effective-partnerships

31. Impact planning template for GCRF projects

32.  Visual map of causal links that take you from
the research to impact
Evaluating ImpactTheory of Change

34.  Bottom-up:
 Complete impact planning templates with in-country
partners across all WPs
 Thematically group impact goals
Evaluating ImpactProcess
 Arrange impact
goals in causal
chains
 Look back to
activities in
templates to further
trace back to
research

35.  Top-down:
 Overall project goal
 Missing impact goals
 Missing links in causal chains
 Balance of activities across WPs, themes, countries
etc
 Go back to impact templates to write impact
summary (linking goals to beneficiaries) and
write up more detailed pathways to impact
(including risks and mitigation, impact track
record, resources and approach to monitoring,
evaluation, accountability and learning
Evaluating ImpactProcess

36. Evaluating impact

37.  Tracking
 Evaluating
 Evidencing
Evaluating ImpactScope

38. Read and discuss
 Hands up: do you systematically track
the impact of your research?
 University system…
 What are the barriers to impact tracking
and how can you overcome them?
Tracking impacts

39. Read and discuss
 Track indicators/milestones identified a
priori
 Formative feedback
 Summative feedback
Tracking impacts

40. Read and discuss

41. Read and discuss
 Capture other impacts as they occur
opportunistically…
Tracking impacts

42. Read and discussReporting impacts
 Find a way to continually track your
impacts easily to take the pain out of
reporting via your University system
 Email impacts/evidence to yourself and file
 Ring binder/scrap book
 Evernote: enable team members from any
institution to collate impacts in a shared
notebook without having to log into anything

43. www.fasttrackimpact.com/evernote

44.  The process of analysing, monitoring and
managing the intended and unintended
consequences, both positive and negative, of
research
 Seeks to identify causal links between:
 New knowledge
 Knowledge exchange and co-production activities
 Benefits for specific groups
 Typically significant contribution rather
than sole attribution
Evaluating ImpactWhat is impact evaluation?

45.  Significance and reach
 Design of your pathway to impact, delivery of
activities and mpacts
Evaluating ImpactWhat to evaluate?

46. 1. Know what impacts you are looking for
2. Select an evaluation design to establish the
significance of the impact
3. Determine the reach of the impact
4. Communicate the findings of your evaluation
as evidence of impact
Evaluating ImpactHow to evaluate?

47.  Based on your impact planning template and
Theory of Change you should have selected
ambitious, yet feasible SMART impact goals
that you can evaluate
 An evaluation is typically driven by an
assessment of whether these goals were
reached, but you can identify and claim other
unplanned impacts
Evaluating Impact1. What are your impacts?

48.  Evaluation designs lie along three continua:
 Summative – formative
 Sole attribution – significant contribution
 A priori (what works in theory) – ex ante (effective
assessment)
 Factors influencing choice:
 Type of impact
 Data availability
 Resources
Evaluating Impact2. Select an evaluation design

49.  Eight types of evaluation design:
 Theory/logic-driven
 Experimental and quantitative survey work
 Statistical
 Contribution and pathway analysis
 Case-based and narrative analysis
 Participatory
 Evidence synthesis
 Arts-based
Evaluating Impact2. Select an evaluation design

50.  Impacts may:
 “Scale-out” if impacts spread from a one individual
or community to another
 “Scale-up” when an impact reaches a higher
institutional or governance level
 For example:
 Analyse how new ideas based on
research spread between communities
 Analyse how those ideas influence
decisions at different governance levels
Evaluating Impact3. Methods for determining reach

51.  Start early, use formative feedback to
shape your work
 Build in opportunities for longitudinal
evaluation
 Monitor indicators and progress towards goals,
and collect data opportunistically for other
benefits as they arise
 Be proportionate and credible: not all impacts
need an RCT to prove that benefits were felt
Evaluating Impact4. Collecte and analyse data

52. Read and discuss
 Choose a project you are familiar with
 Identify impact goals (consider by type)
 Identify activity indicators
 Identify impact indicators
 Prioritise relevant impact evaluation
methods, discussing their pros and cons
Small group exercise

53. Read and discuss
 How will you communicate your
evaluation findings?
 Write a peer-reviewed paper
 Include in a third party report
 Researching international policy impacts
 Getting powerful testimonials
Evidencing impact

54. Find out more
www.fasttrackimpact.com/resources

55. Fast Track Impact
Training by researchers for
researchers
www.fasttrackimpact.com
@fasttrackimpact