This document discusses the Wellcome Trust's approach to monitoring and evaluating the impact of its funding initiatives. It outlines the Trust's key outcomes of discoveries, applications, engagement, research leaders, research environment, and influence. It then describes the Trust's key indicators to track progress towards these outcomes. The document also discusses challenges to impact evaluation such as ensuring complete self-reporting, tracking career progression over time, attributing outcomes to specific funding, determining individual contributions, and accounting for unpredictable timeframes and unexpected impacts. It proposes that initiatives like ORCID could help address some of these challenges by better connecting researchers, funding, and outputs over time.

1. tracking impact: a funder
perspective
Liz Allen
May 2013

2. the rationale for monitoring & evaluation
o accountability & validation
o management of funding initiatives
o strategy & planning
o policy & advocacy
o learning
monitoring & evaluation are activities central to the management
& improvement of an organisation or process

3. Wellcome Trust outcomes
o Discoveries
o Applications
o Engagement
o Research leaders
o Research environment
o Influence

4. Monitoring progress: WT’s key indicators
Outcomes Key indicators of progress
Discoveries
Applications
Engagement
Research leaders
Research
environment
Influence
1. significant advances in the generation of new knowledge
2. contribute to discoveries with tangible impacts on health
3. contribute to the development of enabling technologies,
products and devices
4. uptake of research into policy and practice
5. enhanced level of informed debate in biomedicine
6. significant engagement of key audiences & increased reach
7. develop a cadre of research leaders
8. evidence of significant career progression among those we
support
9. key contributions to the creation, development and maintenance
of major research resources
10. contributions to the growth of centres of excellence
11. significant impact on science funding & policy developments
12. significant impact on global research priorities and processes

5. Wellcome Trust outcomes
o Discoveries
o Applications
o Engagement
o Research leaders
o Research environment
o Influence
Where do
we get data
on these?

6. Key data sources for M & E
What? Data source Data provided
WT e-Val Online grant reporting tool Progress & various impacts (including
datasets)
Career Tracker Annual, panel-based cohort
survey
Career destination
Output Metrics Web of Science/Scopus/
PubMed
Paper-based citations; Normalised
Citation Impact (NCIs); other ‘indicators’
F1000 ‘qualitative’ assessment of published
work
Alt Metric/ImpactStory et al Other ‘measures’ of re-use; social media
Clinical Guidelines PubMed (via [GR] search) Papers cited on clinical guidelines
Stakeholder
consultations
Surveys; interviews Survey data; narrative

7. Monitoring & tracking impact: challenges
Issue Key challenge
Self reporting How to ensure completeness?
How to minimise reporting burdens?

8. Monitoring & tracking impact: challenges
Issue Key challenge
Self reporting How to ensure completeness?
How to minimise reporting burdens?
Career tracking How to stay in touch?
How to find people?

9. Monitoring & tracking impact: challenges
Issue Key challenge
Self reporting How to ensure completeness?
How to minimise reporting burdens?
Career tracking How to stay in touch?
How to find people?
Attribution What did the funding deliver?
What did you fund?

10. Data citation
Cited 3139 times

11. Monitoring & tracking impact: challenges
Issue Key challenge
Self reporting How to ensure completeness?
How to minimise reporting burdens?
Career tracking How to stay in touch?
How to find people?
Attribution What did the funding deliver?
What did you fund?
Contribution Who did what?
What were the key contributions? (e.g. data curator)

12. 1991
Senior Clinical
Fellowship –
University of
Cambridge
1997
First description of
single gene defects
(leptin and PC 1)
leading to human
obesity
1998 2002
1)Discovery of a form
of diabetes in humans
that results from
mutation in two
genes(digenic disease)
2) Heinrich Wieland
Prize
2002 2004 2006 2007 2009
O’Rahilly S, Farooqi IS
Human obesity: a heritable
neurobehavioral disorder that
is highly sensitive to
environmental conditions.
Diabetes 2008
2011 2013
Coll AP, Farooqi
IS, O’Rahilly S.
The hormonal
control of food
intake. Cell.
2007
EUFp7
funding
Research Leaders:
Stephen O’Rahilly

13. Monitoring & tracking impact: challenges
Issue Key challenge
Self reporting How to ensure completeness?
How to minimise reporting burdens?
Career tracking How to stay in touch?
How to find people?
Attribution What did the funding deliver?
What did you fund?
Contribution Who did what?
What were the key contributions? (e.g. data curator)
Timeframe/lag-time Road to impact can be long & unpredictable …
Impact story evolves over time …

14. 18 years between Robert
Edwards first cultivating &
maturing human eggs in the
lab & first test tube baby
being born
1978
early 1960s
the long road to discovery
In vitro fertilisation (IVF)
Monoclonal antibodies
early 1970s early 2000s
30 years between Milstein &
Kohler manufacturing first
monoclonal antibodies &
widespread adoption in
therapies

15. Monitoring & tracking impact: challenges
Issue Key challenge
Self reporting How to ensure completeness?
How to minimise reporting burdens?
Career tracking How to stay in touch?
How to find people?
Attribution What did the funding deliver?
What did you fund?
Contribution Who did what?
What were the key contributions? (e.g. data curator)
Timeframe/lag-time Road to impact can be long & unpredictable …
Impact story evolves over time …
Serendipity / ripple effects Unexpected impacts & outside the field

16. Monitoring & tracking impact: challenges
Issue Key challenge
Self reporting How to ensure completeness? ORCiD
Career tracking How to stay in touch? ORCiD
Attribution What did the funding deliver?
What did you fund?
Fund Ref – standardising
funding
acknowledgements;
ORCiD
Contribution Who did what?
What were the key
contributions?
Contributor roles
project – can we move
beyond authorship?
ORCiD
Timeframe/lag-time Impact story evolves over
time …
ORCiD
Serendipity / ripple effects Unexpected impacts &
outside the field
ORCiD

17. Liz Allen
Head of Evaluation
Wellcome Trust
l.allen@wellcome.ac.uk