The document discusses the MRC Evaluation Programme and its priorities for 2011/12. It aims to capture and analyze the outputs, outcomes, and impacts of MRC-funded research through its online database MRC e-Val. MRC e-Val collects data on publications, collaborations, patents, spin-off companies, citations in policy, and products/interventions resulting from MRC research. The data allows MRC to assess research quality and productivity, and provides evidence of research impact to support funding decisions. Priorities for 2011/12 include strengthening evaluation to support strategic planning and better understanding economic, academic, and societal impacts of MRC research.

1. Capturing and analysing the impact of
MRC funded research
Ian Viney
Head of Evaluation, MRC Strategy Group
March 2011

2. MRC Evaluation Programme
Progress so far
• Capture and analyse the results of MRC funding
– Develop approaches to assess progress, productivity and
quality of MRC research output at an aggregated
(portfolio) level
• Provide information to strengthen the case for medical
research
– Develop quantitative and qualitative information about the
impact of MRC research

3. MRC Evaluation Programme
Priorities for 2011/12
• Strengthen strategy development and decision making
– Support discussions at Strategy Board, measure progress
with the strategic plan
• Improve understanding of economic/academic/societal impact
and what might lead to it
– Identify improved indicators, encourage research on the
“science of science”
• Support partnership working
– Collaborate with other funding agencies internationally

4. OUTPUTS/OUTCOMES IMPACTS
• Generation of new
knowledge
• Improvements to health (living
longer and with better quality of
life)
• Development of
collaborative networks
• Improving the performance of
existing businesses
• Leveraging funding • Attracting R&D investment from
global business
• Dissemination of research • Improving public policy and
public services
• Research materials • Creating new businesses
• Intellectual
property/licensing
• Delivering highly skilled people
to the labour market
• New
products/interventions
• Academic impact (effects on
further research including other
disciplines)
• Changes to policy
MRC Evaluation Framework
INPUT
FundingforResearchandTraining

5. MRC e-Val
Capturing output/outcome/impact
• MRC e-Val is an online database used to gather information about
outputs/outcomes/impacts arising from MRC research
• MRC e-Val is building a robust dataset of evidenced output which
allows progress/productivity and quality of MRC research to be
analysed, and this information to be utilised in decision making
• MRC e-Val is used each year to collect updated, structured feedback
from researchers throughout the lifetime of their MRC funding, and
beyond
• Take a holistic approach, include papers and patents, but aim for a
much wider range of output types
• E-Val data can “set case studies in context” of the productivity and
quality of all MRC output. Comprehensive data obtained across the
whole MRC portfolio
• A baseline dataset against which progress and trends can be
prospectively tracked over time

6. MRC e-Val 2010
• Feedback from all researchers that have held MRC funding since
2006 (3714 MRC awards to over 2800 scientists)
• Includes fellowships, grants and Unit/Institute programmes
• Significant new system developments between 2009 and 2010
have made data entry and administration of the system easier,
speeding up data cleaning, analysis and publication of results
• Submitted responses for 91.4% of funding agreements
(3339/3655) obtained in e-Val
• These responses correspond to 92.3% of MRC spend between
2006/07 and 2009/10 (£1.88/£2.0 billion in RDW)
• STFC and Chief Scientist Office (CSO) Scotland have
implemented e-Val

9. • 37,500 publications reported (30,000 papers between 2006-2010)
• Details of 5,000 active collaborations involving 6,000 partners
• Over 200 published patents (roughly 30% are licensed)
• Over 50 spin out companies with an evidenced link to MRC research
since 2006
• Over 130 citations in policy documents, including 30 citations in NICE
guidelines since 2006
• 360 new products and interventions in development, around 40
launched onto the market since 2006, including around 10 new drugs
• £340m of inward investment to UK research and development from
overseas and private sector research funding between 2006 and 2010
MRC e-Val 2010 dataset
Example Outputs/Outcomes/Impacts

10. Impact Profile of MRC Publications 2006-2008
(Rebased Impact to end 2009 Data & analysis: Evidence,
Thomson Reuters)

11. Network analysis of collaborations within MRC
Regenerative Medicine portfolio

12. MRC Research has contributed to the
creation and growth of new companies

13. Examples of changes in policy/practice
(2009)
• Martin Dennis, Edinburgh – findings of the CLOTS trial; stockings not effective at
preventing venous thromboembolism in acute stroke patients - impact on NICE and
international clinical guidelines and treatment affecting approximately 80,000 people a
year in the UK
• Paresh Vyas, MRC Molecular Haematology Unit (British Journal of Haematology paper)
recommended newborns with Down’s Syndrome should have a full blood count to
screen for a preleukaemic condition – now most do (approximately 750 each year in the
UK).
• Simon Thompson, MRC Biostatistics Unit, helped to develop the basis for the UK national
screening programme for abdominal aortic aneurysms (AAA). AAAs affect 4% of
men aged 65-74 (approximately 80,000).
• Ray Waters, University of Cardiff, is deputy chair of the Committee on Medical Aspects of
Radiation in the Environment (COMARE) - which has advised on the risks of sunbeds
and made recommendations regarding legislation for the use of sun parlours – the misuse
of which is estimated to lead to 100 additional deaths a year from skin cancer in the UK.

14. Influences on policy/practice
Pathways to Impact
Research
paper
Citation in policy
document
Informs changes to
Clinical practice/
economy etc.
• ROD
• PubMed
• Over 200 guideline
issuing organisations
• NLM/UKPMC digitising
NICE guidance
• E.g. ERNIE database
• Researchers participate in policy setting processes and/or
• Research is cited in policy documents
Potential pathways
Specific pathway
Approaches for analysis

15. Products/interventions (2010)

16. Results from MRC e-Val
http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/MRCe-Val2009/index.htm
Published
summaries of 6
sections from
MRC e-Val 2009
on the MRC
website
Reports e.g.
“Impact of MRC
Research” used as
briefing for
BIS/new ministers
early in 2010

17. University/MRC Unit/Institute
“dashboard” contrasting local MRC
e-Val results with the whole MRC
portfolio
In addition to spreadsheets with all
raw data, arranged by section and
principal investigator

18. Capturing output in 2011/12
• MRC will continue to use MRC e-Val to gather evidence of progress,
productivity and quality of MRC output. Data gathering is planned
again for October 2011
• CSO (Scotland) is piloting e-Val and STFC are currently gathering data
using e-Val
• NIHR have just completed an output survey using a different approach
• RCUK is re-developing ESRC’s “Society Today” system to collect data
for AHRC, EPSRC, ESRC and BBSRC
• MRC is in discussion with medical research charities to pilot the next
phase of output gathering approaches which will encourage adoption
of a “federated” approach between output datasets

19. •Consistent time series for medical research funding
in CVD and MH from 1975-1992
•Clear conceptual framework relating to GDP gain
from “spillovers”
•Estimation from literature of the magnitude of this
GDP return
•Development and application to CVD and MH, of a
‘bottom-up’ approach to estimate health gain in
terms of QALYs
•Analysis of UK guidelines (5 CVD and 12 MH) to
provide indicators of lags and proportion of benefits
attributable to UK
•Suggestions for developing research agenda
•Strong quantitative argument for investment
in medical research
Medical Research: What’s it worth?
(2008)