1) The document discusses current limitations in measuring research impact and proposes a solution called OpenUP Impact Data Services to address these limitations. 2) OpenUP aims to create an open platform for metrics to understand, reproduce, and create new metrics to account for a broader range of research outputs in a transparent way. 3) An example use case shows how OpenUP could enable measurement of research that utilizes a Leica microscope as an input by mining text and data to track mentions and use of the microscope across publications over time.

1. 1
Impact Data Services
on the Cloud
Athens, 07 September 2017

2. 22017-09-07

3. 3
Discussion today
• Define impact, uses and requirements
• Measuring research results today
• The opportunities to improve
• OpenUP Impact Data Services on the
cloud
2017-09-07

4. 4
• Attention
• Presence in media
• Immediate scientific citations
• Short term, measured in day, weeks, months
• Scientific impact
• Substantial changes in scientific theories, methods, etc.
• Citations long after original publication
• Medium to long term, usually takes years
• Social and economic impact
• Substantial changes in products, production processes,
employment, environment, transport, health
• Long term, usually take several decade
What is impact?
2017-09-07

5. 5
• Choosing where to publish/ methods of dissemination
• Short term Journal impact factor
• Academic selection and recruitment
• Short term citations
• Aggregate citation metrics e.g. H-index
• Other attention metrics (Altmetrics)
• Patents and other research outputs, not widely used but useful
• References in patents not widely used but useful
• Allocation of funding
• Formal study of long term metrics
• Long term policy and historical studies
• Long term citations and metrics
Impact metrics: uses & requirements
2017-09-07

6. OpenUP Research Impact Goals
6
• Presents an analysis of how research
results, are measured today
• The limitations of current approaches
• A proposed solution:
• a platform for Open Metrics
• to understand and reproduce current
metrics,
• to create and share their own metrics
• to take account of research outputs
2017-09-07

7. 7
Measuring Research Impact today
2017-09-07

8. 8
Impact Data Services providers today
2017-09-07

9. Opportunities for improvement
9
• Open it up.
• The market is dominated by a small number of metrics
providers.
• Own it.
• Scientific communities do not own the production process.
• One metric does not fit all.
• The available impact factors present a limited view to the
science communities.
• Metrics create distorted market incentives.
• Current metrics do not give proper credit for scientific outputs
other than publications.
• Transparency. Reproducibility.
• Metrics are easily gamed.
2017-09-07

10. Impact Data Services on the Cloud
102017-09-07

11. Features
11
• Collect Impact Data
• E.g. citations, mentions, views, downloads,
references in patents
• Give researchers access to the impact data they
require
• to reproduce current metrics
• E.g. simulate impact metrics of journal
• to produce their own metrics
• E.g. compare citations produced by research in two
different research programs
• Broader coverage of research outputs
2017-09-07

12. 12
This is where the footer goes
Sample: Research Impact on Leica Microscope
• Research Impact today
measures publication related
results.
• Research results can also
reference other forms of input
and output
• Using the Leica microscope, as
a prototype, this use case shows
how a Text & Data mining
platform enables the
measurement of research results
that are input to research

13. 13
Research impact on Leica Microscope
AnnotateAnnotate
Annotated
Articles
Annotated
Articles
OMTD
Data Repository
OMTD
Data Repository
Article name, domain,
institution, year,
Licensing, Article link
Article name, domain,
institution, year,
Licensing, Article link
Leica Microscope
Text & Data Mining
Leica Microscope
Text & Data Mining
2
3
4
5
6
Dictionary of
Keywords,
Synonyms & rules
Dictionary of
Keywords,
Synonyms & rules
1
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Article
Using Leica
Microscope
Review and approveReview and approve
Research Impact
Leica Microscope
Research Impact
Leica Microscope
7
8

14. 2017-09-19 14
OpenUP: Impact Data Architecture

15. 15
• An open science system should be grounded in a mix of expert
judgement, quantitative and qualitative measures
• provides collaboration across stakeholders
• Transparency and accuracy are crucial
• the ability to create, share, reuse metrics and data
• Make better use of existing metrics for open science
• an innovation that allows creation of new metrics where
there is a gap
• Next generation metrics should be underpinned by an open,
transparent, and linked data infrastructure
• use EOSC and existing e-infrastructures
• Measure what matters
• generate fit for purpose qualitative measures
OpenUP Response to the
HLEG Next Generation Metrics
2017-09-07

16. What next?
16
• Continue the engagements, conversations and
use-case development activities
• Finalize the OpenUP research impact
recommendation
• Seek partnerships and collaboration
2017-09-07

17. 17The OpenUP project received funding through the H2020 Framework Programme, GA No: 710722
Thank you for your attention!
Mappet Walker