George Papastefanatos presents the EOSCpilot monitoring framework | OSFair2017 Workshop Workshop title: Open Science Monitor Workshop overview: Which are the measurable components of Open Science? How do we build a trustworthy, global open science monitor? This workshop will discuss a potential framework to measure Open Science, including the path from the publishing of an open policy (registries of policies and how these are represented or machine read), to the use of open methodologies, and the opening up of research results, their recording and measurement. DAY 2 - PARALLEL SESSION 5

1. EOSCPilot Monitoring Framework
Overview and goals
George Papastefanatos
RC Athena
gpapas@imis.athena-innovation.gr
OSFair 2017 | 6-8 September, Athens
Open Science Monitor Workshop
OpenAIRE, EOSCpilot, Jisc, DANS

2. Science 2.0 Processes & Practices
http://ec.europa.eu/research/consultations/science-2.0/background.pdf

3. Open Science Characteristics
● Research results artefacts
– Open Access to Publications (pre-prints, post-prints)
– Open Datasets (FAIR data, data repositories)
– Open Source Software (algorithms, tools, workflows, protocols, services)
● Open Scholarly Communication (open peer-review, open citations)
● Open Educational resources (open courses, presentations, events)
● Citizen Science (scientific blogs, social media engagement)
● …

4. Stakeholders

5. Benefits of monitoring Open Science
Open Science and Research Initiative (2014). Open Science and
Research Handbook.

6. Measuring Open Science
…multiple efforts so far

7. OA to publications
SPARC’s HowOpenIsIt? Guide

8. OSI 2016 “openness score”
http://journals.gmu.edu/osi/article/download/1381/1149

9. FAIR Principles→ Metrics
Towards metrics to assess
and encourage FAIRness
A FAIRness Index is a collection of metrics that
are aligned to the FAIR principles and can be
consistently and transparently evaluated.
Dumontier@Elixir 2017
https://www.slideshare.net/micheldumontier/towards-metrics-to-assess-and-encourage-fairness

10. Example of a FAIRness Metric
F1 (meta)data are assigned a globally unique and persistent identifier
Aspect: Identifier Persistence
Rationale: An identifier can be used to find, access, and reuse a resource. As such, it
must be available to users in the longest term possible otherwise we will not be able to
perform those functions with the identifier in hand.
Relevant FAIR Principles: F,A,I,R
Metric: Availability of data management plan, which includes a section dealing with
continuity and contingencies related to the persistence of identifiers. The value of the
metric is true or false.
Procedure: Check and verify the URL in the resource metadata points to a data
management plan with continuity section. Document should follow a community
standard, or recommend a basic structure.
Dumontier@Elixir 2017

11. FAIR research data and repositories

12. Certification &Badge systems
GEO label facets
Data Seal of Approval
ODI’s Open Data Certificate

13. EU Open Science Monitor Study (by RAND)

14. OpenAIRE usage statistics

15. Open Data Monitor
http://opendatamonitor.eu

16. Open Data

17. Monitoring Open Science - Challenges
•…more than identifying appropriate metrics per each
characteristic
•Transparent & Open Processes of monitoring
–E.g., appropriate sources (availability, trustworthiness)
•Connection & Validation with current OS trends
•Dynamic and Extensible to new OS practices
–New characteristics, indicators,…
•Address diverse analysis needs
–Researchers, Funders, Service Providers, …

18. EOSCPilot Monitoring Framework - Goals
• User driven methodology
– Stakeholders → Goals → Questions → Indicators
• Harmonized Indicators on open research results
– Aspects of openness, transparency & ownership
– Analysis of Practices and Policy-relevant trends
• Open Specifications: models, standards and processes
– Collection of the appropriate data
– Validation & Aggregation services for monitoring
• Added value services on top of the framework
– Analytics, Visualization, Certificates & Comparison tools
• Dynamic & Extensible observatory

19. EOSCPilot Monitoring Framework – Approach
Concept
Data
Gathering
AnalysisPublication
Review
Research Outputs
Educational Resources
Policies
Scholarly Communication
…
OA to Publications
•#OA repositories
•%green access
•#Pre-Prints
Research Data
•Fair metrics
•Linked Science
cloud
OS Software
•#downloads
•#commits
•…
Analysis
Validation
Collection
Monitoring Framework
Components
OS Activities
Indicators
Processes
OS Characteristics
Data sources &
Indicators
OS Characteristics
Users
Stakeholders
Certificates &
Badges
Services &
Statistics
Policies
Specifications &
APIs
Validation and Update
EOSCPilot Pilots

20. Workshop Agenda
• EOSC Pilot Monitoring Framework - Overview and goals (George
Papastefanatos - EOSC Pilot)
• OpenAIRE monitoring - Stakeholders, Vision (Natalia Manola - OpenAIRE)
• Introduction of the DANS Approach to FAIR Metrics (Peter Doorn and Elly
Dijk - DANS)
• Policies: representation & registries from a UK perspective from Jisc’s open
access services (Frank Manista - JISC)
• Monitoring open science trends in Europe (Salil Gunashekar - RAND
Europe)
• Open Data Monitor: Monitoring Open governmental data (Dimitris
Skoutas - OpenDataMonitor)

21. Thanks you for your Attention
OSFair 2017 | 6-8 September, Athens
Open Science Monitor Workshop
OpenAIRE, EOSCpilot, Jisc, DANS