Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

20190527_Diego Chialva_ Research evaluation: the unseized opportunities ...

127 views

Published on

Presented by Diego-Valerio Chialva (ERC)

during the OpenAIRE workshop "Research policy monitoring in the era of Open Science and Big Data" taking place in Ghent, Belgium on May 27th and 28th 2019

Day 1: Monitoring and Infrastructure for Open Science

https://www.openaire.eu/research-policy-monitoring-in-the-era-of-open-science-and-big-data-the-what-indicators-and-the-how-infrastructures

Published in: Science
  • Be the first to comment

  • Be the first to like this

20190527_Diego Chialva_ Research evaluation: the unseized opportunities ...

  1. 1. Diego Chialva 27-05-2019 Diego Chialva ERC Executive Agency, Unit A1 Research evaluation: the unseized opportunities of the open science era and the possible technology and methodology approaches 27 May 2019 The European Research Council
  2. 2. Diego Chialva 27-05-2019 │ 2 Outline Motivation and Perspective Issues and Opportunities Technological and Methodological Aspects
  3. 3. Diego Chialva 27-05-2019 Motivation and Perspective
  4. 4. Diego Chialva 27-05-2019 Research and research policy: evaluation and monitoring The perspective I will adopt here: From workshop abstract: “explore mechanisms for research policy monitoring and indicators, and how to link these to infrastructure and services. [….] first day will focus on open science indicators” The perspective here (a slightly tilted one): let’s look at the potentialities from Open Science, Open Data and the interconnected communication landscape (Internet → Linked (Open) Data) for monitoring and evaluation → effects at large spectrum: data collection, management and analysis, for Open Science monitoring and evaluation, but also beyond that
  5. 5. Diego Chialva 27-05-2019 Research and research policy: evaluation and monitoring OECD, 2009, “Measuring Government Activity”, OECD Publishing, Paris Monitoring: “a continuing function that uses systematic collection of data on specified indicators to provide [….] stakeholders of an ongoing [….] intervention with indications [….]” Evaluation: “the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, [....] efficiency, effectiveness, impact and sustainability.”
  6. 6. Diego Chialva 27-05-2019 Issues and Opportunities
  7. 7. Diego Chialva 27-05-2019 Research evaluation: a few notable features Two directions in the study for the assessment of efficiency, effectiveness, impact and sustainability Vertical: funding-laboratory-society/world/…. Horizontal: bench-marking
  8. 8. Diego Chialva 27-05-2019 Vertical direction: examples An examples of questions for the “vertical” direction: what research and research policy contributed most to those medicines and the related societal improvement?
  9. 9. Diego Chialva 27-05-2019 Horizontal direction: examples Example of questions for the “horizontal” direction: how does the results of research activity (or policy!) implemented by A compare to the activity (or policy) of B? A B
  10. 10. Diego Chialva 27-05-2019 Monitoring and evaluation and Linked Open Data Monitoring and evaluation Collecting and processing large amount of data A multiplicity of questions A multiplicity of stakeholders Working with data from different sources Relationships between data form complex networks
  11. 11. Diego Chialva 27-05-2019 The issues Open Science as a paradigm helps in dealing with these issues (open publications, data, lab notes allow to track and create chains of evidence). But relevant issues appear at a more core level. (And Open Science needs to be monitored and evaluated as well).
  12. 12. Diego Chialva 27-05-2019 The issues Indeed, the above situation brings upon several (related) issues, such as:  data available to a single actor/analyst is limited and often specific to a single entity/operation  data processing done in isolation by each actor and on a ad- hoc basis → often it is re-processing difficult contextualisation, duplication of efforts
  13. 13. Diego Chialva 27-05-2019 The issues  data produced/collected/organised by different entities may be categorised/classified differently  data models and formats are not standardized difficult contextualisation, barriers to automation and to re-use and interoperability of data
  14. 14. Diego Chialva 27-05-2019 Monitoring and Evaluation today Today: 1) looking at repositories and other sources
  15. 15. Diego Chialva 27-05-2019 Monitoring and Evaluation today Today: 2) processing the raw data
  16. 16. Diego Chialva 27-05-2019 Monitoring and Evaluation today Today: 3) obtaining useful data
  17. 17. Diego Chialva 27-05-2019 Monitoring and Evaluation today Today: 4) perform the analysis and find their connections
  18. 18. Diego Chialva 27-05-2019 Monitoring and Evaluation today Today: somebody else has to do it again for him/herself
  19. 19. Diego Chialva 27-05-2019 Monitoring and Evaluation today Today: but crucial/useful (part of the) data and information are easily lost/inaccessible
  20. 20. Diego Chialva 27-05-2019 The issues Overall consequences: limits to the analysis reach, ineffective evaluation
  21. 21. Diego Chialva 27-05-2019 Monitoring and evaluation and Open Linked Data But there are new unseized opportunities and solutions to each of these issues Linked Open Data Semantic Web Knowledge Graphs
  22. 22. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web How do we move toward Linked Open Data, and in particular the Semantic Web?
  23. 23. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web Creating Linked Open Data and in particular a Semantic Web is a task that has technical aspects to it, but which is largely a responsibility of data owners (research institutes), creators, curators, and policy makers (that is, overall, data publishers)
  24. 24. Diego Chialva 27-05-2019 Technological and Methodological Aspects
  25. 25. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web The Semantic Web does not only yield a bunch of interconnected data, but it adds advantages: the possibility to uniquely identify a resource (owl:sameAs, skos:**Match) the possibility to identify/map concepts, classes, specifications the possibility to locate the resource information about the resource, and the data itself relationships and location of related resources that can be queried (federatively) the elimination of data storage and maintenance on a single site, by a single actor the possibility to automatise the data collection and analysis
  26. 26. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web The advantages are bound to Openness and Web Tim Berner Lee’s four qualities of Semantic Web https://www.w3.org/DesignIssues/ LinkedData.html 1) Use URIs as names for things. 2) Use HTTP URIs so that people can look up those names. 3) When someone looks up a URI, provide useful information using the standards 4) Include links to other URIs. so that they can discover more things. Metadata, semantics Link, discover and locate other data/resources on different sites Identify and locate the resource
  27. 27. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web: technology and methodology approaches For all data publishers and policy analysts: 5-star system road to good data Available on the web (whatever format) but with an open licence, to be Open Data Available as machine-readable structured data (e.g. excel instead of image scan of a table) [→ and also eas-ish to automatise ] As (2) plus non-proprietary format (e.g. CSV instead of excel) All the above plus, Use open standards from W3C (RDF and SPARQL) to identify things, so that people can point at your stuff [→also truly easi-er to automatise ] All the above, plus: Link your data to other people’s data to provide context
  28. 28. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web The FAIR principles (a suggestion for good general database practices) also help, but they are not equivalent to  Linked Data or Semantic Web  Open [Wilkinson et al., 2016 ; Mons et al., 2017]
  29. 29. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web The three main steps for a data publisher are: Map your data using some ontology  an ontology is a sort of “enhanced thesaurus”  ontologies are also a piece of data, so even if I and you make two different ones for the same concepts, we can link them and clarify semantic Write out your data in RDF format (“there’s an app for that”) Publish the data (no need to maintain it alone) There are toolchains and best practice guidelines for publishing data in Linked Open Data format (but not enough time to present them here!).
  30. 30. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web https://lod-cloud.net/
  31. 31. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web: at the ERC What are we doing for this at the ERC? Created an ontology for area of interest: DINGO (http://w3id.org/dingo)  extensible interoperable framework  DINGO conceptualizes and expresses relevant parts of the research and research funding landscape  aims at minimising effort by users: minimum effort for integration in data systems, satisfy complex requirements by easy extensions  already linked to other ontologies, and also extended for Wikidata use
  32. 32. Diego Chialva 27-05-2019 The DINGO ontology graph
  33. 33. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web: at the ERC What are we doing for this at the ERC? Liased with relevant stakeholders, for example  Wikidata: also creating models/ontologies for related knowledge/conceptual areas (for impact analysis)  OpenAire: modeling and publishing data on EC research funding
  34. 34. Diego Chialva 27-05-2019 Linked Open Data and the Semantic Web: at the ERC What are we doing for this at the ERC? Established best practices and toolchains Organising a workshop (expected ~ October 2019)
  35. 35. Diego Chialva 27-05-2019 │ Thank you ! diego-valerio.chialva@ec.europa.eu .

×