Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Application of recently developed FAIR metrics to the ELIXIR Core Data Resources

The FAIR (Findable, Accessible, Interoperable and Reusable) principles aim to maximize the discovery and reuse of digital resources. Using recently developed​ software and metrics to assess FAIRness and supported through an ELIXIR Implementation Study, Michel worked with a subset of ELIXIR Core Data Resources to apply these technologies. In this webinar, he will discuss their approach, findings, and lessons learned towards the understanding and promotion of the FAIR principles.

  • Be the first to comment

  • Be the first to like this

Application of recently developed FAIR metrics to the ELIXIR Core Data Resources

  1. 1. Application of recently developed FAIR metrics to the ELIXIR Core Data Resources Ricardo de Miranda Azevedo & Michel Dumontier Institute of Data Science (IDS) Maastricht University, the Netherlands
  2. 2. An international, bottom-up paradigm for the discovery and reuse of digital content for the machines that people use
  3. 3.
  4. 4. • DATA FAIRPORT workshop aimed to define a minimal (yet comprehensive) framework for data discoverability, access, annotation and authoring • FAIR acronym was created and guiding principles drafted • for comment on FORCE11 website • Principles were revised during the 2015 BioHackathon in Japan FAIR: History
  5. 5. FAIR in a nutshell FAIR aims to create social and economic impact by facilitating the discovery and reuse of digital resources through a set of requirements: • unique identifiers to retrieve all forms of digital content and knowledge • high quality meta(data) to enhance discovery of digital resources • use of common vocabularies to share terms and facilitate query • use of community standards for more facile knowledge utilisation • detailed provenance to provide context and reproducibility • simpler terms of use to clarify expectations and intensify innovation • deposited in appropriate repositories with high quality metadata for future content seekers • social and technological commitments to realize reliable access
  6. 6. • 14 universal metrics covering each of the FAIR sub-principles. The metrics don’t dictate any particular standards. They simply demand evidence (using protocols of the Web) that you have met community expectations. • Digital resource providers must provide at least one web-accessible document with machine-readable metadata (FM-F2, FM-F3), resource management plan (FM-A2), and any additional authorization procedures (FM-A1.2). • They must use publically registered: identifier schemes (FM-F1A), (secure) access protocols (FM-A1.1), knowledge representation languages (FM-I1), licenses (FM-R1.1), provenance specifications (FM-R1.2), and community standards (FM-R1.3) • They must evidence that their resource can be located in search results (FM-F4), that it provides links to other (FAIR) resources (FM-I3; FM-I2), and it validates against community standards (FM-R1.3)
  7. 7. ELIXIR Core Data Resources • ELIXIR Core Data Resources (CDRs) are a set of European data resources of fundamental importance to the wider life-science community and the long-term preservation of biological data. • CDRs are assessed across several categories: • Scientific focus and quality of science • Community served by the resource • Quality of service • Legal and funding infrastructure, and governance • Impact and translational stories • Details in F1000R ELIXIR track article 'Identifying ELIXIR Core Data Resources'. • ELIXIR webinar webinar-elixir-core-data-resources-selection-process-and- outcomes
  8. 8. Elixir Implementation Study: FAIRness of the current ELIXIR Core resources Objectives 1. Develop a shared understanding of the FAIR principles 2. Apply newly available FAIR metrics/FAIR evaluation software 3. Get feedback on the evaluation procedure 4. Identify actions that would increase the FAIRness of CDRs
  9. 9. Key Deliverables 1.Workshops and materials including FAIR implementation guide 2.Report on the analysis of the FAIRness of each participating CDR 3.Update records in and TeSS with results of the study 4.Develop a vocabulary to represent and publish FAIR assessments FAIRness of the current ELIXIR Core resources:
  10. 10. FAIRness of the current ELIXIR Core resources: 1st Workshop: European Bioinformatics Institute (Hinxton-UK) – 01/10/2018 • Introduction of FAIR maturity indicators (aka FAIRmetrics) • Instructions on conducting manual FAIRness assessments using FAIRshake • Representatives 8 ELIXIR CDRs submitted an assessment • The assessments were reviewed by experts from the FAIRmetrics group • Feedback was provided for each of the 8 participating ELIXIR CDRs Materials:
  11. 11.
  12. 12. Digite para inserir uma legenda. Item Protocol to access restricted content 0.5 Persistence of resource and metadata 0.5 Provenance scheme 0.5 Persistent identifier 0.38 Metadata format 0.38 Certificate of compliance to community standard 0.25 Linked 0 Distribution sum score of the participating CDRs N = 8 Median = 12
  13. 13. Workshop Outcomes • Substantive discussions about FAIR in the context of repositories! – What is being evaluated: repositories or the records within? • Domain entity descriptions are of high quality owing to depth of curation • Repository metadata could to be improved – structured repository metadata altogether missing (bioschemas) – Unable to locate documentation regarding the persistence of identifiers, and the maintenance of resources in the long term – Licenses for repository metadata, as well as for their records • Concern on how FAIRness assessments will be interpreted by outside parties – FAIRShake did not have the capability to keep assessments private, until completed – Anybody could perform manual assessments, that could be incomplete or wrong, and show a lower compliance than was actually there – Summary scores are not particularly informative – producer and consumer
  14. 14. FAIRness of the current ELIXIR Core resources: 2nd Workshop: European Bioinformatics Institute (Hinxton-UK) – 13/05/2019 • Preliminary results for the first round of assessments • Presentation on the role of • Representatives of 5 CDRs (that did not take part on the first workshop) • Breakout groups to promote discussion on FAIR data stewardship topics • Minimal and ideal metadata for repositories and data records (bioschemas) • Licensing and data stewardship plans • Data standards, vocabularies, and participating in their evolution • Substantial input generated from the breakout groups!
  15. 15. Automated FAIRness Assessments • Powered using smartAPI and semantic web technologies • Harvests a diverse set of metadata through HTTP operations and links in documents • Open source and extensible!
  16. 16. { "@context": "", "@id": "", "@type": [""," ontology/FAIR-Evaluation-Output"], "collection": "", "primaryTopic": "", "title": "FAIRness evaluation of CHEMBL resource", "creator": "", "": "2019-10-17T08:22:05.000Z", "": "FAIR Metrics Evaluation: FAIRness evaluation of CHEMBL resource; Tested identifier:; generated by", "": " 9435", "": " ons/801", "": "", "evaluationInput": “…", "evaluationResult": “…" }
  17. 17. Evaluator Schema
  18. 18. Evaluation Input "evaluationInput": "{"resource": "", "executor": "0000-0003-4727-9435", "title": "FAIRness evaluation of CHEMBL resource"}", { "resource": "10.25504/FAIRsharing.m3jtpg", "executor": "0000-0003-4727-9435", "title": "Evaluation of CHEMBL using FAIRsharing DOI (no http)“ }
  19. 19. Evaluation Output { "": [{ "@id": "", "": [{ "@value": "2019-10-17T10:20:59+00:00", "@type": "" }], "": [{ "@value": "SUCCESS: Found an identifier of type 'doi'", "@language": "en" }], "": [{ "@value": "1", "@type": "" }], "": [{ "@value": "10.25504/FAIRsharing.m3jtpg", "@type": "" }], "@type": [""] }],
  20. 20. FAIRness of the current ELIXIR Core resources: Lessons learned: • The implementation study facilitated valuable interaction between ELIXIR curators and FAIR data experts; FAIRness assessments offer an opportunity to improve • The ELIXIR CDRs exhibited substantial FAIRness in their records they maintain, but metadata about the CDRs need more attention • The ELIXIR CDRs identified areas for improvement in the FAIRness assessment • Questionnaires are time consuming, prone to error, and need proper management • Coupling guidance with the results of the assessment could fuel improvements • The FAIRCDR IS has directly contributed to “FAIR Evaluator Service”, an automated state-of-art tool for FAIRness assessment
  21. 21. FAIRness of the current ELIXIR Core resources: Next steps: • A manuscript on the implementation study is under preparation. All representatives involved in the implementation study will be invited to be co-authors (target journal: F1000) • Manuscript will provide a user-friendly guide for the implementation of the FAIR principles for the ELIXIR CDR community
  22. 22. @ELIXIREurope /company/elixir-europe Thank you! Acknowledgements: Rob Hooft, Rachel Drysdale, Mark Wilkinson, Susanna Sansone, Peter McQuilton, Avi Ma’ayan, Daniel Clarke, and all representatives of the ELIXIR CDRs that promptly collaborated