Recommended
More Related Content
What's hot
What's hot (20)
Similar to Understanding Information Professionals: A Survey on the Quality of Linked Data Sources for Digital Libraries
Similar to Understanding Information Professionals: A Survey on the Quality of Linked Data Sources for Digital Libraries (20)
Recently uploaded
Recently uploaded (20)
Understanding Information Professionals: A Survey on the Quality of Linked Data Sources for Digital Libraries
- 1. Understanding Information Professionals: A Survey on the Quality of Linked Data Sources for Digital Libraries Jeremy Debattista, Lucy McKenna, Rob Brennan ADAPT Centre, Trinity College Dublin, Ireland This research has received funding from the Irish Research Council Government of Ireland Postdoctoral Fellowship award (GOIPD/2017/1204) and theADAPT Centre for Digital Content Technology, funded under the SFI Research Centres Programme(Grant 13/RC/2106) andco-funded by theEuropeanRegionalDevelopmentFund.
- 2. www.adaptcentre.ieWhat is a good digital library? • Literature: Success of DL depends on the quality of available metadata • How do you define good quality metadata? It’s subjective, there is no definite answer • Potentially, an easy question to answer, but definitely not generic
- 3. www.adaptcentre.ieLinked Data in Digital Libraries • Data interoperability & re-usability • Resource discoverability & visibility • Data interlinking
- 4. www.adaptcentre.ieSo why the slow uptake? • Linked Data is not a solution that solves all problems • Quality issues as noted by various literature
- 5. www.adaptcentre.ieThe Aims of this Study • What quality measures do IPs consider important? o Why? Can we identify the generic quality measures for the task at hand? • What quality problems do IPs face when using Linked Data? o Why? Focus quality assessment on Digital Library Linked Datasets
- 6. www.adaptcentre.ieSurvey Methodology • Online questionnaire o Snowball Sampling (Twitter, Email, Mailing lists) • 50 Questions o Primarily multiple choice – able to add own observations o Partially based on: Previous surveys and analysis of projects in domain 2 Data quality focused questions
- 7. www.adaptcentre.ieSurvey Methodology • 185 participants o Split in 2 groups: G1: Participants who have experience working in LD (n=54) G2: Participants who do not have experience working in LD (n=131) • Academic Library (56%), Research Institution (7%), Public Library (7%), Special Library (6%), Archive (6%), National Library (5%), Museum (4%), and Special Archive (1%) • 20 countries o Ireland (28%), the USA (23%) and the UK (20%)
- 8. www.adaptcentre.ieResults and discussion of the whole survey McKenna, L., Debruyne, C., O’Sullivan, D.: Understanding the position of information professionals with regards to linked data: A survey of libraries, archives and museums. In: Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries (JCDL 2018), Fort Worth, Texas, USA, June 3rd-7th, 2018. pp. 7–16 (2018)
- 9. www.adaptcentre.ieThe Questions Q1. When completing different metadata tasks, what evaluation criteria do you apply when using, or searching for, external data sources? Q2. Can you give an example of a data quality issue or concern you experience frequently?
- 10. www.adaptcentre.ieKey Findings – Q1 Q1. When completing different metadata tasks, what evaluation criteria do you apply when using, or searching for, external data sources? Q2. Can you give an example of a data quality issue or concern you experience frequently?
- 11. www.adaptcentre.ieKey Findings – Q1 GOAL: Understand what fitness for use means for the survey participants in a digital library scenario. • 11 dimensions and 2 generic options (none, other) o Trustworthiness, Interoperability, Licensing, Completeness, Understandability, Provenance, Timeliness, Syntactic Validity, Availability, Conciseness, Versatility
- 13. www.adaptcentre.ieKey Findings – Q1 • Statistical Testing: Do both groups consider each measure to be of equal importance or otherwise? • Z-score (α = 0.05) • Reject null hypothesis for: Trustworthiness, Interoperability and Availability.
- 14. www.adaptcentre.ieKey Findings – Q2 Q1. When completing different metadata tasks, what evaluation criteria do you apply when using, or searching for, external data sources? Q2. Can you give an example of a data quality issue or concern you experience frequently?
- 15. www.adaptcentre.ieKey Findings – Q2 GOAL: Understanding quality pitfalls in Linked Data datasets for Digital Libraries. • Open question: o 92 responses => 77 quality problems o 14 different quality measures
- 17. www.adaptcentre.ieKey Findings – Q2 • Semantic Accuracy o Incorrect DOIs o Wrong ISBNs, URI references • Completeness / Data Coverage o Incomplete crowdsource efforts o Incomplete important fields (e.g. publication date) o Using old standards hence having incomplete obligatory fields.
- 18. www.adaptcentre.ieKey Findings – Q2 • Interoperability o Lacks structured standards o Metadata formats changing constantly • Data formatting o Inconsistent formatting of dates o Naming inconsistencies (e.g. first name, last name vs last name, first name)
- 19. www.adaptcentre.ieKey Findings – Q2 • Other problems o Conciseness - duplication o Language Versatility – encoding problems o Availability – resources are not always available o Trustworthiness – credibility of the information on the Web o Licensing – using open datasets freely
- 20. www.adaptcentre.ieNext Steps • Assess the quality of LD digital libraries o Started a monthly assessment in August 2018 o Some results can be seen at http://luzzu.adaptcentre.ie • Identify a quality profile to generalise an answer for “What is a good digital library?”
- 21. www.adaptcentre.ieConclusion • Discussed and identified the quality measures an IP considers for finding external sources o no agreement on importance or otherwise for 3 metrics (trustworthiness, interoperability, and availability) • Discussed quality problems as identified by the IPs in the currently available data sources o Mostly intrinsic in nature jeremy.debattista@adaptcentre.ie twitter: @jerdeb
Editor's Notes
- say that we use the term IPs for people working the the DL domain
- ASK the room: what is a good digital library? most literature state that the success of digital libraries is mostly dependent on the quality of the available metadata however, this is quite ambigious, as defining quality is subjective and mostly depends on the task at hand, in this case different institutions have different needs which are mostly coupled with information professional experiences and roles. an easy question to answer, however cannot generalise which library is best for all cases and suitable everyone
- IPs realised that LD offers many benifits: since we are using a standardised data model sharing and re-use of metadata across DLs, potentially reducing record duplication discoverability by various agents (e.g. using rdfa within html pages would enable search engines such as google to retrieve your data in a meaningful manner) interlink related resources
- - challenges in LD, mostly wrt quality
- we did a survey to understand: (1) what kind of quality measures IPs consider important; (2) what are the problems they face when using linked data for (1) we want to build a system that automatically suggest what quality measures one need for a particular task at hand – which is what we are working on; (2) figure out what kinds of quality issues they encounter and try to validate these issues by actually assessing various linked data digital libraries
- Results of OCLC survey & Library survey Analysis of LAM LD projects & LD tooling Prior work with Digital Repository of Trinity College Dublin (McKenna et al, 2017)
- The 185 questionnaires that were analysed were classified into two groups: participants who have experience working with Linked Data (N = 54) (group 1), and participants who do not have experience working with Linked Data (N = 131) (group 2).
- our goal: understand what fitness for use mean to the participants in different groups in DLs provided a list of quality measures taken from various literature, trying to represent the important measures for both DL and LD – Trustworthiness (e.g. Can this provider be trusted that all data is correct?) – Interoperability (e.g. Does the external source use well-known standard schemas to represent the data?) – Licensing issues (e.g. Can I use this external source freely?) – Completeness (e.g. Do all external metadata fields have values?) – Understandability (e.g. Are all records in the external source labelled and ready for human consumption?) – Provenance (e.g. Does the external source provide provenance/origin information on the data?) – Timeliness (e.g. Are all records up to date?) – Syntactic validity (e.g. Are dates in the correct format, correct spelling?) – Availability of the external source (e.g. SPARQL endpoint is accessible) – Conciseness (e.g. Is there any redundancy within the external source?) – Versatility (e.g. Is the data available in different languages?)
- all participants answered this question, 16 of them being unsure or do not care about quality at all. We also had 9 participants mentioned dimensions different that those listed – marked in a brackets in the table and in italics font The aggregated results show that trustworthiness seems to be the most frequently selected criteria, around 67%, followed by interoperability and licensing however we cannot statistically or scientifically state that trustworthiness is the most important criteria as we cannot assume that participants chose on basis of whats the most important
- statistical test to find evidence whether the two groups consider a measures to be of equal importance or otherwise for this we defined a null hypothesis and an alternative hypothesis and used z-score with a significance level of 0.05 to identify whether there is enough strong evidence to reject of accept the null hypothesis the tests show that there is no supporting evidence that suggest the measures trustworthiness, interoperability , and availability are of equal importance.
- the goal of this 2nd survey was the understand better the quality problems IPs find in LD and hence are contributing to the slow uptake this question was an open question and was answered by 92 participants, of which 15 were out of scope the rest where classified into 14 different measures, including semantic accuracy, completeness, and conciseness
- most problems were intrinsic in nature, meaning that problems where related to the data in the dataset itself followed by representational problems, meaning the way how data is represented for consumption
- not going through all problems, but will mention some interesting ones in semantic accuracy, most participants complained about the presence of incorrect values in various fields of a catalogue resource, mostly due to mispellings in completeness, a number of participants shed doubt to whether information in crowdsource efforts such as wikipedia (and hence dbpedia) are correct and complete. furthermore, participants also complained datasets that are still using old best practices and standards and hence having incomplete fields, such as the publication data
- wrt interoperability, participants mostly noted that there is a lack of consensus on what standards to use and even when there seem to be an agreement, the formats are constantly changing on similar lines to interoperability, participants noted that there are also inconsistnecies on how to represent the data within fields, such as dates and naming standards
- there were other problems highlighted by the participants, for example the duplication of records, introducing redundancy and increasing errors encoding of characters, e.g. usage of cyrillic alphabet in international authority data the 24/7 availability of data and reliability of online services a participant noted that he/she would trust more datasets that were published by his/her own institutions rather than rely on information readily available on the web to what extent can I use a particular external dataset, if the license is not clear or readily available in the dataset?
- - Discussed and identified the quality measures an IP considers for finding external sources - no agreement on importance for 3 metrics (trustworthiness, interoperability, and availability) Discussed quality problems as identified by the IPs in the current available data sources problems are mostly intrisic in nature, identifying semantic accuracy and interoperability as worrying dimensions in which LD should excel in should serve as a starting point for LD publishers to update their publishing mechanisms