SlideShare a Scribd company logo
Validating 126 million
MARC records
DATeCH 2019, Brussels, 2019-05-10.
Péter Király
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Card catalog at Gent University Library, photo: Pieter Morlion, 2010 CC-BY 4.0
https://commons.wikimedia.org/wiki/File:Boekentoren_2010PM_1179_21H9015.JPGhttp://bit.ly/qa-datech2019
part I. short introduction to MARC
❏ MAchine Readable Cataloging
❏ format and semantic specification
❏ comes from the age of punch cards – information compression required
❏ invented in early 60’s
❏ love to hate criticise it: “MARC must die”*, “Stockholm syndrome of MARC”**
❏ “There are only two kinds of people who believe themselves able to read a
MARC record without referring to a stack of manuals: a handful of our top
catalogers and those on serious drugs.”
* Roy Tennnant http://lj.libraryjournal.com/2002/10/ljarchives/marc-must-die/
** Niklas Lindström at ELAG 2019 https://twitter.com/cm_harlow/status/1126068414928293888
2
http://bit.ly/qa-datech2019
a (pretty printed) example
LDR 01136cnm a2200253ui 4500
001 002032820
005 20150224114135.0
008 031117s2003 gw 000 0 ger d
020 $a3805909810
100 1 $avon Staudinger, Julius, $d1836-1902$0(viaf)14846766
245 10$aJ. von Staudingers Kommentar zum ... / $cJ. von Staudinger.
250 $aNeubearb. 2003 $bvon Jörn Eckert
260 $aBerlin :$bSellier-de Gruyter, $c2003.
300 $a534 p. ;.
500 $aCiteertitel: BGB.
500 $aBandtitel: Staudinger BGB.
700 1 $aEckert, Jörn
852 4 $xRE$bRE55$cRBIB$jRBIB.BUR 011 DE 021 $p000000800147
3
http://bit.ly/qa-datech2019
looks like rocket science...
Apollo 11 (moon landing) source code
https://github.com/chrislgarry/Apollo-11
4
http://bit.ly/qa-datech2019
positional fields - 008
‘801003s1958 ja 000 0 jpn ‘
0 1 2 3
0123456789012345678901234567890123456789
aaaaaabccccddddeeefffgh All materials
IIIIjkLLLLmnopqr Books
ijklmnOOOpqrs Continuing Resources
iijklmNNNNNNOOp Music
IIIIjjklmnOO Maps
Iiijklmn Visual Materials
ijkl Computer Files
i Mixed Materials
lower case = distinct units
upper case = repeatable units
 = undefined position
depends on record
type (calculated from
Leader values)
5
http://bit.ly/qa-datech2019
datafields
repeatable/non-repeatable
Indicator1
Indicator2
Subfield1
, ... , Subfieldn
always 1 char long dictionary term
❏ code
❏ value
❏ free text
❏ dictionary term
❏ fixed format (e.g. yymmdd)
❏ fixed format + dictionary terms (d7i2)
❏ fixed positions + dictionary terms
❏ repeatable/non-repeatable
6
http://bit.ly/qa-datech2019
versions
❏ changes of the standard
❏ no versioning
❏ new, deleted and changed elements every year
❏ localized versions
❏ introducing new fields
❏ overwriting existing fields
❏ mixing localized versions
❏ no notion about the localization
❏ 50+ localizations (international, national, consortial)
7
http://bit.ly/qa-datech2019
size – number of data elements implemented
8
MARC 21 versions total
control fields 7 7
control subfields 211 211
data fields 215 68 283
indicators 175 8 183
subfields 2259 344 2603
3287
http://bit.ly/qa-datech2019
Java classes
qa-metadata-marc.jar
Avram JSON
data model
export
machine readable standard
Remember heroines!
9
http://bit.ly/qa-datech2019
Margaret Hamilton
https://qz.com/726338/
Henriette D. Avram
smithsonianmag.com
Part II.
record validation
and quality assessment
Boekentoren UGent - de belvedère, photo: Michiel Hendryckx, 2013, CC-BY-SA 3.0
https://commons.wikimedia.org/wiki/File:Boekentoren_ugent_belvedere_675.jpg10
http://bit.ly/qa-datech2019
quality assessment workflow
1. ingest
2. measure records
3. aggregate
4. report
5. evaluate with experts (feedback loop)
11
http://bit.ly/qa-datech2019
Improve records
1. ingest data
Bavarian union catalogue (bay) – 27.3 million records; Baden-Würtemberg
union catalogue (bzb) – 23.1 m; Columbia (col) – 6.0 m; Heritage of the
Printed Book Database, CERL (cer) – 6.7 m; German National Bibliography
(dnb) – 16.7 m; Gent (gen) – 1.8 m; Harvard (har) – 13.7 m; Library of
Congress (loc) – 10.1 m; Michigan (mic) – 1.3 m; Finnish National
Bibliography (nfi) – 1.0 m; Repertoire International des Sources Musicales
(ris) – 1.3 m; San Francisco Public Library (sfp) – 0.9 m; Stanford (sta) – 9.4
m; Szeged (szt) – 1.2 m; TIB Hannover (tib) – 3.5 m; Toronto Public Library
(tor) – 2.5 m
union catalogues – national libraries – university libraries – public libraries
12
http://bit.ly/qa-datech2019
2. measure records
$ ./validator [options] [file]
001999999 852 undefined subfield L https://www.loc.gov/...
002000005 035 undefined subfield 9 https://www.loc.gov/...
002000005 852 undefined subfield L https://www.loc.gov/...
002000005 852 undefined subfield L https://www.loc.gov/...
002000008 035 undefined subfield 9 https://www.loc.gov/…
002000011 Leader/17 (leader17) invalid value M https://www.loc.gov/…
002000012 Leader/17 (leader17) invalid value I https://www.loc.gov/...
13
http://bit.ly/qa-datech2019
3. aggregating results – records with issues
14
all filtered
bay 100.0 18.8
bzb 100.0 76.1
cer 2.8 2.8
col 90.4 66.0
dnb 13.9 0.2
gen 40.8 27.3
har 100.0 97.3
loc 30.5 29.3
all filtered
mic 80.8 67.5
nfi 62.1 58.1
ris 99.7 57.1
sfp 82.7 60.4
sta 92.7 92.5
szt 30.8 30.6
tib 100.0 100.0
tor 100.0 74.2
Filtered = issues excluding the undocumented tags and subfields
http://bit.ly/qa-datech2019
issue types
issues on record level
❏ R1 ambiguous linkage
❏ R2 invalid linkage
❏ R3 type error
control field issues
❏ C1 invalid code
❏ C2 invalid value
15
field issues
❏ F1 missing reference
subfield (880$6)
❏ F2 non-repeatable field
❏ F3 undefined field
indicator issues
❏ I1 invalid value
❏ I2 non-empty value
❏ I3 obsolete value
subfield issues
❏ S1 classification
❏ S2 invalid ISBN
❏ S3 invalid ISSN
❏ S4 invalid length
❏ S5 invalid value
❏ S6 repetition
❏ S7 undefined subfield
❏ S8 non well-formatted
value
http://bit.ly/qa-datech2019
number of subfields in catalogues
total 1% 10%
bay 854 144 51
bzb 522 144 65
crl 169 65 39
col 1862 196 59
dnb 575 186 97
gnt 955 122 47
har 2024 154 49
loc 1156 128 40
16
total 1% 10%
mic 1233 138 37
nfi 811 145 54
ris 138 88 52
sfp 1046 125 37
sta 2997 225 64
szt 1210 74 42
tib 46 41 35
tor 1733 163 46
The tool has 2600+ subfield definitions
total: total number of fields, 1% fields availabe in at least 1% of the records, 10%: fields available in at
least 10% of the records.
Top fields (not in the table) – 50%: 13-25 fields, 80%: 4-18 fields, 90%: 0-16 fields
http://bit.ly/qa-datech2019
completeness by field groups
17
summary of errors
18
K-means clustering
Spark (Scala)
increasing number of clusters
decreasing the distance from
the centroids
after a point this gain is not
so big (“elbow effect”) -- in
theory
Big number or low
quality records
small clusters with ‘in
between’ quality records
the acceptable average
clusters with good quality
records
19
http://bit.ly/qa-datech2019 Thompson and Traill (2017) http://journal.code4lib.org/articles/12828
4. report (web UI)
20
http://bit.ly/qa-datech2019
21
http://bit.ly/qa-datech2019
22
http://bit.ly/qa-datech2019
23
http://bit.ly/qa-datech2019
Finding problems with facets
Vandenhoeck und Ruprecht
Vandenhoeck & Ruprecht
Vandenhoeck u. Ruprecht
Vandenhoeck
Vandenhoek & Ruprecht
Vandenhoek und Ruprecht
Bandenhoed und Ruprecht
Vandenhoeck et Ruprecht
Vandenhoeck & Reprecht
Vandenhoed und Ruprecht
V&R unipress
V&R Unipress
V & R Unipress
V & R unipress
24
http://bit.ly/qa-datech2019
est. 1735
cataloging
frontline
intensive backward
cataloging -
maybe importing?
backward
cataloging is still
intensive, the
tendency continues
peak is > 13K
2000-07-10, the “golden day”:
95K new records
forward cataloging
25
http://bit.ly/qa-datech2019
everything else
… at least regarding to this project
code & docs: https://github.com/pkiraly/metadata-qa-marc
Web UI source code: https://github.com/pkiraly/metadata-qa-marc-web
Avram Specification (Jakob Voß):
http://format.gbv.de/schema/avram/specification
https://twitter.com/kiru
peter.kiraly@gwdg.de
26
http://bit.ly/qa-datech2019

More Related Content

Similar to Session7 02.peter kiraly

MARC
MARCMARC
Odp
OdpOdp
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
Jen Aman
 
cse6339-spring15-02.pptx
cse6339-spring15-02.pptxcse6339-spring15-02.pptx
cse6339-spring15-02.pptx
Paul832
 
STM Week: Demonstrating bringing publications to life via an End-to-end XML p...
STM Week: Demonstrating bringing publications to life via an End-to-end XML p...STM Week: Demonstrating bringing publications to life via an End-to-end XML p...
STM Week: Demonstrating bringing publications to life via an End-to-end XML p...
GigaScience, BGI Hong Kong
 
Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...
Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...
Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...
Martin Krallinger
 
Computer Interface for Electroluminescence (EL)
Computer Interface for Electroluminescence (EL)Computer Interface for Electroluminescence (EL)
Computer Interface for Electroluminescence (EL)
Editor IJCATR
 
A Hacking Toolset for Big Tabular Files (3)
A Hacking Toolset for Big Tabular Files (3)A Hacking Toolset for Big Tabular Files (3)
A Hacking Toolset for Big Tabular Files (3)
Toshiyuki Shimono
 
ACS San Francisco 2010 CINF Talk
ACS San Francisco 2010 CINF TalkACS San Francisco 2010 CINF Talk
ACS San Francisco 2010 CINF Talk
Markus Sitzmann
 
Benchmarking Commercial RDF Stores with Publications Office Dataset
Benchmarking Commercial RDF Stores with Publications Office DatasetBenchmarking Commercial RDF Stores with Publications Office Dataset
Benchmarking Commercial RDF Stores with Publications Office Dataset
Ghislain Atemezing
 
Dublin Core Intro
Dublin Core IntroDublin Core Intro
Dublin Core Intro
Rich Wisneski
 
IDW2022: A decades experiences in transparent and interactive publication of ...
IDW2022: A decades experiences in transparent and interactive publication of ...IDW2022: A decades experiences in transparent and interactive publication of ...
IDW2022: A decades experiences in transparent and interactive publication of ...
GigaScience, BGI Hong Kong
 
New ways to communicate in science: perspectives from biodiversity research
New ways to communicate in science: perspectives from biodiversity researchNew ways to communicate in science: perspectives from biodiversity research
New ways to communicate in science: perspectives from biodiversity research
Vince Smith
 
VTD-XML: The Future of XML Processing
VTD-XML: The Future of XML ProcessingVTD-XML: The Future of XML Processing
VTD-XML: The Future of XML Processing
Guo Albert
 
Beyond the Record : OCLC & the Future of MARC
Beyond the Record : OCLC & the Future of MARCBeyond the Record : OCLC & the Future of MARC
Beyond the Record : OCLC & the Future of MARC
tfons
 
MOAR RDA For Systems Folks
MOAR RDA For Systems FolksMOAR RDA For Systems Folks
MOAR RDA For Systems Folks
Alison Hitchens
 
SC4 Workshop 2: Soren Auer BDE project Overview
SC4 Workshop 2: Soren Auer BDE project OverviewSC4 Workshop 2: Soren Auer BDE project Overview
SC4 Workshop 2: Soren Auer BDE project Overview
BigData_Europe
 
Change Tracking in Knowledge Organization Systems with skos-history
Change Tracking in Knowledge Organization Systems with skos-historyChange Tracking in Knowledge Organization Systems with skos-history
Change Tracking in Knowledge Organization Systems with skos-history
Joachim Neubert
 
Xbfs HPDC'2019
Xbfs HPDC'2019Xbfs HPDC'2019
Xbfs HPDC'2019
Anil Gaihre
 
Exploring the Great Olympian Graph
Exploring the Great Olympian GraphExploring the Great Olympian Graph
Exploring the Great Olympian Graph
Neo4j
 

Similar to Session7 02.peter kiraly (20)

MARC
MARCMARC
MARC
 
Odp
OdpOdp
Odp
 
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
 
cse6339-spring15-02.pptx
cse6339-spring15-02.pptxcse6339-spring15-02.pptx
cse6339-spring15-02.pptx
 
STM Week: Demonstrating bringing publications to life via an End-to-end XML p...
STM Week: Demonstrating bringing publications to life via an End-to-end XML p...STM Week: Demonstrating bringing publications to life via an End-to-end XML p...
STM Week: Demonstrating bringing publications to life via an End-to-end XML p...
 
Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...
Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...
Named Entity Recognition, Concept Normalization and Clinical Coding: Overview...
 
Computer Interface for Electroluminescence (EL)
Computer Interface for Electroluminescence (EL)Computer Interface for Electroluminescence (EL)
Computer Interface for Electroluminescence (EL)
 
A Hacking Toolset for Big Tabular Files (3)
A Hacking Toolset for Big Tabular Files (3)A Hacking Toolset for Big Tabular Files (3)
A Hacking Toolset for Big Tabular Files (3)
 
ACS San Francisco 2010 CINF Talk
ACS San Francisco 2010 CINF TalkACS San Francisco 2010 CINF Talk
ACS San Francisco 2010 CINF Talk
 
Benchmarking Commercial RDF Stores with Publications Office Dataset
Benchmarking Commercial RDF Stores with Publications Office DatasetBenchmarking Commercial RDF Stores with Publications Office Dataset
Benchmarking Commercial RDF Stores with Publications Office Dataset
 
Dublin Core Intro
Dublin Core IntroDublin Core Intro
Dublin Core Intro
 
IDW2022: A decades experiences in transparent and interactive publication of ...
IDW2022: A decades experiences in transparent and interactive publication of ...IDW2022: A decades experiences in transparent and interactive publication of ...
IDW2022: A decades experiences in transparent and interactive publication of ...
 
New ways to communicate in science: perspectives from biodiversity research
New ways to communicate in science: perspectives from biodiversity researchNew ways to communicate in science: perspectives from biodiversity research
New ways to communicate in science: perspectives from biodiversity research
 
VTD-XML: The Future of XML Processing
VTD-XML: The Future of XML ProcessingVTD-XML: The Future of XML Processing
VTD-XML: The Future of XML Processing
 
Beyond the Record : OCLC & the Future of MARC
Beyond the Record : OCLC & the Future of MARCBeyond the Record : OCLC & the Future of MARC
Beyond the Record : OCLC & the Future of MARC
 
MOAR RDA For Systems Folks
MOAR RDA For Systems FolksMOAR RDA For Systems Folks
MOAR RDA For Systems Folks
 
SC4 Workshop 2: Soren Auer BDE project Overview
SC4 Workshop 2: Soren Auer BDE project OverviewSC4 Workshop 2: Soren Auer BDE project Overview
SC4 Workshop 2: Soren Auer BDE project Overview
 
Change Tracking in Knowledge Organization Systems with skos-history
Change Tracking in Knowledge Organization Systems with skos-historyChange Tracking in Knowledge Organization Systems with skos-history
Change Tracking in Knowledge Organization Systems with skos-history
 
Xbfs HPDC'2019
Xbfs HPDC'2019Xbfs HPDC'2019
Xbfs HPDC'2019
 
Exploring the Great Olympian Graph
Exploring the Great Olympian GraphExploring the Great Olympian Graph
Exploring the Great Olympian Graph
 

More from IMPACT Centre of Competence

Session6 01.helmut schmid
Session6 01.helmut schmidSession6 01.helmut schmid
Session6 01.helmut schmid
IMPACT Centre of Competence
 
Session1 03.hsian-an wang
Session1 03.hsian-an wangSession1 03.hsian-an wang
Session1 03.hsian-an wang
IMPACT Centre of Competence
 
Session7 03.katrien depuydt
Session7 03.katrien depuydtSession7 03.katrien depuydt
Session7 03.katrien depuydt
IMPACT Centre of Competence
 
Session6 04.giuseppe celano
Session6 04.giuseppe celanoSession6 04.giuseppe celano
Session6 04.giuseppe celano
IMPACT Centre of Competence
 
Session6 03.sandra young
Session6 03.sandra youngSession6 03.sandra young
Session6 03.sandra young
IMPACT Centre of Competence
 
Session6 02.jeremi ochab
Session6 02.jeremi ochabSession6 02.jeremi ochab
Session6 02.jeremi ochab
IMPACT Centre of Competence
 
Session5 04.evangelos varthis
Session5 04.evangelos varthisSession5 04.evangelos varthis
Session5 04.evangelos varthis
IMPACT Centre of Competence
 
Session5 03.george rehm
Session5 03.george rehmSession5 03.george rehm
Session5 03.george rehm
IMPACT Centre of Competence
 
Session5 02.tom derrick
Session5 02.tom derrickSession5 02.tom derrick
Session5 02.tom derrick
IMPACT Centre of Competence
 
Session5 01.rutger vankoert
Session5 01.rutger vankoertSession5 01.rutger vankoert
Session5 01.rutger vankoert
IMPACT Centre of Competence
 
Session4 04.senka drobac
Session4 04.senka drobacSession4 04.senka drobac
Session4 04.senka drobac
IMPACT Centre of Competence
 
Session3 04.arnau baro
Session3 04.arnau baroSession3 04.arnau baro
Session3 04.arnau baro
IMPACT Centre of Competence
 
Session3 03.christian clausner
Session3 03.christian clausnerSession3 03.christian clausner
Session3 03.christian clausner
IMPACT Centre of Competence
 
Session3 02.kimmo ketunnen
Session3 02.kimmo ketunnenSession3 02.kimmo ketunnen
Session3 02.kimmo ketunnen
IMPACT Centre of Competence
 
Session3 01.clemens neudecker
Session3 01.clemens neudeckerSession3 01.clemens neudecker
Session3 01.clemens neudecker
IMPACT Centre of Competence
 
Session2 04.ashkan ashkpour
Session2 04.ashkan ashkpourSession2 04.ashkan ashkpour
Session2 04.ashkan ashkpour
IMPACT Centre of Competence
 
Session2 03.juri opitz
Session2 03.juri opitzSession2 03.juri opitz
Session2 03.juri opitz
IMPACT Centre of Competence
 
Session2 02.christian reul
Session2 02.christian reulSession2 02.christian reul
Session2 02.christian reul
IMPACT Centre of Competence
 
Session2 01.emad mohamed
Session2 01.emad mohamedSession2 01.emad mohamed
Session2 01.emad mohamed
IMPACT Centre of Competence
 
Session1 04.florian fink
Session1 04.florian finkSession1 04.florian fink
Session1 04.florian fink
IMPACT Centre of Competence
 

More from IMPACT Centre of Competence (20)

Session6 01.helmut schmid
Session6 01.helmut schmidSession6 01.helmut schmid
Session6 01.helmut schmid
 
Session1 03.hsian-an wang
Session1 03.hsian-an wangSession1 03.hsian-an wang
Session1 03.hsian-an wang
 
Session7 03.katrien depuydt
Session7 03.katrien depuydtSession7 03.katrien depuydt
Session7 03.katrien depuydt
 
Session6 04.giuseppe celano
Session6 04.giuseppe celanoSession6 04.giuseppe celano
Session6 04.giuseppe celano
 
Session6 03.sandra young
Session6 03.sandra youngSession6 03.sandra young
Session6 03.sandra young
 
Session6 02.jeremi ochab
Session6 02.jeremi ochabSession6 02.jeremi ochab
Session6 02.jeremi ochab
 
Session5 04.evangelos varthis
Session5 04.evangelos varthisSession5 04.evangelos varthis
Session5 04.evangelos varthis
 
Session5 03.george rehm
Session5 03.george rehmSession5 03.george rehm
Session5 03.george rehm
 
Session5 02.tom derrick
Session5 02.tom derrickSession5 02.tom derrick
Session5 02.tom derrick
 
Session5 01.rutger vankoert
Session5 01.rutger vankoertSession5 01.rutger vankoert
Session5 01.rutger vankoert
 
Session4 04.senka drobac
Session4 04.senka drobacSession4 04.senka drobac
Session4 04.senka drobac
 
Session3 04.arnau baro
Session3 04.arnau baroSession3 04.arnau baro
Session3 04.arnau baro
 
Session3 03.christian clausner
Session3 03.christian clausnerSession3 03.christian clausner
Session3 03.christian clausner
 
Session3 02.kimmo ketunnen
Session3 02.kimmo ketunnenSession3 02.kimmo ketunnen
Session3 02.kimmo ketunnen
 
Session3 01.clemens neudecker
Session3 01.clemens neudeckerSession3 01.clemens neudecker
Session3 01.clemens neudecker
 
Session2 04.ashkan ashkpour
Session2 04.ashkan ashkpourSession2 04.ashkan ashkpour
Session2 04.ashkan ashkpour
 
Session2 03.juri opitz
Session2 03.juri opitzSession2 03.juri opitz
Session2 03.juri opitz
 
Session2 02.christian reul
Session2 02.christian reulSession2 02.christian reul
Session2 02.christian reul
 
Session2 01.emad mohamed
Session2 01.emad mohamedSession2 01.emad mohamed
Session2 01.emad mohamed
 
Session1 04.florian fink
Session1 04.florian finkSession1 04.florian fink
Session1 04.florian fink
 

Recently uploaded

Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
chetankumar9855
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
HackersList
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 

Recently uploaded (20)

Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 

Session7 02.peter kiraly

  • 1. Validating 126 million MARC records DATeCH 2019, Brussels, 2019-05-10. Péter Király Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG) Card catalog at Gent University Library, photo: Pieter Morlion, 2010 CC-BY 4.0 https://commons.wikimedia.org/wiki/File:Boekentoren_2010PM_1179_21H9015.JPGhttp://bit.ly/qa-datech2019
  • 2. part I. short introduction to MARC ❏ MAchine Readable Cataloging ❏ format and semantic specification ❏ comes from the age of punch cards – information compression required ❏ invented in early 60’s ❏ love to hate criticise it: “MARC must die”*, “Stockholm syndrome of MARC”** ❏ “There are only two kinds of people who believe themselves able to read a MARC record without referring to a stack of manuals: a handful of our top catalogers and those on serious drugs.” * Roy Tennnant http://lj.libraryjournal.com/2002/10/ljarchives/marc-must-die/ ** Niklas Lindström at ELAG 2019 https://twitter.com/cm_harlow/status/1126068414928293888 2 http://bit.ly/qa-datech2019
  • 3. a (pretty printed) example LDR 01136cnm a2200253ui 4500 001 002032820 005 20150224114135.0 008 031117s2003 gw 000 0 ger d 020 $a3805909810 100 1 $avon Staudinger, Julius, $d1836-1902$0(viaf)14846766 245 10$aJ. von Staudingers Kommentar zum ... / $cJ. von Staudinger. 250 $aNeubearb. 2003 $bvon Jörn Eckert 260 $aBerlin :$bSellier-de Gruyter, $c2003. 300 $a534 p. ;. 500 $aCiteertitel: BGB. 500 $aBandtitel: Staudinger BGB. 700 1 $aEckert, Jörn 852 4 $xRE$bRE55$cRBIB$jRBIB.BUR 011 DE 021 $p000000800147 3 http://bit.ly/qa-datech2019
  • 4. looks like rocket science... Apollo 11 (moon landing) source code https://github.com/chrislgarry/Apollo-11 4 http://bit.ly/qa-datech2019
  • 5. positional fields - 008 ‘801003s1958 ja 000 0 jpn ‘ 0 1 2 3 0123456789012345678901234567890123456789 aaaaaabccccddddeeefffgh All materials IIIIjkLLLLmnopqr Books ijklmnOOOpqrs Continuing Resources iijklmNNNNNNOOp Music IIIIjjklmnOO Maps Iiijklmn Visual Materials ijkl Computer Files i Mixed Materials lower case = distinct units upper case = repeatable units = undefined position depends on record type (calculated from Leader values) 5 http://bit.ly/qa-datech2019
  • 6. datafields repeatable/non-repeatable Indicator1 Indicator2 Subfield1 , ... , Subfieldn always 1 char long dictionary term ❏ code ❏ value ❏ free text ❏ dictionary term ❏ fixed format (e.g. yymmdd) ❏ fixed format + dictionary terms (d7i2) ❏ fixed positions + dictionary terms ❏ repeatable/non-repeatable 6 http://bit.ly/qa-datech2019
  • 7. versions ❏ changes of the standard ❏ no versioning ❏ new, deleted and changed elements every year ❏ localized versions ❏ introducing new fields ❏ overwriting existing fields ❏ mixing localized versions ❏ no notion about the localization ❏ 50+ localizations (international, national, consortial) 7 http://bit.ly/qa-datech2019
  • 8. size – number of data elements implemented 8 MARC 21 versions total control fields 7 7 control subfields 211 211 data fields 215 68 283 indicators 175 8 183 subfields 2259 344 2603 3287 http://bit.ly/qa-datech2019 Java classes qa-metadata-marc.jar Avram JSON data model export machine readable standard
  • 10. Part II. record validation and quality assessment Boekentoren UGent - de belvedère, photo: Michiel Hendryckx, 2013, CC-BY-SA 3.0 https://commons.wikimedia.org/wiki/File:Boekentoren_ugent_belvedere_675.jpg10 http://bit.ly/qa-datech2019
  • 11. quality assessment workflow 1. ingest 2. measure records 3. aggregate 4. report 5. evaluate with experts (feedback loop) 11 http://bit.ly/qa-datech2019 Improve records
  • 12. 1. ingest data Bavarian union catalogue (bay) – 27.3 million records; Baden-Würtemberg union catalogue (bzb) – 23.1 m; Columbia (col) – 6.0 m; Heritage of the Printed Book Database, CERL (cer) – 6.7 m; German National Bibliography (dnb) – 16.7 m; Gent (gen) – 1.8 m; Harvard (har) – 13.7 m; Library of Congress (loc) – 10.1 m; Michigan (mic) – 1.3 m; Finnish National Bibliography (nfi) – 1.0 m; Repertoire International des Sources Musicales (ris) – 1.3 m; San Francisco Public Library (sfp) – 0.9 m; Stanford (sta) – 9.4 m; Szeged (szt) – 1.2 m; TIB Hannover (tib) – 3.5 m; Toronto Public Library (tor) – 2.5 m union catalogues – national libraries – university libraries – public libraries 12 http://bit.ly/qa-datech2019
  • 13. 2. measure records $ ./validator [options] [file] 001999999 852 undefined subfield L https://www.loc.gov/... 002000005 035 undefined subfield 9 https://www.loc.gov/... 002000005 852 undefined subfield L https://www.loc.gov/... 002000005 852 undefined subfield L https://www.loc.gov/... 002000008 035 undefined subfield 9 https://www.loc.gov/… 002000011 Leader/17 (leader17) invalid value M https://www.loc.gov/… 002000012 Leader/17 (leader17) invalid value I https://www.loc.gov/... 13 http://bit.ly/qa-datech2019
  • 14. 3. aggregating results – records with issues 14 all filtered bay 100.0 18.8 bzb 100.0 76.1 cer 2.8 2.8 col 90.4 66.0 dnb 13.9 0.2 gen 40.8 27.3 har 100.0 97.3 loc 30.5 29.3 all filtered mic 80.8 67.5 nfi 62.1 58.1 ris 99.7 57.1 sfp 82.7 60.4 sta 92.7 92.5 szt 30.8 30.6 tib 100.0 100.0 tor 100.0 74.2 Filtered = issues excluding the undocumented tags and subfields http://bit.ly/qa-datech2019
  • 15. issue types issues on record level ❏ R1 ambiguous linkage ❏ R2 invalid linkage ❏ R3 type error control field issues ❏ C1 invalid code ❏ C2 invalid value 15 field issues ❏ F1 missing reference subfield (880$6) ❏ F2 non-repeatable field ❏ F3 undefined field indicator issues ❏ I1 invalid value ❏ I2 non-empty value ❏ I3 obsolete value subfield issues ❏ S1 classification ❏ S2 invalid ISBN ❏ S3 invalid ISSN ❏ S4 invalid length ❏ S5 invalid value ❏ S6 repetition ❏ S7 undefined subfield ❏ S8 non well-formatted value http://bit.ly/qa-datech2019
  • 16. number of subfields in catalogues total 1% 10% bay 854 144 51 bzb 522 144 65 crl 169 65 39 col 1862 196 59 dnb 575 186 97 gnt 955 122 47 har 2024 154 49 loc 1156 128 40 16 total 1% 10% mic 1233 138 37 nfi 811 145 54 ris 138 88 52 sfp 1046 125 37 sta 2997 225 64 szt 1210 74 42 tib 46 41 35 tor 1733 163 46 The tool has 2600+ subfield definitions total: total number of fields, 1% fields availabe in at least 1% of the records, 10%: fields available in at least 10% of the records. Top fields (not in the table) – 50%: 13-25 fields, 80%: 4-18 fields, 90%: 0-16 fields http://bit.ly/qa-datech2019
  • 19. K-means clustering Spark (Scala) increasing number of clusters decreasing the distance from the centroids after a point this gain is not so big (“elbow effect”) -- in theory Big number or low quality records small clusters with ‘in between’ quality records the acceptable average clusters with good quality records 19 http://bit.ly/qa-datech2019 Thompson and Traill (2017) http://journal.code4lib.org/articles/12828
  • 20. 4. report (web UI) 20 http://bit.ly/qa-datech2019
  • 24. Finding problems with facets Vandenhoeck und Ruprecht Vandenhoeck & Ruprecht Vandenhoeck u. Ruprecht Vandenhoeck Vandenhoek & Ruprecht Vandenhoek und Ruprecht Bandenhoed und Ruprecht Vandenhoeck et Ruprecht Vandenhoeck & Reprecht Vandenhoed und Ruprecht V&R unipress V&R Unipress V & R Unipress V & R unipress 24 http://bit.ly/qa-datech2019 est. 1735
  • 25. cataloging frontline intensive backward cataloging - maybe importing? backward cataloging is still intensive, the tendency continues peak is > 13K 2000-07-10, the “golden day”: 95K new records forward cataloging 25 http://bit.ly/qa-datech2019
  • 26. everything else … at least regarding to this project code & docs: https://github.com/pkiraly/metadata-qa-marc Web UI source code: https://github.com/pkiraly/metadata-qa-marc-web Avram Specification (Jakob Voß): http://format.gbv.de/schema/avram/specification https://twitter.com/kiru peter.kiraly@gwdg.de 26 http://bit.ly/qa-datech2019