Linked Library Data in the wild

Linked Library Datain the wild

Technical Lead for Prism
Phil John
Introductions...

So, what’s Prism then?
Introductions...

a next generation discovery interface
Prism
Introductions

(yes…even configuration settings)
Built entirely on Linked Data
Prism

Discovery of library
catalogue resources
Prism
but grander plans afoot...

...some future sources...
Prism
journal metadata
archives/records (e.g. DS Calm)
thesis repositories
rare items/special collections
and more!

SaaS/Cloud Based
Prism

MARC 21 RDF
Performs data conversion
Prism

this ensures it keeps in sync with the LMS
Initial “bulk” conversion then periodic “delta” files
Prism

provided by a suite of RESTful web services
Borrower/Availability data pulled from LMS “live”
Prism

just add .rss to collectionsor .rdf/.nt/.ttl/.json to items
Linked Data API
Prism

The Challenges
Prism

Extracting data from MARC 21
The Challenges

Some quotes...
Extracting Data from MARC 21
...cataloguers may want to look away now

...and even if it does, there are millions of existing records that we’ll want to convert
MARC 21 is not going
away anytime soon...

How are we approaching it?

By tackling it in small chunks!

We’ve created a solution that...
allows us to build the model iteratively
compartmentalises code for different sections
provides robustness
is performant
allows us to experiment

Parser
Observer
Handlers
Our conversion pipeline

fires events when it encounters a MARC 21 data structure; very strict with syntax
MARC 21 Parser

listens for MARC 21 data structures and hands control over to one or more handlers
Event Observer

know how to convert MARC 21structures and fields into linked data
Bibliographic Handlers

So, where are we up to?

we tackled this one first as it allows us to reason more fully about the record
Format (and duration)

In theory quite easy...
Format

...in practice not so much...
Format
no code for CD (12cm sound disk, 1.4m/s)
DVD and LaserDisc share(d) a code
LC slow(ish) to support new formats in M21
limited use of control field (007) codings...
...so need to parse text from 3xx, 5xx fields

LDR: 01425ngm a22005058 4504
001: 750785
003: xxxxxxx
005: 20090824164118.0
007: vd||s||||
008: 080623s2007 enk||| e v|eng d
020: , | $c Retail (S24.99) |
024: 3, | $a 7321900108089 |
028: 4, 0 | $a BDY10808 | $b Warner Home Video |
029: , | $a 7321900108089 |
082: , | $a 812
245: 0, 0 | $a Goodfellas | $h [videorecording] / | $c directed by Martin Scorsese ; music by Christopher Brooks
260: , | $b Warner Home Video, | $c 2007. |
300: , | $a 1 Blu-Ray (139 min.) : | $b col. |
306: , | $a 021900 |
366: , | $b 20070611 |
511: , | $a Starring Robert De Niro, Ray Liotta and Joe Pesci
521: 8, | $a BBFC code: 18. |
538: , | $a Blu-Ray. |
700: 1, | $a Scorsese, Martin |
700: 1, | $a Brooks, Christopher |
852: , | $b John Harvard | $c BLU-RAY DISC | $m 18 | $z , $z Blu Ray Disc. 18Cert
Teasing format from a MARC 21 Record

Which gives us...

an important part of the recordto model, or so I’ve been told
Title

Quite tricky because...
Title
don’t want to duplicate data that appears elsewhere (e.g. in 100/700)
‡c must be last subfield in a 245...
...so sometimes data from ‡n / ‡p is in ‡c instead...
...which means we can’t just drop the ‡c 

http://journal.code4lib.org/articles/3832
Got a helping hand from Code4Lib Journal (thanks!)
Title

Now with more title

sounds easy...acronyms from EAN to UPC describing 13 digit codes...right?
Identifier

what are all those other things doing in the ‡a?
...STOP!
Identifier

Identifier
“For a hardbound resource, there is no attempt to use a consistent term other than to use one that conveys the condition intelligibly.”
Library of Congress Rule Interpretation 1.8

(and then validate whatever’s left)
So we need to parse them out
Identifier

LDR: 01425ngm a22005058 4504
001: 750785
003: xxxxxxx
005: 20090824164118.0
007: vd||s||||
008: 080623s2007 enk||| e v|eng d
020: , | $c Retail (S24.99) |
024: 3, | $a 7321900108089 |
029: , | $a 7321900108089 |
082: , | $a 812
300: , | $a 1 Blu-Ray (139 min.) : | $b col. |
306: , | $a 021900 |
366: , | $b 20070611 |
521: 8, | $a BBFC code: 18. |
538: , | $a Blu-Ray. |
700: 1, | $a Brooks, Christopher |
Phew, this one’s easy, no (pbk), (hbk) or even (pbk. , alk. paper) to contend with

Now we can start performing lookups against other sources!

hardest of the lot...
Author

...why?
Author
Newt Scamander
Rowling, J.K. vs Rowling, Joanne K.
Few records with relator term in 100/700 ‡e...
...so we have to parse that from the 245 ‡c...
...and we don’t just deal with English records.

we’ve licensed the names/subjects authority files, and created RDF from them
Library of Congress
to the rescue!
Author

LDR: 01425ngm a22005058 4504
001: 750785
003: xxxxxxx
005: 20090824164118.0
007: vd||s||||
008: 080623s2007 enk||| e v|eng d
020: , | $c Retail (S24.99) |
024: 3, | $a 7321900108089 |
029: , | $a 7321900108089 |
082: , | $a 812
300: , | $a 1 Blu-Ray (139 min.) : | $b col. |
306: , | $a 021900 |
366: , | $b 20070611 |
521: 8, | $a BBFC code: 18. |
538: , | $a Blu-Ray. |
700: 1, | $a Brooks, Christopher | $e music
A contrived example (sorry!) with and without relator terms

Hope you can all read this at the back!

A closer look at
Authority Matching
Author

Some requirements:
Author
needs to be fast...
...(able to process 2M records in several hours)
requires accuracy
must handle pseudonyms and variant spellings

which means that for bulk conversions we aren’t incurring HTTP overhead millions of times
So we store as RDF,
but index in SOLR
Author

You can tell J.K. Rowling is successful, she’s been translated lots

Language/Alternate Graphical Representation

Nice “high impact” feature
Language
allows switching between representations
both forms can be searched for
uses RDF content language feature, so useful for people using machine readable RDF

001: | 3013197
008: | 080624s2007cca0000chid
041: , | $a chi
043: , | $a a-cc--- |
050: , 4 | $a NE1300.8.C6 | $b S48 2007 |
100: 1, | $6 880-01 | $a Shu, Huifang. |
245: 1, 0 | $6 880-02 | $a Fan chensuzi : | $b Min jiannianhuazhong de wenqingfengsu / | $c ShuHuifang, Shen Hong zhu. |
246: 3, 1 | $6 880-03 | $a Min jiannianhuazhong de wenqingfengsu |
250: , | $6 880-04 | $a Di 1 ban. |
260: , | $6 880-05 | $a Beijing : | $b Zhongguo gong renchu ban she, | $c 2007. |
300: , | $a 3, 3, 229 p. : | $b col. ill. ; | $c 24 cm. |
440: , 0 | $6 880-06 | $a Zhongguo min suwenhua cong shu |
700: 1, | $6 880-07 | $a Shen, Hong. |
880: 1, | $6 100-01/$1 | $a 舒惠芳. |
880: 1, 0 | $6 245-02/$1 | $a 凡尘俗子 : | $b 民间年画中的温情风俗 / | $c 舒惠芳, 沈泓著.
880: 3, 1 | $6 246-03/$1 | $a 民间年画中的温情风俗 |
880: , | $6 250-04/$1 | $a 第1版. |
880: , | $6 260-05/$1 | $a 北京 : | $b 中国工人出版社 | $c 2007. |
880: , 0 | $6 440-06/$1 | $a 中国民俗文化丛书 |
880: 1, | $6 700-07/$1 | $a 沈泓. |
852: , | $b Main Library | $c East Asian Coll.,Purple 2 | $h 398.351 | $m S4 |
Dealing with language in MARC 21

tagged with an ISO-639-2 language and masquerading as the field listed in ‡6
Passes 880s back into Observer
Language

Which gives us...

it’s part of the reason we use Linked Data...but it’s got some challenges at the moment
Using/Linking to
External Datasets
The Challenges

Pitfalls:
Language
what if a datasource suffers downtime...
...or worse, is taken offline permanently?
can we trust this data?
can we display it, or is it susceptible to vandalism?

Potential solutions (YMMV):
Language
harvest datasets and keep them close to the app...
...or, if that’s not practical, proxy requests using a caching proxy such as Squid
if using Wikipedia and worried about vandalism...
...check for lots of rapid edits, consider caching (or turning off temporarily)

...or – what we’d like to seehappen to Linked Library Data
The Future...

especially on the peripheries – authority data, author information, links to other resources
More library data as LOD
The Future

seriously – this would makeour lives so much simpler
LMS vendors adopting LOD
The Future

LOD replacing MARC 21 as the standard representation of bibliographic records
The Future

Photo Credits
Slide 15 - http://www.flickr.com/photos/gammaman/5241860326/
Slide 21 - http://www.flickr.com/photos/agizienski/3778965891/
Slide 40 - http://www.flickr.com/photos/54409200@N04/5070012761/
Slide 42 - http://www.flickr.com/photos/proimos/4199675334/
Slide 48 - http://www.flickr.com/photos/maveric2003/91198458/
Slide 63 - http://richard.cyganiak.de/2007/10/lod/
Slide 67 - http://www.flickr.com/photos/markchapmanphoto/5139429152/
Slide 72 - http://www.flickr.com/photos/-bast-/349497988/

http://consulting.talis.com	426
http://gavialib.com	368
http://talis-systems.com	196
http://www.thelibrarynews.com	5
http://www.linkedin.com	3
http://twitter.com	1
http://a0.twimg.com	1

Linked Library Data in the wild

by Phil John

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 1,000

Statistics

Linked Library Data in the wild Presentation Transcript