The Semantic Web in Practice: A Case Study at the Metropolitan Museum of Art

THE

SEMANTIC

WEB IN

PRACTICE

Koven J. Smith and Don Undeen The Metropolitan Museum of Art, NYC

Large amounts of data, multiple sources Collections Management System Digital Asset Management System Bibliographic Records Word Documents Archival Materials Artist Letters Publications Didactic Text/Labels

Madame X: depicts Virginie Amelie Avegno Gautreau, wife of Pierre Gautreau was first shown at the Paris Salon in 1884 is a portrait was created by John Singer Sargent was originally titled “Portrait de Mme ***” is related to a portrait by Gustave Courtois, who painted the same subject is 82.5” by 43.5” was acquired by MMA at the same time as “Elijah On the Fiery Chariot” by William Blake

The Semantic Web

An information network in which the nodes are linked at the DATA level, rather than at the PRESENTATION level.

Primary Problems, or, um, “Goals”

Store our unstructured content, and harvest usable data from it

2. Pull records and documents from multiple sources together into a single, query-able data store

Structured Content Collections Management System Object Record Creator Record Creator Name: John Singer Sargent

Semantic MediaWiki

Triple SUBJECT OBJECT PREDICATE (PROPERTY) “ Madame X” “ Elijah In the Fiery Chariot” acquiredConcurrentlyWith

How it works

The Process

Calais accepts unstructured text and uses sophisticated NLP and machine learning techniques to return intelligent metadata

“ Madame X” John Singer Sargent paintedBy

“ Madame X” 1884 paintedIn

“ Madame X” John Singer Sargent paintedBy 1884 paintedIn NODE NODE NODE PROPERTY PROPERTY

“ Madame X” painting is A John Singer Sargent painter is A 1884 date is A INSTANCES CLASSES

paintedBy 1884 paintedIn “ Madame X” John Singer Sargent painting isA painter isA date isA

painting artwork subClassOf painter artist subClassOf paintedBy madeBy subPropertyOf

paintedBy 1884 paintedIn “ Madame X” John Singer Sargent painting isA painter isA date isA artwork subClassOf artist subClassOf madeBy subPropertyOf

“ Madame X” INFERRED TRIPLE painting isA artwork subClassOf isA “ Madame X” artwork isA(n)

paintedBy 1884 paintedIn “ Madame X” John Singer Sargent painting isA painter isA date isA artwork subClassOf artist subClassOf madeBy subPropertyOf isA isA ONTOLOGY

Using Inference for Data Integration

In previous examples, we’ve built up an ad-hoc ontology of artists and artworks, with some Class and Property definitions.

“ Madame X ” John Singer Sargent Painted By paintedIn 1884 painting painter Is A Is A date Is A artwork artist SubClass Of SubClass Of Made by SubProperty of Is A Is A

<marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> This portion of a MARC XML format represents a book’s Title <IMAGE> <PARAM> <LABEL>Object_Title</LABEL> <VALUE> <STRING>Madame X (Madame Pierre Gautreau)</STRING> </VALUE> </PARAM> </IMAGE> MARC MediaBin TMS Ontologies Can also be IMPORTED from other formats, into triples. AccNo ID 16.53 12 Table OBJECTS “ Madame X” 12 Title ObjectID Table OBJECT_TITLES 33 12 ConstituentID ObjectID Table CONXREFS John Singer Sargent 33 Name ID Table CONSTITUENTS

<marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> XML Import: MARC XML

XML Import: MARC XML <marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> marc_record marc_leader marc_subfield marc_datafield Element Names become CLASSES

XML Import: MARC XML <marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 isA isA isA isA Element Names become CLASSES Individual Elements become INSTANCES Of those classes

XML Import: MARC XML <marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 isA isA isA isA child child child Element Names become CLASSES Individual Elements become INSTANCES Of those classes Parent Elements connected to children Via child relationship

XML Import: MARC XML <marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” code isA isA isA isA child child child tag Element Names become CLASSES Individual Elements become INSTANCES Of those classes Parent Elements connected to children Via child relationship Attributes become Properties

XML Import: MARC XML <marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ 00259nz a2200109n 4500” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text text tag Element Names become CLASSES Individual Elements become INSTANCES Of those classes Parent Elements connected to children Via child property Attributes become Properties Text is connected with the text property

XML Import: MARC XML <marc_record> <marc_leader>00259nz a2200109n 4500</marc_leader> <marc_datafield tag="245" > <marc_subfield code="a">John Singer Sargent and the fall of Madame X.</marc_subfield> </marc_datafield> </marc_record> marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ 00259nz a2200109n 4500” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text text tag

XML Import: MediaBin XML <IMAGE> <PARAM> <LABEL>Object_Title</LABEL> <VALUE> <STRING>Madame X (Madame Pierre Gautreau)</STRING> </VALUE> </PARAM> </IMAGE> This portion of a MediaBin XML record denotes an image’s Title IMAGE PARAM LABEL VALUE STRING IMAGE1 PARAM1 LABEL1 VALUE1 STRING1 “ Object_Title” “ Madame X (Madame Pierre Gautreau)” isA isA isA isA isA text text child child child child

RDB Import: TMS This Portion of TMS database records Represents the Title and Artist of “Madame X” Tools like D2RQ ( free) make it possible to do this translation In real-time, from the SQL database. Data does not need to be “Imported.” AccNo ID 16.53 12 Table OBJECTS “ Madame X” 12 Title ObjectID Table OBJECT_TITLES 33 12 ConstituentID ObjectID Table CONXREFS John Singer Sargent 33 Name ID Table CONSTITUENTS

RDB Import: TMS OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS This Portion of TMS database records Represents the Title and Artist of “Madame X” Tools like D2RQ make it possible to do this translation In real-time, from the SQL database. Data does not need to be “Imported.” Tables Become CLASSES AccNo ID 16.53 12 Table OBJECTS “ Madame X” 12 Title ObjectID Table OBJECT_TITLES 33 12 ConstituentID ObjectID Table CONXREFS John Singer Sargent 33 Name ID Table CONSTITUENTS

RDB Import: TMS OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 isA isA isA isA This Portion of TMS database records Represents the Title and Artist of “Madame X” Tools like D2RQ make it possible to do this translation In real-time, from the SQL database. Data does not need to be “Imported.” Tables Become CLASSES Individual rows become INSTANCES AccNo ID 16.53 12 Table OBJECTS “ Madame X” 12 Title ObjectID Table OBJECT_TITLES 33 12 ConstituentID ObjectID Table CONXREFS John Singer Sargent 33 Name ID Table CONSTITUENTS

RDB Import: TMS OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 ObjectID ObjectID ConstituentID isA isA isA isA This Portion of TMS database records Represents the Title and Artist of “Madame X” Tools like D2RQ make it possible to do this translation In real-time, from the SQL database. Data does not need to be “Imported.” Tables Become CLASSES Individual rows become INSTANCES Relational Keys become Properties connecting INSTANCES AccNo ID 16.53 12 Table OBJECTS “ Madame X” 12 Title ObjectID Table OBJECT_TITLES 33 12 ConstituentID ObjectID Table CONXREFS John Singer Sargent 33 Name ID Table CONSTITUENTS

RDB Import: TMS OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ 12” “ 33” “ John Singer Sargent” “ 16.53” ObjectID ObjectID ConstituentID ID ID AccNo Title Name isA isA isA isA This Portion of TMS database records Represents the Title and Artist of “Madame X” Tools like D2RQ make it possible to do this translation In real-time, from the SQL database. Data does not need to be “Imported.” Tables Become CLASSES Individual rows become INSTANCES Relational Keys become Properties connecting INSTANCES All other columns become Properties AccNo ID 16.53 12 Table OBJECTS “ Madame X” 12 Title ObjectID Table OBJECT_TITLES 33 12 ConstituentID ObjectID Table CONXREFS John Singer Sargent 33 Name ID Table CONSTITUENTS

RDB Import: TMS OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ 12” “ 33” “ John Singer Sargent” “ 16.53” ObjectID ObjectID ConstituentID ID ID AccNo Title Name isA isA isA isA This Portion of TMS database records Represents the Title and Artist of “Madame X” Tools like D2RQ make it possible to do this translation In real-time, from the SQL database. Data does not need to be “Imported.” AccNo ID 16.53 12 Table OBJECTS “ Madame X” 12 Title ObjectID Table OBJECT_TITLES 33 12 ConstituentID ObjectID Table CONXREFS John Singer Sargent 33 Name ID Table CONSTITUENTS

marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text tag IMAGE PARAM LABEL VALUE STRING IMAGE1 PARAM1 LABEL1 VALUE1 STRING1 “ Object_Title” “ Madame X (Madame Pierre Gautreau)” isA isA isA isA isA text text child child child child OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ 12” “ 33” “ John Singer Sargent” ObjectID ObjectID ConstituentID ID ID Title Name isA isA isA isA MARC MediaBin TMS

marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text tag IMAGE PARAM LABEL VALUE STRING IMAGE1 PARAM1 LABEL1 VALUE1 STRING1 “ Object_Title” “ Madame X (Madame Pierre Gautreau)” isA isA isA isA isA text text child child child child OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ 12” “ 33” “ John Singer Sargent” ObjectID ObjectID ConstituentID ID ID Title Name isA isA isA isA Titles

Image of madame x

Object madame x

Book with madame x as subect

Existing Triple-Based Ontologies: CIDOC

Existing Triple-Based Ontologies: E71.Man-Made Thing E35.Title E12.Production Event E39.Actor P11B.participated_in Thing1 Event1 Title1 Actor1 P108B.was_produced_by P131F.is_identified_by P102F.has_title “ Madame X” “ John Singer Sargent” P3F.has_note CIDOC

marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text tag

marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text tag E31.Document subClassOf

marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text tag E35.Title subClassOf E31.Document subClassOf

marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text tag E35.Title P102F.has_title subClassOf SubPropertyOf SubPropertyOf E31.Document subClassOf

marc_record marc_datafield1 marc_leader marc_subfield marc_record1 marc_datafield marc_leader1 marc_subfield1 “ 245” “ a” “ John Singer Sargent and the fall of Madame X” code isA isA isA isA child child child text tag E35.Title P102F.has_title P3F.has_note subClassOf SubPropertyOf SubPropertyOf E31.Document subClassOf

marc_record1 marc_subfield1 “ John Singer Sargent and the fall of Madame X” E35.Title P102F.has_title P3F.has_note E31.Document isA

IMAGE PARAM LABEL VALUE STRING IMAGE1 PARAM1 LABEL1 VALUE1 STRING1 “ Object_Title” “ Madame X (Madame Pierre Gautreau)” isA isA isA isA isA text text child child child child

IMAGE PARAM LABEL VALUE STRING IMAGE1 PARAM1 LABEL1 VALUE1 STRING1 “ Object_Title” “ Madame X (Madame Pierre Gautreau)” isA isA isA isA isA text text child child child child E35.Title P102F.has_title P3F.has_note SubPropertyOf E38.Image subClassOf subClassOf SubPropertyOf

IMAGE1 STRING1 “ Madame X (Madame Pierre Gautreau)” isA E35.Title P102F.has_title P3F.has_note E38.Image isA

OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ 12” “ 33” “ John Singer Sargent” “ 16.53” ObjectID ObjectID ConstituentID ID ID AccNo Title Name isA isA isA isA

OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ John Singer Sargent” ObjectID ObjectID ConstituentID Title Name isA isA isA isA

E71.Man-Made Thing E35.Title E12.Production Event E39.Actor P108B.was_produced_by P131F.is_identified_by P102F.has_title P3F.has_note OBJECTS OBJECT_TITLES CONXREFS CONSTITUENTS Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ John Singer Sargent” ObjectID ObjectID ConstituentID Title Name isA isA isA isA P102B.is_title_of P108F.produced P11F.had_participant SubClassOf SubClassOf SubClassOf SubClassOf subPropertyOf subPropertyOf inversePropertyOf inversePropertyOf subPropertyOf subPropertyOf subPropertyOf

E71.Man-Made Thing E35.Title E12.Production Event E39.Actor P102F.has_title P3F.has_note Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ John Singer Sargent” isA isA P108B.was_produced_by P11F.had_participant isA P131F.is_identified_by isA

E71.Man-Made Thing E35.Title E12.Production Event E39.Actor E38.Image E31.Document has_title has_note Object12 ObjectTitle12 ConXRefs1233 Constituents33 “ Madame X” “ John Singer Sargent” was_produced_by had_participant is_identified_by IMAGE1 STRING1 “ Madame X (Madame Pierre Gautreau)” has_title has_note marc_record1 marc_subfield1 “ John Singer Sargent and the fall of Madame X” has_title has_note isA isA isA isA isA isA isA isA

SELECT DISTINCT ?found ?node ?rootNode ?rootText WHERE{ FILTER(fn:matches(?found, ‘madame x’,’I’)). ?node has_note ?found . ?node composite:hasRootNode ?rootNode . ?rootNode has_title ?rootTitle . ?rootTitle has_note ?rootText . }

Resources - Tools

Installing Semantic MediaWiki using Halo - http:// semanticweb.org/wiki/Halo_Extension_Installation

D2RQ (SQL to RDF tool) - http://www4.wiwiss.fu-berlin.de/bizer/d2rq/

TopQuadrant - http:// www.topquadrant.com / (some of the ontology modeling for this pres. was done using TopBraid Composer)

Protégé (nice free modeling tool) - http:// protege.stanford.edu /

Sesame (RDF triple store) - http:// www.openrdf.org /

Mulgara (RDF triple store) - http:// www.mulgara.org /

Resources – Further Reading

Dean Allemang & Jim Hendler, Semantic Web for the Working Ontologist

RDF Primer - http://www.w3.org/TR/REC-rdf-syntax/

SPARQL - http://www.w3.org/TR/rdf-sparql-query/

Jena (application framework) - http:// jena.sourceforge.net /

Additional Resources

Semantic Museum discussion group:

http:// groups.google.com/group/semuse

Semantic Museum wiki:

http://semuse.org

These slides:

http://kovenjsmith.com/pres/mcn_2008.ppt

The Semantic Web in Practice: A Case Study at the Metropolitan Museum of Art

2 comments

Notes on slide 1

12 Favorites & 1 Event