A Semantic Multimedia Web: Create, Annotate, Present and Share your Media Raphaël Troncy, < [email_address] > CWI, Interactive Information Access

Agenda The Three Ingredients for Designing Interfaces Semantic Annotation of Multimedia Content Multimedia metadata formats: use cases and requirements Towards an ontology for media objects Interacting with Linked Data Interactive exploration of heterogeneous cultural heritage collections Searching and Browsing Multimedia Semantic Datasets with Cliopatria Towards Configuring Semantic Web Interfaces by Data Mapping

The Three Ingredients for Designing Interfaces

Semantic Annotation of Multimedia Content

Multimedia metadata formats: use cases and requirements

Towards an ontology for media objects

Interacting with Linked Data

Interactive exploration of heterogeneous cultural heritage collections

Searching and Browsing Multimedia Semantic Datasets with Cliopatria

Towards Configuring Semantic Web Interfaces by Data Mapping

The problem Too many users need their own information for their level of expertise using appropriate media in an appropriate style displayed on their own device Multimedia information design is expensive There has to be some automation in the process

Too many users need their own information

for their level of expertise

using appropriate media

in an appropriate style

displayed on their own device

Multimedia information design is expensive

There has to be some automation in the process

Three ingredients Content

Content of example Clair-obscur (Frans) en chiaroscuro (Italiaans) betekenen 'licht-donker'. Beide termen worden gebruikt om sterke licht-donkercontrasten in schilderijen, tekeningen en prenten aan te duiden. Hoewel het effect al eerder werd toegepast, is de term pas sinds het einde van de 16de eeuw in zwang. De oorsprong van het woord ligt in Italië. De schilder Caravaggio (1573-1610) maakte het chiaroscuro-effect tot zijn handelsmerk. Hij was een meester in het schilderen van donkere taferelen met één felle lichtbundel .

Three ingredients Content Presentation structure

Presentation structure of example Clair-obscur (Frans) en chiaroscuro (Italiaans) betekenen 'licht-donker'. Beide termen worden gebruikt om sterke licht-donkercontrasten in schilderijen, tekeningen en prenten aan te duiden. Hoewel het effect al eerder werd toegepast, is de term pas sinds het einde van de 16de eeuw in zwang. De oorsprong van het woord ligt in Italië. De schilder Caravaggio (1573-1610) maakte het chiaroscuro-effect tot zijn handelsmerk. Hij was een meester in het schilderen van donkere taferelen met één felle lichtbundel . Chiaroscuro & Rembrandt Harmensz. Van Rijn description examples title

Three ingredients Content Presentation structure Aesthetics abcdefghij abcdefghijklmn ABCDEFGHIJK abcdefghijklm abcdefghijklm

Aesthetics of example Colours Fonts Layout Colours

Design dependencies Content Presentation structure Aesthetics abcdefghij abcdefghijk ABCDEFGHIJK abcdefghijk abcdefghijk

Presentation Structure depends on Layout description title title description description examples title description examples title

Style Depends on Content Chiaroscuro & Rembrandt The Stone Bridge (1638) Chiaroscuro & Rembrandt The Stone Bridge (1638)

Presentation structure depends on content Rembrandt Caravaggists chiaroscuro 1631 1628 1638 1623 1628

Example Presentation Structure Rembrandt Chiaroscuro Caravaggists 1631 1628 1638 1623 1628

Agenda The Three Ingredients for Designing Interfaces Semantic Annotation of Multimedia Content Multimedia metadata formats: use cases and requirements Towards an ontology for media objects Interacting with Linked Data Interactive exploration of heterogeneous cultural heritage collections Searching and Browsing Multimedia Semantic Datasets with Cliopatria Towards Configuring Semantic Web Interfaces by Data Mapping

The Three Ingredients for Designing Interfaces

Semantic Annotation of Multimedia Content

Multimedia metadata formats: use cases and requirements

Towards an ontology for media objects

Interacting with Linked Data

Interactive exploration of heterogeneous cultural heritage collections

Searching and Browsing Multimedia Semantic Datasets with Cliopatria

Towards Configuring Semantic Web Interfaces by Data Mapping

Multimedia Web Tagging Semantic Web

Making Music Out Of The Social Noise

W3C Multimedia Semantics XG (2007) W3C Multimedia Semantics XG http://www.w3.org/2005/Incubator/mmsem/

Managing Personal Photos Interoperable Image Metadata Combining EXIF, MPEG-7, IPTC and DIG35 metadata using RDF and OWL schemas

Interoperable Image Metadata

Combining EXIF, MPEG-7, IPTC and DIG35 metadata using RDF and OWL schemas

Facetting Music Songs Interoperable Music and Social Metadata ID3 Tags + low-level features extraction + lastFM recommendations + FOAF profiles + ... Auto-construction of playlist (similar bit rate), Personalization, Browsing music store

Interoperable Music and Social Metadata

ID3 Tags + low-level features extraction + lastFM recommendations + FOAF profiles + ...

Auto-construction of playlist (similar bit rate), Personalization, Browsing music store

Metadata Working Group

W3C Media Annotations WG (2009) http://www.w3.org/2008/WebVideo/Annotations/

W3C Media Annotations WG Many standards: EXIF, ID3, MPEG 7, SMIL, iTunes XML, Yahoo! MediaRSS, CableLabs VOD Metadata Content, Video sitemaps, IPTC, EBU, etc. Develop a simple lingua-franca ontology

Many standards: EXIF, ID3, MPEG 7, SMIL, iTunes XML, Yahoo! MediaRSS, CableLabs VOD Metadata Content, Video sitemaps, IPTC, EBU, etc.

Develop a simple lingua-franca ontology

W3C Media Annotations WG Output Use Cases and Requirements for Ontology and API for Media Object 1.0 Working Draft, 19 January 2009 Giant Table of mapping between multimedia formats with XMP as the pivot format Common Table, 15 April 2009 Ontology for Media Objects 1.0 Table, 28 April, 2009

Use Cases and Requirements for Ontology and API for Media Object 1.0

Working Draft, 19 January 2009

Giant Table of mapping between multimedia formats with XMP as the pivot format

Common Table, 15 April 2009

Ontology for Media Objects 1.0

Table, 28 April, 2009

Agenda The Three Ingredients for Designing Interfaces Semantic Annotation of Multimedia Content Multimedia metadata formats: use cases and requirements Towards an ontology for media objects Interacting with Linked Data Interactive exploration of heterogeneous cultural heritage collections Searching and Browsing Multimedia Semantic Datasets with Cliopatria Towards Configuring Semantic Web Interfaces by Data Mapping

The Three Ingredients for Designing Interfaces

Semantic Annotation of Multimedia Content

Multimedia metadata formats: use cases and requirements

Towards an ontology for media objects

Interacting with Linked Data

Interactive exploration of heterogeneous cultural heritage collections

Searching and Browsing Multimedia Semantic Datasets with Cliopatria

Towards Configuring Semantic Web Interfaces by Data Mapping

Design interactive interfaces to explore a large heterogeneous graph Presentation @ Talis December 2008

Browsing Linked Data Tabulator (CSAIL, MIT)

Browsing Linked Data Disco (Free University of Berlin)

Browsing Linked Data Marbles (Free University of Berlin)

Browsing Linked Data Zitgist (Zitgist LLC)

Browsing Linked Data OpenLink Data Explorer (OpenLink Software)

Browsing Linked Data (more) RKBExplorer: http://www.rkbexplorer.com/ Humboldt: http://humboldt-project.org/ Fenfire: http://fenfire.org/ Cliopatria: http://e-culture.multimedian.nl/software/ClioPatria.shtml

RKBExplorer: http://www.rkbexplorer.com/

Humboldt: http://humboldt-project.org/

Fenfire: http://fenfire.org/

Cliopatria: http://e-culture.multimedian.nl/software/ClioPatria.shtml

cartoons videos

animations blogs

News Production

News Production: Complex & Dynamic Workflows

News Production: As is Example: VRT, a Flemish broadcaster Classical broadcast: linear, 5 broadcasts a day On-demand (“ooit gemist”): replay of the linear broadcasts Web (“deredactie.be”): news clips, continuous updates Linear broadcasts have a rigid structure: Max 20 news items & Max 40 minutes More content available than broadcasted News feeds (EBU/Eurovision, Reuters, …) Produced items never or only a few times broadcasted User’s wishes: enable a news consumer to compose his own news program without interfering with current news production processes

Example: VRT, a Flemish broadcaster

Classical broadcast: linear, 5 broadcasts a day

On-demand (“ooit gemist”): replay of the linear broadcasts

Web (“deredactie.be”): news clips, continuous updates

Linear broadcasts have a rigid structure:

Max 20 news items & Max 40 minutes

More content available than broadcasted

News feeds (EBU/Eurovision, Reuters, …)

Produced items never or only a few times broadcasted

User’s wishes: enable a news consumer to compose his own news program without interfering with current news production processes

Metadata

Metadata Information on the news story Content description (who, what, where, when, why, …) Administrative information (reporter, cameraman, …) Rights information (usage terms, copyright statement, …) Technical information (video format, audio format, …) Past: plain text/proprietary structured text (e.g. telex, XML) Present: News-ML / News-ML G2 (XML-based) Future: Semantic News-ML (RDF-based)

Information on the news story

Content description (who, what, where, when, why, …)

Administrative information (reporter, cameraman, …)

Rights information (usage terms, copyright statement, …)

Technical information (video format, audio format, …)

Past: plain text/proprietary structured text (e.g. telex, XML)

Present: News-ML / News-ML G2 (XML-based)

Future: Semantic News-ML (RDF-based)

News Workflow Interoperability No integration of media (stories, photo, animation, video) Little (or no) context in the news presentation Lack of interoperability in the current workflow NAR Schema Controlled Vocabularies Broadcaster Schema NewsCodes User Vocabulary

No integration of media (stories, photo, animation, video)

Little (or no) context in the news presentation

Lack of interoperability in the current workflow

Metadata is Key (Ultimate) Goal: Provide an environment for searching and browsing contextualized multimedia news information Required integration: Data: various media, different forms, various sources Metadata: schema integration, semantic models Influence and implications of UI: How to represent semantic multimedia metadata to facilitate presenting information? in other words ... What constraints do end-user interfaces put on the modeling of the metadata?

(Ultimate) Goal:

Provide an environment for searching and browsing contextualized multimedia news information

Required integration:

Data: various media, different forms, various sources

Metadata: schema integration, semantic models

Influence and implications of UI:

How to represent semantic multimedia metadata to facilitate presenting information?

in other words ... What constraints do end-user interfaces put on the modeling of the metadata?

News and Multimedia Formats (NAR) News Architecture NewsML G2 EventsML G2 SportsML G2

Porting Schemas and Thesauri to the Semantic Web Methodologies and tools for building ontologies: ... from scratch ʺ SKOSification ʺ of thesauri in the CH domain: preparation, syntactic and semantic conversion, standardization  Lack of best practices for modeling ontologies from UML diagrams, integrating ontologies with various thesauri, while taking the end-user interface into account

Methodologies and tools for building ontologies: ... from scratch

ʺ SKOSification ʺ of thesauri in the CH domain:

preparation, syntactic and semantic conversion, standardization

Building a Semantic Web Infrastructure for News 1 2 3 4 Modeling the NAR ontology Linking with media ontologies Building SKOS thesauri Enriching the metadata

Step 1: Modeling the NAR Ontology  focus on reuse of XML types leading to multiple repetition resulting in overly complex nested XML structures

Step 1: Modeling the NAR Ontology Flattening the XML structure PhotoNewsItem NewsItem

Flattening the XML structure

Step 1: Modeling the NAR Ontology Modeling unique identifiers Use of dereferencable URIs for any resources (news items + vocabularies) Future: Use of URIs for resource fragments http://www.youtube.com/watch?v=1bibCui3lFM#t=1m45s Modeling the provenance of the information Reification Named (and Networked) Graphs {<> nar:subject cat:11002000} dc:creator team:md ; dc:modified ‘‘2005-11-11T08:00:00Z’’.

Modeling unique identifiers

Use of dereferencable URIs for any resources (news items + vocabularies)

Future: Use of URIs for resource fragments http://www.youtube.com/watch?v=1bibCui3lFM#t=1m45s

Modeling the provenance of the information

Reification

Named (and Networked) Graphs

Step 2: Linking with Media Ontologies foaf:Person ≈ nar:Person dc:Subject ≈ nar:Subject sioc:Item ≈ nar:Item geo:lat geo:long +

Step 3: Getting SKOS Vocabularies

Step 3: Getting SKOS Vocabularies

Step 4: Enriching the News Metadata Concepts/Entities that are subject of news Thematic categories People Organizations Geopolitical Areas Points of Interest Events Products or artefacts

Concepts/Entities that are subject of news

Thematic categories

People

Organizations

Geopolitical Areas

Points of Interest

Events

Products or artefacts

Step 4: Enriching the News Metadata NAR Ontology NewsCodes Thesaurus Named Entity Recognition Domain Ontologies

Step 4: Enriching the News Metadata NAR Ontology NewsCodes Thesaurus Domain Ontologies Concept Detectors

Web of Data and Linked Data dbpedia:Zidane foaf:depicts nar:location geonames:2950159 nar:subject nc:15054000 events:id wp:2006_FIFA_World_Cup#Final

Presenting News Information Dimensions used for searching news items When time 10/07/2006 Where location Paris What is depicted J. Chirac, Z. Zidane Why event WC 2006 Who photographer Bertrand Guay, AFP Metadata

Dimensions used for searching news items

When time 10/07/2006

Where location Paris

What is depicted J. Chirac, Z. Zidane

Why event WC 2006

Who photographer Bertrand Guay, AFP

Semantic Search of Multimedia News Powered by ClioPatria 1.0 alpha 3 1,067,754 Total 1,932 INA Broadcast Video (June/July 2006) 61,311 AFP Photos (June/July 2006) 804,446 AFP News Feed (June/July 2006) 53,468 DBpedia, Geonames 34,903 Thesauri: newscodes 104,358 Domain Specific Ontologies: football 7,336 General Ontologies: NAR, DC, FOAF Number of RDF Triples Description

Provide new dimensions for browsing Distinguish field images vs stadium and street images with a grass detector for the World Cup dataset

Credits Datasets: People: More info: http:// newsml.cwi.nl

Datasets:

People:

More info: http:// newsml.cwi.nl

ABC Event Model

CIDOC CRM Event Model

CultureSampo Event Model

Framenet Event Model

Motools Event Model

Towards an Linked Events Model?

Configuring Semantic Web Interfaces by Data Mapping Presentation @ VISSW Visual Interfaces to the Social and the Semantic Web February 2009

Literature Michiel Hildebrand, Jacco van Ossenbruggen and Lynda Hardman: /facet: A Browser for Heterogeneous Semantic Web Repositories . In 5th International Semantic Web Conference (ISWC'2006) , pages 272-285, Athens (GA), USA, November 5-9, 2006. Jan Wielemaker, Michiel Hildebrand, Jacco van Ossenbruggen and Guus Schreiber: Infrastructure for thesaurus-based search and annotation: evauating the standards . In 7th International Semantic Web Conference (ISWC'2008) , pages ?-?, Karlsruhe, Germany, October 26-30, 2008. Raphaël Troncy: Bringing the IPTC News Architecture into the Semantic Web . In 7th International Semantic Web Conference (ISWC'2008) , pages 483-498, Karlsruhe, Germany, October 26-30, 2008.

Michiel Hildebrand, Jacco van Ossenbruggen and Lynda Hardman: /facet: A Browser for Heterogeneous Semantic Web Repositories . In 5th International Semantic Web Conference (ISWC'2006) , pages 272-285, Athens (GA), USA, November 5-9, 2006.

Jan Wielemaker, Michiel Hildebrand, Jacco van Ossenbruggen and Guus Schreiber: Infrastructure for thesaurus-based search and annotation: evauating the standards . In 7th International Semantic Web Conference (ISWC'2008) , pages ?-?, Karlsruhe, Germany, October 26-30, 2008.

Raphaël Troncy: Bringing the IPTC News Architecture into the Semantic Web . In 7th International Semantic Web Conference (ISWC'2008) , pages 483-498, Karlsruhe, Germany, October 26-30, 2008.

Assignment (follow-up)

Assignment (task) Build your ontology Pick a domain Choose its relevant concepts and properties Sketch a first model (draw on paper, using online tool) Search for existing concepts and properties and refine your ontology by asserting equivalences: link your model! Use Semantic Web Search Engines Populate your ontology with samples from the Linked Data Cloud Look up entities from the cloud Report your experience Send the ontology and the knowledge base: owl, rdf, skos files Check for their validity: visualize the data in some tools Write a short report (txt, html) summarizing the whole process

Build your ontology

Pick a domain

Choose its relevant concepts and properties

Sketch a first model (draw on paper, using online tool)

Search for existing concepts and properties and refine your ontology by asserting equivalences: link your model!

Use Semantic Web Search Engines

Populate your ontology with samples from the Linked Data Cloud

Look up entities from the cloud

Report your experience

Send the ontology and the knowledge base: owl, rdf, skos files

Check for their validity: visualize the data in some tools

Write a short report (txt, html) summarizing the whole process

What are the messages? Features can be extracted and used to describe multimedia content (metadata) Multimedia presentations embody messages Media, structure and aesthetics all contribute to conveying the message The message can be made explicit (more metadata) Media content and metadata can be passed around and among systems We need to agree on how to do this (e.g. canonical processes; COMM; W3C working groups; IPTC) Users can be given much richer and more flexible access to (semantically annotated) content, but … ...we are still figuring out how to do this.

Features can be extracted and used to describe multimedia content (metadata)

Multimedia presentations embody messages

Media, structure and aesthetics all contribute to conveying the message

The message can be made explicit (more metadata)

Media content and metadata can be passed around and among systems

We need to agree on how to do this (e.g. canonical processes; COMM; W3C working groups; IPTC)

Users can be given much richer and more flexible access to (semantically annotated) content, but … ...we are still figuring out how to do this.

Thanks for your attention http:// www.cwi.nl/~troncy