SAMT09 - Web of Data Tutorial - Part 2

Web of Data in the Context of Multimedia
Part 2: Linked Multimedia

Bernhard Haslhofer, Bernhard Schandl, Andreas Langegger,
Wolfgang Halb, Tobias Bürger

1 SAMT 2009 – Tutorial Web of Data in the Context of Multimedia (WoDMM) Graz, Austria - 2 Dec 2009

Agenda
  Multimedia Interlinking: Concept + Vision
  Multimedia Annotations
  Enriching Personal Media Collections


Multimedia Interlinking
  Concept and Vision of a Web of Video Data
  Applying the Linked Data principles to multimedia
content
  W3C Video on the Web activity
  Fragment Identification
  Media Annotation


Vision of a Web of Video Data
  Aim: Make video an integral part of the Web in which
users can seamlessly consume and create videos
  Requirements:
  Deep Linking: timed URIs
  Metadata: Time-aligned annotations & tags
  Search-ability: Searching media using text
  Dynamic composition of video mashups
  Discoverability: Hyperlinking from inside media to other Web
resources
  Share-ability: Integrated handling of metadata with the content
  Personalisation: Dynamic composition of media web content

cf. Silvia Pfeiffer “Architecture of a Video Web - Experience with Annodex” W3C Video on the Web workshop, 13.12.2008


Why?

Source: Cisco's Global Consumer Internet Traffic Forecast, Cisco, 2009


Example 1: Where is Danny?


Example 2: Who is Jack?
  Region-based annotation in Flickr
  Cannot be taken outside of Flickr
  Can only be accessed via the Flickr API


Example 3: Complex Query
1.  show me photos of actors participating in the
SHINING movie visiting a country in Europe
  LinkedMovieDB: list of actors participating in SHINING-: [L-
ACT]
  Geonames: list European countries -: [L-EC]
  Google: list photos taken in a country of [L-EC] -: [L-ECP]
  Google: in [L-ECP] find regions that depict members of [L-
ACT] -: result
2.  give me a summary of all scenes from arbitraty
videos where actors talk to a film producer


Linked Data: Current State of
Practice
  Linked Data currently mostly applied for textual
resources (cf. LOD cloud)
  If any media resources are interlinked then on a
global level (cf. BBC)
  Some communities demand for methods to link on a
more fine grained level (cineasts, biologists, etc.)
[Schroeter and Hunter, 2007]


Problems
  How to address and retrieve parts of multimedia
content (e.g., a region of an image, sequence of a
video)
  How to describe parts of multimedia content in the
Web of Data?

cf. M. Hausenblas, R. Troncy, T. Bürger, and Y. Raimond “How to apply Linked Data principles to multimedia fragments”
Proceedings of LDOW 2009, 2009.


Interlinking Multimedia (iM)
1.  Apply linked data principles for fine-grained
identification and description of multimedia
fragments -> W3C Media Fragments WG
2.  Describe multimedia content -> W3C Media
Annotations WG
3.  Deploy legacy multimedia metadata formats on the
Web of Data -> ramm.x
4.  Develop specialized interlinking methods for
multimedia
  User-contributed, game-based interlinking
  Collaborative interlinking, cf. MetaVidWiki
  (Semi-)automatic methods: Multimedia analysis


Media Fragments WG Goal

Provide URI-based mechanisms for
uniquely identifying fragments for
media objects on the Web, such as
video, audio, and images.

Credits: Slides about Media Fragments WG by the Media Fragments WG.

Media Fragments: Use Case
and Requirements
  Document:
http://www.w3.org/2008/WebVideo/Fragments/wiki/
Use_Cases_&_Requirements_Draft
  Link to, Display, Bookmark, Annotate, Search, etc. media
fragments
  Type of fragments: four dimensions considered
  time ([point?] interval), space ([point?] rectangle), track and id
(named fragment)
  Protocols covered: HTTP(S), RTSP


Technological Survey
  Media Fragments URI already present on the Web ... but with
very different syntax and behaviour
  Google Video: (#50m16s)
  YouTube: (#t=1m45s)
  Archive.org powered by TemporalURI: (?t=74.5 )
  Videosurf: (?t=140&e=184)
  Microsoft IIS 7.0: (Fragments(video=20000000))
  Other time-clipping and spatial clipping methods:
  MPEG-21: time, space, track, id (mp(/CD/track/~time(%27npt
%27,%270%27,%2760%27))
  SVG: space (svgView(14.64,15.73,146.98,147.48))
  SMIL: time (clipBegin, clipEnd), space (panZoom, area), media markers
  MPEG-7: time, space, modality


Media Fragments 1.0
  Syntax
  http://www.example.com/movie.mov#t=12.33,21.16
  http://www.example.com/
movie.mov#t=smpte-25:00:12:33:06,00:21:16:00
  http://www.example.com/movie.mov#xywh=20,20,40,40
  http://www.example.com/movie.mov#xywh=pixel:20,20,40,40
  http://www.example.com/movie.mov#track='audio1'
  http://www.example.com/movie.mov#id='the%20kiss%20scene'
  http://www.example.com/
movie.mov#track='audio2'&t=12.33,21.16


Media Annotation WG: Purpose


Media Annotations WG: Core
Ontology
  Goal: to develop a simple lingua franca


Bringing Legacy Metadata to
the Web of Data: ramm.x
  Problem: How to make existing multimedia metadata in legacy formats (e.g.,
MPEG-7) available on the Semantic Web?
  Aim: To deploy existing legacy metadata in HTML documents and by that
integrate external descriptions in a RICO by reference.
  Ramm.X uses RDFa, formalisations of metadata standards, and the
Ramm.X vocabulary to deploy multimedia metadata.


Interlinking Methods
1.  Automatic Interlinking
2.  Emergent Interlinking
x
3.  User Contributed Interlinking UCI
4.  Game based Interlinking GBI
x
Quality
AI
EI x
x
# Annotations

Time

References
  Michael Hausenblas, Raphael Troncy, Tobias Bürger, and Yves
Raimond "Interlinking Multimedia: How to Apply Linked Data
Principles to Multimedia Fragments" In: Proceedings of Linked Data
on the Web (LDOW2009), co-located with the 18th International
World Wide Web Conference (WWW2009), Madrid, Spain, 2009
  WonSuk Lee, Tobias Bürger, Felix Sasaki, Véronique Malaisé,
Florian Stegmaier, and Joakim Söderberg (eds.) "Ontology for
Media Resource 1.0" W3C Media Annotation Working Group, W3C
Working Draft 18 June 2009, 2009.
  Raphaël Troncy, Erik Mannens, Michael Hausenblas, Jack Jansen,
Yves Lafon, Silvia Pfeiffer, and Davy van Deursen "Media
Fragments URI 1.0" W3C Working Draft 18 November 2009, 2009.
  Silvia Pfeiffer “Architecture of a Video Web - Experience with
Annodex” W3C Video on the Web workshop, 13.12.2008, 2008.


What’ll come next
  Multimedia annotations: The LEMO framework
  Enriching personal media collections


Web of Data in the Context of Multimedia
Part 1: Linked Open Data: Vision, Concepts and Technologies

Bernhard Haslhofer, Bernhard Schandl, Andreas Langegger,
Wolfgang Halb, Tobias Bürger


Agenda
  Introduction   Multimedia Interlinking

  Producing Linked Data   Multimedia Annotations

  Existing Data Sets   Enriching Personal
Media Collections
  Linking Data

  Consuming Linked Data


Many interpretations
  An annotation is…
  … a remark, explanation, or interpretation added to the original
document
  … a means to make implicit structures explicit
  … a datum created by a third party that can take the form of a
written note, a symbol, a drawing, or a multimedia clip
  ...

  There are different interpretation of the term
Annotation in different domains


Many solutions


What is LEMO?
  A Web-based, multimedia annotation framework

  Consisting of
  A semantic backend-end storage
  Front-ends for image, audio, and video annotations

  Online Demo at:
http://dme.arcs.ac.at/annotation/econnect-
annotation-showcase.html


Driving requirements
  Objects to be annotated are multimedia
  Need a uniform data model, no content-type specific solution

  Annotations often address specific content parts
(fragements)
  Need a uniform fragment identification approach

  Shift towards the Web
  Do no lock annotation data in closed data stores but…
  Expose annotation data as Linked Data on the Web (following the
Linked Data Principles)


Driving Requirements
  Reuse of existing standards

  (Semantic) Web Standards
  URI, HTTP, RDF, RDF/S, OWL

  RESTful Web Service Design

  Annotea data model & protocol

  Xpointer, MPEG21, SVG, etc. for fragment
identification


Why not use Annotea as it is?
  Annotea is for the annotation of Web pages; LEMO
should also support annotation of embedded content
items

  Annotea doesn’t assign datatypes to its properties;
LEMO uses OWL’s (data)typing features

  Annotea doesn’t support multimedia fragment
identification; LEMO uses MPEG-21 (multimedia)
fragment identification


A Uniform Annotation Model
  annotation profiles; cf. application profiles for
metadata interoperability

  an Annotea-based core model

  with content- and annotation-type specific model
extensions

  implemented as light-weight software components
(add-ons)


Uniform Fragment Identification
  introduce a fragment element with URI range
  follow MPEG-21 media pointer scheme
  an annotation can define multiple fragments

  URI-encoding of fragments
  minimal consensus
  content-type specific extensions


Annotations as Web Resources


References
  The LEMO Annotation Framework (paper)
  http://www.cs.univie.ac.at/upload//550/papers/
lemo_framework_final.pdf

  The LEMO Annotation Framework (code)
  http://europeanalabs.eu

  Open Annotation Collaboration
  http://www.openannotation.org/


Agenda
  Introduction   Multimedia Interlinking

  Producing Linked Data   Multimedia Annotations

  Existing Data Sets   Enriching Personal
Media Collections
  Linking Data

  Consuming Linked Data


Enriching Personal
Media Collections


Goal
  to raise problem awareness for personal
semantic multimedia
  to introduce the basic concepts of semantic
personal information management
  to introduce recent research and tools from
the semantic desktop community
  to indicate overlaps and potential for cross-
fertilization


The Web
  The World Wide Web is designed for
humans, not machines.
  Automatic information processing is hard.
  But at least, there is a common technology
infrastructure (URIs, HTTP, HTML,
browsers, etc.).

  This is a solid basis to build upon: the
Semantic Web and Linked Data reuse and
extend the human-centric Web.

The Desktop
  On your desktop, this problem is even
worse:

  No unified identification of data items.
  No standardized, simple communication
protocol.
  No agreed, open representation format for
data.
  No means to store and retrieve semantic
annotations.

How does your desktop look like?
  files
  emails
  documents
  contacts
  media
  ...

  at most: fulltext search ... which doesn’t
work well for multimedia data 

… organize
Invitation E-Mail
meeting in
Belfast …

Time Schedule

Attendee Dirk Belfast Weather Meeting Agenda


How are data organized?
  Hierarchical structures (aka data silos) for
files, emails, bookmarks, applications, ...
  Separate organization metaphors for
contacts, calendars, communication, ...
  Multimedia data: media-dependent
metadata (look at your music player or your
photo manager), locked away in files or in
application-specific data bases.
  The walled personal information garden ...


Personal Multimedia

  For the organization of multimedia data,
metadata is key.

  How to manage your personal metadata?


The Semantic Desktop

  The vision of the Semantic Desktop is to
create an infrastructure for the
management of meaningful metadata about
resources that covers the entire personal
sphere of information.

  In fact, the Semantic Desktop is your
personal Semantic Web.


How?
  assign globally unique, immutable URIs to
all resources.
  collect / store / index resource descriptions
and relations between resources.
  provide uniform access to these
descriptions.
  integrate these descriptions into
applications and make use of them.
  social semantic desktop: share and
communicate personal resources.

Why?

  The Semantic Desktop reflects the user’s
Personal Information Model — i.e., the
user’s view on their data.

  This allows tools to deliver information
when it is actually needed.


Topic Project
… organize
Invitation E-Mail
meeting in Management CID

Belfast …

Time Schedule

Attendee Dirk Belfast Weather Meeting Agenda


What can the Semantic Desktop deliver?
  Contextualized, semantic search
  find information more efficiently

  Similarity metrics and recommendations
  show the user what they already have in their
data space

  Data analysis and information extractions
  understand and interpret what is there


What can the Semantic Desktop deliver?
  Semantic Tagging
  relate things and concepts

  Semantic Personal Wikis
  useful notetaking and semistructured resource
description

  User Interfaces
  faceted search, drag+drop annotations, etc.


Recent Research Projects
  NEPOMUK (EU FP6) [Gro07]
 has (amongst others) defined a reference
architecture and ontologies for the
representation of personal information
 reference implementation (desktop
service), based on Java/Eclipse
 goal: infrastructure for existing OS
 integration in KDE 4
 raised significant public awareness (2
times slashdotted)

Commercial Break

  Results from NEPOMUK and other
projects are commercialized, have a
look at www.gnowsis.com


Recent Research Projects
 Haystack [Kar07]
 has presented a new metaphor for
interacting with data
 dynamically assembled user
interfaces
 dynamically assembled workflows
and data conversions
 basically an OS modeled in RDF

Many more
  DeepaMehta
  DBin
  Chandler
  Semex
  iMeMex
  IRIS
  X-COSIM
  ...


Semantic Desktop Infrastructure
  SemDAV [Sch09]
  combines aspects from the Semantic Web and
file systems into a unified data model
  provides storage and query mechanism for
semantic files
  provides virtual file system access for file-based
applications
  provides an API for “semantic” applications
  provides file browser-like user interaction
metaphors


  TripFS [Sch09a]
 exposes a file system as Linked Data
 assigns immutable URIs to files and
traces file system modifications
 extracts metadata from files and exposes
them as RDF
 provides a SPARQL endpoint over the file
system
 links files to external data sources (e.g.,
MusicBrainz, ACM, ...)

  Aperture [Sau06]
 a generic framework for extraction of
metadata from digital resources
 provides extractors, crawlers, monitors, ...
for different kinds of data sources (e.g.,
file system, email, web pages, web
services, ...)
 can be extended and customized
 open-source


  Nepomuk-KDE
 core components from NEPOMUK
(resource identification, triple store,
ontologies) have been integrated in KDE 4
and are now part of the standard
distribution.
 first implemented application: tags,
comments and ratings for files
 much more to follow (e.g., Gwenview
image viewer, Bangarang media player,...)

Summary

  The Semantic Desktop is highly relevant for
the management of personal media
collections — metadata is key!
  It is there! – just use it.

  Questions? bernhard.schandl@univie.ac.at


References & Credits
  [Gro07] Tudor Groza et al., The NEPOMUK Project –
On the Way to the Social Semantic Desktop. Proc. I-
Semantics 2007
  [Kar07] David Karger, Haystack: Per-User
Information Environments Based on Semistructured
Data, in: Beyond the Desktop Metaphor, MIT Press,
2007.
  [Sch09] Bernhard Schandl et al., The Sile Model — A
Semantic File System Infrastructure for the Desktop.
Proc. ESWC 2009


References & Credits
  [Sch09] Bernhard Schandl, TripFS: Exposing File
Systems as Linked Data. Triplification Challenge
2009.
  [Sau06] Leo Sauermann et al., Semantic Desktop
2.0: The Gnowsis Experience. Proc. ISWC 2006


SAMT09 - Web of Data Tutorial - Part 2

SAMT09 - Web of Data Tutorial - Part 2

Recommended

More Related Content

Similar to SAMT09 - Web of Data Tutorial - Part 2

Similar to SAMT09 - Web of Data Tutorial - Part 2(20)

More from Bernhard Haslhofer

More from Bernhard Haslhofer(20)

Recently uploaded

Recently uploaded(20)

SAMT09 - Web of Data Tutorial - Part 2