Semantic annotation of media resources has been a focus in research since many years, the closing of the "semantic gap" being seen as key to signicant improvements in media retrieval and browsing and
enabling new media applications and services. However, current tools and services exhibit varied approaches which do not easily integrate and
act as a barrier to wider uptake of semantic annotation of online multimedia.
In this paper, we outline the Linked Media principles which can help form a consensus on media annotation approaches, survey current media annotation tools against these principles and present two emerging
toolsets which can support Linked Media conformant annotation, closing with a call to future semantic media annotation tools and services to follow the same principles and ensure the growth of a Linked Media
layer of semantic descriptions of online media which can be an enabler to richer future online media services.
Survey of Semantic Media Annotation Tools - towards New Media Applications with Linked Media
1. Television Linked To The Web
www.linkedtv.eu
Lyndon Nixon, MODUL University Vienna GmbH
Raphael Troncy, EURECOM
Survey of Semantic Media Annotation
Tools for the Web: towards New Media
Applications with Linked Media
25 May 2014 LiME Workshop ESWC2014
2. 2
www.linkedtv.eu
Overview
Why Linked Media?
What is Linked Media?
Requirements on online media annotation
Survey of existing tools
LinkedTV services and applications for Linked Media
Summary and Outlook
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
3. 3
www.linkedtv.eu
Why Linked Media?
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
„It is growing at more than 20% per annum, fuelled by increased
demands for new programming and the huge saving it represents
compared with shooting new footage. Interactive technology and the
Internet will further contribute to the growth of the market as it makes
stock footage cheaper and easier to locate and license.“
- http://moneyam.uk-wire.com/cgi-bin/articles/200201020827103514P.html
4. 4
www.linkedtv.eu
Why Linked Media?
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
Dublin Core Metadata Initiative
(DCMI)
Metadata Encoding and Transmission
Standard (METS)
6. 6
www.linkedtv.eu
What is Linked Media?
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
• What is being described?
• The media object – what is it?
Video uploaded to YouTube 2011
ZDF documentary broadcast 1980
Rembrandt self-portrait from 1639
Date?
Is Rembrandt a subject of the whole
video?
Or only a specific temporal
segment of the video?
Is the whole frame important or only
the part with Rembrandt‘s face?
7. 7
www.linkedtv.eu
What is Linked Media?
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
Media fragment players:
http://ninsuna.elis.ugent.be/MFPlayer/html5
http://linkedtv.eurecom.fr/mediafragmentplayer
Emerging support in Web browsers:
Firefox, Safari, Chrome
8. 8
www.linkedtv.eu
What is Linked Media?
The object of annotation
• What is the description of the media?
"Rembrandt"
"Rembrandt self-
portrait" from 1639
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
9. 9
www.linkedtv.eu
What is Linked Media?
Digital identifiers for non-
addressable resources
• “URIs for concepts” since the Semantic Web
• Linked Data has popularized the notion
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
10. 10
www.linkedtv.eu
What is Linked Media?
1. Web media descriptions need a
common representation of media
structure
2. Web media descriptions need a
common representation of media
content
3. Web media descriptions need to use
a media ontology which supports
description of both the structure
and content of media
4. The descriptions of media in terms of
common representations of structure
and content are the basis for deriving
links across media on the Web (Linked
Media)
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
11. 11
www.linkedtv.eu
Requirements on online media annotation
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
“Just as Web browsers could never have been successful without a
core consensus on the Web page mark up (HTML), Web media can not fulfil
its potential without a core consensus on the available descriptive metadata.”
(1) tool or service is online and allows to annotate media of different types via
URL
(2) media can be annotated in terms of its parts and the media fragmentation
follows a non-proprietary approach supported by other tools
(3) media can be annotated by concepts where a shared understanding is
possible across systems
(4) media ontology used for the generated descriptions is interoperable
across systems
12. 12
www.linkedtv.eu
Survey: overview of annotation tools
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
EuropeanaConnect Media Annotation
Suite
(http://dme.arcs.ac.at/annotation/)
"Developed in the
EuropeanaConnect project, it
supports image,audio and video.
Both DBPedia and Geonames
resources can be annotation
targets, and are suggested from
free text or location references
respectively. Annotations can be
exported as RDF using a tool-
specific vocabulary."
13. 13
www.linkedtv.eu
Survey: overview of annotation tools
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
ImageSnippets
(http://www.imagesnippets.com/)
Open Video Annotation
(http://www.openvideoannotation.o
rg/)
14. 14
www.linkedtv.eu
Survey: overview of annotation tools
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
Annotorius
(http://annotorious.github.io/)
Pundit and SemTube
(http://thepund.it and
http://www.semedia.dibet.univpm.it/semtube/)
15. 15
www.linkedtv.eu
Survey: overview of annotation tools
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
Web based; open videos from online or the repository; term
suggestion; export annotations as file or to repository.
ConnectME annotation tool
https://github.com/lyndonnixon/
annotationtool
Contact lyndon.nixon@modul.ac.at
GNU GPL 3.0 license
16. 16
www.linkedtv.eu
LinkedTV: background
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
40% of TV viewers are using a companion
device alongside the TV program.*
* J. Abreu, P. Almeida, B. Teles, and M. Reis. Viewer behaviors and practices in the (new)
television environment. In Proceedings of the 11th European Conference on Interactive TV
and Video, EuroITV '13.
http://www.linkedtv.eu
Ever saw something
on TV and wanted to
know more about it,
but didn‘t even know
how to search for it?
17. 17
www.linkedtv.eu
LinkedTV technology
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
“...schilderij van Jan Sluijters....”
Video object and
word detection
dbpedia.org/resource/Jan_Sluyters
Connection to
concepts
Paintings by Jan Sluijters
Selection of related
concepts
Selection of related
content
Presentation
engine
18. 18
www.linkedtv.eu
Annotation task: automatic approaches
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
http://nerd.eurecom.fr
19. 19
www.linkedtv.eu
Media Fragment + Open Annotation + NERD
08.11.
2012
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
21. 21
www.linkedtv.eu
Annotation task: automatic approaches
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
http://linkedtv.eurecom.fr/tv2rdf/
22. 22
www.linkedtv.eu
Annotation task: manual approaches
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
The LinkedTV Editor Tool will have a public code release in late
2014 (planned GNU GPL 3.0), but a test installation can be used at
http://editortool.linkedtv.eu (Enquiries: jblom@beeldengeluid.nl)
23. 23
www.linkedtv.eu
Enrichment task: associating media
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
Edward Snowden asks for asylum in Russia (04 / 07 / 2013)
24. 24
www.linkedtv.eu
Enrichment task: associating media
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
In which Russian airport is he exactly?
LSCOM:Face
LSCOM:Building
?
25. 25
www.linkedtv.eu
Enrichment task: associating media
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
List of Relevant Named Entities (1) Named Entity
(2) Filtering and Ranking
b) Expanded Entities
b) Re-ranked Entities
a) Entities from Video
26. 26
www.linkedtv.eu
Enrichment task: associating media
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
http://linkedtv.project.cwi.nl/news/
mainscreen.html
http://linkedtv.project.cwi.nl/news/
Second Screen @
Main Screen @
27. 27
www.linkedtv.eu
Outlook
Survey of Semantic Media Annotation Tools for the Web:
Towards new Media Applications with Linked Media
Tool Input Fragment support Conceptual
annotation
Ontology
Europeana
Connect
Audio, image or video.
By URL
Spatial & temporal
(SVG)
DBPedia and
GeoNames
Own vocabulary
ImageSnippets Image only. By URL Spatial. Free to import from
LOD or create own
LIO ontology
OpenVideo Annotation Video. Unclear if
support URL
Only temporal in the
demo, unknown
syntax
Free text or tags Open Annotation
Model
Annotorius Image only. By URL. Spatial, proprietary
syntax
DBPedia No ontology (JS data
model)
Pundit and SemTube Image and video. By
URL
Spatial, proprietary
syntax
Freebase. Terms can
be imported.
Own RDF vocabulary
ConnectME Video. By URL Spatial & temporal
using MF URIs
DBPedia and
Geonames
ConnectME Ontology
LinkedTV TV2RDF
and Editor Tool
Video. By URL (Editor
Tool uses resources
on the LinkedTV
platform)
Spatial & temporal
using MF URIs
DBPedia. Editor tool
suggests DBPedia but
allows any URI.
LinkedTV Ontology