The second session looks at how using Linked Data principles for media fragment annotation publication and retrieval (Linked Media) can enable online media fragment re-use:
Introducing the Linked Media principles
Publishing Linked Media using dedicated multimedia RDF repositories
Retrieval of media resources that illustrate linked data concepts
Using the Linked Data graph to find relevant links between distinct media assets (examples with SPARQL)
Retrieval of links between annotated media to enable topical browsing (using the TVEnricher service)
Examples of Linked Media at scale: VideoLyzard and HyperTED
Remixing Media on the Semantic Web (ISWC2014 Tutorial) Pt 2 Linked Media: An approach to online media annotation and re-use
1. 1
LinkedMedia:
An approach to online media re-use
Lyndon Nixon
MODUL University Vienna
lyndon.nixon@modul.ac.at
RE-USING MEDIA ON THE (SEMANTIC) WEB
ISWC2014 Tutorial, Riva de Garda, Italy, October 20 2014
2. 20.10.14 Slide 2 of 50
Agenda
• Session 1: Media fragment specification and semantics
• Summary: Introduce the W3C Media Fragment URI specification and the Open
Annotation model. Highlight how media fragments can be annotated using NER
tools.
• Session 2: Linked Media principles
• Summary: Introduce the Linked Media principles. How to
publish Linked Media in RDF and how to retrieve media
enrichments. Illustration with Linked Media applications.
• Session 3: User experience driven design of Linked Media applications
• Summary: Present the Web and TV convergence. Describe LinkedTV experience
via two innovative applications.
2
3. 2/3: LinkedMedia
• Introducing the Linked Media principles
• Publishing Linked Media
• LinkedTV Platform (Virtuoso)
• Retrieval of Linked Media
• Enrichment of media (LinkedTV)
• Browsing and linking media
(VideoLyzard, HyperTED)
3
20.10.14 Slide 3 of 50
5. Why is Online Media important?
„It is growing at more than 20% per annum, fuelled by increased
demands for new programming and the huge saving it represents
compared with shooting new footage. Interactive technology and the
Internet will further contribute to the growth of the market as it makes
stock footage cheaper and easier to locate and license.“
- http://moneyam.uk-wire.com/cgi-bin/articles/200201020827103514P.html
20.10.14 Slide 5 of 50
10. 20.10.14 Slide 10 of 50
Why Linked Media?
From Lyndon Nixon, „The importance of Linked Media to the Future Web“
slideshare.net/linkedtv/www-linked-media-keynote
11. Linked Media Principles
1. Web media descriptions
need a common
representation of media
structure
22.. WWeebb mmeeddiiaa ddeessccrriippttiioonnss
nneeeedd aa ccoommmmoonn
rreepprreesseennttaattiioonn ooff mmeeddiiaa
ccoonntteenntt
33.. WWeebb mmeeddiiaa ddeessccrriippttiioonnss nneeeedd ttoo uussee aa
mmeeddiiaa oonnttoollooggyy wwhhiicchh ssuuppppoorrttss ddeessccrriippttiioonn
ooff bbootthh tthhee ssttrruuccttuurree aanndd ccoonntteenntt ooff mmeeddiiaa
20.10.14 Slide 11 of 50
15. Linked Media is Linked Data
Good news: Linked Media can be published as Linked
Data!
•The media resource has a globally (Web wide) unique
identifier
•Metadata about the media resource can be accessed via
its identifier
•Media identifier can‘t be = media locator!
20.10.14 Slide 15 of 50
16. 20.10.14 Slide 16 of 50
LinkedTV Platform
Web administration interface & REST API at
http://api.linkedtv.eu
18. Media Metadata view
http://data.linkedtv.eu is the base of the RDF graph of
LinkedTV content, and via Virtuoso, every LinkedTV
instance URI can return HTML or RDF, e.g.
http://data.linkedtv.eu/mediaresource/8a8187f2-3fc8-cb54-
0140-7dccd76f0001
20.10.14 Slide 18 of 50
19. Media Metadata view
Media fragments of a media resource:
http://data.linkedtv.eu/mediaresource/8a8187f2-3fc8-cb54-
0140-7dccd76f0001/mediafragment
20.10.14 Slide 19 of 50
20. Media Metadata view
Annotations of a media fragment:
http://data.linkedtv.eu/mediafragment/3d5b11e0-f6df-11e3-
b0fe-005056a7235c%23t=1,299/annotation
20.10.14 Slide 20 of 50
22. 20.10.14 Slide 22 of 50
SPARQL endpoint
SPARQL queries allow us to connect metadata across the
Linked Media store AND to connect it to other metadata
outside in the Linked Data cloud.
Linked Media queries courtesy LinkedTV deliverable 2.4
„Annotation and retrieval module of media fragments“ (Jose
Luis Redondo Garcia & Raphael Troncy) available from:
http://de.slideshare.net/linkedtv/annotation-and-retrieval-module-
of-media-fragments
23. 20.10.14 Slide 23 of 50
SPARQL queries
Get all Shots or Chapters for a media resource.
24. 20.10.14 Slide 24 of 50
SPARQL queries
Return all entities of type nerd:Person in the annotation of a media resource.
25. 20.10.14 Slide 25 of 50
SPARQL queries
Return all entities occurring within a temporal boundary of a media resource.
27. Why Linked Television?
http://www.linkedtv.eu
20.10.14 Slide 27 of 50
40% of TV viewers are using a
companion device alongside the TV
program.*
* J. Abreu, P. Almeida, B. Teles, and M. Reis. Viewer behaviors and practices
in the (new) television environment. In Proceedings of the 11th European
Conference on Interactive TV and Video, EuroITV '13.
Ever saw
something on TV
and wanted to know
more about it, but
didn‘t even know
how to search for
it?
28. LinkedTV Technology
Paintings by Jan Sluijters
Selection of
related concepts
Selection of
related content
20.10.14 Slide 28 of 50
Video object and
word detection
Connection to
concepts
“...schilderij van Jan Sluijters....”
dbpedia.org/resource/Jan_Sluyters
Presentation
engine
29. The use of Linked Data in the identification
of concepts in LinkedTV means we can
expand concepts along different facets, i.e.
allow users to explore in terms of their
different interests in a given concept.
For example, for an artist like Jan Sluijters,
has art style
Leo Gestel
20.10.14 Slide 29 of 50
Related concepts
Additional information, e.g.
biography of artist
style of painting
Related information, e.g.
artists from same
period
paintings in similar style
related styles
Paintings by Jan Sluijters
Expansion of
related concepts
Jan Sluijters
luminism
has art style
Piet Mondriaan
has art style
LinkedTV can link into:
30. 20.10.14 Slide 30 of 50
Related content
LinkedTV provides enrichment services
which provide recommendations for
related online content (Web pages,
images, audio, video) pertaining to the
concepts in the TV program:
– Fresh Social Web content coming
from e.g. Twitter and Facebook
– User Generated content coming
from e.g. Flickr and YouTube
– Whitelist content coming from
partner Websites like public
broadcasters in Germany or cultural
heritage archives in the Netherlands
– Extendable and configurable by
source and media type
… and their digital images
Linking to
related content
31. LinkedTV „enrichment“
Base enrichments: title, thumbnail (poster), description (abstract)
Information cards: set of properties and values according to the entity type
Linksets: links to online content determined by queries
over Web content sources using a group of entities as
search term
•Linksets can be split along enrichment dimensions
•Dimensions are distinct aspects of interest to viewers
•They map to different queries & services in LinkedTV
20.10.14 Slide 31 of 50
32. LinkedCulture enrichment
TKK Video
Chapter segmentation + tagging as „Art Object“
TV2RDF incl. Art Object annotation
Entity Proxy
20.10.14 Slide 32 of 50
Related
TKK
chapters
(Solr)
Related
art objects
(Europeana)
Related
White List
media
(IRAPI)
TVEnricher (TKK configuration)
Editor Tool LinkedTV Player
34. Art object: semantic model
20.10.14 Slide 34 of 50
http://data.linke
dtv.eu/object/a
vro/8a8187f2-
3fc8-cb54-
0140-
7dccd76f0001/
2138
A silver tea jar
RDF Is-a Container http://vocab.getty.ed
u/aat/300045611
CRM Consists-of Silver http://vocab.getty.ed
u/aat/300010975
VRA
locationCreationSite
Friesland http://www.geoname
s.org/2755812
DCT temporal Start: 1690, End: 1742
35. Mapping to Europeana API
what:
(container+OR+houder+OR+bak+OR+ta
nk+OR+blik)
proxy_dc_format:zilver
where:Friesland
20.10.14 Slide 35 of 50
http://data.linke
dtv.eu/object/a
vro/8a8187f2-
3fc8-cb54-
0140-
7dccd76f0001/
2138
A silver tea jar
RDF Is-a Container
CRM Consists-of Silver
VRA
Friesland
locationCreationSite
DCT temporal Start: 1690, End: 1742
YEAR:[1690+TO+1742]
36. 20.10.14 Slide 36 of 50
Enrichment results
Friesian silver from
1690 to 1742 to 1742
37. LinkedNews enrichment
RBB Video
Chapter segmentation + tagging as „News Item“
Named Entity Expansion over news items
Entity Proxy
20.10.14 Slide 37 of 50
Related
news
items
(Solr)
Related
news articles
(TVNews-Enricher)
Related
White List
media
(IRAPI)
TVEnricher (RBB configuration)
Editor Tool LinkedTV Player
38. 20.10.14 Slide 38 of 50
Entity Proxy
Fill information cards for entities giving values
for their most relevant properties.
39. 20.10.14 Slide 39 of 50
TVNewsEnricher
Returns news media sources related to given
entity sets, powered by Google CSE
40. IRAPI: Web media crawler
Extract descriptions of media on websites so that
related media can be found
Video
Title
Description
Entities, e.g.
http://dbpedia.org/
resource/Edward_
Snowden
20.10.14 Slide 40 of 50
47. 20.10.14 Slide 47 of 50
198
4
.com
200
6
CHAPTERS
201
4
HOT SPOTS
ENTITIES
RELATED TED’S
CHAPTERS
THE
MYSTERIOUS
FIELD OF
ENGINEERING
SYSTEMS
UNDERSTANDI
NG
ENVIRONMEN
T:
A SYSTEM
APPROACH
SYSTEMS
PRACTICE:
MANAGING
SUSTAINABIL
ITY
COURSES
51. MediaMixer
community portal
Introduction to all technologies at
community.mediamixer.eu/technology
Updated with latest materials on all
Media Mixer topics:
Technology use cases
Demonstrators
Tutorials
Presentations
Software
Specifications
http://community.mediamixer.eu
20.10.14 Slide 51 of 50
54. Video fragment creation
Fragments were created
based on the slide
synchronisation timeline.
Transcripts (auto-generated
by speech-to-text
technology where
necessary) were parsed and
… there are three split across fragments.
Kingdoms of Life,
Bacteria, Archaea
and Eukaryota...
20.10.14 Slide 54 of 50
55. Video fragment annotation
Fragments were then annotated by
extracting topics from their textual
metadata (slide OCR or speaker
transcription).
Topics are connected to a global
knowledge model (DBPedia).
20.10.14 Slide 55 of 50
Video Fragment
(4:41-5:12)
Archaea
56. 20.10.14 Slide 56 of 50
Video fragment
management
Annotations are managed in
a separate metadata store.
The store provides a
semantic query endpoint
returning lists of video
fragments matching a query
topic (including semantically
related topics)
Acidiplasma „type“ relation
Archaea
Video
Fragment
(4:41-5:12)
57. Video fragment playback
The front end uses HTML5
or Flash. Both codebases
are extended to support
video fragment playout.
Individual playback can be
modified to linear or non-linear
channels (for e.g. a
TV or mobile video
experience)
20.10.14 Slide 57 of 50
64. SPARQL RDF query
language
RDF data is a „labeled, directed graph“
without any fixed vocabulary (this is defined
in RDF Schema). A query language for RDF
needs to support this data model:
•Tranversing paths in a RDF graph
•Being independent of any defined
vocabulary
•Able to query over the data or the schema
SPARQL („sparkle“) is a W3C standard
supported in most RDF tools
•SELECT... WHERE.... constructions like in
SQL
•Path expressions
•Additional keywords for more query
expressiveness
20.10.14 Slide 64 of 50
65. Examples: triple patterns
The basic idea in SPARQL is to match graph patterns against RDF graphs.To
understand graph patterns we must first define triple patterns:
• A triple pattern is similar to a triple (in RDF) but with one or more variables in
the place of a RDF resource (URI or literal)
Triple: dbpedia:Lou_Reed foaf:givenName „Lewis Allen Reed“ .
Triple pattern: dbpedia:Lou_Reed foaf:givenName ?name .
?name is the variable. If a RDF graph with the above triple is queried with the
below triple pattern, then in the SPARQL results the variable ?name would be
‚bound‘ to the value „Lewis Allen Reed“.
• A SPARQL query result is a set of bindings for the variables appearing in the
SELECT clause, based on matching RDF resources to variables in triple
patterns in the WHERE clause. .
20.10.14 Slide 65 of 50
66. Examples: conjunctive and
disjunctive query
In graph patterns, several triple patterns are listed within braces { ... } and
these are interpreted conjunctively:
Ex: { ?what tech:noOfWheels „4“ .
?what tech:minSpeed „180“ . }
The variable ?what will only be bound to resources which BOTH have 4 wheels
AND a minimum speed of 180.
You can also join results from distinct graph patterns using the UNION
keyword. Note that result sets from graph patterns and from UNIONs are
different, since UNION works disjunctively:
Ex: { ?what tech:noOfWheels „4“ . }
UNION { ?what tech:minSpeed „180“ . }
20.10.14 Slide 66 of 50
67. Examples: FILTER, ORDER
BY
SPARQL has many other keywords. FILTER restricts variable bindings
returned in query results to those for which the filter expression evaluates to
true. A filter applies to solutions over the entire graph pattern it is contained in.
Ex: { ?what tech:noOfWheels „4“ .
?what tech:minSpeed ?speed .
FILTER ( ?speed > 170 ) }
The ORDER BY keyword determines the sequence of query results returned
according to a sort on the referenced variable‘s bindings, ascending by default:
Ex: { ?what tech:minSpeed ?speed . }
20.10.14 Slide 67 of 50
ORDER BY ?speed
Or ORDER BY DESC(?speed)
68. Query for media fragments:
SPARQL-MM
Research work in progress: extending SPARQL to Media Fragments by adding
spatio-temporal filter and aggregration functions.
Relation
Function
Aggregation
Function
Spatial mm:rightBeside mm:spatialIntersection
mm:spatialOverlaps mm:spatialBoundingBo
x
… …
Temporal mm:after mm:temporalIntersecti
on
mm:temoralOverlaps mm:temporalIntermedi
ate
… …
Courtesy Thomas Kurz (Salzburg Research) & MICO EU project
http://demos.mico-project.eu/sparql-mm/sparql-mm/demo/index.html
Combined mm:overlaps mm:boundingBox
mm:contains mm:intersection
20.10.14 Slide 68 of 50
Editor's Notes
Part I: LinkedMedia, publishing online, retrieval (20 slides, 30 min)
Part II: Examples of LinkedMedia retrieval: enrichment, browsing, linking (20 slides, 30 min)
Part I: LinkedMedia, publishing online, retrieval (20 slides, 30 min)
Part II: Examples of LinkedMedia retrieval: enrichment, browsing, linking (20 slides, 30 min)
Growing amount of digital multimedia, metadata is very basic and limiting scalable retrieval and re-use of the media which is put online.
Growing amount of digital multimedia, metadata is very basic and limiting scalable retrieval and re-use of the media which is put online.
Compare with Linked Data Publishing rules.
Compare with Apache Marmotta: is there a repository that truly supports „Linked Media Publishing“?
NB: the actual media resource URI is media/UUID it seems Elda forces everything to go through a redirect on mediaresource/ so not sure if this is „true“ Linked Data
http://data.linkedtv.eu/sparql
PREFIX linkedtv: <http://data.linkedtv.eu/ontologies/core#>
PREFIX ma: <http://www.w3.org/ns/ma-ont#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX oa: <http://www.w3.org/ns/oa#>
SELECT ?mediaFragment
FROM <http://data.linkedtv.eu/graph/linkedtv>
WHERE {
?mediaFragment ma:isFragmentOf <http://data.linkedtv.eu/media/8a8187f2-3fc8-cb54-0140-7dd151100003>.
?mediaFragment a ma:MediaFragment .
?annotation a oa:Annotation .
?annotation oa:hasBody ?shot .
?annotation oa:hasTarget ?mediaFragment .
?shot a linkedtv:Shot .
}
Running example: „geoengineering“, 1 YouTube video in the Sept/Oct 2013 search period.
Currently only has videos annotated from 1 Sep to 30 Oct 2013
I still have to use http://ccc.modul.ac.at/dev/youtube/search.php to show geoengineering with 6 matches incl 3 videos without the term in their title.
Working example is „hydropower“ which digs up a transcript out of ONE video
In the DBpedia search, „hydroelectricity“ (no matches on the portal) gets 22 fragments (all mapped to DBPedia:Hydropower)
SPARQL, the query language for RDF
SQL vs SPARQL
Guide by example. Triple patterns
Guide by example. JOINs and UNIONs
Guide by example. FILTERs and ORDERs
SPARQL in Sesame: testing with saved queries
SPARQL-MM: a SPARQL for Media Fragments (demos)
SPARQL, the query language for RDF
SQL vs SPARQL
Guide by example. Triple patterns
Guide by example. JOINs and UNIONs
Guide by example. FILTERs and ORDERs
SPARQL in Sesame: testing with saved queries
SPARQL-MM: a SPARQL for Media Fragments (demos)
SPARQL, the query language for RDF
SQL vs SPARQL
Guide by example. Triple patterns
Guide by example. JOINs and UNIONs
Guide by example. FILTERs and ORDERs
SPARQL in Sesame: testing with saved queries
SPARQL-MM: a SPARQL for Media Fragments (demos)
SPARQL, the query language for RDF
SQL vs SPARQL
Guide by example. Triple patterns
Guide by example. JOINs and UNIONs
Guide by example. FILTERs and ORDERs
SPARQL in Sesame: testing with saved queries
SPARQL-MM: a SPARQL for Media Fragments (demos)