Your SlideShare is downloading. ×
0
Deep-linking into Media Assets at
the Fragment Level: Specification,
Model and Applications
Raphaël Troncy <raphael.troncy...
21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

-2
Once upon a time …

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

-3
… leading to sharing Media Fragments
 Publishing status message containing
a Media Fragment URI
 Use a ‘#’ !
 Highlight...
W3C Video on the Web Workshop - 2007

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

...
Key topics
 Addressing: having global identifiers for identifying
spatial and temporal clips (for deep linking,
bookmarki...
Making video a "first class citizen"

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

...
Flickr Notes

http://www.flickr.com/photos/mhausenblas/2883727293/
21/10/2013 -

International Workshop on Semantic Music ...
YouTube Temporal Addressing (Sept 2008)

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 201...
Media Fragments Use Cases
 Bookmark / Share parts (fragments) of
audio/video content
 Annotate media fragments
 Search ...
What are Media Fragments?

0

20

“Scared Scene”

t

35

temporal media fragment

named media fragment

spatial media frag...
Media Fragments Dimensions
 r01: Temporal fragments:
 a clipping along the time dimension from a start to an end time th...
Media Fragments (temporal)

Original resource
length

Fragment beginning
21/10/2013 -

Playback progress
International Wor...
Media Fragments (spatial)

highlighted
fragment
semi-opaque
overlay

http://ninsuna.elis.ugent.be/MFPlayer/html5
21/10/201...
URI Scheme
 Using URI query part:
http://www.example.org/video.ogv?t=60,100

 Using URI fragment part:
http://www.exampl...
URI Fragments vs. URI Queries
#t=20,30
?t=20,30
secondary resource,
primary resource,
notion of context
no notion of conte...
Media Fragments Resolution
 For the URI query part:
 The media file is only processed on server side
 The UA receives a...
Media Fragments Resolution
 2 ways
handshake

 4 ways
handshake

21/10/2013 -

International Workshop on Semantic Music ...
Influence of Media Formats
 Fragment extraction needs to be expressible in
terms of byte ranges
 Requirements for the di...
Media Fragment Clients
 Web Browsers
 Firefox (since version 9, now version 23)
 Safari (since Jan 2012, announcement)
...
Media Fragment Servers
 Ninsuna: http://ninsuna.elis.ugent.be/MediaFragmentsServer
 Southampton-Eurecom: node.js based
i...
21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 22
Media Fragment Semantic Annotation
Reg1
The "Big Three" at the Yalta
Conference (Wikipedia)

 Media Fragment creation: lo...
Media Fragment Semantic Annotation
A history of G8 violence (video)
(© Reuters)

Seq4
Seq1

 Media Fragment creation:
loc...
Media Fragment Semantic Annotation
 Things, not strings!
http://googleblog.blogspot.fr/2012/05/introducing-knowledgegraph...
Open Annotation Data Model
 Specification developed in the W3C Open Annotation
Community Group
http://www.openannotation....
Semantic Annotation of an Image

http://www.w3.org/community/openannotation/wiki/
SE_Semantically_Tagging_an_Image
21/10/2...
Maphub: http://maphub.github.io/

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 28
Open Video: Annotation Project

http://openvideoannotation.org/
21/10/2013 -

International Workshop on Semantic Music and...
YouTube Annotations

 Annotations are clickable text overlays on YouTube videos
 Annotations are used to boost engagemen...
YouTube Annotations: How To

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 31
LinkedTV: automatic annotations ...

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

-...
... and enrichment for hypervideos

CONCEPT IN
PLAYER
Cubism

Expressionism

Fauvism

FACETS / PROPERTIES OF CONCEPT
21/10...
Media Fragments and Annotations

http://data.linkedtv.eu/medi
a/e2899e7f#t=840,900

nerd:Location
Casablanca

nerd:Locatio...
Enrichment and Hypervideos

nerd:Location
Casablanca

nerd:Location
Cafe Rick

nerd:Person
H. Bogart

Nerd:Person
E. Tiern...
21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 36
NER Tools and Web APIs
 Standalone software
 GATE
 Stanford CoreNLP
 Temis

http://nerd.eurecom.fr/

 Web APIs

21/10...
NERD: Named Entity Recognition and
Disambiguation
 Compare performances of
NER and NEL tools
 Understand strengths and w...
NERD User Interface

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 39
Media Fragment + Open Annotation + NERD
Locator

MediaResource

Annotation

MediaFragment

Entity
Type

URL (hyperlink)

2...
Media Fragment Enricher:
http://mfe.synote.org/mfe/

21/10/2013 -

International Workshop on Semantic Music and Media (SMA...
Linking pieces of knowledge

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 42
Linking pieces of knowledge

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 43
Towards a Linked Media Layer
 Enriching media with media from a closed collection
(e.g. BBC archive)
 The MediaEval scen...
Media Collector

 Composition of media item extractors (12 SNs)
 Rely on search APIs (+ a fix 30s timeout window to prov...
Deep link
Permalink

Clean text for NLP
processing
Aggregate view of ALL
social interactions

12 Social Networks
Media Collector + White List web sites
Q:http://ir.lmcloud.vse.cz:8080/irapi/mediaserver?q=img_title:*Merkel*

21/10/2013 ...
Seed video enriched with web content
rbbaktuell_20120809

nerd:Location
Brandenburg
oa
Enrichments are Annotations too

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM) @ISWC 2013

- 49
http://linkedtv.project.cwi.nl/news/

20.10.2013

WP2 - Linking hypervideos to Web content
TV + Kinect + Web experience

http://www.youtube.com/watch?v=4mSC685AG7k

21/10/2013 -

International Workshop on Semantic...
Take Away Summary
 Video is a first class citizen on the Web
 Annotations: Ontology and API for Media Resources,
Open An...
Winter School:
http://winterschool.mediamixer.eu/

21/10/2013 -

International Workshop on Semantic Music and Media (SMAM)...
Credits
 Giuseppe Rizzo, Vuk Milicic, José Luis Redondo Garcia (EURECOM)
 Thomas Steiner (Google Inc.), Yunjia Li (Unive...
Upcoming SlideShare
Loading in...5
×

Deep-linking into Media Assets at the Fragment Level SMAM 2013

625

Published on

"Deep-linking into Media Assets at the Fragment Level: Specification, Model and Applications" - Keynote Talk given at the International Workshop on Semantic Music and Media (SMAM), 21 October 2013

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
625
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Deep-linking into Media Assets at the Fragment Level SMAM 2013"

  1. 1. Deep-linking into Media Assets at the Fragment Level: Specification, Model and Applications Raphaël Troncy <raphael.troncy@eurecom.fr>
  2. 2. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -2
  3. 3. Once upon a time … 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -3
  4. 4. … leading to sharing Media Fragments  Publishing status message containing a Media Fragment URI  Use a ‘#’ !  Highlight a video sequence  Highlight a region to pay attention to 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -4
  5. 5. W3C Video on the Web Workshop - 2007 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -5
  6. 6. Key topics  Addressing: having global identifiers for identifying spatial and temporal clips (for deep linking, bookmarking, caching and indexing)  Metadata: searching and discovering video is difficult with the volume of online video  Video codec: recommending a baseline (open) video codec for the World Wide Web  Content protection: managing digital rights associated with the media is key: W3C should look into metadata for digital rights 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -6
  7. 7. Making video a "first class citizen" 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -7
  8. 8. Flickr Notes http://www.flickr.com/photos/mhausenblas/2883727293/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -8
  9. 9. YouTube Temporal Addressing (Sept 2008) 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -9
  10. 10. Media Fragments Use Cases  Bookmark / Share parts (fragments) of audio/video content  Annotate media fragments  Search for media fragments  Develop Mash-ups/Collage  Conserve bandwidth http://www.w3.org/TR/media-frags-reqs/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 10
  11. 11. What are Media Fragments? 0 20 “Scared Scene” t 35 temporal media fragment named media fragment spatial media fragment track media fragment 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 11
  12. 12. Media Fragments Dimensions  r01: Temporal fragments:  a clipping along the time dimension from a start to an end time that are within the duration of the media resource  r02: Spatial fragments:  a clipping of an image region, only consider rectangular regions  r03: Track fragments:  a track as exposed by a container format of the media resource  r04: Named fragments:  A temporal media fragment that has been given a name through some sort of annotation mechanism 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 12
  13. 13. Media Fragments (temporal) Original resource length Fragment beginning 21/10/2013 - Playback progress International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 Fragment end - 13
  14. 14. Media Fragments (spatial) highlighted fragment semi-opaque overlay http://ninsuna.elis.ugent.be/MFPlayer/html5 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 14
  15. 15. URI Scheme  Using URI query part: http://www.example.org/video.ogv?t=60,100  Using URI fragment part: http://www.example.org/video.ogv#t=60,100  Mixing both: http://www.example.org/video.ogv?t=60,100 #t=10,15 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 15
  16. 16. URI Fragments vs. URI Queries #t=20,30 ?t=20,30 secondary resource, primary resource, notion of context no notion of context extraction needs to be no adaptation restrictions expressible in byte ranges key-value pairs are sent to no provisions for the server communi-cating fragments to the server potentially cacheable not cacheable  The media fragment URI syntax can be used for URI queries  We will focus on URI fragments 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 16
  17. 17. Media Fragments Resolution  For the URI query part:  The media file is only processed on server side  The UA receives a new video file  For the URI fragment part:  Smart UA will strip out the fragment definition and encode it into custom http headers (Range header)  (Media) Servers will handle the request, slice the media content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 17
  18. 18. Media Fragments Resolution  2 ways handshake  4 ways handshake 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 18
  19. 19. Influence of Media Formats  Fragment extraction needs to be expressible in terms of byte ranges  Requirements for the different axes  temporal: presence of intra-coded frames (i.e., random access points)  spatial: presence of independently coded spatial regions  track: need to be identifiable by a name  Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 19
  20. 20. Media Fragment Clients  Web Browsers  Firefox (since version 9, now version 23)  Safari (since Jan 2012, announcement)  Chrome (since Jan 2012, announcement)  Library (or Polyfill)  mediafragment.js: https://github.com/tomayac/Media-Fragments-URI  xywh.js: https://github.com/tomayac/xywh.js  Custom Players:  Ligne de Temps: http://ldt.iri.centrepompidou.fr/ldtplatform/ldt/  Synote: http://smfplayer.synote.org/smfplayer/  Noterik, Condat, JSI, etc. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 20
  21. 21. Media Fragment Servers  Ninsuna: http://ninsuna.elis.ugent.be/MediaFragmentsServer  Southampton-Eurecom: node.js based implementation  YouTube: partial support, syntax difference  Dailymotion: partial support, syntax difference 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 21
  22. 22. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 22
  23. 23. Media Fragment Semantic Annotation Reg1 The "Big Three" at the Yalta Conference (Wikipedia)  Media Fragment creation: localize a region (person)  Media Fragment annotation (tagging) = interpretation Winston Churchill, UK Prime Minister, Allied Forces, WWII  Media Fragment semantic annotation :Reg1 foaf:depicts dbpedia:WinstonChurchill. dbpedia:Churchill rdfs:label "Winston Churchill"; rdf:type foaf:Person dbprop:order dbpedia:Prime_Minister_(UK). 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 23
  24. 24. Media Fragment Semantic Annotation A history of G8 violence (video) (© Reuters) Seq4 Seq1  Media Fragment creation: localize a temporal sequence  Media Fragment annotation (tagging) = interpretation G8 Summit, EU Summit, Heiligendamm, 2007, Gothenburg, 2001  Media Fragment semantic annotation :Seq1 foaf:depicts dbpedia:33rd_G8_Summit. :Seq4 foaf:depicts dbpedia:EU_Summit. dbpedia:33rd_G8_Summit rdfs:label "33rd G8 summit"@en ; grs:point "54.143055555555556 11.841666666666667". 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 24
  25. 25. Media Fragment Semantic Annotation  Things, not strings! http://googleblog.blogspot.fr/2012/05/introducing-knowledgegraph-things-not.html  Use knowledge bases (LOD)  Use common vocabularies (LOV)  Follow the 4 Linked Data principles  Refine the 4 Linked Media principles 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 25
  26. 26. Open Annotation Data Model  Specification developed in the W3C Open Annotation Community Group http://www.openannotation.org/spec/core/  Core model  OWL vocabulary for representing and sharing annotation of digital resources (and their fragment) … in RDF  A body is related to a target  Nature of the annotation changes according to intention (motivation)  How to annotate this image? 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 26
  27. 27. Semantic Annotation of an Image http://www.w3.org/community/openannotation/wiki/ SE_Semantically_Tagging_an_Image 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 27
  28. 28. Maphub: http://maphub.github.io/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 28
  29. 29. Open Video: Annotation Project http://openvideoannotation.org/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 29
  30. 30. YouTube Annotations  Annotations are clickable text overlays on YouTube videos  Annotations are used to boost engagement, give more information, and aid in navigation 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 30
  31. 31. YouTube Annotations: How To 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 31
  32. 32. LinkedTV: automatic annotations ... 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 32
  33. 33. ... and enrichment for hypervideos CONCEPT IN PLAYER Cubism Expressionism Fauvism FACETS / PROPERTIES OF CONCEPT 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 CONTENT ENRICHMENT - 33
  34. 34. Media Fragments and Annotations http://data.linkedtv.eu/medi a/e2899e7f#t=840,900 nerd:Location Casablanca nerd:Location Cafe Rick nerd:Person H. Bogart nerd:Person I. Bergman  Media Fragment URI 1.0     21/10/2013 - Chapters Scenes Shots etc… International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 34
  35. 35. Enrichment and Hypervideos nerd:Location Casablanca nerd:Location Cafe Rick nerd:Person H. Bogart Nerd:Person E. Tierney 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 nerd:Person I. Bergman nerd:Location China - 35
  36. 36. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 36
  37. 37. NER Tools and Web APIs  Standalone software  GATE  Stanford CoreNLP  Temis http://nerd.eurecom.fr/  Web APIs 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 37
  38. 38. NERD: Named Entity Recognition and Disambiguation  Compare performances of NER and NEL tools  Understand strengths and weaknesses of different Web APIs  Adapt NER processing to different context  (Learn how to) Combine NER (/ NEL) tools What is NERD? ontology1 REST API2 UI3 1 2 21/10/2013 - http://nerd.eurecom.fr/ontology http://nerd.eurecom.fr/api/application.wadl 3 http://nerd.eurecom.fr International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 38
  39. 39. NERD User Interface 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 39
  40. 40. Media Fragment + Open Annotation + NERD Locator MediaResource Annotation MediaFragment Entity Type URL (hyperlink) 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 40
  41. 41. Media Fragment Enricher: http://mfe.synote.org/mfe/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 41
  42. 42. Linking pieces of knowledge 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 42
  43. 43. Linking pieces of knowledge 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 43
  44. 44. Towards a Linked Media Layer  Enriching media with media from a closed collection (e.g. BBC archive)  The MediaEval scenario (~ 1697 hours of archived BBC video) http://www.multimediaeval.org/mediaeval2013/hyper2013/  Enriching media with content from the open web  LinkedTV scenarios: white listed web sites for each program  Media Collector for Social Media 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 44
  45. 45. Media Collector  Composition of media item extractors (12 SNs)  Rely on search APIs (+ a fix 30s timeout window to provide results)  Fallback on screen scraping when necessary (Twitter ecosystem)  Implemented as a NodeJS server  Serialize results in a common schema (JSON) 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 45
  46. 46. Deep link Permalink Clean text for NLP processing Aggregate view of ALL social interactions 12 Social Networks
  47. 47. Media Collector + White List web sites Q:http://ir.lmcloud.vse.cz:8080/irapi/mediaserver?q=img_title:*Merkel* 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 47
  48. 48. Seed video enriched with web content rbbaktuell_20120809 nerd:Location Brandenburg oa
  49. 49. Enrichments are Annotations too 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 49
  50. 50. http://linkedtv.project.cwi.nl/news/ 20.10.2013 WP2 - Linking hypervideos to Web content
  51. 51. TV + Kinect + Web experience http://www.youtube.com/watch?v=4mSC685AG7k 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 51
  52. 52. Take Away Summary  Video is a first class citizen on the Web  Annotations: Ontology and API for Media Resources, Open Annotation Data Model  Access: Media Fragments URI  NERD platform for extracting key information from textual resources including video subtitles and microposts  Embrace the Linked Media vision  Publish, re-use, re-purpose and remix media descriptions  Develop links between (part of) media items via their descriptions 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 52
  53. 53. Winter School: http://winterschool.mediamixer.eu/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 53
  54. 54. Credits  Giuseppe Rizzo, Vuk Milicic, José Luis Redondo Garcia (EURECOM)  Thomas Steiner (Google Inc.), Yunjia Li (University of Southampton)  Marieke van Erp (Free University of Amsterdam)  Erik Mannens, Davy ven Deursen (iMinds, Uni. Ghent)  Paolo Ciccarese, Robert Sanderson, Herbert Van de Sompel and all the members of the W3C Open Annotation Community Group  … and many other students 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 54
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×