Deep-linking into Media Assets at the Fragment Level SMAM 2013

  • 470 views
Uploaded on

"Deep-linking into Media Assets at the Fragment Level: Specification, Model and Applications" - Keynote Talk given at the International Workshop on Semantic Music and Media (SMAM), 21 October 2013

"Deep-linking into Media Assets at the Fragment Level: Specification, Model and Applications" - Keynote Talk given at the International Workshop on Semantic Music and Media (SMAM), 21 October 2013

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
470
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
14
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Deep-linking into Media Assets at the Fragment Level: Specification, Model and Applications Raphaël Troncy <raphael.troncy@eurecom.fr>
  • 2. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -2
  • 3. Once upon a time … 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -3
  • 4. … leading to sharing Media Fragments  Publishing status message containing a Media Fragment URI  Use a ‘#’ !  Highlight a video sequence  Highlight a region to pay attention to 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -4
  • 5. W3C Video on the Web Workshop - 2007 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -5
  • 6. Key topics  Addressing: having global identifiers for identifying spatial and temporal clips (for deep linking, bookmarking, caching and indexing)  Metadata: searching and discovering video is difficult with the volume of online video  Video codec: recommending a baseline (open) video codec for the World Wide Web  Content protection: managing digital rights associated with the media is key: W3C should look into metadata for digital rights 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -6
  • 7. Making video a "first class citizen" 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -7
  • 8. Flickr Notes http://www.flickr.com/photos/mhausenblas/2883727293/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -8
  • 9. YouTube Temporal Addressing (Sept 2008) 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 -9
  • 10. Media Fragments Use Cases  Bookmark / Share parts (fragments) of audio/video content  Annotate media fragments  Search for media fragments  Develop Mash-ups/Collage  Conserve bandwidth http://www.w3.org/TR/media-frags-reqs/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 10
  • 11. What are Media Fragments? 0 20 “Scared Scene” t 35 temporal media fragment named media fragment spatial media fragment track media fragment 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 11
  • 12. Media Fragments Dimensions  r01: Temporal fragments:  a clipping along the time dimension from a start to an end time that are within the duration of the media resource  r02: Spatial fragments:  a clipping of an image region, only consider rectangular regions  r03: Track fragments:  a track as exposed by a container format of the media resource  r04: Named fragments:  A temporal media fragment that has been given a name through some sort of annotation mechanism 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 12
  • 13. Media Fragments (temporal) Original resource length Fragment beginning 21/10/2013 - Playback progress International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 Fragment end - 13
  • 14. Media Fragments (spatial) highlighted fragment semi-opaque overlay http://ninsuna.elis.ugent.be/MFPlayer/html5 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 14
  • 15. URI Scheme  Using URI query part: http://www.example.org/video.ogv?t=60,100  Using URI fragment part: http://www.example.org/video.ogv#t=60,100  Mixing both: http://www.example.org/video.ogv?t=60,100 #t=10,15 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 15
  • 16. URI Fragments vs. URI Queries #t=20,30 ?t=20,30 secondary resource, primary resource, notion of context no notion of context extraction needs to be no adaptation restrictions expressible in byte ranges key-value pairs are sent to no provisions for the server communi-cating fragments to the server potentially cacheable not cacheable  The media fragment URI syntax can be used for URI queries  We will focus on URI fragments 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 16
  • 17. Media Fragments Resolution  For the URI query part:  The media file is only processed on server side  The UA receives a new video file  For the URI fragment part:  Smart UA will strip out the fragment definition and encode it into custom http headers (Range header)  (Media) Servers will handle the request, slice the media content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 17
  • 18. Media Fragments Resolution  2 ways handshake  4 ways handshake 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 18
  • 19. Influence of Media Formats  Fragment extraction needs to be expressible in terms of byte ranges  Requirements for the different axes  temporal: presence of intra-coded frames (i.e., random access points)  spatial: presence of independently coded spatial regions  track: need to be identifiable by a name  Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 19
  • 20. Media Fragment Clients  Web Browsers  Firefox (since version 9, now version 23)  Safari (since Jan 2012, announcement)  Chrome (since Jan 2012, announcement)  Library (or Polyfill)  mediafragment.js: https://github.com/tomayac/Media-Fragments-URI  xywh.js: https://github.com/tomayac/xywh.js  Custom Players:  Ligne de Temps: http://ldt.iri.centrepompidou.fr/ldtplatform/ldt/  Synote: http://smfplayer.synote.org/smfplayer/  Noterik, Condat, JSI, etc. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 20
  • 21. Media Fragment Servers  Ninsuna: http://ninsuna.elis.ugent.be/MediaFragmentsServer  Southampton-Eurecom: node.js based implementation  YouTube: partial support, syntax difference  Dailymotion: partial support, syntax difference 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 21
  • 22. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 22
  • 23. Media Fragment Semantic Annotation Reg1 The "Big Three" at the Yalta Conference (Wikipedia)  Media Fragment creation: localize a region (person)  Media Fragment annotation (tagging) = interpretation Winston Churchill, UK Prime Minister, Allied Forces, WWII  Media Fragment semantic annotation :Reg1 foaf:depicts dbpedia:WinstonChurchill. dbpedia:Churchill rdfs:label "Winston Churchill"; rdf:type foaf:Person dbprop:order dbpedia:Prime_Minister_(UK). 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 23
  • 24. Media Fragment Semantic Annotation A history of G8 violence (video) (© Reuters) Seq4 Seq1  Media Fragment creation: localize a temporal sequence  Media Fragment annotation (tagging) = interpretation G8 Summit, EU Summit, Heiligendamm, 2007, Gothenburg, 2001  Media Fragment semantic annotation :Seq1 foaf:depicts dbpedia:33rd_G8_Summit. :Seq4 foaf:depicts dbpedia:EU_Summit. dbpedia:33rd_G8_Summit rdfs:label "33rd G8 summit"@en ; grs:point "54.143055555555556 11.841666666666667". 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 24
  • 25. Media Fragment Semantic Annotation  Things, not strings! http://googleblog.blogspot.fr/2012/05/introducing-knowledgegraph-things-not.html  Use knowledge bases (LOD)  Use common vocabularies (LOV)  Follow the 4 Linked Data principles  Refine the 4 Linked Media principles 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 25
  • 26. Open Annotation Data Model  Specification developed in the W3C Open Annotation Community Group http://www.openannotation.org/spec/core/  Core model  OWL vocabulary for representing and sharing annotation of digital resources (and their fragment) … in RDF  A body is related to a target  Nature of the annotation changes according to intention (motivation)  How to annotate this image? 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 26
  • 27. Semantic Annotation of an Image http://www.w3.org/community/openannotation/wiki/ SE_Semantically_Tagging_an_Image 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 27
  • 28. Maphub: http://maphub.github.io/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 28
  • 29. Open Video: Annotation Project http://openvideoannotation.org/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 29
  • 30. YouTube Annotations  Annotations are clickable text overlays on YouTube videos  Annotations are used to boost engagement, give more information, and aid in navigation 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 30
  • 31. YouTube Annotations: How To 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 31
  • 32. LinkedTV: automatic annotations ... 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 32
  • 33. ... and enrichment for hypervideos CONCEPT IN PLAYER Cubism Expressionism Fauvism FACETS / PROPERTIES OF CONCEPT 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 CONTENT ENRICHMENT - 33
  • 34. Media Fragments and Annotations http://data.linkedtv.eu/medi a/e2899e7f#t=840,900 nerd:Location Casablanca nerd:Location Cafe Rick nerd:Person H. Bogart nerd:Person I. Bergman  Media Fragment URI 1.0     21/10/2013 - Chapters Scenes Shots etc… International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 34
  • 35. Enrichment and Hypervideos nerd:Location Casablanca nerd:Location Cafe Rick nerd:Person H. Bogart Nerd:Person E. Tierney 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 nerd:Person I. Bergman nerd:Location China - 35
  • 36. 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 36
  • 37. NER Tools and Web APIs  Standalone software  GATE  Stanford CoreNLP  Temis http://nerd.eurecom.fr/  Web APIs 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 37
  • 38. NERD: Named Entity Recognition and Disambiguation  Compare performances of NER and NEL tools  Understand strengths and weaknesses of different Web APIs  Adapt NER processing to different context  (Learn how to) Combine NER (/ NEL) tools What is NERD? ontology1 REST API2 UI3 1 2 21/10/2013 - http://nerd.eurecom.fr/ontology http://nerd.eurecom.fr/api/application.wadl 3 http://nerd.eurecom.fr International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 38
  • 39. NERD User Interface 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 39
  • 40. Media Fragment + Open Annotation + NERD Locator MediaResource Annotation MediaFragment Entity Type URL (hyperlink) 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 40
  • 41. Media Fragment Enricher: http://mfe.synote.org/mfe/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 41
  • 42. Linking pieces of knowledge 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 42
  • 43. Linking pieces of knowledge 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 43
  • 44. Towards a Linked Media Layer  Enriching media with media from a closed collection (e.g. BBC archive)  The MediaEval scenario (~ 1697 hours of archived BBC video) http://www.multimediaeval.org/mediaeval2013/hyper2013/  Enriching media with content from the open web  LinkedTV scenarios: white listed web sites for each program  Media Collector for Social Media 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 44
  • 45. Media Collector  Composition of media item extractors (12 SNs)  Rely on search APIs (+ a fix 30s timeout window to provide results)  Fallback on screen scraping when necessary (Twitter ecosystem)  Implemented as a NodeJS server  Serialize results in a common schema (JSON) 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 45
  • 46. Deep link Permalink Clean text for NLP processing Aggregate view of ALL social interactions 12 Social Networks
  • 47. Media Collector + White List web sites Q:http://ir.lmcloud.vse.cz:8080/irapi/mediaserver?q=img_title:*Merkel* 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 47
  • 48. Seed video enriched with web content rbbaktuell_20120809 nerd:Location Brandenburg oa
  • 49. Enrichments are Annotations too 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 49
  • 50. http://linkedtv.project.cwi.nl/news/ 20.10.2013 WP2 - Linking hypervideos to Web content
  • 51. TV + Kinect + Web experience http://www.youtube.com/watch?v=4mSC685AG7k 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 51
  • 52. Take Away Summary  Video is a first class citizen on the Web  Annotations: Ontology and API for Media Resources, Open Annotation Data Model  Access: Media Fragments URI  NERD platform for extracting key information from textual resources including video subtitles and microposts  Embrace the Linked Media vision  Publish, re-use, re-purpose and remix media descriptions  Develop links between (part of) media items via their descriptions 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 52
  • 53. Winter School: http://winterschool.mediamixer.eu/ 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 53
  • 54. Credits  Giuseppe Rizzo, Vuk Milicic, José Luis Redondo Garcia (EURECOM)  Thomas Steiner (Google Inc.), Yunjia Li (University of Southampton)  Marieke van Erp (Free University of Amsterdam)  Erik Mannens, Davy ven Deursen (iMinds, Uni. Ghent)  Paolo Ciccarese, Robert Sanderson, Herbert Van de Sompel and all the members of the W3C Open Annotation Community Group  … and many other students 21/10/2013 - International Workshop on Semantic Music and Media (SMAM) @ISWC 2013 - 54