Addressing and Annotating Multimedia Fragments

  • 1,186 views
Uploaded on

Talk given at the "Séminaire Muséologie: Le Web devient audivisuel" at IRI on Tuesday 10/04/2012, Paris, France

Talk given at the "Séminaire Muséologie: Le Web devient audivisuel" at IRI on Tuesday 10/04/2012, Paris, France

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,186
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
15
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Addressing and annotating multimedia fragmentsRaphaël Troncy <raphael.troncy@eurecom.fr>
  • 2. Use Case 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -2
  • 3. Use Case Aidem received on her Facebook wall a status message containing a Media Fragment URI  Use a ‘#’ !  Highlight a video sequence  Highlight a region to pay attention to 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -3
  • 4. What are Media Fragments?0 20 temporal media fragment 35 t spatial media fragment track media fragment 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -4
  • 5. Temporal Media FragmentsOriginal resource lengthFragment beginning Playback progress Fragment end 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -5
  • 6. Spatial Media Fragments highlighted fragmentsemi-opaque overlay 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -6
  • 7. Media Fragments Use Cases Bookmark / Share parts (fragments) of audio/video content Annotate media fragments Mash-ups Conserve bandwidth http://www.w3.org/TR/media-frags-reqs/ 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -7
  • 8. Requirements r01: Temporal fragments:  a clipping along the time dimension from a start to an end time that are within the duration of the media resource r02: Spatial fragments:  a clipping of an image region, only consider rectangular regions r03: Track fragments:  a track as exposed by a container format of the media resource r04: Named fragments:  A temporal media fragment that has been given a name through some sort of annotation mechanism 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -8
  • 9. Media URIs Using URI query part: http://www.example.org/video.ogv?t=60,100 Using URI fragment part: http://www.example.org/video.ogv#t=60,100 Mixing both: http://www.example.org/video.ogv?t=60,100 #t=10,15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -9
  • 10. URI Fragments vs. URI Queries #t=20,30 ?t=20,30 secondary resource, primary resource, notion of context no notion of context extraction needs to be no adaptation restrictions expressible in byte ranges no provisions for key-value pairs are sent to communi-cating the server fragments to the server potentially cacheable not cacheable The media fragment URI syntax can be used for URI queries We will focus on URI fragments 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 10
  • 11. Media Fragments Resolution For the URI query part:  The media file is only processed on server side  The UA receives a new video file For the URI fragment part:  Smart UA may strip out the fragment definition and encode it into custom http headers (Range header)  (Media) Servers will handle the request, slice the media content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 11
  • 12. Influence of Media Formats Fragment extraction needs to be expressible in terms of byte ranges Requirements for the different axes  temporal: presence of intra-coded frames (i.e., random access points)  spatial: presence of independently coded spatial regions  track: need to be identifiable by a name Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 12
  • 13. Can I use Media Fragments now? <video id=v src=AudioAPI.webm#t=50,100 onloadedmetadata=update() onpause=update() onplay=update() onseeked=update() controls></video>  Firefox 9+ (or WebKit)  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 13
  • 14. Can I use Media Fragments now?  Sysnote over YouTube  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 14
  • 15. Can I use Media Fragments now?  Ninsuna  Bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 15
  • 16. Examining HTTP Traffic HTTP request:  retrieving URI  parsing key=values pairs from fragment part  setting Range header HTTP response:  checking Content-Type and Content-Range-Mapping headers values  attaching custom playback controls to page  creating spatial dimension overlay (if specified) 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 16
  • 17. Example: Requesting a Time Fragment A web developer specifies a video source with a temporal fragment URI:http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15 key=value pair is analyzed, fragment begin and end time are matched t=5,15 Media Fragments Extension analyses the fragment part, retrieves beginning and end time and sets proper Range header value: Range: t:npt=5-15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 17
  • 18. Example: Requesting a Time Fragment The NinSuna server responds with the 206 Partial Content response and Content-Range- Mapping header showing the mapped time ranges and media fragment in the message payload:HTTP/1.1 206 Partial ContentContent-Type: video/oggAccept-Ranges: bytes, t, track, idContent-Range: bytes 629578-1690588/4055466Content-Range-Mapping:{t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466} 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 18
  • 19. Example: Requesting a Time Fragment ... and it won’t work! Because the player does not have the media file header 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 19
  • 20. Example: Requesting a Time Fragment Player needs to be initialized, thus bare fragment is not playable To fix this, we add ;include-setup to the Range header value: Range: t:npt=5-15;include-setup 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 20
  • 21. Example: Requesting a Time Fragment The response from the server is slightly different: HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33;include-setup} ={bytes 0-5998,629578-1690588/4055466} ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 21
  • 22. Example: Requesting a Time Fragment This is why we need to attach a Stream Listener to the HTTP channel:HTTP/1.1 206 Partial ContentContent-Type:multipart/byteranges;boundary=End... HTTP/1.1 206 Partial Content--EndContent-Type: video/ogg Content-Type: video/oggContent-Range: bytes 0-5997/4055466 ...{binary data} {binary data}--End {binary data}Content-Type: video/oggContent-Range: bytes 629578-1690588/4055466{binary data}--End--End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 22
  • 23. Spatial Fragments Without additional information send to server Overlay is created by appending four additional DIVs to the webpage They are styled to create the impression of semi-opaque layer over the video element 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 23
  • 24. NinSuna: Overview Fully integrated media adaptation and delivery platform  media adaptation and media packaging core is independent of media formats based on model-driven content adaptation & delivery technique  only high-level adaptation operations such as scene selection and frame rate scaling no transcoding is applied ideal candidate for MF implementation  more information: http://ninsuna.elis.ugent.be 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 24
  • 25. MF-specific Features of NinSuna Support for Media Fragment URI queries  both HTTP and RTSP implementation  try some URIs at http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t railer_2.mp4?track=5;6 Support for Media Fragment URI fragments  i.e., support for the MF-specific HTTP headers Combining Media Fragment URI queries and fragments  e.g., http://foo.com/media.mp4?t=10,40#t=5,10 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 25
  • 26. Impact on the Existing Web Infrastructure Updating existing infrastructure is not trivial  web servers need to be extended with media extractors MF-specific HTTP communication  existing web caches are only effective with HTTP byte ranges specialized media caches need to be developed in the future to cache the other cases  user agents need to be extended with a Media Fragment URI parser and interpreter MF-specific HTTP communication MF-specific visualization 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 26
  • 27. Media Fragments Proxy Goal: make existing media resources, served by generic HTTP Web servers, available as Media Fragments Biggest problem for content providers willing to support Media Fragments  media extractor (dependent on underlying media formats) Solution: an approach that  works with existing HTTP Web servers (apache, IIS)  works with existing Web caches  works with not so smart user agents 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 27
  • 28. Media Fragments Proxy http://foo.com/video.ogv#t=11,19 GET /?url=http://foo.com/video.ogv HTTP/1.1 Host: MFProxy.com HTTP Accept: video/* server Range: t:npt=11-19 Accept-Range-Redirect: bytes HTTP/1.1 307 Temporary Redirect Location: http://foo.com/video.ogv -get (only) the header info Accept-Ranges: bytes, t, track Content-Length:a fragment-to-byte mapping -try to find 0 Content-Type: video/ogg -construct the redirect response Content-Range-Mapping: t:npt 10-20/0-50 Range-Redirect: 24000-32000 Vary: Accept-Range-Redirect GET /video.ogv HTTP/1.1 MF Proxy Host: foo.com MF User Accept: video/* Agent Range: bytes=24000-32000http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI> 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 28
  • 29. Annotating Media Fragments yuma.min.js 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 29
  • 30. Annotating Media Fragments Benchmarking: Sphinx, HTK, Julius  NER + full text index with the transcription  Interlinking with the Linked Data Cloud to enable semantic search 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 30
  • 31. 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 31
  • 32. http://www.slideshare.net/troncy10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 32