Addressing and Annotating Multimedia Fragments

Addressing and annotating multimedia fragmentsRaphaël Troncy <raphael.troncy@eurecom.fr>

Use Case 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -2

Use Case Aidem received on her Facebook wall a status message containing a Media Fragment URI  Use a ‘#’ !  Highlight a video sequence  Highlight a region to pay attention to 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -3

What are Media Fragments?0 20 temporal media fragment 35 t spatial media fragment track media fragment 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -4

Temporal Media FragmentsOriginal resource lengthFragment beginning Playback progress Fragment end 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -5

Spatial Media Fragments highlighted fragmentsemi-opaque overlay 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -6

Media Fragments Use Cases Bookmark / Share parts (fragments) of audio/video content Annotate media fragments Mash-ups Conserve bandwidth http://www.w3.org/TR/media-frags-reqs/ 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -7

Requirements r01: Temporal fragments:  a clipping along the time dimension from a start to an end time that are within the duration of the media resource r02: Spatial fragments:  a clipping of an image region, only consider rectangular regions r03: Track fragments:  a track as exposed by a container format of the media resource r04: Named fragments:  A temporal media fragment that has been given a name through some sort of annotation mechanism 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -8

Media URIs Using URI query part: http://www.example.org/video.ogv?t=60,100 Using URI fragment part: http://www.example.org/video.ogv#t=60,100 Mixing both: http://www.example.org/video.ogv?t=60,100 #t=10,15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -9

URI Fragments vs. URI Queries #t=20,30 ?t=20,30 secondary resource, primary resource, notion of context no notion of context extraction needs to be no adaptation restrictions expressible in byte ranges no provisions for key-value pairs are sent to communi-cating the server fragments to the server potentially cacheable not cacheable The media fragment URI syntax can be used for URI queries We will focus on URI fragments 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 10

Media Fragments Resolution For the URI query part:  The media file is only processed on server side  The UA receives a new video file For the URI fragment part:  Smart UA may strip out the fragment definition and encode it into custom http headers (Range header)  (Media) Servers will handle the request, slice the media content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 11

Influence of Media Formats Fragment extraction needs to be expressible in terms of byte ranges Requirements for the different axes  temporal: presence of intra-coded frames (i.e., random access points)  spatial: presence of independently coded spatial regions  track: need to be identifiable by a name Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 12

Can I use Media Fragments now? <video id=v src=AudioAPI.webm#t=50,100 onloadedmetadata=update() onpause=update() onplay=update() onseeked=update() controls></video>  Firefox 9+ (or WebKit)  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 13

Can I use Media Fragments now?  Sysnote over YouTube  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 14

Can I use Media Fragments now?  Ninsuna  Bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 15

Examining HTTP Traffic HTTP request:  retrieving URI  parsing key=values pairs from fragment part  setting Range header HTTP response:  checking Content-Type and Content-Range-Mapping headers values  attaching custom playback controls to page  creating spatial dimension overlay (if specified) 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 16

Example: Requesting a Time Fragment A web developer specifies a video source with a temporal fragment URI:http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15 key=value pair is analyzed, fragment begin and end time are matched t=5,15 Media Fragments Extension analyses the fragment part, retrieves beginning and end time and sets proper Range header value: Range: t:npt=5-15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 17

Example: Requesting a Time Fragment The NinSuna server responds with the 206 Partial Content response and Content-Range- Mapping header showing the mapped time ranges and media fragment in the message payload:HTTP/1.1 206 Partial ContentContent-Type: video/oggAccept-Ranges: bytes, t, track, idContent-Range: bytes 629578-1690588/4055466Content-Range-Mapping:{t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466} 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 18

Example: Requesting a Time Fragment ... and it won’t work! Because the player does not have the media file header 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 19

Example: Requesting a Time Fragment Player needs to be initialized, thus bare fragment is not playable To fix this, we add ;include-setup to the Range header value: Range: t:npt=5-15;include-setup 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 20

Example: Requesting a Time Fragment The response from the server is slightly different: HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33;include-setup} ={bytes 0-5998,629578-1690588/4055466} ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 21

Example: Requesting a Time Fragment This is why we need to attach a Stream Listener to the HTTP channel:HTTP/1.1 206 Partial ContentContent-Type:multipart/byteranges;boundary=End... HTTP/1.1 206 Partial Content--EndContent-Type: video/ogg Content-Type: video/oggContent-Range: bytes 0-5997/4055466 ...{binary data} {binary data}--End {binary data}Content-Type: video/oggContent-Range: bytes 629578-1690588/4055466{binary data}--End--End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 22

Spatial Fragments Without additional information send to server Overlay is created by appending four additional DIVs to the webpage They are styled to create the impression of semi-opaque layer over the video element 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 23

NinSuna: Overview Fully integrated media adaptation and delivery platform  media adaptation and media packaging core is independent of media formats based on model-driven content adaptation & delivery technique  only high-level adaptation operations such as scene selection and frame rate scaling no transcoding is applied ideal candidate for MF implementation  more information: http://ninsuna.elis.ugent.be 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 24

MF-specific Features of NinSuna Support for Media Fragment URI queries  both HTTP and RTSP implementation  try some URIs at http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t railer_2.mp4?track=5;6 Support for Media Fragment URI fragments  i.e., support for the MF-specific HTTP headers Combining Media Fragment URI queries and fragments  e.g., http://foo.com/media.mp4?t=10,40#t=5,10 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 25

Impact on the Existing Web Infrastructure Updating existing infrastructure is not trivial  web servers need to be extended with media extractors MF-specific HTTP communication  existing web caches are only effective with HTTP byte ranges specialized media caches need to be developed in the future to cache the other cases  user agents need to be extended with a Media Fragment URI parser and interpreter MF-specific HTTP communication MF-specific visualization 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 26

Media Fragments Proxy Goal: make existing media resources, served by generic HTTP Web servers, available as Media Fragments Biggest problem for content providers willing to support Media Fragments  media extractor (dependent on underlying media formats) Solution: an approach that  works with existing HTTP Web servers (apache, IIS)  works with existing Web caches  works with not so smart user agents 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 27

Media Fragments Proxy http://foo.com/video.ogv#t=11,19 GET /?url=http://foo.com/video.ogv HTTP/1.1 Host: MFProxy.com HTTP Accept: video/* server Range: t:npt=11-19 Accept-Range-Redirect: bytes HTTP/1.1 307 Temporary Redirect Location: http://foo.com/video.ogv -get (only) the header info Accept-Ranges: bytes, t, track Content-Length:a fragment-to-byte mapping -try to find 0 Content-Type: video/ogg -construct the redirect response Content-Range-Mapping: t:npt 10-20/0-50 Range-Redirect: 24000-32000 Vary: Accept-Range-Redirect GET /video.ogv HTTP/1.1 MF Proxy Host: foo.com MF User Accept: video/* Agent Range: bytes=24000-32000http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI> 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 28

Annotating Media Fragments yuma.min.js 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 29

Annotating Media Fragments Benchmarking: Sphinx, HTK, Julius  NER + full text index with the transcription  Interlinking with the Linked Data Cloud to enable semantic search 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 30

10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 31

http://www.slideshare.net/troncy10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 32

Addressing and Annotating Multimedia Fragments

by Raphael Troncy

Accessibility

Categories

Upload Details

Usage Rights

Statistics