Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Addressing and annotating
     multimedia fragments
Raphaël Troncy <raphael.troncy@eurecom.fr>
Use Case




  10/04/2012 -   Séminaire Muséologie - Le Web devient audiovisuel   -2
Use Case

 Aidem received on her Facebook
  wall a status message containing
  a Media Fragment URI
   Use a ‘#’ !
   H...
What are Media Fragments?




0                                  20            temporal media fragment               35   ...
Temporal Media Fragments



Original resource
      length




Fragment beginning     Playback progress                   ...
Spatial Media Fragments


                                                                              highlighted
      ...
Media Fragments Use Cases

 Bookmark / Share parts (fragments) of
  audio/video content
 Annotate media fragments
 Mash...
Requirements
 r01: Temporal fragments:
    a clipping along the time dimension from a start to an end time that
     are...
Media URIs

 Using URI query part:
  http://www.example.org/video.ogv?t=60,100

 Using URI fragment part:
  http://www.e...
URI Fragments vs. URI Queries
                      #t=20,30                                           ?t=20,30
          ...
Media Fragments Resolution

 For the URI query part:
   The media file is only processed on server side
   The UA recei...
Influence of Media Formats

 Fragment extraction needs to be expressible in
  terms of byte ranges
 Requirements for the...
Can I use Media Fragments now?

                                           <video id='v'
                                 ...
Can I use Media Fragments now?




                                                  Sysnote over YouTube
               ...
Can I use Media Fragments now?



                                                                      Ninsuna
         ...
Examining HTTP Traffic

 HTTP request:
   retrieving URI
   parsing key=values pairs from fragment part
   setting Ran...
Example: Requesting a Time Fragment

 A web developer specifies a video source with
  a temporal fragment URI:
http://nin...
Example: Requesting a Time Fragment

 The NinSuna server responds with the 206
  Partial Content response and Content-Ran...
Example: Requesting a Time Fragment

 ... and it won’t work!


 Because the
  player does not
  have the media
  file he...
Example: Requesting a Time Fragment

 Player needs to be initialized, thus bare
  fragment is not playable
 To fix this,...
Example: Requesting a Time Fragment

 The response from the server is slightly different:
 HTTP/1.1 206 Partial Content

...
Example: Requesting a Time Fragment

 This is why we need to attach a Stream Listener
  to the HTTP channel:
HTTP/1.1 206...
Spatial Fragments

 Without additional information send to server
 Overlay is created by appending four additional
  DIV...
NinSuna: Overview

 Fully integrated media adaptation and delivery
  platform
   media adaptation and media packaging co...
MF-specific Features of NinSuna

 Support for Media Fragment URI queries
   both HTTP and RTSP implementation
   try so...
Impact on the Existing Web Infrastructure

 Updating existing infrastructure is not trivial
   web servers need to be ex...
Media Fragments Proxy

 Goal: make existing media resources, served
  by generic HTTP Web servers, available as
  Media F...
Media Fragments Proxy                                                           http://foo.com/video.ogv#t=11,19


       ...
Annotating Media Fragments                                           yuma.min.js




  10/04/2012 -   Séminaire Muséologie...
Annotating Media Fragments

                                                        Benchmarking: Sphinx, HTK,
           ...
10/04/2012 -   Séminaire Muséologie - Le Web devient audiovisuel   - 31
http://www.slideshare.net/troncy

10/04/2012 -   Séminaire Muséologie - Le Web devient audiovisuel   - 32
Upcoming SlideShare
Loading in …5
×

of

Addressing and Annotating Multimedia Fragments Slide 1 Addressing and Annotating Multimedia Fragments Slide 2 Addressing and Annotating Multimedia Fragments Slide 3 Addressing and Annotating Multimedia Fragments Slide 4 Addressing and Annotating Multimedia Fragments Slide 5 Addressing and Annotating Multimedia Fragments Slide 6 Addressing and Annotating Multimedia Fragments Slide 7 Addressing and Annotating Multimedia Fragments Slide 8 Addressing and Annotating Multimedia Fragments Slide 9 Addressing and Annotating Multimedia Fragments Slide 10 Addressing and Annotating Multimedia Fragments Slide 11 Addressing and Annotating Multimedia Fragments Slide 12 Addressing and Annotating Multimedia Fragments Slide 13 Addressing and Annotating Multimedia Fragments Slide 14 Addressing and Annotating Multimedia Fragments Slide 15 Addressing and Annotating Multimedia Fragments Slide 16 Addressing and Annotating Multimedia Fragments Slide 17 Addressing and Annotating Multimedia Fragments Slide 18 Addressing and Annotating Multimedia Fragments Slide 19 Addressing and Annotating Multimedia Fragments Slide 20 Addressing and Annotating Multimedia Fragments Slide 21 Addressing and Annotating Multimedia Fragments Slide 22 Addressing and Annotating Multimedia Fragments Slide 23 Addressing and Annotating Multimedia Fragments Slide 24 Addressing and Annotating Multimedia Fragments Slide 25 Addressing and Annotating Multimedia Fragments Slide 26 Addressing and Annotating Multimedia Fragments Slide 27 Addressing and Annotating Multimedia Fragments Slide 28 Addressing and Annotating Multimedia Fragments Slide 29 Addressing and Annotating Multimedia Fragments Slide 30 Addressing and Annotating Multimedia Fragments Slide 31 Addressing and Annotating Multimedia Fragments Slide 32
Upcoming SlideShare
Intelligent tools-mitja-jermol-2013-bali-7 may2013
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

Addressing and Annotating Multimedia Fragments

Download to read offline

Talk given at the "Séminaire Muséologie: Le Web devient audivisuel" at IRI on Tuesday 10/04/2012, Paris, France

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Addressing and Annotating Multimedia Fragments

  1. 1. Addressing and annotating multimedia fragments Raphaël Troncy <raphael.troncy@eurecom.fr>
  2. 2. Use Case 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -2
  3. 3. Use Case  Aidem received on her Facebook wall a status message containing a Media Fragment URI  Use a ‘#’ !  Highlight a video sequence  Highlight a region to pay attention to 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -3
  4. 4. What are Media Fragments? 0 20 temporal media fragment 35 t spatial media fragment track media fragment 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -4
  5. 5. Temporal Media Fragments Original resource length Fragment beginning Playback progress Fragment end 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -5
  6. 6. Spatial Media Fragments highlighted fragment semi-opaque overlay 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -6
  7. 7. Media Fragments Use Cases  Bookmark / Share parts (fragments) of audio/video content  Annotate media fragments  Mash-ups  Conserve bandwidth http://www.w3.org/TR/media-frags-reqs/ 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -7
  8. 8. Requirements  r01: Temporal fragments:  a clipping along the time dimension from a start to an end time that are within the duration of the media resource  r02: Spatial fragments:  a clipping of an image region, only consider rectangular regions  r03: Track fragments:  a track as exposed by a container format of the media resource  r04: Named fragments:  A temporal media fragment that has been given a name through some sort of annotation mechanism 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -8
  9. 9. Media URIs  Using URI query part: http://www.example.org/video.ogv?t=60,100  Using URI fragment part: http://www.example.org/video.ogv#t=60,100  Mixing both: http://www.example.org/video.ogv?t=60,100 #t=10,15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -9
  10. 10. URI Fragments vs. URI Queries #t=20,30 ?t=20,30 secondary resource, primary resource, notion of context no notion of context extraction needs to be no adaptation restrictions expressible in byte ranges no provisions for key-value pairs are sent to communi-cating the server fragments to the server potentially cacheable not cacheable  The media fragment URI syntax can be used for URI queries  We will focus on URI fragments 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 10
  11. 11. Media Fragments Resolution  For the URI query part:  The media file is only processed on server side  The UA receives a new video file  For the URI fragment part:  Smart UA may strip out the fragment definition and encode it into custom http headers (Range header)  (Media) Servers will handle the request, slice the media content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 11
  12. 12. Influence of Media Formats  Fragment extraction needs to be expressible in terms of byte ranges  Requirements for the different axes  temporal: presence of intra-coded frames (i.e., random access points)  spatial: presence of independently coded spatial regions  track: need to be identifiable by a name  Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 12
  13. 13. Can I use Media Fragments now? <video id='v' src='AudioAPI.webm#t=50,100' onloadedmetadata='update()' onpause='update()' onplay='update()' onseeked='update()' controls></video>  Firefox 9+ (or WebKit)  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 13
  14. 14. Can I use Media Fragments now?  Sysnote over YouTube  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 14
  15. 15. Can I use Media Fragments now?  Ninsuna  Bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 15
  16. 16. Examining HTTP Traffic  HTTP request:  retrieving URI  parsing key=values pairs from fragment part  setting Range header  HTTP response:  checking Content-Type and Content-Range-Mapping headers values  attaching custom playback controls to page  creating spatial dimension overlay (if specified) 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 16
  17. 17. Example: Requesting a Time Fragment  A web developer specifies a video source with a temporal fragment URI: http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15  key=value pair is analyzed, fragment begin and end time are matched t=5,15  Media Fragments Extension analyses the fragment part, retrieves beginning and end time and sets proper Range header value: Range: t:npt=5-15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 17
  18. 18. Example: Requesting a Time Fragment  The NinSuna server responds with the 206 Partial Content response and Content-Range- Mapping header showing the mapped time ranges and media fragment in the message payload: HTTP/1.1 206 Partial Content Content-Type: video/ogg Accept-Ranges: bytes, t, track, id Content-Range: bytes 629578-1690588/4055466 Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466} 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 18
  19. 19. Example: Requesting a Time Fragment  ... and it won’t work!  Because the player does not have the media file header 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 19
  20. 20. Example: Requesting a Time Fragment  Player needs to be initialized, thus bare fragment is not playable  To fix this, we add ;include-setup to the Range header value: Range: t:npt=5-15;include-setup 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 20
  21. 21. Example: Requesting a Time Fragment  The response from the server is slightly different: HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33;include-setup} ={bytes 0-5998,629578-1690588/4055466} ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 21
  22. 22. Example: Requesting a Time Fragment  This is why we need to attach a Stream Listener to the HTTP channel: HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End ... HTTP/1.1 206 Partial Content --End Content-Type: video/ogg Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 ... {binary data} {binary data} --End {binary data} Content-Type: video/ogg Content-Range: bytes 629578- 1690588/4055466 {binary data} --End --End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 22
  23. 23. Spatial Fragments  Without additional information send to server  Overlay is created by appending four additional DIVs to the webpage  They are styled to create the impression of semi-opaque layer over the video element 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 23
  24. 24. NinSuna: Overview  Fully integrated media adaptation and delivery platform  media adaptation and media packaging core is independent of media formats based on model-driven content adaptation & delivery technique  only high-level adaptation operations such as scene selection and frame rate scaling no transcoding is applied ideal candidate for MF implementation  more information: http://ninsuna.elis.ugent.be 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 24
  25. 25. MF-specific Features of NinSuna  Support for Media Fragment URI queries  both HTTP and RTSP implementation  try some URIs at http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t railer_2.mp4?track=5;6  Support for Media Fragment URI fragments  i.e., support for the MF-specific HTTP headers  Combining Media Fragment URI queries and fragments  e.g., http://foo.com/media.mp4?t=10,40#t=5,10 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 25
  26. 26. Impact on the Existing Web Infrastructure  Updating existing infrastructure is not trivial  web servers need to be extended with media extractors MF-specific HTTP communication  existing web caches are only effective with HTTP byte ranges specialized media caches need to be developed in the future to cache the other cases  user agents need to be extended with a Media Fragment URI parser and interpreter MF-specific HTTP communication MF-specific visualization 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 26
  27. 27. Media Fragments Proxy  Goal: make existing media resources, served by generic HTTP Web servers, available as Media Fragments  Biggest problem for content providers willing to support Media Fragments  media extractor (dependent on underlying media formats)  Solution: an approach that  works with existing HTTP Web servers (apache, IIS)  works with existing Web caches  works with not so smart user agents 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 27
  28. 28. Media Fragments Proxy http://foo.com/video.ogv#t=11,19 GET /?url=http://foo.com/video.ogv HTTP/1.1 Host: MFProxy.com HTTP Accept: video/* server Range: t:npt=11-19 Accept-Range-Redirect: bytes HTTP/1.1 307 Temporary Redirect Location: http://foo.com/video.ogv -get (only) the header info Accept-Ranges: bytes, t, track Content-Length:a fragment-to-byte mapping -try to find 0 Content-Type: video/ogg -construct the redirect response Content-Range-Mapping: t:npt 10-20/0-50 Range-Redirect: 24000-32000 Vary: Accept-Range-Redirect GET /video.ogv HTTP/1.1 MF Proxy Host: foo.com MF User Accept: video/* Agent Range: bytes=24000-32000 http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI> 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 28
  29. 29. Annotating Media Fragments yuma.min.js 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 29
  30. 30. Annotating Media Fragments Benchmarking: Sphinx, HTK, Julius  NER + full text index with the transcription  Interlinking with the Linked Data Cloud to enable semantic search 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 30
  31. 31. 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 31
  32. 32. http://www.slideshare.net/troncy 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 32

Talk given at the "Séminaire Muséologie: Le Web devient audivisuel" at IRI on Tuesday 10/04/2012, Paris, France

Views

Total views

2,206

On Slideshare

0

From embeds

0

Number of embeds

135

Actions

Downloads

21

Shares

0

Comments

0

Likes

0

×