Addressing and annotating     multimedia fragmentsRaphaël Troncy <raphael.troncy@eurecom.fr>
Use Case  10/04/2012 -   Séminaire Muséologie - Le Web devient audiovisuel   -2
Use Case Aidem received on her Facebook  wall a status message containing  a Media Fragment URI   Use a ‘#’ !   Highlig...
What are Media Fragments?0                                  20            temporal media fragment               35   t    ...
Temporal Media FragmentsOriginal resource      lengthFragment beginning     Playback progress                             ...
Spatial Media Fragments                                                                              highlighted          ...
Media Fragments Use Cases Bookmark / Share parts (fragments) of  audio/video content Annotate media fragments Mash-ups...
Requirements r01: Temporal fragments:    a clipping along the time dimension from a start to an end time that     are wi...
Media URIs Using URI query part:  http://www.example.org/video.ogv?t=60,100 Using URI fragment part:  http://www.example...
URI Fragments vs. URI Queries                      #t=20,30                                           ?t=20,30            ...
Media Fragments Resolution For the URI query part:   The media file is only processed on server side   The UA receives ...
Influence of Media Formats Fragment extraction needs to be expressible in  terms of byte ranges Requirements for the dif...
Can I use Media Fragments now?                                           <video id=v                                      ...
Can I use Media Fragments now?                                                  Sysnote over YouTube                     ...
Can I use Media Fragments now?                                                                      Ninsuna              ...
Examining HTTP Traffic HTTP request:   retrieving URI   parsing key=values pairs from fragment part   setting Range he...
Example: Requesting a Time Fragment A web developer specifies a video source with  a temporal fragment URI:http://ninsuna...
Example: Requesting a Time Fragment The NinSuna server responds with the 206  Partial Content response and Content-Range-...
Example: Requesting a Time Fragment ... and it won’t work! Because the  player does not  have the media  file header    ...
Example: Requesting a Time Fragment Player needs to be initialized, thus bare  fragment is not playable To fix this, we ...
Example: Requesting a Time Fragment The response from the server is slightly different: HTTP/1.1 206 Partial Content Cont...
Example: Requesting a Time Fragment This is why we need to attach a Stream Listener  to the HTTP channel:HTTP/1.1 206 Par...
Spatial Fragments Without additional information send to server Overlay is created by appending four additional  DIVs to...
NinSuna: Overview Fully integrated media adaptation and delivery  platform   media adaptation and media packaging core i...
MF-specific Features of NinSuna Support for Media Fragment URI queries   both HTTP and RTSP implementation   try some U...
Impact on the Existing Web Infrastructure Updating existing infrastructure is not trivial   web servers need to be exten...
Media Fragments Proxy Goal: make existing media resources, served  by generic HTTP Web servers, available as  Media Fragm...
Media Fragments Proxy                                                           http://foo.com/video.ogv#t=11,19          ...
Annotating Media Fragments                                           yuma.min.js  10/04/2012 -   Séminaire Muséologie - Le...
Annotating Media Fragments                                                        Benchmarking: Sphinx, HTK,              ...
10/04/2012 -   Séminaire Muséologie - Le Web devient audiovisuel   - 31
http://www.slideshare.net/troncy10/04/2012 -   Séminaire Muséologie - Le Web devient audiovisuel   - 32
Upcoming SlideShare
Loading in...5
×

Addressing and Annotating Multimedia Fragments

1,363

Published on

Talk given at the "Séminaire Muséologie: Le Web devient audivisuel" at IRI on Tuesday 10/04/2012, Paris, France

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,363
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Addressing and Annotating Multimedia Fragments

  1. 1. Addressing and annotating multimedia fragmentsRaphaël Troncy <raphael.troncy@eurecom.fr>
  2. 2. Use Case 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -2
  3. 3. Use Case Aidem received on her Facebook wall a status message containing a Media Fragment URI  Use a ‘#’ !  Highlight a video sequence  Highlight a region to pay attention to 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -3
  4. 4. What are Media Fragments?0 20 temporal media fragment 35 t spatial media fragment track media fragment 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -4
  5. 5. Temporal Media FragmentsOriginal resource lengthFragment beginning Playback progress Fragment end 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -5
  6. 6. Spatial Media Fragments highlighted fragmentsemi-opaque overlay 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -6
  7. 7. Media Fragments Use Cases Bookmark / Share parts (fragments) of audio/video content Annotate media fragments Mash-ups Conserve bandwidth http://www.w3.org/TR/media-frags-reqs/ 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -7
  8. 8. Requirements r01: Temporal fragments:  a clipping along the time dimension from a start to an end time that are within the duration of the media resource r02: Spatial fragments:  a clipping of an image region, only consider rectangular regions r03: Track fragments:  a track as exposed by a container format of the media resource r04: Named fragments:  A temporal media fragment that has been given a name through some sort of annotation mechanism 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -8
  9. 9. Media URIs Using URI query part: http://www.example.org/video.ogv?t=60,100 Using URI fragment part: http://www.example.org/video.ogv#t=60,100 Mixing both: http://www.example.org/video.ogv?t=60,100 #t=10,15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel -9
  10. 10. URI Fragments vs. URI Queries #t=20,30 ?t=20,30 secondary resource, primary resource, notion of context no notion of context extraction needs to be no adaptation restrictions expressible in byte ranges no provisions for key-value pairs are sent to communi-cating the server fragments to the server potentially cacheable not cacheable The media fragment URI syntax can be used for URI queries We will focus on URI fragments 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 10
  11. 11. Media Fragments Resolution For the URI query part:  The media file is only processed on server side  The UA receives a new video file For the URI fragment part:  Smart UA may strip out the fragment definition and encode it into custom http headers (Range header)  (Media) Servers will handle the request, slice the media content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 11
  12. 12. Influence of Media Formats Fragment extraction needs to be expressible in terms of byte ranges Requirements for the different axes  temporal: presence of intra-coded frames (i.e., random access points)  spatial: presence of independently coded spatial regions  track: need to be identifiable by a name Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 12
  13. 13. Can I use Media Fragments now? <video id=v src=AudioAPI.webm#t=50,100 onloadedmetadata=update() onpause=update() onplay=update() onseeked=update() controls></video>  Firefox 9+ (or WebKit)  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 13
  14. 14. Can I use Media Fragments now?  Sysnote over YouTube  No bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 14
  15. 15. Can I use Media Fragments now?  Ninsuna  Bandwidth saving 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 15
  16. 16. Examining HTTP Traffic HTTP request:  retrieving URI  parsing key=values pairs from fragment part  setting Range header HTTP response:  checking Content-Type and Content-Range-Mapping headers values  attaching custom playback controls to page  creating spatial dimension overlay (if specified) 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 16
  17. 17. Example: Requesting a Time Fragment A web developer specifies a video source with a temporal fragment URI:http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15 key=value pair is analyzed, fragment begin and end time are matched t=5,15 Media Fragments Extension analyses the fragment part, retrieves beginning and end time and sets proper Range header value: Range: t:npt=5-15 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 17
  18. 18. Example: Requesting a Time Fragment The NinSuna server responds with the 206 Partial Content response and Content-Range- Mapping header showing the mapped time ranges and media fragment in the message payload:HTTP/1.1 206 Partial ContentContent-Type: video/oggAccept-Ranges: bytes, t, track, idContent-Range: bytes 629578-1690588/4055466Content-Range-Mapping:{t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466} 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 18
  19. 19. Example: Requesting a Time Fragment ... and it won’t work! Because the player does not have the media file header 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 19
  20. 20. Example: Requesting a Time Fragment Player needs to be initialized, thus bare fragment is not playable To fix this, we add ;include-setup to the Range header value: Range: t:npt=5-15;include-setup 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 20
  21. 21. Example: Requesting a Time Fragment The response from the server is slightly different: HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33;include-setup} ={bytes 0-5998,629578-1690588/4055466} ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 21
  22. 22. Example: Requesting a Time Fragment This is why we need to attach a Stream Listener to the HTTP channel:HTTP/1.1 206 Partial ContentContent-Type:multipart/byteranges;boundary=End... HTTP/1.1 206 Partial Content--EndContent-Type: video/ogg Content-Type: video/oggContent-Range: bytes 0-5997/4055466 ...{binary data} {binary data}--End {binary data}Content-Type: video/oggContent-Range: bytes 629578-1690588/4055466{binary data}--End--End-- 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 22
  23. 23. Spatial Fragments Without additional information send to server Overlay is created by appending four additional DIVs to the webpage They are styled to create the impression of semi-opaque layer over the video element 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 23
  24. 24. NinSuna: Overview Fully integrated media adaptation and delivery platform  media adaptation and media packaging core is independent of media formats based on model-driven content adaptation & delivery technique  only high-level adaptation operations such as scene selection and frame rate scaling no transcoding is applied ideal candidate for MF implementation  more information: http://ninsuna.elis.ugent.be 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 24
  25. 25. MF-specific Features of NinSuna Support for Media Fragment URI queries  both HTTP and RTSP implementation  try some URIs at http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t railer_2.mp4?track=5;6 Support for Media Fragment URI fragments  i.e., support for the MF-specific HTTP headers Combining Media Fragment URI queries and fragments  e.g., http://foo.com/media.mp4?t=10,40#t=5,10 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 25
  26. 26. Impact on the Existing Web Infrastructure Updating existing infrastructure is not trivial  web servers need to be extended with media extractors MF-specific HTTP communication  existing web caches are only effective with HTTP byte ranges specialized media caches need to be developed in the future to cache the other cases  user agents need to be extended with a Media Fragment URI parser and interpreter MF-specific HTTP communication MF-specific visualization 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 26
  27. 27. Media Fragments Proxy Goal: make existing media resources, served by generic HTTP Web servers, available as Media Fragments Biggest problem for content providers willing to support Media Fragments  media extractor (dependent on underlying media formats) Solution: an approach that  works with existing HTTP Web servers (apache, IIS)  works with existing Web caches  works with not so smart user agents 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 27
  28. 28. Media Fragments Proxy http://foo.com/video.ogv#t=11,19 GET /?url=http://foo.com/video.ogv HTTP/1.1 Host: MFProxy.com HTTP Accept: video/* server Range: t:npt=11-19 Accept-Range-Redirect: bytes HTTP/1.1 307 Temporary Redirect Location: http://foo.com/video.ogv -get (only) the header info Accept-Ranges: bytes, t, track Content-Length:a fragment-to-byte mapping -try to find 0 Content-Type: video/ogg -construct the redirect response Content-Range-Mapping: t:npt 10-20/0-50 Range-Redirect: 24000-32000 Vary: Accept-Range-Redirect GET /video.ogv HTTP/1.1 MF Proxy Host: foo.com MF User Accept: video/* Agent Range: bytes=24000-32000http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI> 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 28
  29. 29. Annotating Media Fragments yuma.min.js 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 29
  30. 30. Annotating Media Fragments Benchmarking: Sphinx, HTK, Julius  NER + full text index with the transcription  Interlinking with the Linked Data Cloud to enable semantic search 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 30
  31. 31. 10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 31
  32. 32. http://www.slideshare.net/troncy10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 32
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×