Evaluation of Bandwidth Performance for Interactive Spherical Video Patrice Rondao Alface, Jean-François Macq, Nico Verzij...
Acknowledgment FP7 FascinatE project
AGENDA <ul><li>Introduction </li></ul><ul><li>Interactive Spherical Video Transmission </li></ul><ul><li>Transmission Opti...
Introduction Interactive Spherical Video   <ul><li>Capture devices </li></ul><ul><ul><li>6 to 12 HD views </li></ul></ul><...
Introduction Interactive Spherical Video - Applications <ul><li>Immersive Media </li></ul><ul><ul><li>Cultural heritage… <...
Interactive Spherical Video Transmission State of the Art <ul><li>Compression of the panorama as a video and implicit mapp...
Interactive Spherical Video Transmission <ul><li>Observations: </li></ul><ul><ul><li>Field-of-View (fov) for a given user-...
Interactive Spherical Video Transmission Spherical Video Random Access <ul><li>Tiling on panorama :  </li></ul><ul><ul><li...
Interactive Spherical Video Transmission Tiling <ul><li>Tile compression with adaptive quality </li></ul><ul><ul><li>Tile ...
Transmission Optimization Tile-Based Quality Adaptation <ul><li>Study of four tile-based adaptation techniques with respec...
Transmission Optimization Proposed probability-based approach <ul><li>Problem of optimizing the quality of the tiles in fu...
Transmission Optimization Proposed probability-based approach <ul><li>Relaxing the problem by considering the MSE of each ...
Transmission Optimization Proposed probabilistic Model of the Tile Visibility <ul><li>Optimization of the tile selection u...
Results  (case 1: no delay) Tiling: client viewport quality vs. bandwidth
Transmission Optimization Probabilistic Model of the Tile Visibility <ul><li>Optimization of the tile selection under 2 cl...
Low delay: 40ms Quality is higher close to the estimated viewport High delay: 1280ms Low probability for any viewport posi...
Results  (case 2: user interaction delay) PSNR vs. Delay for a fixed bandwidth 75KBs
Results  (case 2: user interaction delay) Discrepancy at the client side - delay of 1280ms
Conclusions <ul><li>Analysis of  interactive  spherical video delivery techniques.  </li></ul><ul><li>Received quality opt...
http://belllabs.be/internships
Transmission Optimization Experimental Setting <ul><li>Point Grey Ladybug3 (6 HD cameras)  </li></ul><ul><li>Panorama: 409...
Interactive Spherical Video Transmission State of the Art <ul><li>Dodecahedron-based spherical patches optimized sampling ...
Interactive Spherical Video Transmission 2D Random Access <ul><li>Example: Random Access for H.264 </li></ul><ul><ul><li>I...
Results  (case 2: user interaction delay) Temporal analysis
Results  (case 2: user interaction delay) Delay 1280ms Delay 40ms
 
Upcoming SlideShare
Loading in …5
×

Evaluation of bandwidth performance for interactive spherical video

1,938 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,938
On SlideShare
0
From Embeds
0
Number of Embeds
418
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Evaluation of bandwidth performance for interactive spherical video

  1. 1. Evaluation of Bandwidth Performance for Interactive Spherical Video Patrice Rondao Alface, Jean-François Macq, Nico Verzijp Bell Labs, Alcatel-Lucent ICME’s WoMAN’11, Barcelona, July 2011
  2. 2. Acknowledgment FP7 FascinatE project
  3. 3. AGENDA <ul><li>Introduction </li></ul><ul><li>Interactive Spherical Video Transmission </li></ul><ul><li>Transmission Optimization </li></ul><ul><li>Experimental Results </li></ul><ul><li>Conclusions </li></ul>
  4. 4. Introduction Interactive Spherical Video <ul><li>Capture devices </li></ul><ul><ul><li>6 to 12 HD views </li></ul></ul><ul><li>Panorama/Texture mapping on mesh sphere </li></ul>Heymann et al., “Representation, Coding & Interactive Rendering of High-Resolution Panoramic Images and Video using MPEG-4”, in Proc. Panoramic Photogrammetry Workshop (PPW) , Berlin, Germany, Feb. 2005.
  5. 5. Introduction Interactive Spherical Video - Applications <ul><li>Immersive Media </li></ul><ul><ul><li>Cultural heritage… </li></ul></ul><ul><li>Entertainment – live events </li></ul><ul><ul><li>360 º Media Coverage </li></ul></ul><ul><ul><li>Music concerts </li></ul></ul><ul><ul><li>Sports events… </li></ul></ul><ul><li>Video-surveillance & Security </li></ul><ul><li>Geographical Information Systems (GIS) </li></ul><ul><ul><li>Real Estate </li></ul></ul><ul><ul><li>Google Street View… </li></ul></ul>Image source: www.immersivemedia.com
  6. 6. Interactive Spherical Video Transmission State of the Art <ul><li>Compression of the panorama as a video and implicit mapping on a sphere at the client’s renderer side </li></ul><ul><ul><li>Panorama coding with uniform quality </li></ul></ul><ul><ul><ul><li>MPEG-4 Visual (Smolic et al. 2003) </li></ul></ul></ul><ul><ul><ul><li>Flash/PaperVision3D </li></ul></ul></ul>A. Smolic et al.,”Efficient Representation & Coding of Omni-directional Video using MPEG-4”, Proc. WIAMIS 2003, 4th European Workshop on Image Analysis for Multimedia Interactive Services, London, UK, April 9.-11. 2003.
  7. 7. Interactive Spherical Video Transmission <ul><li>Observations: </li></ul><ul><ul><li>Field-of-View (fov) for a given user-selected orientation is limited to 45 - 60 degrees (vertical axis, aspect ratio gives the horizontal fov width) </li></ul></ul><ul><ul><li>More than the half of the bandwidth is wasted on parts of the panorama or views that are not viewed. </li></ul></ul><ul><li>Contribution: </li></ul><ul><ul><li>Leveraging from the 2D case (Mavlankar et al. 2009), namely a tiling approach, we propose to adapt the encoding for Interactive Random Access Viewing in live video transmission/adaptation scenarios </li></ul></ul>A. Mavlankar, B. Girod, Background extraction and long-term memory motion-compensated prediction for spatial-random-access-enabled video coding, Picture Coding Symposium 2009, pp.1-4, 6-8 May 2009
  8. 8. Interactive Spherical Video Transmission Spherical Video Random Access <ul><li>Tiling on panorama : </li></ul><ul><ul><li>the requested ROI does not map as in 2D to an equal number of tiles. </li></ul></ul><ul><ul><li>Visibility mask </li></ul></ul>
  9. 9. Interactive Spherical Video Transmission Tiling <ul><li>Tile compression with adaptive quality </li></ul><ul><ul><li>Tile Intra compression using JPEG2000 (quality constant on a tile) </li></ul></ul><ul><ul><li>Tiles coded independently at different rates (qualities) depending on their estimated visibility </li></ul></ul><ul><ul><li>Fixed bandwidth budget for the panorama </li></ul></ul>Whole panorama Visible tiles only (RoI) Possible impairment (interaction delay)
  10. 10. Transmission Optimization Tile-Based Quality Adaptation <ul><li>Study of four tile-based adaptation techniques with respect to transmission delay </li></ul><ul><li>The delay induces an uncertainty about the actual requested ROI </li></ul><ul><li>For a given bandwidth budget, bitrate is adapted on tiles based on following approaches: </li></ul><ul><li>Motivation: when delay increases, a probabilistic approach should behave better </li></ul><ul><li>Constant rate </li></ul><ul><li>Direct ROI encoding </li></ul><ul><li>ROI with uniform quality background encoding </li></ul><ul><li>ROI with probability-based adaptive quality background </li></ul>
  11. 11. Transmission Optimization Proposed probability-based approach <ul><li>Problem of optimizing the quality of the tiles in function of the user requests is modeled as an integer linear program </li></ul><ul><ul><li>Given an offered bandwidth BW, the objective is to maximize, for each frame, the total utility of the selected tiles, hence minimizing their global MSE. </li></ul></ul>Tile marginal utility Tile at position i, quality q Tile bitrate Tile MSE Tile marginal cost Tile inserted at quality ≥ q Probability of tile visibility ,
  12. 12. Transmission Optimization Proposed probability-based approach <ul><li>Relaxing the problem by considering the MSE of each tile as a convex function of its cost-bitrate ratio, and dropping the integrality constraint, this fractional knapsack can be solved by a greedy heuristic: </li></ul>
  13. 13. Transmission Optimization Proposed probabilistic Model of the Tile Visibility <ul><li>Optimization of the tile selection under 2 classes of test scenarios </li></ul><ul><li>Ideal case: </li></ul><ul><ul><li>No delay </li></ul></ul><ul><ul><li>Probability of visibility is known </li></ul></ul><ul><ul><li>Knapsack optimization can directly use the visibility information </li></ul></ul><ul><li>User interaction delay case </li></ul>
  14. 14. Results (case 1: no delay) Tiling: client viewport quality vs. bandwidth
  15. 15. Transmission Optimization Probabilistic Model of the Tile Visibility <ul><li>Optimization of the tile selection under 2 classes of test scenarios </li></ul><ul><li>Ideal case </li></ul><ul><li>User interaction delay case </li></ul><ul><ul><li>Delay assumed to be constant in the system </li></ul></ul><ul><ul><li>If delay increases , the uncertainty around requested viewport position also increases </li></ul></ul><ul><ul><li>The uncertainty around the visibility mask for the panorama tiles is characterized by a Gaussian with a standard deviation set proportional to the product of the navigation speed and the delay : σ  v . d </li></ul></ul>
  16. 16. Low delay: 40ms Quality is higher close to the estimated viewport High delay: 1280ms Low probability for any viewport position; quality tends to be constant on the whole panorama 15dB better than constant bitrate 7dB better than constant bitrate Results (case 2: user interaction delay) 75kB for 4Kx2K panorama
  17. 17. Results (case 2: user interaction delay) PSNR vs. Delay for a fixed bandwidth 75KBs
  18. 18. Results (case 2: user interaction delay) Discrepancy at the client side - delay of 1280ms
  19. 19. Conclusions <ul><li>Analysis of interactive spherical video delivery techniques. </li></ul><ul><li>Received quality optimization under bw constraints and interaction delay. </li></ul><ul><li>Proposed probability-based tile quality optimization method behaves better than state-of-the-art. </li></ul><ul><li>Future work: enabling this adaptation for different live delivery architectures. </li></ul>
  20. 20. http://belllabs.be/internships
  21. 21. Transmission Optimization Experimental Setting <ul><li>Point Grey Ladybug3 (6 HD cameras) </li></ul><ul><li>Panorama: 4096x2048 </li></ul><ul><li>Viewport: 1280x960 </li></ul><ul><li>FOV: 45 degrees vertically </li></ul><ul><li>Pan-Tilt interactions only </li></ul><ul><li>Static camera setting </li></ul><ul><li>Live Constant bitrate transmission with smooth user interactivity </li></ul><ul><li>Live coding: JPEG2000 intra-picture coding-only for low delay </li></ul><ul><li>Panorama partitioned into JPEG2000 independently coded tiles </li></ul><ul><li>Analysis in terms of Interaction speed v and transmission delay d </li></ul>
  22. 22. Interactive Spherical Video Transmission State of the Art <ul><li>Dodecahedron-based spherical patches optimized sampling for view coding using MPEG-2 (multi-texture approach) </li></ul>Chi-Wing Fu et al., &quot;The Rhombic Dodecahedron Map: An Efficient Scheme for Encoding Panoramic Video,&quot; Multimedia, IEEE Transactions on , vol.11, no.4, pp.634-644, June 2009
  23. 23. Interactive Spherical Video Transmission 2D Random Access <ul><li>Example: Random Access for H.264 </li></ul><ul><ul><li>Interactive Regions of Interest (ROIs) are usually set at the encoder side by applying a regular grid of tiles on the video frames. </li></ul></ul><ul><ul><li>Content-based approaches (e.g. background/foreground) with FMO slices </li></ul></ul>Content-based Arbitrary
  24. 24. Results (case 2: user interaction delay) Temporal analysis
  25. 25. Results (case 2: user interaction delay) Delay 1280ms Delay 40ms

×