Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enable the next Generation - Interactive video streaming

170 views

Published on

Thèse "Enable the next Generation - Interactive video streaming" par Xavier CORBILLON, lors de la journée Futur & Ruptures du 31 janvier 2019. Une journée scientifique pour présenter l’ensemble des travaux de thèses aboutis portant sur des thématiques prospectives du programme de l’IMT.

Published in: Engineering
  • Be the first to comment

Enable the next Generation - Interactive video streaming

  1. 1. Institut Mines-Télécom IMT Atlantique Bretagne-Pays de la Loire École Mines-Télécom Enable the Next Generation Interactive Video Streaming Xavier Corbillon Thesis defended on October 30, 2018 Rapporteurs: Vincent CHARVILLAT Professor University of Toulouse France Miska M. HANNUKSELA Bell Labs Fellow Nokia Technologies Finland Jury members: President: Patrick LE CALLET Professor University of Nantes France Examiners: Vincent CHARVILLAT Professor University of Toulouse France Miska M. HANNUKSELA Bell Labs Fellow Nokia Technologies Finland Laura TONI Lecturer University College London UK Director: Gwendal SIMON Professor IMT Atlantique, IRISA France Rapporteurs: Vincent CHARVILLAT Professo Miska M. HANNUKSELA Bell Labs Jury members: President: Patrick LE CALLET Professo Examiners: Vincent CHARVILLAT Professo Miska M. HANNUKSELA Bell Labs Laura TONI Lecturer Director: Gwendal SIMON Professo
  2. 2. General Context: Video Streaming 2/12 Paradigm shift: an area of on demand personalized delivery Content Provider Internet Pull 1 2 0 34 2 1 2 0 IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  3. 3. General Context: Video Streaming 2/12 Paradigm shift: an area of on demand personalized delivery Content Provider Internet Pull 1 2 0 34 2 1 2 0 In 2016: 37 TB s−1 (67 % is video) IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  4. 4. General Context: Video Streaming 2/12 Paradigm shift: an area of on demand personalized delivery Users request high quality Content Provider Internet Pull 1 2 0 34 2 1 2 0 In 2021: 105 TB s−1 (80 % is video) IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  5. 5. How to Deliver Traditional Videos Today? 3/12 Main technology: HTTP Adaptive Streaming • MPEG-DASH, Apple HLS, Adobe HDS, MSS IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  6. 6. How to Deliver Traditional Videos Today? 3/12 Main technology: HTTP Adaptive Streaming • MPEG-DASH, Apple HLS, Adobe HDS, MSS Concept: high s1 med low s2 s3 tt high s1 med low s2 s3 server / CDN cache Bandwidth PredictionSegment Generation bw t client connect mpd HTTP requests s1:med s2:high s3:low origin server sent once Original Video user displayed IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  7. 7. How to Deliver Traditional Videos Today? 3/12 Main technology: HTTP Adaptive Streaming • MPEG-DASH, Apple HLS, Adobe HDS, MSS Concept: high s1 med low s2 s3 tt high s1 med low s2 s3 server / CDN cache Bandwidth PredictionSegment Generation bw t client connect mpd HTTP requests s1:med s2:high s3:low origin server sent once Original Video user displayed IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  8. 8. How to Deliver Traditional Videos Today? 3/12 Main technology: HTTP Adaptive Streaming • MPEG-DASH, Apple HLS, Adobe HDS, MSS Concept: high s1 med low s2 s3 tt high s1 med low s2 s3 server / CDN cache Bandwidth PredictionSegment Generation bw t client connect mpd HTTP requests s1:med s2:high s3:low origin server sent once Original Video user displayed IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  9. 9. How to Deliver Traditional Videos Today? 3/12 Main technology: HTTP Adaptive Streaming • MPEG-DASH, Apple HLS, Adobe HDS, MSS Concept: high s1 med low s2 s3 tt high s1 med low s2 s3 server / CDN cache Bandwidth PredictionSegment Generation bw t client connect mpd HTTP requests s1:med s2:high s3:low origin server sent once Original Video user displayed mpd = Manifest file IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  10. 10. How to Deliver Traditional Videos Today? 3/12 Main technology: HTTP Adaptive Streaming • MPEG-DASH, Apple HLS, Adobe HDS, MSS Concept: high s1 med low s2 s3 tt high s1 med low s2 s3 server / CDN cache Bandwidth PredictionSegment Generation bw t med high low client connect mpd HTTP requests s1:med s2:high s3:low origin server sent once Original Video user displayed mpd = Manifest file IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  11. 11. How to Deliver Traditional Videos Today? 3/12 Main technology: HTTP Adaptive Streaming • MPEG-DASH, Apple HLS, Adobe HDS, MSS Concept: high s1 med low s2 s3 tt high s1 med low s2 s3 server / CDN cache Bandwidth PredictionSegment Generation bw t med high low client connect mpd HTTP requests s1:med s2:high s3:low origin server sent once Original Video user displayed DASH design is challenged by new interactive contents mpd = Manifest file IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  12. 12. Omnidirectional Videos 4/12 O #» k #» #»ı IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  13. 13. Omnidirectional Videos 4/12 O #» k #» #»ı IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  14. 14. Omnidirectional Videos 4/12 O #» k #» #»ı IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  15. 15. Omnidirectional Videos 4/12 O #» k #» #»ı#»u #»v Viewport IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  16. 16. Omnidirectional Videos 4/12 O #» k #» #»ı #»u #»v Viewport IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  17. 17. Omnidirectional Videos Streaming 5/12 Thesis target: Stream on demand 360° videos to users’ devices O #» k #» #»ı Internet? ? Storage and Content Representation IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  18. 18. High Feeling of Immersion 6/12 Constraints: High viewport resolution, low motion-to-photon delay Viewport: • 4K resolution in the viewport → 12K resolution for the full video • at least 90 frames per second IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  19. 19. High Feeling of Immersion 6/12 Constraints: High viewport resolution, low motion-to-photon delay Viewport: • 4K resolution in the viewport → 12K resolution for the full video • at least 90 frames per second Latency: Delay between action and visible result • head moves without visible change → motion sickness • ⇒ maximum 10 ms IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  20. 20. High Feeling of Immersion 6/12 Constraints: High viewport resolution, low motion-to-photon delay Viewport: • 4K resolution in the viewport → 12K resolution for the full video • at least 90 frames per second Latency: Delay between action and visible result • head moves without visible change → motion sickness • ⇒ maximum 10 ms 2015 state of the art: • send full 360-video requires >100 Mbit s−1 • send viewport only is not scalable and the latency is not low enough IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  21. 21. Quality Emphasized Region 7/12 Traditional videos are encoded with uniform quality Encoding based: Uniform Quality IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  22. 22. Quality Emphasized Region 7/12 360° videos can be prepared with non uniform spatial quality Encoding based: Quality Emphasized Region good degradation IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  23. 23. Quality Emphasized Region 7/12 360° videos can be prepared with non uniform spatial quality Encoding based, Projection based, Region wise packing Quality Emphasized Region good degradation IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  24. 24. Viewport Adaptive Streaming 8/12 Viewport adaptive streaming high s1 low QER1 high low QER2 high low QER3 s2 s3 t server bw t low high low client connect mpd s1:QER2 lo s2:QER3 hi s3:QER1 lo QER = Quality Emphasized Region IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  25. 25. Viewport Adaptive Streaming 8/12 Viewport adaptive streaming high s1 low QER1 high low QER2 high low QER3 s2 s3 t server bw t low high low client connect mpd s1:QER2 lo s2:QER3 hi s3:QER1 lo QER = Quality Emphasized Region Publication: X. Corbillon, A. Devlic, G. Simon, and J. Chakareski, “Viewport-adaptive navigable 360-degree video delivery”, in IEEE International Conference on Communications (ICC), 2017, pp. 1–7 Best Paper Award IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  26. 26. “Quality” VS Number of QER 9/12 Optimal QER generation for omniscient clients 1 2 3 4 5 6 7 8 9 10 11 0.5 1 1.5 2 surface bit-rate for a uniform quality Number of QER versions offered Visible surfacebit-rate[Mbits−1 m−2 ] roller-coaster diving IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  27. 27. “Quality” VS Number of QER 9/12 Optimal QER generation for omniscient clients 1 2 3 4 5 6 7 8 9 10 11 0.5 1 1.5 2 surface bit-rate for a uniform quality Number of QER versions offered Visible surfacebit-rate[Mbits−1 m−2 ] roller-coaster diving IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  28. 28. “Quality” VS Number of QER 9/12 Optimal QER generation for omniscient clients 1 2 3 4 5 6 7 8 9 10 11 0.5 1 1.5 2 surface bit-rate for a uniform quality Number of QER versions offered Visible surfacebit-rate[Mbits−1 m−2 ] roller-coaster diving IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  29. 29. Takeaway 10/12 Thesis Target: Stream on demand 360° videos to users’ devices Contributions: Recorded one of the first Head Movement dataset Propose a Viewport Adaptive Streaming architecture based on QERs Propose a model to generate optimal set of QERs Extend the evaluation of Facebook’s offset projections Extend Viewport Adaptive Streaming for multi-viewpoint 360° videos IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  30. 30. Future Work 11/12 How to integrate Quality Emphasized Regions (QERs) in live streaming? • Automatic saliency estimation • Distributed computation to feed the origin servers with relevant features IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  31. 31. Future Work 11/12 How to integrate Quality Emphasized Regions (QERs) in live streaming? • Automatic saliency estimation • Distributed computation to feed the origin servers with relevant features We studied static QERs; Are dynamic QERs more efficient? IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  32. 32. Future Work 11/12 How to integrate Quality Emphasized Regions (QERs) in live streaming? • Automatic saliency estimation • Distributed computation to feed the origin servers with relevant features We studied static QERs; Are dynamic QERs more efficient? Caching optimization for / with QERs IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  33. 33. Future Work 11/12 How to integrate Quality Emphasized Regions (QERs) in live streaming? • Automatic saliency estimation • Distributed computation to feed the origin servers with relevant features We studied static QERs; Are dynamic QERs more efficient? Caching optimization for / with QERs Light field / Point Cloud compression for real 6DoF videos IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  34. 34. Publications 12/12 International Conferences: 2018 • H. Hristova, X. Corbillon, G. Simon, V. Swaminathan, and A. Devlic, “Heterogeneous spatial quality for omnidirectional video”, in Proceeding of IEEE International Workshop on Multimedia Signal Processing (MMSP), 2018 Best Paper Award • X. Corbillon, F. De Simone, G. Simon, and P. Frossard, “Dynamic adaptive streaming for multi-viewpoint omnidirectional videos”, in Proceedings of the 9th ACM Multimedia Systems (MMSys), 2018, pp. 237–249 Best Paper Award • J. Chakareski, R. Aksu, X. Corbillon, and G. Simon, “Viewport-driven rate-distortion optimized 360° video streaming”, in Proceedings of IEEE International Conference on Communications (ICC), 2018, pp. 1–7 2017 • X. Corbillon, A. Devlic, G. Simon, and J. Chakareski, “Optimal set of 360-degree videos for viewport-adaptive streaming”, in Proceedings of ACM on Multimedia Conference, MM, 2017, pp. 943–951 • X. Corbillon, F. De Simone, and G. Simon, “360-degree video head movement dataset”, in Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys’17), 2017, pp. 199–204 • X. Corbillon, A. Devlic, G. Simon, and J. Chakareski, “Viewport-adaptive navigable 360-degree video delivery”, in IEEE International Conference on Communications (ICC), 2017, pp. 1–7 Best Paper Award 2016 • X. Corbillon, F. Boyrivent, G. A. D. Williencourt, G. Simon, G. Texier, and J. Chakareski, “Efficient lightweight video packet filtering for large-scale video data delivery”, in IEEE International Conference on Multimedia & Expo Workshops, ICME Workshops, 2016, pp. 1–6 • X. Corbillon, R. Aparicio-Pardo, N. Kuhn, G. Texier, and G. Simon, “Cross-layer scheduler for video streaming over MPTCP”, in Proceedings of the 7th International Conference on Multimedia Systems (MMSys’16), 2016, 7:1–7:12 Journals: 2018 • M. Ben Yahia, Y. Le Louedec, G. Simon, L. Nuaymi, and X. Corbillon, “HTTP/2-based frame discarding for low-latency adaptive video streaming”, IEEE Transactions on Multimedia Computing Communications and Applications, 2018, Accepted for publication IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  35. 35. Submitted Work 12/12 Journals: 2018 • X. Corbillon, H. Hristova, A. Devlic, and G. Simon, “Survey on omnidirectional videos streaming”, IEEE Communications Surveys & Tutorials, 2018, in submission • J. Liu, G. Simon, X. Corbillon, J. Chakareski, and Q. Yang, “Mobile edge caching for viewport-adaptive 360-degree video streaming”, IEEE Transactions on Circuits and Systems for Video Technology, 2018, in submission IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  36. 36. Collaborations 12/12 Francesca De Simone & Pascal Frossard (visiting PhD) Jacob Chakareski Alisa Devlic Vishwanathan Swaminathan Jiayi Liu Hristina Hristova & Mariem Ben Yahia IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  37. 37. Questions 12/12 IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming
  38. 38. Takeaway 12/12 Thesis Target: Stream on demand 360° videos to users’ devices Contributions: Recorded one of the first Head Movement dataset Propose a Viewport Adaptive Streaming architecture based on QERs Propose a model to generate optimal set of QERs Extend the evaluation of Facebook’s offset projections Extend Viewport Adaptive Streaming for multi-viewpoint 360° videos IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

×