Successfully reported this slideshow.

MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices

0

Share

Loading in …3
×
1 of 22
1 of 22

MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices

0

Share

Download to read offline

The advancement of hardware capabilities in recent years made it possible to apply deep neural network (DNN) based approaches on mobile devices. This paper introduces a lightweight super-resolution (SR) network, namely SR-ABR Net, deployed at mobile devices to upgrade low-resolution/low-quality videos and a novel adaptive bitrate (ABR) algorithm, namely WISH-SR, that leverages SR networks at the client to improve the video quality depending on the client's context. WISH-SR takes into account mobile device properties, video characteristics, and user preferences. Experimental results show that the proposed SR-ABR Net can improve the video quality compared to traditional SR approaches while running in real time. Moreover, the proposed WISH-SR can significantly boost the visual quality of the delivered content while reducing both bandwidth consumption and number of stalling events.

The advancement of hardware capabilities in recent years made it possible to apply deep neural network (DNN) based approaches on mobile devices. This paper introduces a lightweight super-resolution (SR) network, namely SR-ABR Net, deployed at mobile devices to upgrade low-resolution/low-quality videos and a novel adaptive bitrate (ABR) algorithm, namely WISH-SR, that leverages SR networks at the client to improve the video quality depending on the client's context. WISH-SR takes into account mobile device properties, video characteristics, and user preferences. Experimental results show that the proposed SR-ABR Net can improve the video quality compared to traditional SR approaches while running in real time. Moreover, the proposed WISH-SR can significantly boost the visual quality of the delivered content while reducing both bandwidth consumption and number of stalling events.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices

  1. 1. All rights reserved. ©2020 All rights reserved. ©2020 A Super-Resolution Based Approach for HTTP Adaptive Streaming for Mobile Devices ACM Mile-High Video 2022 March 03, 2022 Minh Nguyen, Ekrem Çetinkaya, Hermann Hellwagner, Christian Timmerer Christian Doppler Laboratory ATHENA | Alpen-Adria-Universität Klagenfurt | Austria ekrem.cetinkaya@aau.at | athena.itec.aau.at 1
  2. 2. All rights reserved. ©2020 Video Streaming on Mobile Devices 1 “YouTube by the Numbers: Stats, Demographics & Fun Facts”, Omnicore. All rights reserved. ©2020 2 70% of YouTube watch time is from mobile devices 1 70% 30% 2 “Experience Shapes Mobile Customer Loyalty”, Ericsson. 26% of smartphone users encounter video streaming problem every day 2
  3. 3. All rights reserved. ©2020 ML-Benchmark GPU Scores of iPhones 3 ML-Benchmark GPU Scores, Source: https://browser.geekbench.com/ml-benchmarks 1797 1362 858 502 iPhone 13 (2021) iPhone 11 (2019) iPhone 8 (2017) iPhone 6S (2015)
  4. 4. All rights reserved. ©2020 Super-Resolution 4 * Ahn, N., Kang, B., & Sohn, K. A. (2018). Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European conference on computer vision (ECCV) (pp. 252-268) Bilinear CARN* 540p 1080p
  5. 5. All rights reserved. ©2020 5 SR-ABR Net WISH-SR Why? 🔋 Mobile devices are becoming powerful ⏱ Execution time of SR-DNNs is still high What? 🗂 ABR algorithm that considers throughput cost, buffer cost, and quality cost. 🗂 An extension to WISH1 ABR. Trade-off among different factors Why? 💿 Reduce downloaded data while preserving the QoE 🗂 ABR needs to consider when to apply SR What? 🗂 Lightweight SR network that considers the limitations of the mobile environment 🗂 Performance on-par with SoTA SR-DNNs while running on real-time on mobile GPUs Proposed Method 1M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021 IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
  6. 6. SR-ABR All rights reserved. ©2020 6
  7. 7. All rights reserved. ©2020 System Architecture 7 WISH-SR Server Client SR Network Request X2 X3 X4 X2 X3 X4 HR LR HTTP Get Request
  8. 8. All rights reserved. ©2020 SR-ABR Net 8 Convolution ReLU Add Pixel Shuffle Convolution ReLU Add Convolution ReLU Add Convolution ReLU Convolution Clip ReLU LR Frame HR Frame
  9. 9. All rights reserved. ©2020 WISH-SR ABR Algorithm 9 GET High Bitrate Segment More transferred data (higher throughput cost) More download time (higher buffer cost) Higher Quality (lower quality cost)
  10. 10. All rights reserved. ©2020 WISH-SR ABR Algorithm 10 Throughput Cost Buffer Cost Conventional Quality Cost SR-Enabled Quality Cost
  11. 11. All rights reserved. ©2020 WISH-SR ABR Algorithm 11 Throughput Cost Buffer Cost Current bitrate Estimated throughput Download time of current segment Current buffer - low threshold
  12. 12. All rights reserved. ©2020 WISH-SR ABR Algorithm 12 Quality Cost Distortion penalty + Instability penalty Conventional Quality Current bitrate Maximum bitrate SR Quality Improvement in quality level
  13. 13. All rights reserved. ©2020 WISH-SR ABR Algorithm 13 Quality Cost Throughput Cost Buffer Cost WISH-SR ABR Algorithm M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021 IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
  14. 14. Evaluation Setup All rights reserved. ©2020 14
  15. 15. All rights reserved. ©2020 Experimental Setup 15 Testbed 💻 Lenovo Thinkpad P1 (i7 / 16GB) Ubuntu 18.04 📱 Xiaomi Mi 11 (Snapdragon 888) Android 11 - ExoPlayer Dataset - ABR 🗂 HEVC - Segment duration 4s 🗂{100, 145, 900, 2400, 4500} kbps {270p, 360p, 540p, 720p, 1080p} (i) Tears of steel - First 5 mins (ToS1) (Mix 🌍🗂 - 📉 SI 📉 TI) (ii) Tears of steel - Last 5 mins (ToS2) (Mix 🌍🗂 - 📈 SI 📈 TI) (iii) Gameplay - (Generated 🗂 - 📈 SI 📉 TI) (iv) Rally (Natural 🌍 - 📉 SI 📈 TI) 🔗 Linux traffic control tool (tc) 4G Network trace1 Avg. 3787 kbps - Std.dev. 3193 kbps RTT 20ms - Buffer 20s - Low threshold 4s 1D. Raca, J. J. Quinlan, A. H. Zahran, and C. J. Sreenan. “Beyond throughput: a 4G LTE dataset with channel and context metrics”. In Proceedings of the 9th ACM Multimedia Systems Conference, pages 460–465. ACM, 2018. 2T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson. A buffer-based approach to rate adaptation: Evidence from a large video streaming service. In ACM SIGCOMM Computer Communication Review, volume 44, pages 187–198. ACM, 2014. 3C. Wang, A. Rizk, and M. Zink. SQUAD: A spectrum-based quality adaptation for dynamic adaptive streaming over HTTP. In Proceedings of the 7th International Conference on Multimedia Systems, pages 1–12, 2016. 4M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices. In 2021 IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021. BBA-02, ExoPlayer, SQUAD3, WISH4
  16. 16. All rights reserved. ©2020 SR Network Training 16 Dataset 🗂 HEVC - Target Resolution 1080p 270p - X4, 360p - X3, 540p - X2 DIV2K Dataset 1 Frames from around ~ 100 Videos Waterloo 2 - SJTU 3 - Tencent Video Dataset 4 1 Agustsson, Eirikur, and Radu Timofte. "Ntire 2017 challenge on single image super-resolution: Dataset and study." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017. 2 M. Cheon and J.-S. Lee. Subjective and objective quality assessment of compressed 4K UHD videos for immersive experience. IEEE Transactions on Circuits and Systems for Video Technology, 28(7):1467–1480, 2017. 3 L. Song, X. Tang, W. Zhang, X. Yang, and P. Xia. The SJTU 4K video sequence dataset. In 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), pages 34–35, 2013. doi: 10.1109/QoMEX.2013.6603201. 4 X. Xu, S. Liu, and Z. Li. Tencent Video Dataset (TVD): A Video Dataset for Learning-based Visual Data Compression and Analysis. arXiv preprint arXiv:2105.05961, 2021 5 N. Ahn, B. Kang, and K.-A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 252–268, 2018. Training CARN-M5 - SR-ABR Net Train on DIV2K - Finetune on encoded videos Adam optimizer - Learning rate scheduler - MSE Tensorflow-lite Float16 quantization
  17. 17. All rights reserved. ©2020 Evaluation Metrics 17 Average Bitrate # of Stalls and Stall Duration QoE Score - ITU-T P.1203 Extension Mode 0 VMAF VMAF/Bitrate
  18. 18. Results All rights reserved. ©2020 18
  19. 19. All rights reserved. ©2020 SR-DNN Results 19 1 Ekrem Çetinkaya, Minh Nguyen, and Christian Timmerer. "MoViDNN: A Mobile Platform for Evaluating Video Quality Enhancement with Deep Neural Networks." arXiv preprint arXiv:2201.04402 (2022). Execution Speed (FPS) X2 90.93 91.13 82.10 52.83 54.11 42.91 39.00 41.56 24.32 X3 X4 24 30 36 14 9 5 X3 X4 X2 VMAF SR-ABR Net CARN-M Bilinear
  20. 20. All rights reserved. ©2020 SR-ABR Results 20 3098 1818 2670 1748 1738 BBA-0 EP SQUAD WISH WISH-SR Average Bitrate (kbps) 3.54 4.05 3.35 4.06 4.09 BBA-0 EP SQUAD WISH WISH-SR QoE Score (ITU.T P.1203) 90.87 81.75 86.55 81.29 84.91 BBA-0 EP SQUAD WISH WISH-SR VMAF 22 1.85 1 0.3 24 1.8 0 0 BBA-0 EP SQUAD WISH WISH-SR Stall Duration (s) # of Stalls 0.029 0.045 0.032 0.046 0.049 VMAF / Bitrate (1 kbps) BBA-0 EP SQUAD WISH WISH-SR
  21. 21. All rights reserved. ©2020 Conclusion 21 SR-ABR Net WISH-SR Lightweight SR DNN that considers the limitations of the mobile environment Significant improvement (up to 60%) over bilinear interpolation (default in Android) On-par performance with SoTA SR DNNs while running in real time on mobile GPU ABR algorithm that leverages SR networks to improve quality Weighted sum model of throughput cost, buffer cost, and quality cost SR-ABR SR-ABR Net integrated into WISH-SR and deployed on ExoPlayer Significant data reduction (up to 43%) while providing high QoE
  22. 22. All rights reserved. ©2020 Thank you! ekrem.cetinkaya@aau.at minh.nguyen@aau.at @ekremcetinkaya_ @minhkstn linkedin.com/in/ekrcet linkedin.com/in/minhkstn

×