SlideShare a Scribd company logo
1 of 22
All rights reserved. ©2020
All rights reserved. ©2020
A Super-Resolution Based Approach for HTTP
Adaptive Streaming for Mobile Devices
ACM Mile-High Video 2022
March 03, 2022
Minh Nguyen, Ekrem Çetinkaya, Hermann Hellwagner, Christian Timmerer
Christian Doppler Laboratory ATHENA | Alpen-Adria-Universität Klagenfurt | Austria
ekrem.cetinkaya@aau.at | athena.itec.aau.at
1
All rights reserved. ©2020
Video Streaming on Mobile Devices
1 “YouTube by the Numbers: Stats, Demographics & Fun Facts”, Omnicore.
All rights reserved. ©2020
2
70% of YouTube watch time is
from mobile devices 1
70%
30%
2 “Experience Shapes Mobile Customer Loyalty”, Ericsson.
26% of smartphone users encounter
video streaming problem every day 2
All rights reserved. ©2020
ML-Benchmark GPU Scores of iPhones
3
ML-Benchmark GPU Scores, Source: https://browser.geekbench.com/ml-benchmarks
1797
1362
858
502
iPhone 13 (2021)
iPhone 11 (2019)
iPhone 8 (2017)
iPhone 6S (2015)
All rights reserved. ©2020
Super-Resolution
4
* Ahn, N., Kang, B., & Sohn, K. A. (2018). Fast, accurate, and lightweight super-resolution with cascading residual network.
In Proceedings of the European conference on computer vision (ECCV) (pp. 252-268)
Bilinear
CARN*
540p
1080p
All rights reserved. ©2020
5
SR-ABR Net WISH-SR
Why?
🔋 Mobile devices are becoming powerful
⏱ Execution time of SR-DNNs is still high
What?
🗂 ABR algorithm that considers throughput
cost, buffer cost, and quality cost.
🗂 An extension to WISH1 ABR. Trade-off
among different factors
Why?
💿 Reduce downloaded data while preserving
the QoE
🗂 ABR needs to consider when to apply SR
What?
🗂 Lightweight SR network that considers the
limitations of the mobile environment
🗂 Performance on-par with SoTA SR-DNNs
while running on real-time on mobile GPUs
Proposed Method
1M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021
IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
SR-ABR
All rights reserved. ©2020
6
All rights reserved. ©2020
System Architecture
7
WISH-SR
Server
Client
SR Network Request X2 X3 X4
X2 X3 X4
HR LR
HTTP Get Request
All rights reserved. ©2020
SR-ABR Net
8
Convolution
ReLU
Add
Pixel
Shuffle
Convolution
ReLU
Add
Convolution
ReLU
Add
Convolution
ReLU
Convolution
Clip
ReLU
LR Frame HR Frame
All rights reserved. ©2020
WISH-SR ABR Algorithm
9
GET High Bitrate Segment
More transferred data
(higher throughput cost)
More download time
(higher buffer cost)
Higher Quality
(lower quality cost)
All rights reserved. ©2020
WISH-SR ABR Algorithm
10
Throughput
Cost
Buffer
Cost
Conventional
Quality Cost
SR-Enabled
Quality Cost
All rights reserved. ©2020
WISH-SR ABR Algorithm
11
Throughput Cost
Buffer Cost
Current bitrate
Estimated throughput
Download time of current segment
Current buffer - low threshold
All rights reserved. ©2020
WISH-SR ABR Algorithm
12
Quality Cost Distortion penalty + Instability penalty
Conventional
Quality
Current bitrate
Maximum bitrate
SR
Quality
Improvement in quality level
All rights reserved. ©2020
WISH-SR ABR Algorithm
13
Quality Cost
Throughput Cost Buffer Cost
WISH-SR ABR Algorithm
M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021 IEEE
23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
Evaluation Setup
All rights reserved. ©2020
14
All rights reserved. ©2020
Experimental Setup
15
Testbed
💻 Lenovo Thinkpad P1 (i7 / 16GB)
Ubuntu 18.04
📱 Xiaomi Mi 11 (Snapdragon 888)
Android 11 - ExoPlayer
Dataset - ABR
🗂 HEVC - Segment duration 4s
🗂{100, 145, 900, 2400, 4500} kbps
{270p, 360p, 540p, 720p, 1080p}
(i) Tears of steel - First 5 mins (ToS1) (Mix 🌍🗂 - 📉 SI 📉 TI)
(ii) Tears of steel - Last 5 mins (ToS2) (Mix 🌍🗂 - 📈 SI 📈 TI)
(iii) Gameplay - (Generated 🗂 - 📈 SI 📉 TI)
(iv) Rally (Natural 🌍 - 📉 SI 📈 TI)
🔗 Linux traffic control tool (tc)
4G Network trace1
Avg. 3787 kbps - Std.dev. 3193 kbps
RTT 20ms - Buffer 20s - Low threshold 4s
1D. Raca, J. J. Quinlan, A. H. Zahran, and C. J. Sreenan. “Beyond throughput: a 4G LTE dataset with channel and context metrics”. In Proceedings of the 9th ACM
Multimedia Systems Conference, pages 460–465. ACM, 2018.
2T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson. A buffer-based approach to rate adaptation: Evidence from a large video streaming service. In ACM
SIGCOMM Computer Communication Review, volume 44, pages 187–198. ACM, 2014.
3C. Wang, A. Rizk, and M. Zink. SQUAD: A spectrum-based quality adaptation for dynamic adaptive streaming over HTTP. In Proceedings of the 7th International
Conference on Multimedia Systems, pages 1–12, 2016.
4M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices. In 2021 IEEE 23rd Int’l.
Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
BBA-02, ExoPlayer, SQUAD3, WISH4
All rights reserved. ©2020
SR Network Training
16
Dataset
🗂 HEVC - Target Resolution 1080p
270p - X4, 360p - X3, 540p - X2
DIV2K Dataset 1
Frames from around ~ 100 Videos
Waterloo 2 - SJTU 3 - Tencent Video Dataset 4
1 Agustsson, Eirikur, and Radu Timofte. "Ntire 2017 challenge on single image super-resolution: Dataset and study." Proceedings of the IEEE conference on computer
vision and pattern recognition workshops. 2017.
2 M. Cheon and J.-S. Lee. Subjective and objective quality assessment of compressed 4K UHD videos for immersive experience. IEEE Transactions on Circuits and
Systems for Video Technology, 28(7):1467–1480, 2017.
3 L. Song, X. Tang, W. Zhang, X. Yang, and P. Xia. The SJTU 4K video sequence dataset. In 2013 Fifth International Workshop on Quality of Multimedia Experience
(QoMEX), pages 34–35, 2013. doi: 10.1109/QoMEX.2013.6603201.
4 X. Xu, S. Liu, and Z. Li. Tencent Video Dataset (TVD): A Video Dataset for Learning-based Visual Data Compression and Analysis. arXiv preprint arXiv:2105.05961, 2021
5 N. Ahn, B. Kang, and K.-A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on
Computer Vision (ECCV), pages 252–268, 2018.
Training
CARN-M5 - SR-ABR Net
Train on DIV2K - Finetune on encoded videos
Adam optimizer - Learning rate scheduler - MSE
Tensorflow-lite
Float16 quantization
All rights reserved. ©2020
Evaluation Metrics
17
Average Bitrate
# of Stalls and Stall Duration
QoE Score - ITU-T P.1203 Extension Mode 0
VMAF
VMAF/Bitrate
Results
All rights reserved. ©2020
18
All rights reserved. ©2020
SR-DNN Results
19
1 Ekrem Çetinkaya, Minh Nguyen, and Christian Timmerer. "MoViDNN: A Mobile Platform for Evaluating Video Quality Enhancement with Deep Neural Networks." arXiv
preprint arXiv:2201.04402 (2022).
Execution Speed (FPS)
X2
90.93 91.13
82.10
52.83 54.11
42.91
39.00
41.56
24.32
X3 X4
24
30
36
14
9
5
X3 X4
X2
VMAF
SR-ABR Net CARN-M Bilinear
All rights reserved. ©2020
SR-ABR Results
20
3098
1818
2670
1748 1738
BBA-0 EP SQUAD WISH WISH-SR
Average Bitrate (kbps)
3.54
4.05
3.35
4.06
4.09
BBA-0 EP SQUAD WISH WISH-SR
QoE Score (ITU.T P.1203)
90.87
81.75
86.55
81.29
84.91
BBA-0 EP SQUAD WISH WISH-SR
VMAF
22
1.85
1
0.3
24
1.8
0 0
BBA-0 EP SQUAD WISH WISH-SR
Stall Duration (s)
# of Stalls
0.029
0.045
0.032
0.046
0.049
VMAF / Bitrate (1 kbps)
BBA-0 EP SQUAD WISH WISH-SR
All rights reserved. ©2020
Conclusion
21
SR-ABR Net
WISH-SR
Lightweight SR DNN that considers the limitations of the mobile environment
Significant improvement (up to 60%) over bilinear interpolation (default in Android)
On-par performance with SoTA SR DNNs while running in real time on mobile GPU
ABR algorithm that leverages SR networks to improve quality
Weighted sum model of throughput cost, buffer cost, and quality cost
SR-ABR
SR-ABR Net integrated into WISH-SR and deployed on ExoPlayer
Significant data reduction (up to 43%) while providing high QoE
All rights reserved. ©2020
Thank you!
ekrem.cetinkaya@aau.at
minh.nguyen@aau.at
@ekremcetinkaya_
@minhkstn
linkedin.com/in/ekrcet
linkedin.com/in/minhkstn

More Related Content

What's hot

Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...
Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...
Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...Alpen-Adria-Universität
 
Objective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming
Objective and Subjective QoE Evaluation for Adaptive Point Cloud StreamingObjective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming
Objective and Subjective QoE Evaluation for Adaptive Point Cloud StreamingAlpen-Adria-Universität
 
MiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive Streaming
MiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive StreamingMiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive Streaming
MiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive StreamingAlpen-Adria-Universität
 
CAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsCAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsAlpen-Adria-Universität
 
Content Generation and Practical Applications for Dynamic Adaptive Streaming ...
Content Generation and Practical Applications for Dynamic Adaptive Streaming ...Content Generation and Practical Applications for Dynamic Adaptive Streaming ...
Content Generation and Practical Applications for Dynamic Adaptive Streaming ...Förderverein Technische Fakultät
 
Press Release of 131st WG11 (MPEG) Meeting
Press Release of 131st WG11 (MPEG) MeetingPress Release of 131st WG11 (MPEG) Meeting
Press Release of 131st WG11 (MPEG) MeetingAlpen-Adria-Universität
 
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...Alpen-Adria-Universität
 
Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...
Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...
Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...Alpen-Adria-Universität
 
Overview of Selected Current MPEG Activities
Overview of Selected Current MPEG ActivitiesOverview of Selected Current MPEG Activities
Overview of Selected Current MPEG ActivitiesAlpen-Adria-Universität
 
LwTE-Live: Light-weight Transcoding at the Edge for Live Streaming
LwTE-Live: Light-weight Transcoding at the Edge for Live StreamingLwTE-Live: Light-weight Transcoding at the Edge for Live Streaming
LwTE-Live: Light-weight Transcoding at the Edge for Live StreamingAlpen-Adria-Universität
 
Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...
Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...
Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...mgrafl
 
On the Impact of Viewing Distance on Perceived Video Quality
On the Impact of Viewing Distance on Perceived Video QualityOn the Impact of Viewing Distance on Perceived Video Quality
On the Impact of Viewing Distance on Perceived Video QualityAlpen-Adria-Universität
 
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming SystemsAutomated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming SystemsAlpen-Adria-Universität
 
Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression
Towards 6DoF HTTP Adaptive Streaming Through Point Cloud CompressionTowards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression
Towards 6DoF HTTP Adaptive Streaming Through Point Cloud CompressionAlpen-Adria-Universität
 
MPEG-DASH Reference Software and Conformance
MPEG-DASH Reference Software and ConformanceMPEG-DASH Reference Software and Conformance
MPEG-DASH Reference Software and ConformanceAlpen-Adria-Universität
 
Trends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding StandardizationTrends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding StandardizationMathias Wien
 

What's hot (19)

Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...
Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...
Quality Optimization of Live Streaming Services over HTTP with Reinforcement ...
 
Objective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming
Objective and Subjective QoE Evaluation for Adaptive Point Cloud StreamingObjective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming
Objective and Subjective QoE Evaluation for Adaptive Point Cloud Streaming
 
MiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive Streaming
MiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive StreamingMiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive Streaming
MiPSO: Multi-Period Per-Scene Optimization For HTTP Adaptive Streaming
 
CAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsCAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR Systems
 
What will 5G bring to the future of video?
What will 5G bring to the future of video?What will 5G bring to the future of video?
What will 5G bring to the future of video?
 
Content Generation and Practical Applications for Dynamic Adaptive Streaming ...
Content Generation and Practical Applications for Dynamic Adaptive Streaming ...Content Generation and Practical Applications for Dynamic Adaptive Streaming ...
Content Generation and Practical Applications for Dynamic Adaptive Streaming ...
 
Press Release of 131st WG11 (MPEG) Meeting
Press Release of 131st WG11 (MPEG) MeetingPress Release of 131st WG11 (MPEG) Meeting
Press Release of 131st WG11 (MPEG) Meeting
 
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
 
Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...
Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...
Relevance-Based Compression of Cataract Surgery Videos Using Convolutional Ne...
 
Overview of Selected Current MPEG Activities
Overview of Selected Current MPEG ActivitiesOverview of Selected Current MPEG Activities
Overview of Selected Current MPEG Activities
 
LwTE-Live: Light-weight Transcoding at the Edge for Live Streaming
LwTE-Live: Light-weight Transcoding at the Edge for Live StreamingLwTE-Live: Light-weight Transcoding at the Edge for Live Streaming
LwTE-Live: Light-weight Transcoding at the Edge for Live Streaming
 
AVSTP2P: Welcome Message from the Chairs
AVSTP2P: Welcome Message from the ChairsAVSTP2P: Welcome Message from the Chairs
AVSTP2P: Welcome Message from the Chairs
 
Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...
Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...
Scalable Video Coding Guidelines and Performance Evaluations for Adaptive Me...
 
On the Impact of Viewing Distance on Perceived Video Quality
On the Impact of Viewing Distance on Perceived Video QualityOn the Impact of Viewing Distance on Perceived Video Quality
On the Impact of Viewing Distance on Perceived Video Quality
 
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming SystemsAutomated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
 
Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression
Towards 6DoF HTTP Adaptive Streaming Through Point Cloud CompressionTowards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression
Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression
 
ITEC DASH
ITEC DASHITEC DASH
ITEC DASH
 
MPEG-DASH Reference Software and Conformance
MPEG-DASH Reference Software and ConformanceMPEG-DASH Reference Software and Conformance
MPEG-DASH Reference Software and Conformance
 
Trends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding StandardizationTrends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding Standardization
 

Similar to MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices

Video Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive StreamingVideo Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive StreamingAlpen-Adria-Universität
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfVignesh V Menon
 
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive StreamingQoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive StreamingDanieleLorenzi6
 
MMSys'21 - Multi-access edge computing for adaptive bitrate video streaming
MMSys'21 - Multi-access edge computing for adaptive bitrate video streamingMMSys'21 - Multi-access edge computing for adaptive bitrate video streaming
MMSys'21 - Multi-access edge computing for adaptive bitrate video streamingJesus Aguilar
 
HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)Alpen-Adria-Universität
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningEkrem Çetinkaya
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningAlpen-Adria-Universität
 
HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?Alpen-Adria-Universität
 
Policy-driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-driven Dynamic HTTP Adaptive Streaming Player EnvironmentMinh Nguyen
 
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video StreamingCollaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video StreamingAlpen-Adria-Universität
 
USuurey_Presentation__CollaborativeHASSystems.pdf
USuurey_Presentation__CollaborativeHASSystems.pdfUSuurey_Presentation__CollaborativeHASSystems.pdf
USuurey_Presentation__CollaborativeHASSystems.pdfReza Farahani
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfVignesh V Menon
 
Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingAlpen-Adria-Universität
 
Optimizing User QoE through Overlay Routing, Bandwidth ...
Optimizing User QoE through Overlay Routing, Bandwidth ...Optimizing User QoE through Overlay Routing, Bandwidth ...
Optimizing User QoE through Overlay Routing, Bandwidth ...Videoguy
 
Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingVignesh V Menon
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfVignesh V Menon
 
Virtual Reality in 5G Networks
Virtual Reality in 5G NetworksVirtual Reality in 5G Networks
Virtual Reality in 5G NetworksGwendal Simon
 
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and SolutionsHow to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and SolutionsAlpen-Adria-Universität
 
VCIP_MCBE_presentation.pdf
VCIP_MCBE_presentation.pdfVCIP_MCBE_presentation.pdf
VCIP_MCBE_presentation.pdfVignesh V Menon
 

Similar to MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices (20)

Video Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive StreamingVideo Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive Streaming
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
 
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive StreamingQoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
 
MMSys'21 - Multi-access edge computing for adaptive bitrate video streaming
MMSys'21 - Multi-access edge computing for adaptive bitrate video streamingMMSys'21 - Multi-access edge computing for adaptive bitrate video streaming
MMSys'21 - Multi-access edge computing for adaptive bitrate video streaming
 
HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?
 
Policy-driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-driven Dynamic HTTP Adaptive Streaming Player Environment
 
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video StreamingCollaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
 
USuurey_Presentation__CollaborativeHASSystems.pdf
USuurey_Presentation__CollaborativeHASSystems.pdfUSuurey_Presentation__CollaborativeHASSystems.pdf
USuurey_Presentation__CollaborativeHASSystems.pdf
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
 
Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive Streaming
 
Optimizing User QoE through Overlay Routing, Bandwidth ...
Optimizing User QoE through Overlay Routing, Bandwidth ...Optimizing User QoE through Overlay Routing, Bandwidth ...
Optimizing User QoE through Overlay Routing, Bandwidth ...
 
Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC Streaming
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
 
PEMWN'21 - ANGELA
PEMWN'21 - ANGELAPEMWN'21 - ANGELA
PEMWN'21 - ANGELA
 
Virtual Reality in 5G Networks
Virtual Reality in 5G NetworksVirtual Reality in 5G Networks
Virtual Reality in 5G Networks
 
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and SolutionsHow to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
 
VCIP_MCBE_presentation.pdf
VCIP_MCBE_presentation.pdfVCIP_MCBE_presentation.pdf
VCIP_MCBE_presentation.pdf
 

More from Minh Nguyen

Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentMinh Nguyen
 
CADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive Streaming
CADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive StreamingCADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive Streaming
CADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive StreamingMinh Nguyen
 
CAdViSE or how to find the sweet spots of ABR systems
CAdViSE or how to find the sweet spots of ABR systemsCAdViSE or how to find the sweet spots of ABR systems
CAdViSE or how to find the sweet spots of ABR systemsMinh Nguyen
 
Video streaming using light-weight transcoding and in-network intelligence
Video streaming using light-weight transcoding and in-network intelligenceVideo streaming using light-weight transcoding and in-network intelligence
Video streaming using light-weight transcoding and in-network intelligenceMinh Nguyen
 
Efficient bitrate ladder construction for live video streaming
Efficient bitrate ladder construction for live video streamingEfficient bitrate ladder construction for live video streaming
Efficient bitrate ladder construction for live video streamingMinh Nguyen
 
RICHTER: hybrid P2P-CDN architecture for low latency live video streaming
RICHTER: hybrid P2P-CDN architecture for low latency live video streamingRICHTER: hybrid P2P-CDN architecture for low latency live video streaming
RICHTER: hybrid P2P-CDN architecture for low latency live video streamingMinh Nguyen
 
EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...
EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...
EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...Minh Nguyen
 
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...Minh Nguyen
 

More from Minh Nguyen (8)

Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
 
CADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive Streaming
CADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive StreamingCADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive Streaming
CADLAD: Device-aware Bitrate Ladder Construction for HTTP Adaptive Streaming
 
CAdViSE or how to find the sweet spots of ABR systems
CAdViSE or how to find the sweet spots of ABR systemsCAdViSE or how to find the sweet spots of ABR systems
CAdViSE or how to find the sweet spots of ABR systems
 
Video streaming using light-weight transcoding and in-network intelligence
Video streaming using light-weight transcoding and in-network intelligenceVideo streaming using light-weight transcoding and in-network intelligence
Video streaming using light-weight transcoding and in-network intelligence
 
Efficient bitrate ladder construction for live video streaming
Efficient bitrate ladder construction for live video streamingEfficient bitrate ladder construction for live video streaming
Efficient bitrate ladder construction for live video streaming
 
RICHTER: hybrid P2P-CDN architecture for low latency live video streaming
RICHTER: hybrid P2P-CDN architecture for low latency live video streamingRICHTER: hybrid P2P-CDN architecture for low latency live video streaming
RICHTER: hybrid P2P-CDN architecture for low latency live video streaming
 
EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...
EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...
EPIQ'21: Days of Future Past: An Optimization-based Adaptive Bitrate Algorith...
 
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...
 

Recently uploaded

Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvLewisJB
 
Solving The Right Triangles PowerPoint 2.ppt
Solving The Right Triangles PowerPoint 2.pptSolving The Right Triangles PowerPoint 2.ppt
Solving The Right Triangles PowerPoint 2.pptJasonTagapanGulla
 
lifi-technology with integration of IOT.pptx
lifi-technology with integration of IOT.pptxlifi-technology with integration of IOT.pptx
lifi-technology with integration of IOT.pptxsomshekarkn64
 
Arduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptArduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptSAURABHKUMAR892774
 
Call Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile serviceCall Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile servicerehmti665
 
8251 universal synchronous asynchronous receiver transmitter
8251 universal synchronous asynchronous receiver transmitter8251 universal synchronous asynchronous receiver transmitter
8251 universal synchronous asynchronous receiver transmitterShivangiSharma879191
 
Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024hassan khalil
 
Electronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdfElectronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdfme23b1001
 
complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...asadnawaz62
 
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsyncWhy does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsyncssuser2ae721
 
computer application and construction management
computer application and construction managementcomputer application and construction management
computer application and construction managementMariconPadriquez1
 
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort serviceGurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort servicejennyeacort
 
Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.eptoze12
 
Risk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfRisk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfROCENODodongVILLACER
 
welding defects observed during the welding
welding defects observed during the weldingwelding defects observed during the welding
welding defects observed during the weldingMuhammadUzairLiaqat
 
Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...VICTOR MAESTRE RAMIREZ
 
Introduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECHIntroduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECHC Sai Kiran
 
Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)
Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)
Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)dollysharma2066
 

Recently uploaded (20)

Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvv
 
Solving The Right Triangles PowerPoint 2.ppt
Solving The Right Triangles PowerPoint 2.pptSolving The Right Triangles PowerPoint 2.ppt
Solving The Right Triangles PowerPoint 2.ppt
 
9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf
9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf
9953056974 Call Girls In South Ex, Escorts (Delhi) NCR.pdf
 
lifi-technology with integration of IOT.pptx
lifi-technology with integration of IOT.pptxlifi-technology with integration of IOT.pptx
lifi-technology with integration of IOT.pptx
 
Arduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.pptArduino_CSE ece ppt for working and principal of arduino.ppt
Arduino_CSE ece ppt for working and principal of arduino.ppt
 
Call Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile serviceCall Girls Delhi {Jodhpur} 9711199012 high profile service
Call Girls Delhi {Jodhpur} 9711199012 high profile service
 
8251 universal synchronous asynchronous receiver transmitter
8251 universal synchronous asynchronous receiver transmitter8251 universal synchronous asynchronous receiver transmitter
8251 universal synchronous asynchronous receiver transmitter
 
Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024Architect Hassan Khalil Portfolio for 2024
Architect Hassan Khalil Portfolio for 2024
 
Electronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdfElectronically Controlled suspensions system .pdf
Electronically Controlled suspensions system .pdf
 
complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...complete construction, environmental and economics information of biomass com...
complete construction, environmental and economics information of biomass com...
 
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsyncWhy does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
Why does (not) Kafka need fsync: Eliminating tail latency spikes caused by fsync
 
computer application and construction management
computer application and construction managementcomputer application and construction management
computer application and construction management
 
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort serviceGurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
Gurgaon ✡️9711147426✨Call In girls Gurgaon Sector 51 escort service
 
Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptx
Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptxExploring_Network_Security_with_JA3_by_Rakesh Seal.pptx
Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptx
 
Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.Oxy acetylene welding presentation note.
Oxy acetylene welding presentation note.
 
Risk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdfRisk Assessment For Installation of Drainage Pipes.pdf
Risk Assessment For Installation of Drainage Pipes.pdf
 
welding defects observed during the welding
welding defects observed during the weldingwelding defects observed during the welding
welding defects observed during the welding
 
Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...Software and Systems Engineering Standards: Verification and Validation of Sy...
Software and Systems Engineering Standards: Verification and Validation of Sy...
 
Introduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECHIntroduction to Machine Learning Unit-3 for II MECH
Introduction to Machine Learning Unit-3 for II MECH
 
Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)
Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)
Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)
 

MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streaming for Mobile Devices

  • 1. All rights reserved. ©2020 All rights reserved. ©2020 A Super-Resolution Based Approach for HTTP Adaptive Streaming for Mobile Devices ACM Mile-High Video 2022 March 03, 2022 Minh Nguyen, Ekrem Çetinkaya, Hermann Hellwagner, Christian Timmerer Christian Doppler Laboratory ATHENA | Alpen-Adria-Universität Klagenfurt | Austria ekrem.cetinkaya@aau.at | athena.itec.aau.at 1
  • 2. All rights reserved. ©2020 Video Streaming on Mobile Devices 1 “YouTube by the Numbers: Stats, Demographics & Fun Facts”, Omnicore. All rights reserved. ©2020 2 70% of YouTube watch time is from mobile devices 1 70% 30% 2 “Experience Shapes Mobile Customer Loyalty”, Ericsson. 26% of smartphone users encounter video streaming problem every day 2
  • 3. All rights reserved. ©2020 ML-Benchmark GPU Scores of iPhones 3 ML-Benchmark GPU Scores, Source: https://browser.geekbench.com/ml-benchmarks 1797 1362 858 502 iPhone 13 (2021) iPhone 11 (2019) iPhone 8 (2017) iPhone 6S (2015)
  • 4. All rights reserved. ©2020 Super-Resolution 4 * Ahn, N., Kang, B., & Sohn, K. A. (2018). Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European conference on computer vision (ECCV) (pp. 252-268) Bilinear CARN* 540p 1080p
  • 5. All rights reserved. ©2020 5 SR-ABR Net WISH-SR Why? 🔋 Mobile devices are becoming powerful ⏱ Execution time of SR-DNNs is still high What? 🗂 ABR algorithm that considers throughput cost, buffer cost, and quality cost. 🗂 An extension to WISH1 ABR. Trade-off among different factors Why? 💿 Reduce downloaded data while preserving the QoE 🗂 ABR needs to consider when to apply SR What? 🗂 Lightweight SR network that considers the limitations of the mobile environment 🗂 Performance on-par with SoTA SR-DNNs while running on real-time on mobile GPUs Proposed Method 1M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021 IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
  • 7. All rights reserved. ©2020 System Architecture 7 WISH-SR Server Client SR Network Request X2 X3 X4 X2 X3 X4 HR LR HTTP Get Request
  • 8. All rights reserved. ©2020 SR-ABR Net 8 Convolution ReLU Add Pixel Shuffle Convolution ReLU Add Convolution ReLU Add Convolution ReLU Convolution Clip ReLU LR Frame HR Frame
  • 9. All rights reserved. ©2020 WISH-SR ABR Algorithm 9 GET High Bitrate Segment More transferred data (higher throughput cost) More download time (higher buffer cost) Higher Quality (lower quality cost)
  • 10. All rights reserved. ©2020 WISH-SR ABR Algorithm 10 Throughput Cost Buffer Cost Conventional Quality Cost SR-Enabled Quality Cost
  • 11. All rights reserved. ©2020 WISH-SR ABR Algorithm 11 Throughput Cost Buffer Cost Current bitrate Estimated throughput Download time of current segment Current buffer - low threshold
  • 12. All rights reserved. ©2020 WISH-SR ABR Algorithm 12 Quality Cost Distortion penalty + Instability penalty Conventional Quality Current bitrate Maximum bitrate SR Quality Improvement in quality level
  • 13. All rights reserved. ©2020 WISH-SR ABR Algorithm 13 Quality Cost Throughput Cost Buffer Cost WISH-SR ABR Algorithm M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. “WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices.” In 2021 IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021.
  • 14. Evaluation Setup All rights reserved. ©2020 14
  • 15. All rights reserved. ©2020 Experimental Setup 15 Testbed 💻 Lenovo Thinkpad P1 (i7 / 16GB) Ubuntu 18.04 📱 Xiaomi Mi 11 (Snapdragon 888) Android 11 - ExoPlayer Dataset - ABR 🗂 HEVC - Segment duration 4s 🗂{100, 145, 900, 2400, 4500} kbps {270p, 360p, 540p, 720p, 1080p} (i) Tears of steel - First 5 mins (ToS1) (Mix 🌍🗂 - 📉 SI 📉 TI) (ii) Tears of steel - Last 5 mins (ToS2) (Mix 🌍🗂 - 📈 SI 📈 TI) (iii) Gameplay - (Generated 🗂 - 📈 SI 📉 TI) (iv) Rally (Natural 🌍 - 📉 SI 📈 TI) 🔗 Linux traffic control tool (tc) 4G Network trace1 Avg. 3787 kbps - Std.dev. 3193 kbps RTT 20ms - Buffer 20s - Low threshold 4s 1D. Raca, J. J. Quinlan, A. H. Zahran, and C. J. Sreenan. “Beyond throughput: a 4G LTE dataset with channel and context metrics”. In Proceedings of the 9th ACM Multimedia Systems Conference, pages 460–465. ACM, 2018. 2T.-Y. Huang, R. Johari, N. McKeown, M. Trunnell, and M. Watson. A buffer-based approach to rate adaptation: Evidence from a large video streaming service. In ACM SIGCOMM Computer Communication Review, volume 44, pages 187–198. ACM, 2014. 3C. Wang, A. Rizk, and M. Zink. SQUAD: A spectrum-based quality adaptation for dynamic adaptive streaming over HTTP. In Proceedings of the 7th International Conference on Multimedia Systems, pages 1–12, 2016. 4M. Nguyen, E. Çetinkaya, H. Hellwagner, and C. Timmerer. WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile Devices. In 2021 IEEE 23rd Int’l. Workshop on Multimedia Signal Processing (MMSP). IEEE, 2021. BBA-02, ExoPlayer, SQUAD3, WISH4
  • 16. All rights reserved. ©2020 SR Network Training 16 Dataset 🗂 HEVC - Target Resolution 1080p 270p - X4, 360p - X3, 540p - X2 DIV2K Dataset 1 Frames from around ~ 100 Videos Waterloo 2 - SJTU 3 - Tencent Video Dataset 4 1 Agustsson, Eirikur, and Radu Timofte. "Ntire 2017 challenge on single image super-resolution: Dataset and study." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017. 2 M. Cheon and J.-S. Lee. Subjective and objective quality assessment of compressed 4K UHD videos for immersive experience. IEEE Transactions on Circuits and Systems for Video Technology, 28(7):1467–1480, 2017. 3 L. Song, X. Tang, W. Zhang, X. Yang, and P. Xia. The SJTU 4K video sequence dataset. In 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), pages 34–35, 2013. doi: 10.1109/QoMEX.2013.6603201. 4 X. Xu, S. Liu, and Z. Li. Tencent Video Dataset (TVD): A Video Dataset for Learning-based Visual Data Compression and Analysis. arXiv preprint arXiv:2105.05961, 2021 5 N. Ahn, B. Kang, and K.-A. Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 252–268, 2018. Training CARN-M5 - SR-ABR Net Train on DIV2K - Finetune on encoded videos Adam optimizer - Learning rate scheduler - MSE Tensorflow-lite Float16 quantization
  • 17. All rights reserved. ©2020 Evaluation Metrics 17 Average Bitrate # of Stalls and Stall Duration QoE Score - ITU-T P.1203 Extension Mode 0 VMAF VMAF/Bitrate
  • 19. All rights reserved. ©2020 SR-DNN Results 19 1 Ekrem Çetinkaya, Minh Nguyen, and Christian Timmerer. "MoViDNN: A Mobile Platform for Evaluating Video Quality Enhancement with Deep Neural Networks." arXiv preprint arXiv:2201.04402 (2022). Execution Speed (FPS) X2 90.93 91.13 82.10 52.83 54.11 42.91 39.00 41.56 24.32 X3 X4 24 30 36 14 9 5 X3 X4 X2 VMAF SR-ABR Net CARN-M Bilinear
  • 20. All rights reserved. ©2020 SR-ABR Results 20 3098 1818 2670 1748 1738 BBA-0 EP SQUAD WISH WISH-SR Average Bitrate (kbps) 3.54 4.05 3.35 4.06 4.09 BBA-0 EP SQUAD WISH WISH-SR QoE Score (ITU.T P.1203) 90.87 81.75 86.55 81.29 84.91 BBA-0 EP SQUAD WISH WISH-SR VMAF 22 1.85 1 0.3 24 1.8 0 0 BBA-0 EP SQUAD WISH WISH-SR Stall Duration (s) # of Stalls 0.029 0.045 0.032 0.046 0.049 VMAF / Bitrate (1 kbps) BBA-0 EP SQUAD WISH WISH-SR
  • 21. All rights reserved. ©2020 Conclusion 21 SR-ABR Net WISH-SR Lightweight SR DNN that considers the limitations of the mobile environment Significant improvement (up to 60%) over bilinear interpolation (default in Android) On-par performance with SoTA SR DNNs while running in real time on mobile GPU ABR algorithm that leverages SR networks to improve quality Weighted sum model of throughput cost, buffer cost, and quality cost SR-ABR SR-ABR Net integrated into WISH-SR and deployed on ExoPlayer Significant data reduction (up to 43%) while providing high QoE
  • 22. All rights reserved. ©2020 Thank you! ekrem.cetinkaya@aau.at minh.nguyen@aau.at @ekremcetinkaya_ @minhkstn linkedin.com/in/ekrcet linkedin.com/in/minhkstn