SlideShare a Scribd company logo
1 of 17
Download to read offline
Video Coding Enhancements for HTTP Adaptive Streaming
Vignesh V Menon
Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria
10-14 Oct 2022
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 1
Outline
1 Introduction
2 Research Questions
3 Target of this study
4 Outcomes
5 Summary
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 2
Introduction
Introduction
HTTP Adaptive Streaming (HAS)1
Why Adaptive Streaming?
Adapt for a wide range of devices.
Adapt for a broad set of Internet speeds.
What HAS does?
Each source video is split into segments.
Encoded at multiple bitrates, resolutions, and codecs.
Delivered to the client based on the device capability, network speed etc.
1
Abdelhak Bentaleb et al. “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP”. In: IEEE Communications Surveys Tutorials 21.1
(2019), pp. 562–585. doi: 10.1109/COMST.2018.2862938.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 3
Introduction
Introduction
Video Coding in HTTP Adaptive Streaming (HAS)
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 4
Research Questions
Research Questions
RQ-1 Introduction
How to efficiently provide multi-bitrate, multi-resolution representations for HAS?
The demand for high resolution and high bitrate content is rising.
More devices are introduced to cater to the demand for high-quality video content at various
resolutions and bitrates.
Goal: Reduce the increasing computational complexity (i.e., runtime) while maintaining
the compression efficiency of standalone encodings.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 5
Research Questions
Research Questions
RQ-1 Introduction
Relative encoding time of the representations normalized to encoding time of the 2160p
25Mbps representation.
500 1000 1500 2000 3000 4500 5800 7000 11600 16800 20000 25000
Bitrate (in kbps)
0
20
40
60
80
100
Relative Time Complexity (in percentage)
540p
1080p
2160p
As resolution doubles, encoding time complexity doubles!
Many encoder analysis decisions are redundant across the representations.
Multi-rate: Exploit this redundancy across representations of a resolution.
Multi-resolution: Exploit this redundancy across resolutions.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 6
Research Questions
Research Questions
RQ-1 Literature Review
How to efficiently provide multi-bitrate, multi-resolution representations for HAS?
Limit block partitioning search levels.2,3
Reuse additional available information to help with other encoding decisions.4
Multi-encoding methods combine fast multi-rate methods and multi-resolution encoding
optimizations.5
2
Damien Schroeder, Adithyan Ilangovan, and Eckehard Steinbach. “Multi-rate encoding for HEVC-based adaptive HTTP streaming with multiple resolutions”.
In: 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP). 2015, pp. 1–6. doi: 10.1109/MMSP.2015.7340822.
3
Hadi Amirpour et al. “Fast Multi-rate Encoding for Adaptive HTTP Streaming”. In: 2020 Data Compression Conference (DCC). 2020, pp. 358–358.
4
Damien Schroeder et al. “Efficient Multi-Rate Video Encoding for HEVC-Based Adaptive HTTP Streaming”. In: IEEE Transactions on Circuits and Systems
for Video Technology 28.1 (2018), pp. 143–157. doi: 10.1109/TCSVT.2016.2599028.
5
Kalyan Goswami et al. “Adaptive Multi-Resolution Encoding for ABR Streaming”. In: 2018 25th IEEE International Conference on Image Processing (ICIP).
2018, pp. 1008–1012. doi: 10.1109/ICIP.2018.8451485.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 7
Research Questions
Research Questions
RQ-2 Introduction
How to efficiently provide multi-codec representations for HAS?
HAS serves multiple types of devices:
A subset of devices can only decode the previous generation codec (e.g., AVC6
).
A subset of devices can only decode the current generation codec (e.g., HEVC7
).
A subset of devices can decode both codecs and can also seamlessly switch between them.
Representations of both codecs should be stored to address multiple clients.8
Goal: Reduce the increased computational complexity of encoding representations of mul-
tiple codecs without compromising the overall quality of such a system.
6
Thomas Wiegand et al. “Overview of the H.264/AVC video coding standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 13.7
(2003), pp. 560–576. doi: 10.1109/TCSVT.2003.815165.
7
Gary J. Sullivan et al. “Overview of the high efficiency video coding (HEVC) standard”. In: IEEE Transactions on circuits and systems for video technology
22.12 (2012), pp. 1649–1668.
8
Yury Reznik et al. “Optimal Multi-Codec Adaptive Bitrate Streaming”. In: 2019 IEEE International Conference on Multimedia Expo Workshops (ICMEW).
2019, pp. 348–353. doi: 10.1109/ICMEW.2019.00066.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 8
Research Questions
Research Questions
RQ-3 Introduction
How to improve the encoder performance using content-adaptive algorithms?
Quest to achieve a perfect trade-off between perceptual quality and compression efficiency.
Allocating only the required bits for a given video based on its complexity, Content Adaptive
Encoding (CAE) can drive significant bitrate savings.9
Goal: Derive content-adaptive spatial and temporal features for the video, which we can
later use to influence encoder decisions like slice-type, quantization parameter, block par-
titioning, framerate, and much more.
9
J. Shingala and P. Dixit. “Content Adaptive Encoding: Key Decisions for an Effective Solution”. In: “online“. Feb. 2018.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 9
Research Questions
Research Questions
RQ-3 Literature Review
How to improve the encoder performance using content-adaptive algorithms?
Shot detection, content-adaptive block partitioning prediction,10 adaptive quantization pa-
rameter prediction, and much more.11
Content-adaptive controls for video encoding.12,13
10
Y. Zhang et al. “Statistical Early Termination and Early Skip Models for Fast Mode Decision in HEVC INTRA Coding”. In: ACM Trans. Multimedia Comput.
Commun. Appl. 15.3 (July 2019). issn: 1551-6857. doi: 10.1145/3321510.
11
Shingala and Dixit, “Content Adaptive Encoding: Key Decisions for an Effective Solution”.
12
Pradeep Ramachandran et al. “Content adaptive live encoding with open source codecs”. In: May 2020, pp. 345–348. doi: 10.1145/3339825.3393580.
13
Tamar Shoham et al. “Content-adaptive frame level rate control for video encoding using a perceptual video quality measure”. In: Applications of Digital
Image Processing XLII. vol. 11137. SPIE, 2019, pp. 164 –177.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 10
Research Questions
Research Questions
RQ-4 Introduction
How to improve the efficiency of per-title encoding for live-streaming applications?
For live streaming applications, the same video content is encoded (also called ”title”) at
multiple bitrates, resolutions, codecs, etc., though we are constrained in terms of encoding
time.
Encoders have presets with conservative settings biased towards the compression effi-
ciency.14
Goal:
1 Decrease the time taken to compute the convex hull for each video by analyzing it using certain
features (e.g., SI, TI information15
).
2 Study dynamically configuring the encoder parameters on the fly to sustain a target encoding
speed according to the content for efficient live streaming.
14
Ramachandran et al., “Content adaptive live encoding with open source codecs”.
15
https://github.com/Telecommunication-Telemedia-Assessment/SITI.git
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 11
Target of this study
Target of this study
Figure: The ideal video compression system for HAS envisioned in this doctoral study.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 12
Outcomes
Outcomes
Video Complexity Feature Extraction
Video Complexity Analyzer (VCA) open-source project16,17 is released, which is a practical
implementation of the video complexity feature extraction proposed in this doctoral research.
Figure: VCA in content-adaptive encoding systems.
16
https://github.com/cd-athena/VCA, last access: Aug 20, 2022
17
Vignesh V Menon et al. “VCA: Video Complexity Analyzer”. In: 13th ACM Multimedia Systems Conference (MMSys ’22). MMSys ’22. New York, NY, USA:
Association for Computing Machinery, 2022. isbn: 9781450392839. doi: 10.1145/3524273.3532896. url: https://doi.org/10.1145/3524273.3532896.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 13
Outcomes
Outcomes
Perceptually-aware Bitrate Ladder Prediction18
Figure: Perceptually-aware per-title encoding scheme.
b0 b1 b2 b3 b4 b5 b6
Bitrate
v0
v1=v0 + vJ(v0)
v2=v1 + vJ(v1)
v3=v2 + vJ(v2)
v4=v3 + vJ(v3)
v5=v4 + vJ(v4)
v6=v5 + vJ(v5)
VMAF
vmax
r1
r0
r2
r3
r4
r5
r6
18
Vignesh V Menon et al. “Perceptually-aware Per-title Encoding for Adaptive Video Streaming”. In: 2022 IEEE International Conference on Multimedia and
Expo (ICME). ICME ’22. Taiwan, 2022.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 14
Outcomes
Outcomes
Dynamic Framerate Prediction19
Figure: Block diagram of the complete variable framerate (VFR) coding scheme in the context of
per-title encoding. Here, f and ˆ
f denotes the original framerate of the title and the framerate at which
the title is encoded. d represents the frame dropping factor.
19
Vignesh V Menon et al. “CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming”. In: 2022 Data Compression Conference
(DCC). 2022, pp. 475–475. doi: 10.1109/DCC52660.2022.00086.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 15
Summary
Summary
Table: List of Related Publications
Row Title Venue
1 Efficient Multi-Encoding Algorithms for HTTP Adaptive
Bitrate Streaming
Picture Coding Symposium (PCS) 2021
2 Efficient Content-Adaptive Feature-based Shot Detection
for HTTP Adaptive Streaming
International Conference on Image Processing (ICIP) 2021
3 INCEPT: Intra CU depth Prediction for HEVC International Workshop on Multimedia Signal Processing
(MMSP) 2021
4 OPTE: Online Per-title Encoding International Conference on Acoustics, Speech, and Signal
Processing (ICASSP) 2022
5 CODA: Content-Adaptive Frame dropping Algorithm Data Compression Conference (DCC) 2022
6 Live-PSTR: Live Per-title Encoding for Ultra HD Adaptive
Streaming
NAB Broadcast Engineering and Information Technology
(BEIT) Conference 2022
7 Perceptually-aware Per-title Encoding for Adaptive Video
Streaming
International Conference on Multimedia & Expo (ICME) 2022
8 OPSE: Online Per-Scene Encoding for Adaptive HTTP Live
Streaming
International Conference on Multimedia & Expo Workshop
(ICMEW) 2022
9 VCA: Video Complexity Analyzer Multimedia Systems Conference (MMSys) 2022
10 ETPS: Efficient Two-Pass encoding Scheme for Live Adap-
tive Streaming
ICIP 2022
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 16
Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 17

More Related Content

Similar to Doctoral Symposium presentation.pdf

HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)
Alpen-Adria-Universität
 
HTTP Adaptive Streaming – Quo Vadis?
HTTP Adaptive Streaming – Quo Vadis?HTTP Adaptive Streaming – Quo Vadis?
HTTP Adaptive Streaming – Quo Vadis?
Alpen-Adria-Universität
 
CAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsCAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR Systems
Alpen-Adria-Universität
 
A Study on FFmpeg Multimedia Framework
A Study on FFmpeg Multimedia FrameworkA Study on FFmpeg Multimedia Framework
A Study on FFmpeg Multimedia Framework
ijtsrd
 
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - Poster
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - PosterQoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - Poster
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - Poster
DanieleLorenzi6
 

Similar to Doctoral Symposium presentation.pdf (20)

Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
 
HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
OPTE: Online Per-title Encoding for Live Video Streaming
OPTE: Online Per-title Encoding for Live Video StreamingOPTE: Online Per-title Encoding for Live Video Streaming
OPTE: Online Per-title Encoding for Live Video Streaming
 
OPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdfOPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdf
 
HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?
 
Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...
Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...
Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, S...
 
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
 
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
 
HTTP Adaptive Streaming – Quo Vadis?
HTTP Adaptive Streaming – Quo Vadis?HTTP Adaptive Streaming – Quo Vadis?
HTTP Adaptive Streaming – Quo Vadis?
 
Analyzing Video Streaming Quality by Using Various Error Correction Methods o...
Analyzing Video Streaming Quality by Using Various Error Correction Methods o...Analyzing Video Streaming Quality by Using Various Error Correction Methods o...
Analyzing Video Streaming Quality by Using Various Error Correction Methods o...
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
CAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsCAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR Systems
 
12 11 aug17 29may 7301 8997-1-ed edit satria
12 11 aug17 29may 7301 8997-1-ed edit satria12 11 aug17 29may 7301 8997-1-ed edit satria
12 11 aug17 29may 7301 8997-1-ed edit satria
 
A Study on FFmpeg Multimedia Framework
A Study on FFmpeg Multimedia FrameworkA Study on FFmpeg Multimedia Framework
A Study on FFmpeg Multimedia Framework
 
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - Poster
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - PosterQoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - Poster
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming - Poster
 
Policy-driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-driven Dynamic HTTP Adaptive Streaming Player Environment
 
A04840107
A04840107A04840107
A04840107
 

More from Vignesh V Menon

Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Vignesh V Menon
 
TQPM.pdf
TQPM.pdfTQPM.pdf
TQPM.pdf
Vignesh V Menon
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
Vignesh V Menon
 

More from Vignesh V Menon (16)

Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
 
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
 
Green_VCA_presentation.pdf
Green_VCA_presentation.pdfGreen_VCA_presentation.pdf
Green_VCA_presentation.pdf
 
TQPM.pdf
TQPM.pdfTQPM.pdf
TQPM.pdf
 
CAPS_Presentation.pdf
CAPS_Presentation.pdfCAPS_Presentation.pdf
CAPS_Presentation.pdf
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
 
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdfETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
 
OPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdf
OPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdfOPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdf
OPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdf
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
 

Recently uploaded

The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
heathfieldcps1
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
中 央社
 
SURVEY I created for uni project research
SURVEY I created for uni project researchSURVEY I created for uni project research
SURVEY I created for uni project research
CaitlinCummins3
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
中 央社
 
MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...
MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...
MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...
Krashi Coaching
 

Recently uploaded (20)

The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
 
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUMDEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
DEMONSTRATION LESSON IN ENGLISH 4 MATATAG CURRICULUM
 
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
 
How to Manage Closest Location in Odoo 17 Inventory
How to Manage Closest Location in Odoo 17 InventoryHow to Manage Closest Location in Odoo 17 Inventory
How to Manage Closest Location in Odoo 17 Inventory
 
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
 
Championnat de France de Tennis de table/
Championnat de France de Tennis de table/Championnat de France de Tennis de table/
Championnat de France de Tennis de table/
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).
 
male presentation...pdf.................
male presentation...pdf.................male presentation...pdf.................
male presentation...pdf.................
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
 
SURVEY I created for uni project research
SURVEY I created for uni project researchSURVEY I created for uni project research
SURVEY I created for uni project research
 
An Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge AppAn Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge App
 
Improved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio AppImproved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio App
 
The Liver & Gallbladder (Anatomy & Physiology).pptx
The Liver &  Gallbladder (Anatomy & Physiology).pptxThe Liver &  Gallbladder (Anatomy & Physiology).pptx
The Liver & Gallbladder (Anatomy & Physiology).pptx
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
 
Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"
 
e-Sealing at EADTU by Kamakshi Rajagopal
e-Sealing at EADTU by Kamakshi Rajagopale-Sealing at EADTU by Kamakshi Rajagopal
e-Sealing at EADTU by Kamakshi Rajagopal
 
MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...
MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...
MSc Ag Genetics & Plant Breeding: Insights from Previous Year JNKVV Entrance ...
 
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading RoomSternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
Sternal Fractures & Dislocations - EMGuidewire Radiology Reading Room
 
IPL Online Quiz by Pragya; Question Set.
IPL Online Quiz by Pragya; Question Set.IPL Online Quiz by Pragya; Question Set.
IPL Online Quiz by Pragya; Question Set.
 

Doctoral Symposium presentation.pdf

  • 1. Video Coding Enhancements for HTTP Adaptive Streaming Vignesh V Menon Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria 10-14 Oct 2022 Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 1
  • 2. Outline 1 Introduction 2 Research Questions 3 Target of this study 4 Outcomes 5 Summary Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 2
  • 3. Introduction Introduction HTTP Adaptive Streaming (HAS)1 Why Adaptive Streaming? Adapt for a wide range of devices. Adapt for a broad set of Internet speeds. What HAS does? Each source video is split into segments. Encoded at multiple bitrates, resolutions, and codecs. Delivered to the client based on the device capability, network speed etc. 1 Abdelhak Bentaleb et al. “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP”. In: IEEE Communications Surveys Tutorials 21.1 (2019), pp. 562–585. doi: 10.1109/COMST.2018.2862938. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 3
  • 4. Introduction Introduction Video Coding in HTTP Adaptive Streaming (HAS) Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 4
  • 5. Research Questions Research Questions RQ-1 Introduction How to efficiently provide multi-bitrate, multi-resolution representations for HAS? The demand for high resolution and high bitrate content is rising. More devices are introduced to cater to the demand for high-quality video content at various resolutions and bitrates. Goal: Reduce the increasing computational complexity (i.e., runtime) while maintaining the compression efficiency of standalone encodings. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 5
  • 6. Research Questions Research Questions RQ-1 Introduction Relative encoding time of the representations normalized to encoding time of the 2160p 25Mbps representation. 500 1000 1500 2000 3000 4500 5800 7000 11600 16800 20000 25000 Bitrate (in kbps) 0 20 40 60 80 100 Relative Time Complexity (in percentage) 540p 1080p 2160p As resolution doubles, encoding time complexity doubles! Many encoder analysis decisions are redundant across the representations. Multi-rate: Exploit this redundancy across representations of a resolution. Multi-resolution: Exploit this redundancy across resolutions. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 6
  • 7. Research Questions Research Questions RQ-1 Literature Review How to efficiently provide multi-bitrate, multi-resolution representations for HAS? Limit block partitioning search levels.2,3 Reuse additional available information to help with other encoding decisions.4 Multi-encoding methods combine fast multi-rate methods and multi-resolution encoding optimizations.5 2 Damien Schroeder, Adithyan Ilangovan, and Eckehard Steinbach. “Multi-rate encoding for HEVC-based adaptive HTTP streaming with multiple resolutions”. In: 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP). 2015, pp. 1–6. doi: 10.1109/MMSP.2015.7340822. 3 Hadi Amirpour et al. “Fast Multi-rate Encoding for Adaptive HTTP Streaming”. In: 2020 Data Compression Conference (DCC). 2020, pp. 358–358. 4 Damien Schroeder et al. “Efficient Multi-Rate Video Encoding for HEVC-Based Adaptive HTTP Streaming”. In: IEEE Transactions on Circuits and Systems for Video Technology 28.1 (2018), pp. 143–157. doi: 10.1109/TCSVT.2016.2599028. 5 Kalyan Goswami et al. “Adaptive Multi-Resolution Encoding for ABR Streaming”. In: 2018 25th IEEE International Conference on Image Processing (ICIP). 2018, pp. 1008–1012. doi: 10.1109/ICIP.2018.8451485. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 7
  • 8. Research Questions Research Questions RQ-2 Introduction How to efficiently provide multi-codec representations for HAS? HAS serves multiple types of devices: A subset of devices can only decode the previous generation codec (e.g., AVC6 ). A subset of devices can only decode the current generation codec (e.g., HEVC7 ). A subset of devices can decode both codecs and can also seamlessly switch between them. Representations of both codecs should be stored to address multiple clients.8 Goal: Reduce the increased computational complexity of encoding representations of mul- tiple codecs without compromising the overall quality of such a system. 6 Thomas Wiegand et al. “Overview of the H.264/AVC video coding standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 13.7 (2003), pp. 560–576. doi: 10.1109/TCSVT.2003.815165. 7 Gary J. Sullivan et al. “Overview of the high efficiency video coding (HEVC) standard”. In: IEEE Transactions on circuits and systems for video technology 22.12 (2012), pp. 1649–1668. 8 Yury Reznik et al. “Optimal Multi-Codec Adaptive Bitrate Streaming”. In: 2019 IEEE International Conference on Multimedia Expo Workshops (ICMEW). 2019, pp. 348–353. doi: 10.1109/ICMEW.2019.00066. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 8
  • 9. Research Questions Research Questions RQ-3 Introduction How to improve the encoder performance using content-adaptive algorithms? Quest to achieve a perfect trade-off between perceptual quality and compression efficiency. Allocating only the required bits for a given video based on its complexity, Content Adaptive Encoding (CAE) can drive significant bitrate savings.9 Goal: Derive content-adaptive spatial and temporal features for the video, which we can later use to influence encoder decisions like slice-type, quantization parameter, block par- titioning, framerate, and much more. 9 J. Shingala and P. Dixit. “Content Adaptive Encoding: Key Decisions for an Effective Solution”. In: “online“. Feb. 2018. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 9
  • 10. Research Questions Research Questions RQ-3 Literature Review How to improve the encoder performance using content-adaptive algorithms? Shot detection, content-adaptive block partitioning prediction,10 adaptive quantization pa- rameter prediction, and much more.11 Content-adaptive controls for video encoding.12,13 10 Y. Zhang et al. “Statistical Early Termination and Early Skip Models for Fast Mode Decision in HEVC INTRA Coding”. In: ACM Trans. Multimedia Comput. Commun. Appl. 15.3 (July 2019). issn: 1551-6857. doi: 10.1145/3321510. 11 Shingala and Dixit, “Content Adaptive Encoding: Key Decisions for an Effective Solution”. 12 Pradeep Ramachandran et al. “Content adaptive live encoding with open source codecs”. In: May 2020, pp. 345–348. doi: 10.1145/3339825.3393580. 13 Tamar Shoham et al. “Content-adaptive frame level rate control for video encoding using a perceptual video quality measure”. In: Applications of Digital Image Processing XLII. vol. 11137. SPIE, 2019, pp. 164 –177. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 10
  • 11. Research Questions Research Questions RQ-4 Introduction How to improve the efficiency of per-title encoding for live-streaming applications? For live streaming applications, the same video content is encoded (also called ”title”) at multiple bitrates, resolutions, codecs, etc., though we are constrained in terms of encoding time. Encoders have presets with conservative settings biased towards the compression effi- ciency.14 Goal: 1 Decrease the time taken to compute the convex hull for each video by analyzing it using certain features (e.g., SI, TI information15 ). 2 Study dynamically configuring the encoder parameters on the fly to sustain a target encoding speed according to the content for efficient live streaming. 14 Ramachandran et al., “Content adaptive live encoding with open source codecs”. 15 https://github.com/Telecommunication-Telemedia-Assessment/SITI.git Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 11
  • 12. Target of this study Target of this study Figure: The ideal video compression system for HAS envisioned in this doctoral study. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 12
  • 13. Outcomes Outcomes Video Complexity Feature Extraction Video Complexity Analyzer (VCA) open-source project16,17 is released, which is a practical implementation of the video complexity feature extraction proposed in this doctoral research. Figure: VCA in content-adaptive encoding systems. 16 https://github.com/cd-athena/VCA, last access: Aug 20, 2022 17 Vignesh V Menon et al. “VCA: Video Complexity Analyzer”. In: 13th ACM Multimedia Systems Conference (MMSys ’22). MMSys ’22. New York, NY, USA: Association for Computing Machinery, 2022. isbn: 9781450392839. doi: 10.1145/3524273.3532896. url: https://doi.org/10.1145/3524273.3532896. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 13
  • 14. Outcomes Outcomes Perceptually-aware Bitrate Ladder Prediction18 Figure: Perceptually-aware per-title encoding scheme. b0 b1 b2 b3 b4 b5 b6 Bitrate v0 v1=v0 + vJ(v0) v2=v1 + vJ(v1) v3=v2 + vJ(v2) v4=v3 + vJ(v3) v5=v4 + vJ(v4) v6=v5 + vJ(v5) VMAF vmax r1 r0 r2 r3 r4 r5 r6 18 Vignesh V Menon et al. “Perceptually-aware Per-title Encoding for Adaptive Video Streaming”. In: 2022 IEEE International Conference on Multimedia and Expo (ICME). ICME ’22. Taiwan, 2022. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 14
  • 15. Outcomes Outcomes Dynamic Framerate Prediction19 Figure: Block diagram of the complete variable framerate (VFR) coding scheme in the context of per-title encoding. Here, f and ˆ f denotes the original framerate of the title and the framerate at which the title is encoded. d represents the frame dropping factor. 19 Vignesh V Menon et al. “CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming”. In: 2022 Data Compression Conference (DCC). 2022, pp. 475–475. doi: 10.1109/DCC52660.2022.00086. Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 15
  • 16. Summary Summary Table: List of Related Publications Row Title Venue 1 Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming Picture Coding Symposium (PCS) 2021 2 Efficient Content-Adaptive Feature-based Shot Detection for HTTP Adaptive Streaming International Conference on Image Processing (ICIP) 2021 3 INCEPT: Intra CU depth Prediction for HEVC International Workshop on Multimedia Signal Processing (MMSP) 2021 4 OPTE: Online Per-title Encoding International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2022 5 CODA: Content-Adaptive Frame dropping Algorithm Data Compression Conference (DCC) 2022 6 Live-PSTR: Live Per-title Encoding for Ultra HD Adaptive Streaming NAB Broadcast Engineering and Information Technology (BEIT) Conference 2022 7 Perceptually-aware Per-title Encoding for Adaptive Video Streaming International Conference on Multimedia & Expo (ICME) 2022 8 OPSE: Online Per-Scene Encoding for Adaptive HTTP Live Streaming International Conference on Multimedia & Expo Workshop (ICMEW) 2022 9 VCA: Video Complexity Analyzer Multimedia Systems Conference (MMSys) 2022 10 ETPS: Efficient Two-Pass encoding Scheme for Live Adap- tive Streaming ICIP 2022 Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 16
  • 17. Q & A Q & A Thank you for your attention! Vignesh V Menon (vignesh.menon@aau.at) Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 17