Abstract: Rapid growth in multimedia streaming traffic over the Internet motivates the research and further investigation of the video coding performance of such services in terms of speed and Quality of Experience (QoE). HTTP Adaptive Streaming (HAS) is today’s de-facto standard to deliver clients the highest possible video quality. In HAS, the same video content is encoded at multiple bitrates, resolutions, framerates, and coding formats called representations. This study aims to (i) provide fast and compression-efficient multi-bitrate, multi-resolution representations, (ii) provide fast and compression-efficient multi-codec representations, (iii) improve the encoding efficiency of Video on Demand (VoD) streaming using content-adaptive encoding optimizations, and (iv) provide encoding schemes with optimizations per-title for live streaming applications to decrease the storage or delivery costs or/and increase QoE.

1. Video Coding Enhancements for HTTP Adaptive Streaming
Vignesh V Menon
Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria
10-14 Oct 2022
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 1

2. Outline
1 Introduction
2 Research Questions
3 Target of this study
4 Outcomes
5 Summary
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 2

3. Introduction
Introduction
HTTP Adaptive Streaming (HAS)1
Why Adaptive Streaming?
Adapt for a wide range of devices.
Adapt for a broad set of Internet speeds.
What HAS does?
Each source video is split into segments.
Encoded at multiple bitrates, resolutions, and codecs.
Delivered to the client based on the device capability, network speed etc.
1
Abdelhak Bentaleb et al. “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP”. In: IEEE Communications Surveys Tutorials 21.1
(2019), pp. 562–585. doi: 10.1109/COMST.2018.2862938.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 3

4. Introduction
Introduction
Video Coding in HTTP Adaptive Streaming (HAS)
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 4

5. Research Questions
Research Questions
RQ-1 Introduction
How to efficiently provide multi-bitrate, multi-resolution representations for HAS?
The demand for high resolution and high bitrate content is rising.
More devices are introduced to cater to the demand for high-quality video content at various
resolutions and bitrates.
Goal: Reduce the increasing computational complexity (i.e., runtime) while maintaining
the compression efficiency of standalone encodings.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 5

6. Research Questions
Research Questions
RQ-1 Introduction
Relative encoding time of the representations normalized to encoding time of the 2160p
25Mbps representation.
500 1000 1500 2000 3000 4500 5800 7000 11600 16800 20000 25000
Bitrate (in kbps)
0
20
40
60
80
100
Relative Time Complexity (in percentage)
540p
1080p
2160p
As resolution doubles, encoding time complexity doubles!
Many encoder analysis decisions are redundant across the representations.
Multi-rate: Exploit this redundancy across representations of a resolution.
Multi-resolution: Exploit this redundancy across resolutions.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 6

7. Research Questions
Research Questions
RQ-1 Literature Review
How to efficiently provide multi-bitrate, multi-resolution representations for HAS?
Limit block partitioning search levels.2,3
Reuse additional available information to help with other encoding decisions.4
Multi-encoding methods combine fast multi-rate methods and multi-resolution encoding
optimizations.5
2
Damien Schroeder, Adithyan Ilangovan, and Eckehard Steinbach. “Multi-rate encoding for HEVC-based adaptive HTTP streaming with multiple resolutions”.
In: 2015 IEEE 17th International Workshop on Multimedia Signal Processing (MMSP). 2015, pp. 1–6. doi: 10.1109/MMSP.2015.7340822.
3
Hadi Amirpour et al. “Fast Multi-rate Encoding for Adaptive HTTP Streaming”. In: 2020 Data Compression Conference (DCC). 2020, pp. 358–358.
4
Damien Schroeder et al. “Efficient Multi-Rate Video Encoding for HEVC-Based Adaptive HTTP Streaming”. In: IEEE Transactions on Circuits and Systems
for Video Technology 28.1 (2018), pp. 143–157. doi: 10.1109/TCSVT.2016.2599028.
5
Kalyan Goswami et al. “Adaptive Multi-Resolution Encoding for ABR Streaming”. In: 2018 25th IEEE International Conference on Image Processing (ICIP).
2018, pp. 1008–1012. doi: 10.1109/ICIP.2018.8451485.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 7

8. Research Questions
Research Questions
RQ-2 Introduction
How to efficiently provide multi-codec representations for HAS?
HAS serves multiple types of devices:
A subset of devices can only decode the previous generation codec (e.g., AVC6
).
A subset of devices can only decode the current generation codec (e.g., HEVC7
).
A subset of devices can decode both codecs and can also seamlessly switch between them.
Representations of both codecs should be stored to address multiple clients.8
Goal: Reduce the increased computational complexity of encoding representations of mul-
tiple codecs without compromising the overall quality of such a system.
6
Thomas Wiegand et al. “Overview of the H.264/AVC video coding standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 13.7
(2003), pp. 560–576. doi: 10.1109/TCSVT.2003.815165.
7
Gary J. Sullivan et al. “Overview of the high efficiency video coding (HEVC) standard”. In: IEEE Transactions on circuits and systems for video technology
22.12 (2012), pp. 1649–1668.
8
Yury Reznik et al. “Optimal Multi-Codec Adaptive Bitrate Streaming”. In: 2019 IEEE International Conference on Multimedia Expo Workshops (ICMEW).
2019, pp. 348–353. doi: 10.1109/ICMEW.2019.00066.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 8

9. Research Questions
Research Questions
RQ-3 Introduction
How to improve the encoder performance using content-adaptive algorithms?
Quest to achieve a perfect trade-off between perceptual quality and compression efficiency.
Allocating only the required bits for a given video based on its complexity, Content Adaptive
Encoding (CAE) can drive significant bitrate savings.9
Goal: Derive content-adaptive spatial and temporal features for the video, which we can
later use to influence encoder decisions like slice-type, quantization parameter, block par-
titioning, framerate, and much more.
9
J. Shingala and P. Dixit. “Content Adaptive Encoding: Key Decisions for an Effective Solution”. In: “online“. Feb. 2018.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 9

10. Research Questions
Research Questions
RQ-3 Literature Review
How to improve the encoder performance using content-adaptive algorithms?
Shot detection, content-adaptive block partitioning prediction,10 adaptive quantization pa-
rameter prediction, and much more.11
Content-adaptive controls for video encoding.12,13
10
Y. Zhang et al. “Statistical Early Termination and Early Skip Models for Fast Mode Decision in HEVC INTRA Coding”. In: ACM Trans. Multimedia Comput.
Commun. Appl. 15.3 (July 2019). issn: 1551-6857. doi: 10.1145/3321510.
11
Shingala and Dixit, “Content Adaptive Encoding: Key Decisions for an Effective Solution”.
12
Pradeep Ramachandran et al. “Content adaptive live encoding with open source codecs”. In: May 2020, pp. 345–348. doi: 10.1145/3339825.3393580.
13
Tamar Shoham et al. “Content-adaptive frame level rate control for video encoding using a perceptual video quality measure”. In: Applications of Digital
Image Processing XLII. vol. 11137. SPIE, 2019, pp. 164 –177.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 10

11. Research Questions
Research Questions
RQ-4 Introduction
How to improve the efficiency of per-title encoding for live-streaming applications?
For live streaming applications, the same video content is encoded (also called ”title”) at
multiple bitrates, resolutions, codecs, etc., though we are constrained in terms of encoding
time.
Encoders have presets with conservative settings biased towards the compression effi-
ciency.14
Goal:
1 Decrease the time taken to compute the convex hull for each video by analyzing it using certain
features (e.g., SI, TI information15
).
2 Study dynamically configuring the encoder parameters on the fly to sustain a target encoding
speed according to the content for efficient live streaming.
14
Ramachandran et al., “Content adaptive live encoding with open source codecs”.
15
https://github.com/Telecommunication-Telemedia-Assessment/SITI.git
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 11

12. Target of this study
Target of this study
Figure: The ideal video compression system for HAS envisioned in this doctoral study.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 12

13. Outcomes
Outcomes
Video Complexity Feature Extraction
Video Complexity Analyzer (VCA) open-source project16,17 is released, which is a practical
implementation of the video complexity feature extraction proposed in this doctoral research.
Figure: VCA in content-adaptive encoding systems.
16
https://github.com/cd-athena/VCA, last access: Aug 20, 2022
17
Vignesh V Menon et al. “VCA: Video Complexity Analyzer”. In: 13th ACM Multimedia Systems Conference (MMSys ’22). MMSys ’22. New York, NY, USA:
Association for Computing Machinery, 2022. isbn: 9781450392839. doi: 10.1145/3524273.3532896. url: https://doi.org/10.1145/3524273.3532896.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 13

14. Outcomes
Outcomes
Perceptually-aware Bitrate Ladder Prediction18
Figure: Perceptually-aware per-title encoding scheme.
b0 b1 b2 b3 b4 b5 b6
Bitrate
v0
v1=v0 + vJ(v0)
v2=v1 + vJ(v1)
v3=v2 + vJ(v2)
v4=v3 + vJ(v3)
v5=v4 + vJ(v4)
v6=v5 + vJ(v5)
VMAF
vmax
r1
r0
r2
r3
r4
r5
r6
18
Vignesh V Menon et al. “Perceptually-aware Per-title Encoding for Adaptive Video Streaming”. In: 2022 IEEE International Conference on Multimedia and
Expo (ICME). ICME ’22. Taiwan, 2022.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 14

15. Outcomes
Outcomes
Dynamic Framerate Prediction19
Figure: Block diagram of the complete variable framerate (VFR) coding scheme in the context of
per-title encoding. Here, f and ˆ
f denotes the original framerate of the title and the framerate at which
the title is encoded. d represents the frame dropping factor.
19
Vignesh V Menon et al. “CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming”. In: 2022 Data Compression Conference
(DCC). 2022, pp. 475–475. doi: 10.1109/DCC52660.2022.00086.
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 15

16. Summary
Summary
Table: List of Related Publications
Row Title Venue
1 Efficient Multi-Encoding Algorithms for HTTP Adaptive
Bitrate Streaming
Picture Coding Symposium (PCS) 2021
2 Efficient Content-Adaptive Feature-based Shot Detection
for HTTP Adaptive Streaming
International Conference on Image Processing (ICIP) 2021
3 INCEPT: Intra CU depth Prediction for HEVC International Workshop on Multimedia Signal Processing
(MMSP) 2021
4 OPTE: Online Per-title Encoding International Conference on Acoustics, Speech, and Signal
Processing (ICASSP) 2022
5 CODA: Content-Adaptive Frame dropping Algorithm Data Compression Conference (DCC) 2022
6 Live-PSTR: Live Per-title Encoding for Ultra HD Adaptive
Streaming
NAB Broadcast Engineering and Information Technology
(BEIT) Conference 2022
7 Perceptually-aware Per-title Encoding for Adaptive Video
Streaming
International Conference on Multimedia & Expo (ICME) 2022
8 OPSE: Online Per-Scene Encoding for Adaptive HTTP Live
Streaming
International Conference on Multimedia & Expo Workshop
(ICMEW) 2022
9 VCA: Video Complexity Analyzer Multimedia Systems Conference (MMSys) 2022
10 ETPS: Efficient Two-Pass encoding Scheme for Live Adap-
tive Streaming
ICIP 2022
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 16

17. Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Vignesh V Menon Video Coding Enhancements for HTTP Adaptive Streaming 17