augraphics/pcs-logo
Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate
Streaming
Vignesh V Menon, Hadi Amirpour, Christian Timmerer, Mohammad Ghanbari
Christian Doppler Laboratory ATHENA, Institute of Information Technology (ITEC), University of Klagenfurt, Austria
Date: 02 July 2021
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 1

Outline
1 Introduction
2 State-of-the-art: Multi-Rate Encoding
3 State-of-the-art: Multi-Encoding
4 Efficient Multi-Rate Encoding
5 Efficient Multi-Encoding
6 Experimental Evaluation
7 Conclusions and Future Directions
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 2

Introduction
Introduction
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 3

Introduction
Introduction
Background of HTTP Adaptive Streaming (HAS)1
Source: https://bitmovin.com/adaptive-streaming/
Why Adaptive Streaming?
Adapt for a wide range of devices
Adapt for a broad set of Internet speeds
What HAS does?
Each source video is split into segments
Encoded at multiple bitrates, resolutions,
and codecs
Delivered to the client based on the device
capability, network speed etc.
1
A. Bentaleb et al. “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP”. In: IEEE Communications Surveys Tutorials 21.1 (2019),
pp. 562–585. doi: 10.1109/COMST.2018.2862938.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 4

Introduction
Introduction
Encoding time of HAS representations
Relative encoding time of the representations normalized to encoding time of the 2160p
25Mbps representation.
500 1000 1500 2000 3000 4500 5800 7000 11600 16800 20000 25000
Bitrate (in kbps)
0
20
40
60
80
100
Relative Time Complexity (in percentage)
540p
1080p
2160p
As resolution doubles, encoding time complexity doubles!
Many encoder analysis decisions are redundant across the representations.
Multi-rate: Exploit this redundancy across representations of a resolution.
Multi-resolution: Exploit this redundancy across resolutions.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 5

State-of-the-art: Multi-Rate Encoding
State-of-the-art: Multi-Rate Encoding
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 6

State-of-the-art: Multi-Rate Encoding
Related Works: Multi-Rate Encoding
Single-bound2
and Double-bound3
for CU depth estimation
3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps
3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps
(a) Single-bound for CU depth estimation
3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps
3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps
(b) Double-bound for CU depth estimation
Motivation: CUs generally tend to have higher depth values (smaller CU sizes) in higher bitrate
representations.
CU depth information from the highest bitrate representation is used as the upper bound
of the CU depth estimation of dependent representations.
In double-bound, additionally, CU depth information from the lowest bitrate representation
is used as the lower bound of the CU depth estimation of dependent representations.
2
D. Schroeder, P. Rehm, and E. Steinbach. “Block structure reuse for multi-rate high efficiency video coding”. In: 2015 IEEE International Conference on
Image Processing (ICIP). Sept. 2015, pp. 3972–3976.
3
H. Amirpour et al. “Fast Multi-rate Encoding for Adaptive HTTP Streaming”. In: 2020 Data Compression Conference (DCC). 2020, pp. 358–358.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 7

State-of-the-art: Multi-Rate Encoding
Related Works: Multi-Rate Encoding
Integrated x265 mode-1 and x265 mode-24
Integrated x265 mode-1
Force the optimal decisions made in the
highest bitrate representation
the slice-type
scene-cut decisions
the quadtree structure
prediction modes
reference indices
Integrated x265 mode-2
Reuse all CUs and PUs from the highest
bitrate representation without additional
RDO. They also employ static refinement
techniques defined in x265.
If the reference depth is d, it is re-
evaluated by using the RDO against the
cost of splitting the CU (i.e., depth d +
1).
The motion vectors from the highest bi-
trate representation are used as Motion
vector predictors (MVPs).
4
K. Goswami et al. “Adaptive Multi-Resolution Encoding for ABR Streaming”. In: 2018 25th IEEE International Conference on Image Processing (ICIP). 2018,
pp. 1008–1012.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 8

State-of-the-art: Multi-Encoding
State-of-the-art: Multi-Encoding
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 9

State-of-the-art: Multi-Encoding
Related Work: Multi-Encoding
1920x1080 @ 7.0Mbps 1920x1080 @ 3.0Mbps
1920x1080 @ 5.8Mbps 1920x1080 @ 4.5Mbps
3840x2160 @ 25.0Mbps 3840x2160 @ 11.6Mbps
3840x2160 @ 20.0Mbps 3840x2160 @ 16.8Mbps
Scaled CU,PU reuse + refinement
Figure: Encoder analysis sharing method in x265 Algorithm.5
5
Aruna Mathesawaran et al. “Open source framework for reduced-complexity multi-rate HEVC encoding”. In: Applications of Digital Image Processing XLIII.
ed. by Andrew G. Tescher and Touradj Ebrahimi. Vol. 11510. International Society for Optics and Photonics. SPIE, 2020, pp. 461 –471.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 10

Efficient Multi-Rate Encoding
Efficient Multi-Rate Encoding
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 11

Efficient Multi-Rate Encoding
Efficient Multi-Rate Encoding
Prediction Mode Heuristics
Prediction Mode Heuristics
If the CU depth chosen for the dependent representations is the same as the reference represen-
tation, the following heuristics are proposed.
If Skip mode was chosen in the highest bitrate representation, RDO is evaluated for only
Merge/Skip modes.
If 2Nx2N mode was chosen in the highest bitrate representation, skip RDO for AMP modes.
If inter-prediction mode was chosen in the highest bitrate representation, skip RDO for
intra-prediction modes.
In double-bound for CU estimation, if the intra-prediction mode was chosen for the highest
and lowest bitrate representations, RDO is evaluated for only intra-prediction modes in the
intermediate representations.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 12

Efficient Multi-Rate Encoding
Efficient Multi-Rate Encoding
Motion Estimation Heuristics
Motion Estimation Heuristics
If the CU depth and PU chosen for the dependent representations is the same as the reference
representation, the following heuristics are proposed.
Force the same reference frame as that of the highest bitrate representation.
The Motion Vector Predictor (MVP) is set to be the Motion Vector (MV) of the highest
bitrate representation.
In double-bound for CU estimation, the motion search range is decreased to a smaller
window if the MV of the highest bitrate representation and the MV of the lowest bitrate
representation are close to each other. The search range is determined to be the maximum
difference between the x coordinates and y coordinates of the MVs.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 13

Efficient Multi-Rate Encoding
Efficient Multi-Rate Encoding
Proposed Multi-rate Algorithm-1 and Algorithm-2
3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps
3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps
(a) Proposed Multi-rate Algorithm-1
3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps
3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps
(b) Proposed Multi-rate Algorithm-2
− > denotes analysis sharing using single-bound for CU depth estimation and proposed
heuristics.
− > denotes analysis sharing using double-bound for CU depth estimation and proposed
heuristics.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 14

Efficient Multi-Encoding
Efficient Multi-Encoding
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 15

Efficient Multi-Encoding
Efficient Multi-Encoding
Proposed Multi-Encoding Algorithm-1
1920x1080 @ 7.0Mbps 1920x1080 @ 3.0Mbps
1920x1080 @ 5.8Mbps 1920x1080 @ 4.5Mbps
3840x2160 @ 25.0Mbps 3840x2160 @ 11.6Mbps
3840x2160 @ 20.0Mbps 3840x2160 @ 16.8Mbps
Figure: Encoder analysis sharing method in Proposed Multi-Encoding Algorithm-1.
Multi-rate Approach: Proposed Multi-rate Algorithm-2
Multi-resolution Approach: scaled CU information from highest bitrate representation of
the previous resolution is used as lower bound of CU depth estimation process.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 16

Efficient Multi-Encoding
Efficient Multi-Encoding
Proposed Multi-Encoding Algorithm-2
1920x1080 @ 7.0Mbps 1920x1080 @ 3.0Mbps
1920x1080 @ 5.8Mbps 1920x1080 @ 4.5Mbps
3840x2160 @ 25.0Mbps 3840x2160 @ 11.6Mbps
3840x2160 @ 20.0Mbps 3840x2160 @ 16.8Mbps
Scaling + Single-bound
Figure: Encoder analysis sharing method in Proposed Multi-Encoding Algorithm-2.
Multi-rate Approach: Proposed Multi-rate Algorithm-2
Multi-resolution Approach: scaled CU information from lowest bitrate representation of
the previous resolution is used as lower bound of CU depth estimation process. More rigid
bound compared to Multi-encoding Algorithm-1.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 17

Experimental Evaluation
Experimental Evaluation
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 18

Experimental Evaluation
Experimental Evaluation
Test Setup
Algorithms presented here are tested using x265 v3.46 with the veryslow preset (highest
quality preset practically usable of x265).
Test PC: Dual-processor server with Intel Xeon Gold 5218R (80 cores, frequency at 2.10
GHz).
Test Sequences: (i) 4 JVET test sequences7 and (ii) 3 professionally produced UltraHD
HDR cinematic content.8
The lowest resolution is 960 × 540, intermediate resolution is 1920 × 1080, and the highest
resolution is 3840 × 2160.
The lower resolution sources were generated from the original video source by applying
bi-cubic scaling using FFmpeg.9
6
MulticoreWare Inc. x265 HEVC Encoder/H.265 Video Codec. url: http://x265.org/.
7
Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018.
8
M. H. Pinson. “The Consumer Digital Video Library [Best of the Web]”. In: IEEE Signal Processing Magazine 30.4 (2013), pp. 172–174. doi:
10.1109/MSP.2013.2258265.
9
FFmpeg. FFmpeg Documentation. url: https://ffmpeg.org/ffmpeg.html.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 19

Experimental Evaluation
Experimental Evaluation
Test Setup
Table: Bitrate representations Bi,j (in kbps) based on HLS authoring specification10
Layer (i) Resolution j = 1 j = 2 j = 3 j = 4
1 960x540 2000 1500 1000 500
2 1920x1080 7000 5800 4500 3000
3 3840x2160 25000 20000 16800 11600
Performance evaluation Metrics:
∆T : overall encoding time saving
BDRP : Average difference in bitrate with respect to reference encode to maintain the
same PSNR
BDRV : Average difference in bitrate with respect to reference encode to maintain the
same VMAF
BDRP
∆T : Trade-off between BDR and ∆T.
10
https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 20

Experimental Evaluation
Experimental Evaluation
Multi-rate Encoding
Table: Results for the multi-rate algorithms
Method ∆T BDRP
BDRP
∆T BDRV
BDRV
∆T
Single-bound for CU estimation 14.50% -0.59% -4.39% 0.10% 1.08%
Double-bound for CU estimation 26.37% 0.73% 3.44% 1.25% 5.54%
x265 Algorithm-1 18.61% 1.67% 9.52% 1.56% 8.85%
x265 Algorithm-2 55.11% 8.42% 15.62% 8.72% 16.10%
Multi-rate Algorithm-1 (ours) 26.50% -0.28% -0.84% 0.43% 2.31%
Multi-rate Algorithm-2 (ours) 37.37% 1.06% 3.36% 1.57% 4.88%
Multi-rate Algorithm-1 improves ∆T of Single-bound for CU estimation by 12% with neg-
ligible increase in BDR.
Multi-rate Algorithm-2 improves ∆T of Double-bound for CU estimation by 11% with
negligible increase in BDR.
Better BDR
∆T metric
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 21

Experimental Evaluation
Experimental Evaluation
Multi-Encoding
Scheme ∆T BDRP BDRV
State-of-the-art scheme 80.05% 13.53% 9.59%
Multi-encoding Scheme-1 39.72% 2.32% 1.55%
Multi-encoding Scheme-2 50.90% 3.45% 2.63%
State-of-the-art Multi-encoding Algorithm-1 Multi-encoding Algorithm-2
0
2
4
6
8
10
12
14
16
BDR/ T (in percentage)
BDRP/ T
BDRV/ T
State-of-the-art algorithm yields the highest ∆T.
Multi-encoding Algorithm-1 has the lowest increase in BDR and the lowest BDR
∆T metric.
Multi-encoding Algorithm-2 improves ∆T of Multi-encoding Algorithm-1 by 11% with neg-
ligible increase in BDR.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 22

Conclusions and Future Directions
Conclusions and Future Directions
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 23

Conclusions and Future Directions
Conclusions and Future Directions
Conclusions
Presented efficient multi-encoding algorithms for HTTP Adaptive Streaming, tested with the
open-source x265 HEVC encoder.
Multi-rate Encoding:
Proposed heuristics improved the encoding speed of the single-bound and double-bound
for CU estimation by an average of 12% and 11%, respectively with a negligible increase
in BD-Rate.
BDR
∆T metric for proposed algorithms is better compared to the state-of-the-art algorithms.
Multi-encoding:
Proposed two algorithms that extended the proposed multi-rate encoding algorithms across
resolutions.
BDR
∆T metric for proposed algorithms are better compared to the state-of-the-art algorithm.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 24

Conclusions and Future Directions
Conclusions and Future Directions
Future Directions
The encoding speedup can be improved by parallelizing the execution of encoding instances.
The dependent representations can be started after a delay of one frame execution time of
the reference representation.
Further work is required to extend the algorithms to resolutions, which are non-integer
power-of-2 multiples of each other.
Further work is required to investigate more recent codecs, e.g., VVC.
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 25

Conclusions and Future Directions
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Hadi Amirpour (hadi.amirpourazarian@aau.at)
Christian Timmerer (Christian.Timmerer@aau.at)
Mohammad Ghanbari (ghan@essex.ac.uk)
Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 26