In live streaming applications, service providers generally use a bitrate ladder with fixed bitrate-resolution pairs instead of optimizing it per title to avoid the additional latency caused to find optimum bitrate-resolution pairs for every video content. This paper introduces an online bitrate ladder construction scheme for live video streaming applications. In this scheme, each target bitrate's optimized resolution is determined from any pre-defined set of resolutions using Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for each video segment. Experimental results show that, on average, the proposed scheme yields significant bitrate savings while maintaining the same quality, compared to the HLS fixed bitrate ladder scheme without any noticeable additional latency in streaming.

1. Efficient Bitrate Ladder Construction for Live Video Streaming
Vignesh V Menon 1
Hadi Amirpour 1
Mohammad Ghanbari 1,2
Christian Timmerer 1
1
Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria
2
School of Computer Science and Electronic Engineering, University of Essex, UK
Submission #144
Introduction
In HTTP Adaptive Streaming (HAS), videos are divided into segments, and each seg‐
ment is encoded at different bitrates and resolutions referred as representations [2]
Figure 1. Rate‐Distortion (RD) curves using VMAF as the quality metric of Beauty and Golf sequences
of UVG and BVI datasets encoded at 540p and 1080p resolutions.
Per‐title encoding schemes are based on the fact that each resolution performs
better than others in a specific region for a given bitrate range, and these regions
depend on the video content.
Each video segment is encoded at several quality levels in this scheme, and bitrate‐
resolution pairs per each quality level and convex‐hull is determined.
The optimal resolution is selected for each bitrate for each segment.
Hence, per‐title encoding schemes lead to a higher Quality of Experience (QoE) or
bitrate saving.
Challenge in per-title encoding for live streaming
Though per‐title encoding schemes [1, 5] enhance the quality of video delivery,
determining convex‐hull is computationally very expensive, making it suitable for
only VoD streaming applications.
Proposed Scheme
Phase 1: Feature extraction: A DCT‐based energy function was introduced in [8, 9]
to determine the block‐wise texture of each frame, which is defined as:
Hp,k =
w
∑
i=1
w
∑
j=1
e|( ij
w2)2
−1|
|DCT(i − 1, j − 1)| (1)
where k is the block address in the pth
frame, w × w pixels is the size of the block,
and DCT(i, j) is the (i, j)th
DCT component when i + j > 2, and 0 otherwise [6].
E =
P−1
∑
p=0
C−1
∑
k=0
Hp,k
P · C · w2
(2)
where C represents the number of blocks per frame, and P denotes the number of
frames in the segment. The block‐wise SAD of the texture energy of each frame
compared to its previous frame is computed and then averaged for each frame of
the segment to obtain the average temporal energy (h) as follows:
h =
P−1
∑
p=1
C−1
∑
k=0
SAD(Hp,k, Hp−1,k)
(P − 1) · C · w2
(3)
Input
Segment
Feature
Extraction
Resolution
Prediction
Resolutions
(R)
Bitrates (B)
Encoding
Figure 2. The architecture of the proposed scheme.
Bitrate-ladder Prediction
Inputs:
f : original framerate
rmax : original spatial resolution
R : set of all resolutions
B : set of all target bitrates
Output: r̂(b) ∀ b ∈ B
Compute E, h features.
for each b ∈ B do
Determine the optimized resolution.
ŝ(b) = 1 − s0e−
ΓMA(rmax,f)·h·b
E
r̂(b) = ŝ(b) · rmax
Map r̂(b) to its closest value in R.
Determining ΓMA
The half‐life of the (1 − s) function is evaluated, i.e., the bitrate when (1 − s)
becomes 1
2. (1 − s) is an exponentially decaying (decreasing) function where:
b1
2
=
ln(2)
K
(4)
ΓMA can thus be determined as:
ΓMA =
ln(2) · E
h · b1
2
(5)
ΓMA values obtained for the training sequences of each resolution and framerate
is averaged to determine ΓMA(rmax, f)
Evaluation
E and h features are extracted using VCA open‐source software. The source‐code
is available at https:/
/github.com/cd‐athena/VCA.
The prediction accuracy of the resolution prediction algorithm is determined by
the L2 norm of the sG (ground truth) and the selected resolution scaling factor
ŝ(b), i.e., || sG − ŝ(b) ||2. It ranges from 0.01 to 0.04, which is negligible.
The resulting quality in PSNR and VMAF [7], and the achieved bitrate (in terms
of Bjøntegaard delta rates [3] BDRP and BDRV ) are compared. The proposed
scheme yields a BDRP of ‐20.45% and BDRV of ‐28.45%, respectively.
Table 1. Results of the proposed scheme against HLS bitrate ladder [10]
Dataset Video f E h || sG − s ||2 BDRV BDRP
MCML [4] Bunny 30 23.03 4.88 0.01 ‐39.48% ‐32.25%
MCML Characters 30 41.44 29.21 0.04 ‐51.90% ‐68.81%
MCML Crowd 30 33.11 12.22 0.01 ‐29.82% ‐14.18%
MCML Dolls 30 10.47 0.27 0.02 ‐1.43% ‐8.49%
SJTU [11] BundNightScape 30 54.90 11.62 0.02 ‐61.22% ‐60.86%
SJTU Fountains 30 60.90 23.02 0.02 ‐32.93% ‐8.49%
VQEG CrowdRun 50 96.55 33.33 0.01 ‐8.50% ‐1.90%
VQEG DucksTakeOff 50 119.12 30.88 0.01 ‐2.99% ‐2.79%
VQEG IntoTree 50 24.41 21.95 0.03 ‐26.50% ‐5.75%
VQEG OldTownCross 50 92.75 22.06 0.02 ‐30.91% ‐22.53%
VQEG ParkJoy 50 102.80 52.15 0.02 ‐12.08% ‐2.62%
Average 0.02 ‐28.45% ‐20.45%
0.0 2.5 5.0 7.5 10.0 12.5 15.0
Bitrate (in Mbps)
30
40
50
60
70
80
90
100
VMAF
Default
OPTE
(a) BundNightScape
0.0 2.5 5.0 7.5 10.0 12.5 15.0
Bitrate (in Mbps)
30
40
50
60
70
80
90
100
VMAF
Default
OPTE
(b) Bunny
Figure 3. Comparison of RD curves for encoding the first segment of BundNightScape and Bunny
sequences using the fixed bitrate ladder and the proposed scheme.
Acknowledgment
The financial support of the Austrian Federal Ministry for Digital and Economic Af‐
fairs, the National Foundation for Research, Technology, and Development, and
the Christian Doppler Research Association is gratefully acknowledged. Christian
Doppler Laboratory ATHENA: https://athena.itec.aau.at/.
References
[1] Hadi Amirpour et al. PSTR: Per‐Title Encoding Using Spatio‐Temporal Resolutions. In 2021 IEEE International
Conference on Multimedia and Expo (ICME), pages 1–6, 2021.
[2] A. Bentaleb et al. A survey on bitrate adaptation schemes for streaming media over http. IEEE Communications
Surveys Tutorials, 21(1):562–585, 2019.
[3] G. Bjontegaard. Calculation of average PSNR differences between RD‐curves. VCEG‐M33, 2001.
[4] Manri Cheon and Jong‐Seok Lee. Subjective and Objective Quality Assessment of Compressed 4K UHD Videos
for Immersive Experience. IEEE Transactions on Circuits and Systems for Video Technology, 28(7):1467–1480, 2018.
[5] J. De Cock et al. Complexity‐based consistent‐quality encoding in the cloud. In 2016 IEEE International Conference
on Image Processing (ICIP), 2016.
[6] Michael King et al. A New Energy Function for Segmentation and Compression. In 2007 IEEE International
Conference on Multimedia and Expo, pages 1647–1650, 2007.
[7] Zhi Li et al. VMAF: The journey continues. Netflix Technology Blog, 25, 2018.
[8] Vignesh V Menon et al. Efficient Content‐Adaptive Feature‐Based Shot Detection for HTTP Adaptive Streaming.
In 2021 IEEE International Conference on Image Processing (ICIP), pages 2174–2178, 2021.
[9] Vignesh V Menon et al. CODA: Content‐aware Frame Dropping Algorithm for High Frame‐rate Video Streaming.
In 2022 Data Compression Conference (DCC), 2022.
[10] R. Pantos, Ed. and W. May. HTTP Live Streaming. RFC 8216, https://www.rfc‐editor.org/info/rfc8216, August 2017.
[11] L. Song et al. The SJTU 4K Video Sequence Dataset. Fifth International Workshop on Quality of Multimedia
Experience (QoMEX2013), July 2013.
https:/
/www.athena.itec.aau.at Mile High Video (MHV) 2022, Denver, USA vignesh.menon@aau.at