In HTTP adaptive live streaming applications, video segments are encoded at a fixed set of bitrate-resolution pairs known as bitrate ladder. Live encoders use the fastest available encoding configuration, referred to as preset, to ensure the minimum possible latency in video encoding. However, an optimized preset and optimized number of CPU threads for each encoding instance may result in (i) increased quality and (ii) efficient CPU utilization while encoding. For low latency live encoders, the encoding speed is expected to be more than or equal to the video framerate. To this light, this paper introduces a Just Noticeable Difference (JND)-Aware Low latency Encoding Scheme (JALE), which uses random forest-based models to jointly determine the optimized encoder preset and thread count for each representation, based on video complexity features, the target encoding speed, the total number of available CPU threads, and the target encoder. Experimental results show that, on average, JALE yield a quality improvement of 1.32 dB PSNR and 5.38 VMAF points with the same bitrate, compared to the fastest preset encoding of the HTTP Live Streaming (HLS) bitrate ladder using x265 HEVC open-source encoder with eight CPU threads used for each representation. These enhancements are achieved while maintaining the desired encoding speed. Furthermore, on average, JALE results in an overall storage reduction of 72.70%, a reduction in the total number of CPU threads used by 63.83%, and a 37.87% reduction in the overall encoding time, considering a JND of six VMAF points.

1. Optimal Quality and Efficiency in Adaptive Live Streaming with
JND-Aware Low latency Encoding
Vignesh V Menon1
, Jingwen Zhu2
, Prajit T Rajendran3
, Samira Afzal4
,
Klaus Schoeffmann4
, Patrick Le Callet2
, Christian Timmerer1
1
Video Communication and Applications Dept., Fraunhofer HHI, Berlin, Germany
2
Ecole Centrale Nantes, CNRS, LS2N, UMR 6004, Nantes Universite, Nantes, France
3
CEA, List, F-91120 Palaiseau, Université Paris-Saclay, France
4
Alpen-Adria-Universität, Klagenfurt, Austria
13 Feb 2024
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 1

2. Outline
1 Introduction
2 JND-Aware Low latency Encoding (JALE)
3 Experimental validation
4 Conclusions
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 2

3. Introduction
Introduction
The cloud server can significantly streamline the encoding process by dynamically adjusting CPU
thread counts based on resolution and bitrate, accommodating diverse video qualities within an
adaptive streaming environment.
01 02 03 04 05 06 07 08 09 10 11 12
Representation ID
15
30
60
120
240
480
Encoding
speed
(in
fps)
4 threads
8 threads
16 threads
Figure: The encoding speed of each representation in HLS bitrate ladder1
for the Wood s000 sequence2
using ultrafast preset of x2653
using 4, 8, and 16 CPU threads for each representation.
1
Apple Inc. “HLS Authoring Specification for Apple Devices”. In: url:
https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices.
2
Hadi Amirpour et al. “VCD: Video Complexity Dataset”. In: Proceedings of the 13th ACM Multimedia Systems Conference. New York, NY, USA: Association
for Computing Machinery, 2022, 234–239. isbn: 9781450392839. doi: 10.1145/3524273.3532892. url: https://doi.org/10.1145/3524273.3532892.
3
VideoLAN. “x265”. In: url: https://www.videolan.org/developers/x265.html.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 3

4. Introduction
Optimized encoding preset
Traditional open-source encoders like x264,4 x265, and VVenC5 have pre-defined sets of
encoding parameters (termed as presets), which present a trade-off between the encoding
time and compression efficiency.6
If the encoder preset is configured such that this higher encoding speed can be reduced while
still being compatible with the expected live encoding speed, the quality of the encoded
content achieved by the encoder can be improved.
Subsequently, when the content becomes complex again, the encoder preset needs to be
reconfigured to move back to the faster configuration that achieves live encoding speed.7
By employing efficient storage techniques and removing unnecessary representations, the energy
consumption associated with storing and transmitting redundant data can be minimized.8
4
VideoLAN. “x264”. In: url: https://www.videolan.org/developers/x264.html.
5
Adam Wieckowski et al. “VVenC: An Open And Optimized VVC Encoder Implementation”. In: Proc. IEEE International Conference on Multimedia Expo
Workshops (ICMEW). 2021, pp. 1–2. doi: 10.1109/ICMEW53276.2021.9455944.
6
Dieison Silveira, Marcelo Porto, and Sergio Bampi. “Performance and energy consumption analysis of the X265 video encoder”. In: 2017 25th European
Signal Processing Conference (EUSIPCO). 2017, pp. 1519–1523. doi: 10.23919/EUSIPCO.2017.8081463.
7
Sergey Zvezdakov, Denis Kondranin, and Dmitriy Vatolin. “Machine-Learning-Based Method for Content-Adaptive Video Encoding”. In: 2021 Picture Coding
Symposium (PCS). 2021, pp. 1–5. doi: 10.1109/PCS50896.2021.9477507.
8
Quortex. Mission: Emission. https://www.quortex.io/wp-content/uploads/2022/05/WhitePaper-Mission_-Emission.pdf. [Accessed May 2023]. 2022.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 4

5. JND-Aware Low latency Encoding (JALE)
JALE
Figure: Live encoding using JALE envisioned in this paper.
JALE is classified into three steps:
1 video complexity feature extraction,
2 joint thread count and preset prediction,
3 perceptually-redundant representation elimination.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 5

6. JND-Aware Low latency Encoding (JALE) Spatiotemporal complexity feature extraction
Spatiotemporal complexity feature extraction
Three DCT-energy-based features, the average luma texture energy EY, the average gradient of
the luma texture energy h, and the average luminescence LYare used.9,10
(a) original frame (b) EY (c) h (d) LY
Figure: Heatmap depiction of the luma texture information {EY, h, LY} extracted from the second frame
of CoverSong 1080P 0a86 video of Youtube UGC Dataset.
9
Vignesh V Menon et al. “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming”. In: Proceedings of the First International
Workshop on Green Multimedia Systems. 2023, 16–18. isbn: 9798400701962. doi: 10.1145/3593908.3593942. url:
https://doi.org/10.1145/3593908.3593942.
10
Vignesh V Menon et al. “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”. In: IEEE Transactions on Circuits and Systems for
Video Technology. 2023, pp. 1–1. doi: 10.1109/TCSVT.2023.3290725. url: https://doi.org/10.1109/TCSVT.2023.3290725.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 6

7. JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction
Joint thread count and preset prediction
Selection of the optimized thread count-preset pair for each segment per representation based
on the video content complexity is decomposed into two parts:
1 train models to predict the encoding speed for each thread count-preset pair,
2 develop a function to obtain the optimized thread count- preset pair for each representation.
The encoding speed of the tth representation of the input video segment (st) is modeled as:
st = fS(EY, h, LY, rt, bt, nt, pt) (1)
We use random forest models11 to predict the encoding speed for each thread count-preset pair.
(PT × CT) models are trained, where PT and CT represent the number of encoding presets and
the number of supported thread count per instance, respectively.
11
Leo Breiman. “Random Forests”. In: Machine Learning. Vol. 45. 2001. doi: 10.1023/A:1010933404324. url:
https://doi.org/10.1023/A:1010933404324.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 7

8. JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction
Joint thread count and preset prediction
The optimized thread count-preset prediction function has a look-up table of (n̂t, p̂t) pairs.
(4, medium)
(8,medium)
(12,medium)
(16,medium)
(20,medium)
(24,medium)
(4,fast)
(8,fast)
(12,fast)
(16,fast)
(20,fast)
(24,fast)
(4,faster)
(8,faster)
(12,faster)
(16,faster)
(20,faster)
(24,faster)
(4,veryfast)
(8,veryfast)
(12,veryfast)
(16,veryfast)
(20,veryfast)
(24,veryfast)
(4,superfast)
(8,superfast)
(12,superfast)
(16,superfast)
(20,superfast)
(24,superfast)
(4,ultrafast)
(8,ultrafast)
(12,ultrafast)
(16,ultrafast)
(20,ultrafast)
(24,ultrafast)
Figure: (n̂, p̂) look-up table used in the experimental validation of this paper.
The priority of (n̂t, p̂t) pairs is decided based on the following constraints:
1 the achieved encoding speed ŝt of the tth representation must be greater than or equal to
the target encoding speed sT, i.e., ŝt ≥ sT.
2 total number of CPU threads used for each representation is minimized.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 8

9. JND-Aware Low latency Encoding (JALE) Perceptually-redundant representation elimination
Perceptually-redundant representation elimination
Input:
q : number of representations in R
R =
Sq
t=1 {(rt , bt , n̂t , p̂t )}: representations with predicted thread count and preset
v̂t ; 1 ≤ t ≤ q: predicted VMAF
vT : maximum VMAF threshold
vJ : average target JND
Output: R̂ = (r, b, n̂, p̂): set of encoding configurations
R̂ ← {(r1, b1, n̂1, p̂1)}
u ← 1
if v̂1 ≥ vT then
return R̂
t ← 2
while t ≤ q do
if v̂t − v̂u ≥ vJ then
R̂ ← R̂ ∪ {(rt , bt , n̂t , p̂t )}
u ← t
if v̂t ≥ vT then
return R̂
t ← t + 1
return R̂
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 9

10. Experimental validation
Experimental parameters
Table: Experimental parameters of JALE used in this paper.
Parameter Symbol Values
Set of representations
Resolution height [pixels]
R
360 432 540 540 540 720 720 1080 1080 1440 2160 2160
Bitrate [Mbps] 0.145 0.300 0.600 0.900 1.600 2.400 3.400 4.500 5.800 8.100 11.600 16.800
Set of presets [x265] P 0 (ultrafast) – 5 (medium)
Set of supported thread counts C 4 8 12 16 20 24
Total CPU threads N 96
Encoding speed threshold [fps] sT 30
Average target JND vJ 2 4 6
Maximum VMAF threshold vT 98 96 94
Benchmark schemes:
1 Default: ultrafast preset with eight threads for each encoding instance.12
2 Bruteforce: optimized thread count-preset pair with and without JND-based representation elimi-
nation when the models are fully accurate. This is accomplished by bruteforce encoding using all
thread count-preset pairs and selecting the optimized pair.13
Hence, it is suitable only for video-on-
demand applications.
3 CAPS14
determines the optimized preset for each representation for a target encoding speed of 30 fps.
We evaluate CAPS where c=4, 8, and 16, respectively.
12
Apple Inc., “HLS Authoring Specification for Apple Devices”.
13
Jan De Cock et al. “Complexity-based consistent-quality encoding in the cloud”. In: 2016 IEEE International Conference on Image Processing (ICIP). 2016,
pp. 1484–1488. doi: 10.1109/ICIP.2016.7532605.
14
Vignesh V Menon et al. “Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming”. In: 2022 Picture Coding Symposium (PCS). 2022,
pp. 253–257. doi: 10.1109/PCS56426.2022.10018034.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 10

11. Experimental validation
Results
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
0
1
2
3
4
5
Preset
4
8
12
16
20
24
Threads
(a) average thread count-preset
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
CAPS (c=16)
JALE
(b) average encoding speed
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
Encoding
speed
(in
fps)
Default
Bruteforce
CAPS (c=8)
CAPS (c=16)
JALE
(c) average VMAF
Figure: Results for each representation in JALE. JND-based representation elimination is not considered
in these plots.
Faster encoding presets and more computational resources are needed to encode high-
bitrate representations so that the encoding speed is above the threshold.
JALE controls the encoding speed to be greater than sT but not significantly higher than
the default encoding. This ensures higher CPU utilization when the encodings are carried
out concurrently during a live feed.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 11

12. Experimental validation
Results
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(a) Basketball s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(b) Characters s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(c) Dolls s001
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(d) Maples s000
Figure: Rate-distortion (RD) curves of representative sequences (segments) for Default encoding (blue
line), CAPS (c = 8) encoding (red line), compared to JALE (vJ =6).
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 12

13. Experimental validation
Results
Table: Average results of the encoding schemes compared to the default encoding.
Method vJ BDRP BDRV BD-PSNR BD-VMAF ∆S ∆N ∆E
[%] [%] [dB] [%] [%] [%]
Bruteforce
2 -23.85 -23.02 1.21 4.61 -51.32 - 1688.91
4 -24.54 -24.27 1.25 5.26 -62.13 - 1688.91
6 -26.30 -25.77 1.32 5.53 -72.03 - 1688.91
CAPS (c = 4) - -12.31 -14.98 0.82 2.91 -0.06 -50.00 8.65
CAPS (c = 8) - -20.26 -20.95 1.06 4.03 -0.09 0 39.68
CAPS (c = 16) - -28.13 -29.35 1.62 6.04 -0.35 100.00 96.84
JALE
2 -23.30 -22.80 1.19 4.47 -51.34 -43.13 -2.69
4 -24.41 -24.08 1.25 5.05 -62.95 -54.08 -22.70
6 -25.93 -25.47 1.32 5.38 -72.70 -63.83 -37.87
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 13

14. Conclusions
Conclusions
We proposed JALE, a JND-aware low latency encoding scheme for adaptive live streaming
applications.
JALE jointly predicts the optimized encoder preset and CPU thread count for a given
representation for each video segment based on the video content complexity features,
target encoding speed, and the total number of available CPU threads.
The JND-based representation elimination algorithm removes perceptually redundant rep-
resentations in the bitrate ladder.
JALE yields an overall average quality improvement of 0.98 dB PSNR and 4.41 VMAF
points at the same bitrate, compared to the x265 ultrafast encoding of the reference HLS
bitrate ladder using eight CPU threads for each representation.
Considering a JND of six VMAF points, storage, thread count, and encoding time reductions
of 72.70 %, 63.83 %, and 37.87 %, respectively, are observed.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 14

15. Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 15