SlideShare a Scribd company logo
1 of 15
Download to read offline
Optimal Quality and Efficiency in Adaptive Live Streaming with
JND-Aware Low latency Encoding
Vignesh V Menon1
, Jingwen Zhu2
, Prajit T Rajendran3
, Samira Afzal4
,
Klaus Schoeffmann4
, Patrick Le Callet2
, Christian Timmerer1
1
Video Communication and Applications Dept., Fraunhofer HHI, Berlin, Germany
2
Ecole Centrale Nantes, CNRS, LS2N, UMR 6004, Nantes Universite, Nantes, France
3
CEA, List, F-91120 Palaiseau, Université Paris-Saclay, France
4
Alpen-Adria-Universität, Klagenfurt, Austria
13 Feb 2024
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 1
Outline
1 Introduction
2 JND-Aware Low latency Encoding (JALE)
3 Experimental validation
4 Conclusions
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 2
Introduction
Introduction
The cloud server can significantly streamline the encoding process by dynamically adjusting CPU
thread counts based on resolution and bitrate, accommodating diverse video qualities within an
adaptive streaming environment.
01 02 03 04 05 06 07 08 09 10 11 12
Representation ID
15
30
60
120
240
480
Encoding
speed
(in
fps)
4 threads
8 threads
16 threads
Figure: The encoding speed of each representation in HLS bitrate ladder1
for the Wood s000 sequence2
using ultrafast preset of x2653
using 4, 8, and 16 CPU threads for each representation.
1
Apple Inc. “HLS Authoring Specification for Apple Devices”. In: url:
https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices.
2
Hadi Amirpour et al. “VCD: Video Complexity Dataset”. In: Proceedings of the 13th ACM Multimedia Systems Conference. New York, NY, USA: Association
for Computing Machinery, 2022, 234–239. isbn: 9781450392839. doi: 10.1145/3524273.3532892. url: https://doi.org/10.1145/3524273.3532892.
3
VideoLAN. “x265”. In: url: https://www.videolan.org/developers/x265.html.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 3
Introduction
Optimized encoding preset
Traditional open-source encoders like x264,4 x265, and VVenC5 have pre-defined sets of
encoding parameters (termed as presets), which present a trade-off between the encoding
time and compression efficiency.6
If the encoder preset is configured such that this higher encoding speed can be reduced while
still being compatible with the expected live encoding speed, the quality of the encoded
content achieved by the encoder can be improved.
Subsequently, when the content becomes complex again, the encoder preset needs to be
reconfigured to move back to the faster configuration that achieves live encoding speed.7
By employing efficient storage techniques and removing unnecessary representations, the energy
consumption associated with storing and transmitting redundant data can be minimized.8
4
VideoLAN. “x264”. In: url: https://www.videolan.org/developers/x264.html.
5
Adam Wieckowski et al. “VVenC: An Open And Optimized VVC Encoder Implementation”. In: Proc. IEEE International Conference on Multimedia Expo
Workshops (ICMEW). 2021, pp. 1–2. doi: 10.1109/ICMEW53276.2021.9455944.
6
Dieison Silveira, Marcelo Porto, and Sergio Bampi. “Performance and energy consumption analysis of the X265 video encoder”. In: 2017 25th European
Signal Processing Conference (EUSIPCO). 2017, pp. 1519–1523. doi: 10.23919/EUSIPCO.2017.8081463.
7
Sergey Zvezdakov, Denis Kondranin, and Dmitriy Vatolin. “Machine-Learning-Based Method for Content-Adaptive Video Encoding”. In: 2021 Picture Coding
Symposium (PCS). 2021, pp. 1–5. doi: 10.1109/PCS50896.2021.9477507.
8
Quortex. Mission: Emission. https://www.quortex.io/wp-content/uploads/2022/05/WhitePaper-Mission_-Emission.pdf. [Accessed May 2023]. 2022.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 4
JND-Aware Low latency Encoding (JALE)
JALE
Figure: Live encoding using JALE envisioned in this paper.
JALE is classified into three steps:
1 video complexity feature extraction,
2 joint thread count and preset prediction,
3 perceptually-redundant representation elimination.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 5
JND-Aware Low latency Encoding (JALE) Spatiotemporal complexity feature extraction
Spatiotemporal complexity feature extraction
Three DCT-energy-based features, the average luma texture energy EY, the average gradient of
the luma texture energy h, and the average luminescence LYare used.9,10
(a) original frame (b) EY (c) h (d) LY
Figure: Heatmap depiction of the luma texture information {EY, h, LY} extracted from the second frame
of CoverSong 1080P 0a86 video of Youtube UGC Dataset.
9
Vignesh V Menon et al. “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming”. In: Proceedings of the First International
Workshop on Green Multimedia Systems. 2023, 16–18. isbn: 9798400701962. doi: 10.1145/3593908.3593942. url:
https://doi.org/10.1145/3593908.3593942.
10
Vignesh V Menon et al. “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”. In: IEEE Transactions on Circuits and Systems for
Video Technology. 2023, pp. 1–1. doi: 10.1109/TCSVT.2023.3290725. url: https://doi.org/10.1109/TCSVT.2023.3290725.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 6
JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction
Joint thread count and preset prediction
Selection of the optimized thread count-preset pair for each segment per representation based
on the video content complexity is decomposed into two parts:
1 train models to predict the encoding speed for each thread count-preset pair,
2 develop a function to obtain the optimized thread count- preset pair for each representation.
The encoding speed of the tth representation of the input video segment (st) is modeled as:
st = fS(EY, h, LY, rt, bt, nt, pt) (1)
We use random forest models11 to predict the encoding speed for each thread count-preset pair.
(PT × CT) models are trained, where PT and CT represent the number of encoding presets and
the number of supported thread count per instance, respectively.
11
Leo Breiman. “Random Forests”. In: Machine Learning. Vol. 45. 2001. doi: 10.1023/A:1010933404324. url:
https://doi.org/10.1023/A:1010933404324.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 7
JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction
Joint thread count and preset prediction
The optimized thread count-preset prediction function has a look-up table of (n̂t, p̂t) pairs.
(4, medium)
(8,medium)
(12,medium)
(16,medium)
(20,medium)
(24,medium)
(4,fast)
(8,fast)
(12,fast)
(16,fast)
(20,fast)
(24,fast)
(4,faster)
(8,faster)
(12,faster)
(16,faster)
(20,faster)
(24,faster)
(4,veryfast)
(8,veryfast)
(12,veryfast)
(16,veryfast)
(20,veryfast)
(24,veryfast)
(4,superfast)
(8,superfast)
(12,superfast)
(16,superfast)
(20,superfast)
(24,superfast)
(4,ultrafast)
(8,ultrafast)
(12,ultrafast)
(16,ultrafast)
(20,ultrafast)
(24,ultrafast)
Figure: (n̂, p̂) look-up table used in the experimental validation of this paper.
The priority of (n̂t, p̂t) pairs is decided based on the following constraints:
1 the achieved encoding speed ŝt of the tth representation must be greater than or equal to
the target encoding speed sT, i.e., ŝt ≥ sT.
2 total number of CPU threads used for each representation is minimized.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 8
JND-Aware Low latency Encoding (JALE) Perceptually-redundant representation elimination
Perceptually-redundant representation elimination
Input:
q : number of representations in R
R =
Sq
t=1 {(rt , bt , n̂t , p̂t )}: representations with predicted thread count and preset
v̂t ; 1 ≤ t ≤ q: predicted VMAF
vT : maximum VMAF threshold
vJ : average target JND
Output: R̂ = (r, b, n̂, p̂): set of encoding configurations
R̂ ← {(r1, b1, n̂1, p̂1)}
u ← 1
if v̂1 ≥ vT then
return R̂
t ← 2
while t ≤ q do
if v̂t − v̂u ≥ vJ then
R̂ ← R̂ ∪ {(rt , bt , n̂t , p̂t )}
u ← t
if v̂t ≥ vT then
return R̂
t ← t + 1
return R̂
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 9
Experimental validation
Experimental parameters
Table: Experimental parameters of JALE used in this paper.
Parameter Symbol Values
Set of representations
Resolution height [pixels]
R
360 432 540 540 540 720 720 1080 1080 1440 2160 2160
Bitrate [Mbps] 0.145 0.300 0.600 0.900 1.600 2.400 3.400 4.500 5.800 8.100 11.600 16.800
Set of presets [x265] P 0 (ultrafast) – 5 (medium)
Set of supported thread counts C 4 8 12 16 20 24
Total CPU threads N 96
Encoding speed threshold [fps] sT 30
Average target JND vJ 2 4 6
Maximum VMAF threshold vT 98 96 94
Benchmark schemes:
1 Default: ultrafast preset with eight threads for each encoding instance.12
2 Bruteforce: optimized thread count-preset pair with and without JND-based representation elimi-
nation when the models are fully accurate. This is accomplished by bruteforce encoding using all
thread count-preset pairs and selecting the optimized pair.13
Hence, it is suitable only for video-on-
demand applications.
3 CAPS14
determines the optimized preset for each representation for a target encoding speed of 30 fps.
We evaluate CAPS where c=4, 8, and 16, respectively.
12
Apple Inc., “HLS Authoring Specification for Apple Devices”.
13
Jan De Cock et al. “Complexity-based consistent-quality encoding in the cloud”. In: 2016 IEEE International Conference on Image Processing (ICIP). 2016,
pp. 1484–1488. doi: 10.1109/ICIP.2016.7532605.
14
Vignesh V Menon et al. “Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming”. In: 2022 Picture Coding Symposium (PCS). 2022,
pp. 253–257. doi: 10.1109/PCS56426.2022.10018034.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 10
Experimental validation
Results
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
0
1
2
3
4
5
Preset
4
8
12
16
20
24
Threads
(a) average thread count-preset
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
CAPS (c=16)
JALE
(b) average encoding speed
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
Encoding
speed
(in
fps)
Default
Bruteforce
CAPS (c=8)
CAPS (c=16)
JALE
(c) average VMAF
Figure: Results for each representation in JALE. JND-based representation elimination is not considered
in these plots.
Faster encoding presets and more computational resources are needed to encode high-
bitrate representations so that the encoding speed is above the threshold.
JALE controls the encoding speed to be greater than sT but not significantly higher than
the default encoding. This ensures higher CPU utilization when the encodings are carried
out concurrently during a live feed.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 11
Experimental validation
Results
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(a) Basketball s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(b) Characters s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(c) Dolls s001
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
Default
Bruteforce (vJ=6)
CAPS (c=8)
JALE (vJ=6)
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
80
160
320
Encoding
speed
(in
fps)
Default
CAPS (c=8)
JALE (vJ=6)
(d) Maples s000
Figure: Rate-distortion (RD) curves of representative sequences (segments) for Default encoding (blue
line), CAPS (c = 8) encoding (red line), compared to JALE (vJ =6).
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 12
Experimental validation
Results
Table: Average results of the encoding schemes compared to the default encoding.
Method vJ BDRP BDRV BD-PSNR BD-VMAF ∆S ∆N ∆E
[%] [%] [dB] [%] [%] [%]
Bruteforce
2 -23.85 -23.02 1.21 4.61 -51.32 - 1688.91
4 -24.54 -24.27 1.25 5.26 -62.13 - 1688.91
6 -26.30 -25.77 1.32 5.53 -72.03 - 1688.91
CAPS (c = 4) - -12.31 -14.98 0.82 2.91 -0.06 -50.00 8.65
CAPS (c = 8) - -20.26 -20.95 1.06 4.03 -0.09 0 39.68
CAPS (c = 16) - -28.13 -29.35 1.62 6.04 -0.35 100.00 96.84
JALE
2 -23.30 -22.80 1.19 4.47 -51.34 -43.13 -2.69
4 -24.41 -24.08 1.25 5.05 -62.95 -54.08 -22.70
6 -25.93 -25.47 1.32 5.38 -72.70 -63.83 -37.87
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 13
Conclusions
Conclusions
We proposed JALE, a JND-aware low latency encoding scheme for adaptive live streaming
applications.
JALE jointly predicts the optimized encoder preset and CPU thread count for a given
representation for each video segment based on the video content complexity features,
target encoding speed, and the total number of available CPU threads.
The JND-based representation elimination algorithm removes perceptually redundant rep-
resentations in the bitrate ladder.
JALE yields an overall average quality improvement of 0.98 dB PSNR and 4.41 VMAF
points at the same bitrate, compared to the x265 ultrafast encoding of the reference HLS
bitrate ladder using eight CPU threads for each representation.
Considering a JND of six VMAF points, storage, thread count, and encoding time reductions
of 72.70 %, 63.83 %, and 37.87 %, respectively, are observed.
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 14
Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 15

More Related Content

Similar to Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding

Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Alpen-Adria-Universität
 
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Vignesh V Menon
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video StreamingPerceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video StreamingAlpen-Adria-Universität
 
Green_VCA_presentation.pdf
Green_VCA_presentation.pdfGreen_VCA_presentation.pdf
Green_VCA_presentation.pdfVignesh V Menon
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfVignesh V Menon
 
Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingVignesh V Menon
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfVignesh V Menon
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Ijripublishers Ijri
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Ijripublishers Ijri
 
Machine Learning approaches at video compression
Machine Learning approaches at video compression Machine Learning approaches at video compression
Machine Learning approaches at video compression Roberto Iacoviello
 
HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?Alpen-Adria-Universität
 
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...Minh Nguyen
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCVignesh V Menon
 
INCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVCINCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVCAlpen-Adria-Universität
 
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...Alpen-Adria-Universität
 
Optimizing QoE and Latency of Live Video Streaming Using Edge Computing a...
Optimizing  QoE and Latency of  Live Video Streaming Using  Edge Computing  a...Optimizing  QoE and Latency of  Live Video Streaming Using  Edge Computing  a...
Optimizing QoE and Latency of Live Video Streaming Using Edge Computing a...Alpen-Adria-Universität
 

Similar to Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding (20)

Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
 
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
 
TQPM.pdf
TQPM.pdfTQPM.pdf
TQPM.pdf
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video StreamingPerceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
 
Green_VCA_presentation.pdf
Green_VCA_presentation.pdfGreen_VCA_presentation.pdf
Green_VCA_presentation.pdf
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
 
Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC Streaming
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
Machine Learning approaches at video compression
Machine Learning approaches at video compression Machine Learning approaches at video compression
Machine Learning approaches at video compression
 
HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?
 
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
 
INCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVCINCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVC
 
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
 
C0161018
C0161018C0161018
C0161018
 
C0161018
C0161018C0161018
C0161018
 
Optimizing QoE and Latency of Live Video Streaming Using Edge Computing a...
Optimizing  QoE and Latency of  Live Video Streaming Using  Edge Computing  a...Optimizing  QoE and Latency of  Live Video Streaming Using  Edge Computing  a...
Optimizing QoE and Latency of Live Video Streaming Using Edge Computing a...
 

More from Vignesh V Menon

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Vignesh V Menon
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfVignesh V Menon
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live StreamingVignesh V Menon
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfVignesh V Menon
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfVignesh V Menon
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVignesh V Menon
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingVignesh V Menon
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...Vignesh V Menon
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...Vignesh V Menon
 

More from Vignesh V Menon (9)

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
 

Recently uploaded

How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17Celine George
 
Application of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matricesApplication of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matricesRased Khan
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Mohamed Rizk Khodair
 
2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptxmansk2
 
How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17Celine George
 
Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxjmorse8
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024CapitolTechU
 
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General QuizPragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General QuizPragya - UEM Kolkata Quiz Club
 
How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17Celine George
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽中 央社
 
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...Nguyen Thanh Tu Collection
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesashishpaul799
 
Behavioral-sciences-dr-mowadat rana (1).pdf
Behavioral-sciences-dr-mowadat rana (1).pdfBehavioral-sciences-dr-mowadat rana (1).pdf
Behavioral-sciences-dr-mowadat rana (1).pdfaedhbteg
 
The Ball Poem- John Berryman_20240518_001617_0000.pptx
The Ball Poem- John Berryman_20240518_001617_0000.pptxThe Ball Poem- John Berryman_20240518_001617_0000.pptx
The Ball Poem- John Berryman_20240518_001617_0000.pptxNehaChandwani11
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfmstarkes24
 
Discover the Dark Web .pdf InfosecTrain
Discover the Dark Web .pdf  InfosecTrainDiscover the Dark Web .pdf  InfosecTrain
Discover the Dark Web .pdf InfosecTraininfosec train
 
The Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. HenryThe Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. HenryEugene Lysak
 
An Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxAn Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxCeline George
 

Recently uploaded (20)

How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17
 
Application of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matricesApplication of Matrices in real life. Presentation on application of matrices
Application of Matrices in real life. Presentation on application of matrices
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).
 
2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx
 
How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17
 
Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptx
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024
 
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General QuizPragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
Pragya Champions Chalice 2024 Prelims & Finals Q/A set, General Quiz
 
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
Operations Management - Book1.p  - Dr. Abdulfatah A. SalemOperations Management - Book1.p  - Dr. Abdulfatah A. Salem
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
 
How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17How to Analyse Profit of a Sales Order in Odoo 17
How to Analyse Profit of a Sales Order in Odoo 17
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
 
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
BỘ LUYỆN NGHE TIẾNG ANH 8 GLOBAL SUCCESS CẢ NĂM (GỒM 12 UNITS, MỖI UNIT GỒM 3...
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
 
Behavioral-sciences-dr-mowadat rana (1).pdf
Behavioral-sciences-dr-mowadat rana (1).pdfBehavioral-sciences-dr-mowadat rana (1).pdf
Behavioral-sciences-dr-mowadat rana (1).pdf
 
The Ball Poem- John Berryman_20240518_001617_0000.pptx
The Ball Poem- John Berryman_20240518_001617_0000.pptxThe Ball Poem- John Berryman_20240518_001617_0000.pptx
The Ball Poem- John Berryman_20240518_001617_0000.pptx
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
Discover the Dark Web .pdf InfosecTrain
Discover the Dark Web .pdf  InfosecTrainDiscover the Dark Web .pdf  InfosecTrain
Discover the Dark Web .pdf InfosecTrain
 
The Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. HenryThe Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. Henry
 
“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
An Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxAn Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptx
 

Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding

  • 1. Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low latency Encoding Vignesh V Menon1 , Jingwen Zhu2 , Prajit T Rajendran3 , Samira Afzal4 , Klaus Schoeffmann4 , Patrick Le Callet2 , Christian Timmerer1 1 Video Communication and Applications Dept., Fraunhofer HHI, Berlin, Germany 2 Ecole Centrale Nantes, CNRS, LS2N, UMR 6004, Nantes Universite, Nantes, France 3 CEA, List, F-91120 Palaiseau, Université Paris-Saclay, France 4 Alpen-Adria-Universität, Klagenfurt, Austria 13 Feb 2024 Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 1
  • 2. Outline 1 Introduction 2 JND-Aware Low latency Encoding (JALE) 3 Experimental validation 4 Conclusions Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 2
  • 3. Introduction Introduction The cloud server can significantly streamline the encoding process by dynamically adjusting CPU thread counts based on resolution and bitrate, accommodating diverse video qualities within an adaptive streaming environment. 01 02 03 04 05 06 07 08 09 10 11 12 Representation ID 15 30 60 120 240 480 Encoding speed (in fps) 4 threads 8 threads 16 threads Figure: The encoding speed of each representation in HLS bitrate ladder1 for the Wood s000 sequence2 using ultrafast preset of x2653 using 4, 8, and 16 CPU threads for each representation. 1 Apple Inc. “HLS Authoring Specification for Apple Devices”. In: url: https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices. 2 Hadi Amirpour et al. “VCD: Video Complexity Dataset”. In: Proceedings of the 13th ACM Multimedia Systems Conference. New York, NY, USA: Association for Computing Machinery, 2022, 234–239. isbn: 9781450392839. doi: 10.1145/3524273.3532892. url: https://doi.org/10.1145/3524273.3532892. 3 VideoLAN. “x265”. In: url: https://www.videolan.org/developers/x265.html. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 3
  • 4. Introduction Optimized encoding preset Traditional open-source encoders like x264,4 x265, and VVenC5 have pre-defined sets of encoding parameters (termed as presets), which present a trade-off between the encoding time and compression efficiency.6 If the encoder preset is configured such that this higher encoding speed can be reduced while still being compatible with the expected live encoding speed, the quality of the encoded content achieved by the encoder can be improved. Subsequently, when the content becomes complex again, the encoder preset needs to be reconfigured to move back to the faster configuration that achieves live encoding speed.7 By employing efficient storage techniques and removing unnecessary representations, the energy consumption associated with storing and transmitting redundant data can be minimized.8 4 VideoLAN. “x264”. In: url: https://www.videolan.org/developers/x264.html. 5 Adam Wieckowski et al. “VVenC: An Open And Optimized VVC Encoder Implementation”. In: Proc. IEEE International Conference on Multimedia Expo Workshops (ICMEW). 2021, pp. 1–2. doi: 10.1109/ICMEW53276.2021.9455944. 6 Dieison Silveira, Marcelo Porto, and Sergio Bampi. “Performance and energy consumption analysis of the X265 video encoder”. In: 2017 25th European Signal Processing Conference (EUSIPCO). 2017, pp. 1519–1523. doi: 10.23919/EUSIPCO.2017.8081463. 7 Sergey Zvezdakov, Denis Kondranin, and Dmitriy Vatolin. “Machine-Learning-Based Method for Content-Adaptive Video Encoding”. In: 2021 Picture Coding Symposium (PCS). 2021, pp. 1–5. doi: 10.1109/PCS50896.2021.9477507. 8 Quortex. Mission: Emission. https://www.quortex.io/wp-content/uploads/2022/05/WhitePaper-Mission_-Emission.pdf. [Accessed May 2023]. 2022. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 4
  • 5. JND-Aware Low latency Encoding (JALE) JALE Figure: Live encoding using JALE envisioned in this paper. JALE is classified into three steps: 1 video complexity feature extraction, 2 joint thread count and preset prediction, 3 perceptually-redundant representation elimination. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 5
  • 6. JND-Aware Low latency Encoding (JALE) Spatiotemporal complexity feature extraction Spatiotemporal complexity feature extraction Three DCT-energy-based features, the average luma texture energy EY, the average gradient of the luma texture energy h, and the average luminescence LYare used.9,10 (a) original frame (b) EY (c) h (d) LY Figure: Heatmap depiction of the luma texture information {EY, h, LY} extracted from the second frame of CoverSong 1080P 0a86 video of Youtube UGC Dataset. 9 Vignesh V Menon et al. “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming”. In: Proceedings of the First International Workshop on Green Multimedia Systems. 2023, 16–18. isbn: 9798400701962. doi: 10.1145/3593908.3593942. url: https://doi.org/10.1145/3593908.3593942. 10 Vignesh V Menon et al. “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”. In: IEEE Transactions on Circuits and Systems for Video Technology. 2023, pp. 1–1. doi: 10.1109/TCSVT.2023.3290725. url: https://doi.org/10.1109/TCSVT.2023.3290725. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 6
  • 7. JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction Joint thread count and preset prediction Selection of the optimized thread count-preset pair for each segment per representation based on the video content complexity is decomposed into two parts: 1 train models to predict the encoding speed for each thread count-preset pair, 2 develop a function to obtain the optimized thread count- preset pair for each representation. The encoding speed of the tth representation of the input video segment (st) is modeled as: st = fS(EY, h, LY, rt, bt, nt, pt) (1) We use random forest models11 to predict the encoding speed for each thread count-preset pair. (PT × CT) models are trained, where PT and CT represent the number of encoding presets and the number of supported thread count per instance, respectively. 11 Leo Breiman. “Random Forests”. In: Machine Learning. Vol. 45. 2001. doi: 10.1023/A:1010933404324. url: https://doi.org/10.1023/A:1010933404324. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 7
  • 8. JND-Aware Low latency Encoding (JALE) Joint thread count and preset prediction Joint thread count and preset prediction The optimized thread count-preset prediction function has a look-up table of (n̂t, p̂t) pairs. (4, medium) (8,medium) (12,medium) (16,medium) (20,medium) (24,medium) (4,fast) (8,fast) (12,fast) (16,fast) (20,fast) (24,fast) (4,faster) (8,faster) (12,faster) (16,faster) (20,faster) (24,faster) (4,veryfast) (8,veryfast) (12,veryfast) (16,veryfast) (20,veryfast) (24,veryfast) (4,superfast) (8,superfast) (12,superfast) (16,superfast) (20,superfast) (24,superfast) (4,ultrafast) (8,ultrafast) (12,ultrafast) (16,ultrafast) (20,ultrafast) (24,ultrafast) Figure: (n̂, p̂) look-up table used in the experimental validation of this paper. The priority of (n̂t, p̂t) pairs is decided based on the following constraints: 1 the achieved encoding speed ŝt of the tth representation must be greater than or equal to the target encoding speed sT, i.e., ŝt ≥ sT. 2 total number of CPU threads used for each representation is minimized. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 8
  • 9. JND-Aware Low latency Encoding (JALE) Perceptually-redundant representation elimination Perceptually-redundant representation elimination Input: q : number of representations in R R = Sq t=1 {(rt , bt , n̂t , p̂t )}: representations with predicted thread count and preset v̂t ; 1 ≤ t ≤ q: predicted VMAF vT : maximum VMAF threshold vJ : average target JND Output: R̂ = (r, b, n̂, p̂): set of encoding configurations R̂ ← {(r1, b1, n̂1, p̂1)} u ← 1 if v̂1 ≥ vT then return R̂ t ← 2 while t ≤ q do if v̂t − v̂u ≥ vJ then R̂ ← R̂ ∪ {(rt , bt , n̂t , p̂t )} u ← t if v̂t ≥ vT then return R̂ t ← t + 1 return R̂ Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 9
  • 10. Experimental validation Experimental parameters Table: Experimental parameters of JALE used in this paper. Parameter Symbol Values Set of representations Resolution height [pixels] R 360 432 540 540 540 720 720 1080 1080 1440 2160 2160 Bitrate [Mbps] 0.145 0.300 0.600 0.900 1.600 2.400 3.400 4.500 5.800 8.100 11.600 16.800 Set of presets [x265] P 0 (ultrafast) – 5 (medium) Set of supported thread counts C 4 8 12 16 20 24 Total CPU threads N 96 Encoding speed threshold [fps] sT 30 Average target JND vJ 2 4 6 Maximum VMAF threshold vT 98 96 94 Benchmark schemes: 1 Default: ultrafast preset with eight threads for each encoding instance.12 2 Bruteforce: optimized thread count-preset pair with and without JND-based representation elimi- nation when the models are fully accurate. This is accomplished by bruteforce encoding using all thread count-preset pairs and selecting the optimized pair.13 Hence, it is suitable only for video-on- demand applications. 3 CAPS14 determines the optimized preset for each representation for a target encoding speed of 30 fps. We evaluate CAPS where c=4, 8, and 16, respectively. 12 Apple Inc., “HLS Authoring Specification for Apple Devices”. 13 Jan De Cock et al. “Complexity-based consistent-quality encoding in the cloud”. In: 2016 IEEE International Conference on Image Processing (ICIP). 2016, pp. 1484–1488. doi: 10.1109/ICIP.2016.7532605. 14 Vignesh V Menon et al. “Content-adaptive Encoder Preset Prediction for Adaptive Live Streaming”. In: 2022 Picture Coding Symposium (PCS). 2022, pp. 253–257. doi: 10.1109/PCS56426.2022.10018034. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 10
  • 11. Experimental validation Results 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 0 1 2 3 4 5 Preset 4 8 12 16 20 24 Threads (a) average thread count-preset 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) CAPS (c=16) JALE (b) average encoding speed 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 Encoding speed (in fps) Default Bruteforce CAPS (c=8) CAPS (c=16) JALE (c) average VMAF Figure: Results for each representation in JALE. JND-based representation elimination is not considered in these plots. Faster encoding presets and more computational resources are needed to encode high- bitrate representations so that the encoding speed is above the threshold. JALE controls the encoding speed to be greater than sT but not significantly higher than the default encoding. This ensures higher CPU utilization when the encodings are carried out concurrently during a live feed. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 11
  • 12. Experimental validation Results 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (a) Basketball s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (b) Characters s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (c) Dolls s001 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF Default Bruteforce (vJ=6) CAPS (c=8) JALE (vJ=6) 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 80 160 320 Encoding speed (in fps) Default CAPS (c=8) JALE (vJ=6) (d) Maples s000 Figure: Rate-distortion (RD) curves of representative sequences (segments) for Default encoding (blue line), CAPS (c = 8) encoding (red line), compared to JALE (vJ =6). Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 12
  • 13. Experimental validation Results Table: Average results of the encoding schemes compared to the default encoding. Method vJ BDRP BDRV BD-PSNR BD-VMAF ∆S ∆N ∆E [%] [%] [dB] [%] [%] [%] Bruteforce 2 -23.85 -23.02 1.21 4.61 -51.32 - 1688.91 4 -24.54 -24.27 1.25 5.26 -62.13 - 1688.91 6 -26.30 -25.77 1.32 5.53 -72.03 - 1688.91 CAPS (c = 4) - -12.31 -14.98 0.82 2.91 -0.06 -50.00 8.65 CAPS (c = 8) - -20.26 -20.95 1.06 4.03 -0.09 0 39.68 CAPS (c = 16) - -28.13 -29.35 1.62 6.04 -0.35 100.00 96.84 JALE 2 -23.30 -22.80 1.19 4.47 -51.34 -43.13 -2.69 4 -24.41 -24.08 1.25 5.05 -62.95 -54.08 -22.70 6 -25.93 -25.47 1.32 5.38 -72.70 -63.83 -37.87 Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 13
  • 14. Conclusions Conclusions We proposed JALE, a JND-aware low latency encoding scheme for adaptive live streaming applications. JALE jointly predicts the optimized encoder preset and CPU thread count for a given representation for each video segment based on the video content complexity features, target encoding speed, and the total number of available CPU threads. The JND-based representation elimination algorithm removes perceptually redundant rep- resentations in the bitrate ladder. JALE yields an overall average quality improvement of 0.98 dB PSNR and 4.41 VMAF points at the same bitrate, compared to the x265 ultrafast encoding of the reference HLS bitrate ladder using eight CPU threads for each representation. Considering a JND of six VMAF points, storage, thread count, and encoding time reductions of 72.70 %, 63.83 %, and 37.87 %, respectively, are observed. Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 14
  • 15. Q & A Q & A Thank you for your attention! Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de) Vignesh V Menon Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resolution Encoding 15