SlideShare a Scribd company logo
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for
Adaptive Video Streaming
Vignesh V Menon1,2, Reza Farahani2, Prajit T Rajendran3,
Samira Afzal2, Klaus Schoeffmann2, Christian Timmerer2
1
Video Communication and Applications Department, Fraunhofer HHI, Berlin, Germany
2
Alpen-Adria-Universität, Klagenfurt, Austria
3
CEA, List, F-91120 Palaiseau, Université Paris-Saclay, France
Dec 6,2023
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 1
Outline
1 Introduction
2 MCBE
3 Evaluation
4 Conclusions
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 2
Introduction
Introduction
Multi-codec streaming ecosystem
In streaming systems, each codec requires its own set of representations, i.e., bitrate ladders.1,2
HEVC
Ladder
AVC
Ladder
AVC
AVC
HEVC
HEVC
Media Server
Original
Video
AV1
AV1
AV1
Ladder
Figure: An example of a multi-codec streaming system.
1
A. Bentaleb et al. “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP”. In: IEEE Communications Surveys Tutorials 21.1 (2019),
pp. 562–585. doi: 10.1109/COMST.2018.2862938.
2
Christian Timmerer, Martin Smole, and Christopher Mueller. “Efficient Multi-Codec Support for OTT Services: HEVC/H. 265 and/or AV1?” In: 2018. url:
http://www.itec.aau.at/bib/files/TimmererC010218.pdf.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 3
Introduction
Introduction
Multi-codec streaming ecosystem
Initially, streaming services used AVC for wider device compatibility.3
As newer devices with HEVC and Alliance for Open Media Video 1 (AV1)4
support becomes preva-
lent, HEVC and AV1-encoded bitrate ladder representations are introduced
Recent years have developed new formats such as Versatile Video Coding (VVC),5
Essential Video
Coding (EVC),6
and Low Complexity Enhancement Video Coding (LCEVC).7
Streaming systems have evolved to accommodate multiple codecs, with older devices relying solely on
AVC, some newer devices using HEVC streams, and certain devices supporting both AVC and HEVC,
including seamlessly switching between them.
3
Yuriy A. Reznik. “Toward Efficient Multicodec Streaming”. In: SMPTE Motion Imaging Journal 132.4 (2023), pp. 16–25. doi: 10.5594/JMI.2023.3263499.
4
Jingning Han et al. “A Technical Overview of AV1”. In: Proceedings of the IEEE 109.9 (2021), pp. 1435–1462. doi: 10.1109/JPROC.2021.3058584.
5
Benjamin Bross et al. “Overview of the Versatile Video Coding (VVC) Standard and its Applications”. In: IEEE Transactions on Circuits and Systems for
Video Technology 31.10 (2021), pp. 3736–3764. doi: 10.1109/TCSVT.2021.3101953.
6
Jonatan Samuelsson et al. “MPEG-5 EVC”. In: SMPTE 2019. 2019, pp. 1–11. doi: 10.5594/M001877.
7
Stefano Battista et al. “Overview of the Low Complexity Enhancement Video Coding (LCEVC) Standard”. In: IEEE Transactions on Circuits and Systems for
Video Technology 32.11 (2022), pp. 7983–7995. doi: 10.1109/TCSVT.2022.3182793.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 4
Introduction
Introduction
Perceptual redundancy
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
40
60
80
100
VMAF
x264
x265
svtav1
(a) Basketball s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
30
40
50
60
70
80
VMAF
x264
x265
svtav1
(b) Riverbank s000
Figure: Rate-distortion (RD) curves of representative sequences of VCD dataset, encoded with JTPS bitrate
ladder8
for x264, x265, and svtav1 encoders.
In some cases, the compression efficiency of AVC is better than new-generation video codecs, i.e.,
at low bitrates.
Compression efficiency of codecs saturates at very high target bitrates, as they become similar to
lossless coding.
The bitrate regions where each codec performs better than others depending on the complexity of
the video content.
8
Vignesh V Menon et al. “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”. In: IEEE Transactions on Circuits and Systems for
Video Technology (2023), pp. 1–1. doi: 10.1109/TCSVT.2023.3290725.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 5
Introduction
Introduction
Energy consumption
Encoding video content into multiple representations in various bitrate-resolution pairs for each
codec results in substantial computational workload and energy consumption.9
Storage and transmission of these representations further contribute to the overall energy consump-
tion.10
When unnecessary high-bitrate representations (of new-generation codecs) are eliminated, the en-
ergy consumption of the streaming system is significantly reduced.11
This is because the energy
consumption of AVC is significantly lower than that of new-generation video codecs.12
As video streaming continues to grow in popularity, finding energy-efficient solutions to optimize the
multi-codec bitrate ladder becomes crucial to mitigate the environmental impact and reduce operational
costs for service providers.
9
Maria G Koziri et al. “Efficient cloud provisioning for video transcoding: Review, open challenges and future opportunities”. In: IEEE Internet Computing 22.5
(2018), pp. 46–55.
10
Jayant Baliga et al. “Green cloud computing: Balancing energy in processing, storage, and transport”. In: Proceedings of the IEEE 99.1 (2010), pp. 149–167.
11
Daniele Lorenzi. “QoE- and Energy-Aware Content Consumption For HTTP Adaptive Streaming”. In: Proceedings of the 14th Conference on ACM
Multimedia Systems. 2023, 348–352. isbn: 9798400701481. doi: 10.1145/3587819.3593029.
12
Thorsten Laude et al. “A Comparison of JEM and AV1 with HEVC: Coding Tools, Coding Efficiency and Complexity”. In: 2018 Picture Coding Symposium
(PCS). 2018, pp. 36–40. doi: 10.1109/PCS.2018.8456291; Isis Bender et al. “Compression Efficiency and Computational Cost Comparison between AV1 and
HEVC Encoders”. In: 2019 27th European Signal Processing Conference (EUSIPCO). 2019, pp. 1–5. doi: 10.23919/EUSIPCO.2019.8903006.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 6
MCBE Architecture
MCBE
Architecture
Input Video
Segment
Encoding
Representations
Spatio-Temporal
Feature Extraction
E h L
Y Y
Set of Target Resolutions (R)
Set of Target Codecs (C)
Redundant Representation
Elimination
r
^
( , b , c
^ ^)
Max Quality and Target JND (v , v )
1
HEVC Bitrate Ladder Estimation AV1 Bitrate Ladder Estimation
AVC Bitrate Ladder Estimation
c
c
c1 2 3
2
3
…
…
max J
Figure: Online encoding using MCBE envisioned in this paper for adaptive video streaming.
MCBE comprises three phases:
1 Spatio-temporal feature extraction
2 Redundant representation elimination
3 Encoding of the segments using the selected bitrate-resolution pairs of each codec
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 7
MCBE Spatio-Temporal Feature Extraction
MCBE
Spatio-Temporal Feature Extraction
MCBE uses the following DCT-energy-based features,13 extracted using open-source VCA v2.0
video complexity analyzer14 for every segment:
1 Average luma texture energy (EY)
2 Average gradient of the luma texture energy (h)
3 Average luminescence (LY)
13
N B Harikrishnan et al. “Comparative evaluation of image compression techniques”. In: 2017 International Conference on Algorithms, Methodology, Models
and Applications in Emerging Technologies (ICAMMAET). 2017, pp. 1–4. doi: 10.1109/ICAMMAET.2017.8186637.
14
Vignesh V Menon et al. “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming”. In: First International ACM Green
Multimedia Systems Workshop (GMSys ’23). 2023. isbn: 9798400701962. doi: 10.1145/3593908.3593942.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 8
MCBE Redundant Representation Elimination
MCBE
Redundant Representation Elimination
VMAF score (vrt ,bt ,c) of the tth representation of the codec c is modeled as a function of
the video content complexity features and the target representation15,16 as shown in the
following equation:
vrt ,bt ,c = fV(EY, h, LY, rt, bt, c) (1)
Random forest models17 which are hyperparameter-tuned with the parameters min samples lea
min samples split=2, n estimators= 100, and max depth=14 are trained for each codec
c ∈ C and resolution r ∈ R to predict VMAF.
15
Vignesh V Menon et al. “Transcoding Quality Prediction for Adaptive Video Streaming”. In: Proceedings of the 2nd Mile-High Video Conference. Denver,
CO, USA, 2023, 103–109. isbn: 9798400701603. doi: 10.1145/3588444.3591012.
16
Vignesh V Menon et al. Video Quality Assessment with Texture Information Fusion for Streaming Applications. 2023. arXiv: 2302.14465 [cs.MM].
17
Leo Breiman. “Random Forests”. In: Machine Learning 45 (2001). doi: 10.1023/A:1010933404324.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 9
MCBE Redundant Representation Elimination
MCBE
Redundant Representation Elimination- Step 1
C : set of all codecs c1, c2...cM in order of priority
Nc : number of representations for codec c
(ˆ
rt, b̂t, c) pairs ∀c ∈ C, t ∈ Nc
Step 1:
for each c ∈ C do
t = 2
while t ≤ Nc do
if v̂c,ˆ
rt ,b̂t
> vmax or v̂c,ˆ
rt ,b̂t
− v̂c,ˆ
rt−1,b̂t−1
< vJ then
Eliminate (ˆ
rt, b̂t, c) from the ladder
N̂c = Nc − 1
t = t + 1
Bitrate ladder representations of each considered codec in C with a perceptual quality difference within a given
Just Noticeable Difference (JND) threshold are eliminated.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 10
MCBE Redundant Representation Elimination
MCBE
Redundant Representation Elimination- Step 2
Q= {(ˆ
rt, b̂t, c1)}, t ∈ N̂c1
Step 2:
for each c ∈ {c2, .., cM} do
for each t ∈ N̂c do
(˜
ri , b̃i , c1) ← arg mini | b̂i,c1 − b̂t | s.t. b̂t ≥ bi,c1
(˜
rj , b̃j , c1) ← arg minj | b̂j,c1 − b̂t | s.t. b̂t ≤ b̂i,c1
RD curve L between (˜
ri , b̃i , c1) and (˜
rj , b̃j , c1): v =
v̂c1,ˆ
rj ,b̂j
−v̂c1,ˆ
ri ,b̂i
b̃j −b̃i
· (b − b̃i ) + v̂c1,ˆ
ri ,b̂i
if (v̂c,ˆ
rt ,b̂t
is above L) then
Add (ˆ
rt, b̂t, c) to Q.
c1 denotes AVC representations. When AVC performs better (in terms of perceptual quality) than or is identical
to HEVC and/or another new-generation codec (e.g., AV1) in a bitrate range, the corresponding new-generation
codec representations are eliminated from the bitrate ladder. This is because clients can be served with AVC
representations with better RD performance.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 11
Evaluation Test Setup
Test Setup
Table: Experimental parameters used to evaluate MCBE.
Parameter Values
C x264 v1.1 x265 v3.5 svtav1 v1.6
R { 360, 432, 540, 720, 1080, 1440, 2160 }
vJ 2 4 6
vmax 98 96 94
CPU threads 8
Bitrate ladder estimation:
1 Default HLS bitrate ladder18
for each codec/encoder.
2 OPTE,19
where optimized resolutions for the set of bitrates in the HLS bitrate ladder are predicted for each
encoder.
3 JTPS,20
where optimized bitrate-resolution pairs are predicted for JND-aware efficient encoding for each
encoder.
18
Apple Inc. “HLS Authoring Specification for Apple Devices”. In: url:
https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices.
19
V. V. Menon et al. “OPTE: Online Per-Title Encoding for Live Video Streaming”. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP). 2022, pp. 1865–1869. doi: 10.1109/ICASSP43922.2022.9746745.
20
Menon et al., “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 12
Evaluation RD curves
Results
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
40
50
60
70
80
90
100
VMAF
x264
x265
svtav1
(a) Bunny s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
30
40
50
60
70
80
90
100
VMAF
x264
x265
svtav1
(b) Characters s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
20
30
40
50
60
70
80
90
100
VMAF
x264
x265
svtav1
(c) RushHour s000
0.2 0.5 1.2 3.0 8.0 16.8
Bitrate (in Mbps)
40
50
60
70
80
90
100
VMAF
x264
x265
svtav1
(d) Wood s000
Figure: RD curves of representative segments encoded using MCBE (x264, x265, svtav1). Here, JTPS is considered
the bitrate ladder prediction method, and vJ = 6. Representations marked using dots indicate the eliminated
representations.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 13
Evaluation Summary
Results
Table: Average performance results using MCBE compared to HLS, OPTE, and JTPS bitrate ladders prediction
methods for various target encoder combinations.
MCBE configuration HLS ladder OPTE JTPS
Target encoders vJ ∆Eenc ∆S ∆Esto ∆Eenc ∆S ∆Esto ∆Eenc ∆S ∆Esto
(x264, x265)
2 -34.05% -43.70% -68.30% -36.03% -46.12% -70.97% -15.82% -10.20% -19.36%
4 -47.72% -60.48% -84.38% -49.96% -63.00% -86.31% -16.07% -9.53% -18.15%
6 -58.09% -69.91% -90.94% -59.75% -72.50% -92.44% -16.18% -11.27% -21.26%
(x264, svtav1)
2 -34.50% -43.31% -67.87% -36.87% -45.73% -70.55% -12.76% -8.79% -16.81%
4 -48.19% -59.95% -83.96% -51.17% -62.82% -86.18% -12.82% -8.27% -15.86%
6 -58.61% -69.18% -90.50% -61.06% -72.60% -92.49% -12.90% -9.38% -17.88%
(x264, x265, svtav1)
2 -20.42% -53.20% -78.10% -18.55% -53.63% -78.50% -22.56% -14.57% -27.01%
4 -41.67% -69.15% -90.49% -39.68% -69.49% -90.69% -23.74% -17.81% -32.45%
6 -56.45% -77.61% -94.99% -54.34% -78.32% -95.30% -27.80% -22.62% -40.12%
As vJ increases, more representations are eliminated, which reduces the encoding energy (Eenc), storage
consumption (S), and storage energy (Esto).
HLS bitrate ladder and OPTE representations have high perceptual redundancy compared to JTPS, as JTPS
representations are predicted with a perceptual gap of one JND. Hence, encoding energy, storage consump-
tion, and storage energy reduction are significantly high with HLS bitrate ladder and OPTE, compared to
JTPS.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 14
Conclusions
Conclusions
This paper proposed MCBE, an online energy-efficient multi-codec JND-aware bitrate ladder
estimation scheme for adaptive streaming applications.
MCBE includes an algorithm to determine an optimized multi-codec encoding bitrate ladder,
where redundant representations of new-generation video codecs are eliminated.
MCBE also minimizes perceptual redundancy within the representations of each codec based
on the JND threshold.
Notably, MCBE can be used in conjunction with any bitrate ladder estimation scheme.
MCBE on average, yields encoding, storage, and transmission energy savings of 56.45%,
77.61%, and 94.99%, respectively, compared to the state-of-the-art HLS bitrate ladder
encoding, for a streaming session with devices supporting AVC, HEVC, and AV1 decoding,
considering a JND of six VMAF points.
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 15
Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
Reza Farahani (reza.farahani@aau.at)
Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 16

More Related Content

Similar to VCIP_MCBE_presentation.pdf

Video Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive StreamingVideo Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive Streaming
Alpen-Adria-Universität
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
Vignesh V Menon
 
TQPM.pdf
TQPM.pdfTQPM.pdf
TQPM.pdf
Vignesh V Menon
 
Green_VCA_presentation.pdf
Green_VCA_presentation.pdfGreen_VCA_presentation.pdf
Green_VCA_presentation.pdf
Vignesh V Menon
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
Vignesh V Menon
 
HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)
Alpen-Adria-Universität
 
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdfETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
Vignesh V Menon
 
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live StreamingETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
Alpen-Adria-Universität
 
HTTP Adaptive Streaming – Quo Vadis (2024)
HTTP Adaptive Streaming – Quo Vadis (2024)HTTP Adaptive Streaming – Quo Vadis (2024)
HTTP Adaptive Streaming – Quo Vadis (2024)
Alpen-Adria-Universität
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Vignesh V Menon
 
Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive Streaming
Alpen-Adria-Universität
 
Machine Learning Based Video Coding Enhancements for HTTP Adaptive Streaming
Machine Learning Based Video Coding Enhancements for HTTP Adaptive StreamingMachine Learning Based Video Coding Enhancements for HTTP Adaptive Streaming
Machine Learning Based Video Coding Enhancements for HTTP Adaptive Streaming
Alpen-Adria-Universität
 
Optimal coding unit decision for early termination in high efficiency video c...
Optimal coding unit decision for early termination in high efficiency video c...Optimal coding unit decision for early termination in high efficiency video c...
Optimal coding unit decision for early termination in high efficiency video c...
IJECEIAES
 
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive StreamingQoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
DanieleLorenzi6
 
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
Minh Nguyen
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
Vignesh V Menon
 
HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?
Alpen-Adria-Universität
 
CODA_presentation.pdf
CODA_presentation.pdfCODA_presentation.pdf
CODA_presentation.pdf
JunZhao68
 
Deep learning-based switchable network for in-loop filtering in high efficie...
Deep learning-based switchable network for in-loop filtering in  high efficie...Deep learning-based switchable network for in-loop filtering in  high efficie...
Deep learning-based switchable network for in-loop filtering in high efficie...
IJECEIAES
 
H43044145
H43044145H43044145
H43044145
IJERA Editor
 

Similar to VCIP_MCBE_presentation.pdf (20)

Video Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive StreamingVideo Coding Enhancements for HTTP Adaptive Streaming
Video Coding Enhancements for HTTP Adaptive Streaming
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
 
TQPM.pdf
TQPM.pdfTQPM.pdf
TQPM.pdf
 
Green_VCA_presentation.pdf
Green_VCA_presentation.pdfGreen_VCA_presentation.pdf
Green_VCA_presentation.pdf
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
 
HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)
 
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdfETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
 
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live StreamingETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
 
HTTP Adaptive Streaming – Quo Vadis (2024)
HTTP Adaptive Streaming – Quo Vadis (2024)HTTP Adaptive Streaming – Quo Vadis (2024)
HTTP Adaptive Streaming – Quo Vadis (2024)
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
 
Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive Streaming
 
Machine Learning Based Video Coding Enhancements for HTTP Adaptive Streaming
Machine Learning Based Video Coding Enhancements for HTTP Adaptive StreamingMachine Learning Based Video Coding Enhancements for HTTP Adaptive Streaming
Machine Learning Based Video Coding Enhancements for HTTP Adaptive Streaming
 
Optimal coding unit decision for early termination in high efficiency video c...
Optimal coding unit decision for early termination in high efficiency video c...Optimal coding unit decision for early termination in high efficiency video c...
Optimal coding unit decision for early termination in high efficiency video c...
 
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive StreamingQoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
 
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
MHV'22 - Super-resolution Based Bitrate Adaptation for HTTP Adaptive Streamin...
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
 
HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?HTTP Adaptive Streaming – Where Is It Heading?
HTTP Adaptive Streaming – Where Is It Heading?
 
CODA_presentation.pdf
CODA_presentation.pdfCODA_presentation.pdf
CODA_presentation.pdf
 
Deep learning-based switchable network for in-loop filtering in high efficie...
Deep learning-based switchable network for in-loop filtering in  high efficie...Deep learning-based switchable network for in-loop filtering in  high efficie...
Deep learning-based switchable network for in-loop filtering in high efficie...
 
H43044145
H43044145H43044145
H43044145
 

More from Vignesh V Menon

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Vignesh V Menon
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
Vignesh V Menon
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
Vignesh V Menon
 
CAPS_Presentation.pdf
CAPS_Presentation.pdfCAPS_Presentation.pdf
CAPS_Presentation.pdf
Vignesh V Menon
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Vignesh V Menon
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
Vignesh V Menon
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Vignesh V Menon
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
Vignesh V Menon
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
Vignesh V Menon
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
Vignesh V Menon
 

More from Vignesh V Menon (10)

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
 
CAPS_Presentation.pdf
CAPS_Presentation.pdfCAPS_Presentation.pdf
CAPS_Presentation.pdf
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
 

Recently uploaded

BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...
BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...
BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...
Nguyen Thanh Tu Collection
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Alvaro Barbosa
 
C# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdfC# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdf
Scholarhat
 
ASP.NET Core Interview Questions PDF By ScholarHat.pdf
ASP.NET Core Interview Questions PDF By ScholarHat.pdfASP.NET Core Interview Questions PDF By ScholarHat.pdf
ASP.NET Core Interview Questions PDF By ScholarHat.pdf
Scholarhat
 
1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx
UmeshTimilsina1
 
Imagination in Computer Science Research
Imagination in Computer Science ResearchImagination in Computer Science Research
Imagination in Computer Science Research
Abhik Roychoudhury
 
11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx
UmeshTimilsina1
 
How To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-marketHow To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-market
Sikandar Ali
 
E-learning Odoo 17 New features - Odoo 17 Slides
E-learning Odoo 17  New features - Odoo 17 SlidesE-learning Odoo 17  New features - Odoo 17 Slides
E-learning Odoo 17 New features - Odoo 17 Slides
Celine George
 
2 Post harvest Physiology of Horticulture produce.pptx
2 Post harvest Physiology of Horticulture  produce.pptx2 Post harvest Physiology of Horticulture  produce.pptx
2 Post harvest Physiology of Horticulture produce.pptx
UmeshTimilsina1
 
Java MCQ Questions and Answers PDF By ScholarHat
Java MCQ Questions and Answers PDF By ScholarHatJava MCQ Questions and Answers PDF By ScholarHat
Java MCQ Questions and Answers PDF By ScholarHat
Scholarhat
 
MATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docxMATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docx
yardenmendoza
 
JavaScript Interview Questions PDF By ScholarHat
JavaScript Interview  Questions PDF By ScholarHatJavaScript Interview  Questions PDF By ScholarHat
JavaScript Interview Questions PDF By ScholarHat
Scholarhat
 
C Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdfC Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdf
Scholarhat
 
DepEd School Calendar 2024-2025 DO_s2024_008
DepEd School Calendar 2024-2025 DO_s2024_008DepEd School Calendar 2024-2025 DO_s2024_008
DepEd School Calendar 2024-2025 DO_s2024_008
Glenn Rivera
 
Our Guide to the July 2024 USPS® Rate Change
Our Guide to the July 2024 USPS® Rate ChangeOur Guide to the July 2024 USPS® Rate Change
Our Guide to the July 2024 USPS® Rate Change
Postal Advocate Inc.
 
Introduction to Banking System in India.ppt
Introduction to Banking System in India.pptIntroduction to Banking System in India.ppt
Introduction to Banking System in India.ppt
Dr. S. Bulomine Regi
 
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
Celine George
 
A beginner’s guide to project reviews - everything you wanted to know but wer...
A beginner’s guide to project reviews - everything you wanted to know but wer...A beginner’s guide to project reviews - everything you wanted to know but wer...
A beginner’s guide to project reviews - everything you wanted to know but wer...
Association for Project Management
 
The Cruelty of Animal Testing in the Industry.pdf
The Cruelty of Animal Testing in the Industry.pdfThe Cruelty of Animal Testing in the Industry.pdf
The Cruelty of Animal Testing in the Industry.pdf
luzmilaglez334
 

Recently uploaded (20)

BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...
BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...
BỘ ĐỀ THI HỌC SINH GIỎI CÁC TỈNH MÔN TIẾNG ANH LỚP 9 NĂM HỌC 2023-2024 (CÓ FI...
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
 
C# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdfC# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdf
 
ASP.NET Core Interview Questions PDF By ScholarHat.pdf
ASP.NET Core Interview Questions PDF By ScholarHat.pdfASP.NET Core Interview Questions PDF By ScholarHat.pdf
ASP.NET Core Interview Questions PDF By ScholarHat.pdf
 
1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx1. Importance_of_reducing_postharvest_loss.pptx
1. Importance_of_reducing_postharvest_loss.pptx
 
Imagination in Computer Science Research
Imagination in Computer Science ResearchImagination in Computer Science Research
Imagination in Computer Science Research
 
11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx11. Post harvest quality, Quality criteria and Judgement.pptx
11. Post harvest quality, Quality criteria and Judgement.pptx
 
How To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-marketHow To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-market
 
E-learning Odoo 17 New features - Odoo 17 Slides
E-learning Odoo 17  New features - Odoo 17 SlidesE-learning Odoo 17  New features - Odoo 17 Slides
E-learning Odoo 17 New features - Odoo 17 Slides
 
2 Post harvest Physiology of Horticulture produce.pptx
2 Post harvest Physiology of Horticulture  produce.pptx2 Post harvest Physiology of Horticulture  produce.pptx
2 Post harvest Physiology of Horticulture produce.pptx
 
Java MCQ Questions and Answers PDF By ScholarHat
Java MCQ Questions and Answers PDF By ScholarHatJava MCQ Questions and Answers PDF By ScholarHat
Java MCQ Questions and Answers PDF By ScholarHat
 
MATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docxMATATAG CURRICULUM sample lesson exemplar.docx
MATATAG CURRICULUM sample lesson exemplar.docx
 
JavaScript Interview Questions PDF By ScholarHat
JavaScript Interview  Questions PDF By ScholarHatJavaScript Interview  Questions PDF By ScholarHat
JavaScript Interview Questions PDF By ScholarHat
 
C Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdfC Interview Questions PDF By Scholarhat.pdf
C Interview Questions PDF By Scholarhat.pdf
 
DepEd School Calendar 2024-2025 DO_s2024_008
DepEd School Calendar 2024-2025 DO_s2024_008DepEd School Calendar 2024-2025 DO_s2024_008
DepEd School Calendar 2024-2025 DO_s2024_008
 
Our Guide to the July 2024 USPS® Rate Change
Our Guide to the July 2024 USPS® Rate ChangeOur Guide to the July 2024 USPS® Rate Change
Our Guide to the July 2024 USPS® Rate Change
 
Introduction to Banking System in India.ppt
Introduction to Banking System in India.pptIntroduction to Banking System in India.ppt
Introduction to Banking System in India.ppt
 
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
 
A beginner’s guide to project reviews - everything you wanted to know but wer...
A beginner’s guide to project reviews - everything you wanted to know but wer...A beginner’s guide to project reviews - everything you wanted to know but wer...
A beginner’s guide to project reviews - everything you wanted to know but wer...
 
The Cruelty of Animal Testing in the Industry.pdf
The Cruelty of Animal Testing in the Industry.pdfThe Cruelty of Animal Testing in the Industry.pdf
The Cruelty of Animal Testing in the Industry.pdf
 

VCIP_MCBE_presentation.pdf

  • 1. Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming Vignesh V Menon1,2, Reza Farahani2, Prajit T Rajendran3, Samira Afzal2, Klaus Schoeffmann2, Christian Timmerer2 1 Video Communication and Applications Department, Fraunhofer HHI, Berlin, Germany 2 Alpen-Adria-Universität, Klagenfurt, Austria 3 CEA, List, F-91120 Palaiseau, Université Paris-Saclay, France Dec 6,2023 Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 1
  • 2. Outline 1 Introduction 2 MCBE 3 Evaluation 4 Conclusions Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 2
  • 3. Introduction Introduction Multi-codec streaming ecosystem In streaming systems, each codec requires its own set of representations, i.e., bitrate ladders.1,2 HEVC Ladder AVC Ladder AVC AVC HEVC HEVC Media Server Original Video AV1 AV1 AV1 Ladder Figure: An example of a multi-codec streaming system. 1 A. Bentaleb et al. “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP”. In: IEEE Communications Surveys Tutorials 21.1 (2019), pp. 562–585. doi: 10.1109/COMST.2018.2862938. 2 Christian Timmerer, Martin Smole, and Christopher Mueller. “Efficient Multi-Codec Support for OTT Services: HEVC/H. 265 and/or AV1?” In: 2018. url: http://www.itec.aau.at/bib/files/TimmererC010218.pdf. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 3
  • 4. Introduction Introduction Multi-codec streaming ecosystem Initially, streaming services used AVC for wider device compatibility.3 As newer devices with HEVC and Alliance for Open Media Video 1 (AV1)4 support becomes preva- lent, HEVC and AV1-encoded bitrate ladder representations are introduced Recent years have developed new formats such as Versatile Video Coding (VVC),5 Essential Video Coding (EVC),6 and Low Complexity Enhancement Video Coding (LCEVC).7 Streaming systems have evolved to accommodate multiple codecs, with older devices relying solely on AVC, some newer devices using HEVC streams, and certain devices supporting both AVC and HEVC, including seamlessly switching between them. 3 Yuriy A. Reznik. “Toward Efficient Multicodec Streaming”. In: SMPTE Motion Imaging Journal 132.4 (2023), pp. 16–25. doi: 10.5594/JMI.2023.3263499. 4 Jingning Han et al. “A Technical Overview of AV1”. In: Proceedings of the IEEE 109.9 (2021), pp. 1435–1462. doi: 10.1109/JPROC.2021.3058584. 5 Benjamin Bross et al. “Overview of the Versatile Video Coding (VVC) Standard and its Applications”. In: IEEE Transactions on Circuits and Systems for Video Technology 31.10 (2021), pp. 3736–3764. doi: 10.1109/TCSVT.2021.3101953. 6 Jonatan Samuelsson et al. “MPEG-5 EVC”. In: SMPTE 2019. 2019, pp. 1–11. doi: 10.5594/M001877. 7 Stefano Battista et al. “Overview of the Low Complexity Enhancement Video Coding (LCEVC) Standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 32.11 (2022), pp. 7983–7995. doi: 10.1109/TCSVT.2022.3182793. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 4
  • 5. Introduction Introduction Perceptual redundancy 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 40 60 80 100 VMAF x264 x265 svtav1 (a) Basketball s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 30 40 50 60 70 80 VMAF x264 x265 svtav1 (b) Riverbank s000 Figure: Rate-distortion (RD) curves of representative sequences of VCD dataset, encoded with JTPS bitrate ladder8 for x264, x265, and svtav1 encoders. In some cases, the compression efficiency of AVC is better than new-generation video codecs, i.e., at low bitrates. Compression efficiency of codecs saturates at very high target bitrates, as they become similar to lossless coding. The bitrate regions where each codec performs better than others depending on the complexity of the video content. 8 Vignesh V Menon et al. “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”. In: IEEE Transactions on Circuits and Systems for Video Technology (2023), pp. 1–1. doi: 10.1109/TCSVT.2023.3290725. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 5
  • 6. Introduction Introduction Energy consumption Encoding video content into multiple representations in various bitrate-resolution pairs for each codec results in substantial computational workload and energy consumption.9 Storage and transmission of these representations further contribute to the overall energy consump- tion.10 When unnecessary high-bitrate representations (of new-generation codecs) are eliminated, the en- ergy consumption of the streaming system is significantly reduced.11 This is because the energy consumption of AVC is significantly lower than that of new-generation video codecs.12 As video streaming continues to grow in popularity, finding energy-efficient solutions to optimize the multi-codec bitrate ladder becomes crucial to mitigate the environmental impact and reduce operational costs for service providers. 9 Maria G Koziri et al. “Efficient cloud provisioning for video transcoding: Review, open challenges and future opportunities”. In: IEEE Internet Computing 22.5 (2018), pp. 46–55. 10 Jayant Baliga et al. “Green cloud computing: Balancing energy in processing, storage, and transport”. In: Proceedings of the IEEE 99.1 (2010), pp. 149–167. 11 Daniele Lorenzi. “QoE- and Energy-Aware Content Consumption For HTTP Adaptive Streaming”. In: Proceedings of the 14th Conference on ACM Multimedia Systems. 2023, 348–352. isbn: 9798400701481. doi: 10.1145/3587819.3593029. 12 Thorsten Laude et al. “A Comparison of JEM and AV1 with HEVC: Coding Tools, Coding Efficiency and Complexity”. In: 2018 Picture Coding Symposium (PCS). 2018, pp. 36–40. doi: 10.1109/PCS.2018.8456291; Isis Bender et al. “Compression Efficiency and Computational Cost Comparison between AV1 and HEVC Encoders”. In: 2019 27th European Signal Processing Conference (EUSIPCO). 2019, pp. 1–5. doi: 10.23919/EUSIPCO.2019.8903006. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 6
  • 7. MCBE Architecture MCBE Architecture Input Video Segment Encoding Representations Spatio-Temporal Feature Extraction E h L Y Y Set of Target Resolutions (R) Set of Target Codecs (C) Redundant Representation Elimination r ^ ( , b , c ^ ^) Max Quality and Target JND (v , v ) 1 HEVC Bitrate Ladder Estimation AV1 Bitrate Ladder Estimation AVC Bitrate Ladder Estimation c c c1 2 3 2 3 … … max J Figure: Online encoding using MCBE envisioned in this paper for adaptive video streaming. MCBE comprises three phases: 1 Spatio-temporal feature extraction 2 Redundant representation elimination 3 Encoding of the segments using the selected bitrate-resolution pairs of each codec Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 7
  • 8. MCBE Spatio-Temporal Feature Extraction MCBE Spatio-Temporal Feature Extraction MCBE uses the following DCT-energy-based features,13 extracted using open-source VCA v2.0 video complexity analyzer14 for every segment: 1 Average luma texture energy (EY) 2 Average gradient of the luma texture energy (h) 3 Average luminescence (LY) 13 N B Harikrishnan et al. “Comparative evaluation of image compression techniques”. In: 2017 International Conference on Algorithms, Methodology, Models and Applications in Emerging Technologies (ICAMMAET). 2017, pp. 1–4. doi: 10.1109/ICAMMAET.2017.8186637. 14 Vignesh V Menon et al. “Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming”. In: First International ACM Green Multimedia Systems Workshop (GMSys ’23). 2023. isbn: 9798400701962. doi: 10.1145/3593908.3593942. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 8
  • 9. MCBE Redundant Representation Elimination MCBE Redundant Representation Elimination VMAF score (vrt ,bt ,c) of the tth representation of the codec c is modeled as a function of the video content complexity features and the target representation15,16 as shown in the following equation: vrt ,bt ,c = fV(EY, h, LY, rt, bt, c) (1) Random forest models17 which are hyperparameter-tuned with the parameters min samples lea min samples split=2, n estimators= 100, and max depth=14 are trained for each codec c ∈ C and resolution r ∈ R to predict VMAF. 15 Vignesh V Menon et al. “Transcoding Quality Prediction for Adaptive Video Streaming”. In: Proceedings of the 2nd Mile-High Video Conference. Denver, CO, USA, 2023, 103–109. isbn: 9798400701603. doi: 10.1145/3588444.3591012. 16 Vignesh V Menon et al. Video Quality Assessment with Texture Information Fusion for Streaming Applications. 2023. arXiv: 2302.14465 [cs.MM]. 17 Leo Breiman. “Random Forests”. In: Machine Learning 45 (2001). doi: 10.1023/A:1010933404324. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 9
  • 10. MCBE Redundant Representation Elimination MCBE Redundant Representation Elimination- Step 1 C : set of all codecs c1, c2...cM in order of priority Nc : number of representations for codec c (ˆ rt, b̂t, c) pairs ∀c ∈ C, t ∈ Nc Step 1: for each c ∈ C do t = 2 while t ≤ Nc do if v̂c,ˆ rt ,b̂t > vmax or v̂c,ˆ rt ,b̂t − v̂c,ˆ rt−1,b̂t−1 < vJ then Eliminate (ˆ rt, b̂t, c) from the ladder N̂c = Nc − 1 t = t + 1 Bitrate ladder representations of each considered codec in C with a perceptual quality difference within a given Just Noticeable Difference (JND) threshold are eliminated. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 10
  • 11. MCBE Redundant Representation Elimination MCBE Redundant Representation Elimination- Step 2 Q= {(ˆ rt, b̂t, c1)}, t ∈ N̂c1 Step 2: for each c ∈ {c2, .., cM} do for each t ∈ N̂c do (˜ ri , b̃i , c1) ← arg mini | b̂i,c1 − b̂t | s.t. b̂t ≥ bi,c1 (˜ rj , b̃j , c1) ← arg minj | b̂j,c1 − b̂t | s.t. b̂t ≤ b̂i,c1 RD curve L between (˜ ri , b̃i , c1) and (˜ rj , b̃j , c1): v = v̂c1,ˆ rj ,b̂j −v̂c1,ˆ ri ,b̂i b̃j −b̃i · (b − b̃i ) + v̂c1,ˆ ri ,b̂i if (v̂c,ˆ rt ,b̂t is above L) then Add (ˆ rt, b̂t, c) to Q. c1 denotes AVC representations. When AVC performs better (in terms of perceptual quality) than or is identical to HEVC and/or another new-generation codec (e.g., AV1) in a bitrate range, the corresponding new-generation codec representations are eliminated from the bitrate ladder. This is because clients can be served with AVC representations with better RD performance. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 11
  • 12. Evaluation Test Setup Test Setup Table: Experimental parameters used to evaluate MCBE. Parameter Values C x264 v1.1 x265 v3.5 svtav1 v1.6 R { 360, 432, 540, 720, 1080, 1440, 2160 } vJ 2 4 6 vmax 98 96 94 CPU threads 8 Bitrate ladder estimation: 1 Default HLS bitrate ladder18 for each codec/encoder. 2 OPTE,19 where optimized resolutions for the set of bitrates in the HLS bitrate ladder are predicted for each encoder. 3 JTPS,20 where optimized bitrate-resolution pairs are predicted for JND-aware efficient encoding for each encoder. 18 Apple Inc. “HLS Authoring Specification for Apple Devices”. In: url: https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices. 19 V. V. Menon et al. “OPTE: Online Per-Title Encoding for Live Video Streaming”. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2022, pp. 1865–1869. doi: 10.1109/ICASSP43922.2022.9746745. 20 Menon et al., “JND-aware Two-pass Per-title Encoding Scheme for Adaptive Live Streaming”. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 12
  • 13. Evaluation RD curves Results 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 40 50 60 70 80 90 100 VMAF x264 x265 svtav1 (a) Bunny s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 30 40 50 60 70 80 90 100 VMAF x264 x265 svtav1 (b) Characters s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 20 30 40 50 60 70 80 90 100 VMAF x264 x265 svtav1 (c) RushHour s000 0.2 0.5 1.2 3.0 8.0 16.8 Bitrate (in Mbps) 40 50 60 70 80 90 100 VMAF x264 x265 svtav1 (d) Wood s000 Figure: RD curves of representative segments encoded using MCBE (x264, x265, svtav1). Here, JTPS is considered the bitrate ladder prediction method, and vJ = 6. Representations marked using dots indicate the eliminated representations. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 13
  • 14. Evaluation Summary Results Table: Average performance results using MCBE compared to HLS, OPTE, and JTPS bitrate ladders prediction methods for various target encoder combinations. MCBE configuration HLS ladder OPTE JTPS Target encoders vJ ∆Eenc ∆S ∆Esto ∆Eenc ∆S ∆Esto ∆Eenc ∆S ∆Esto (x264, x265) 2 -34.05% -43.70% -68.30% -36.03% -46.12% -70.97% -15.82% -10.20% -19.36% 4 -47.72% -60.48% -84.38% -49.96% -63.00% -86.31% -16.07% -9.53% -18.15% 6 -58.09% -69.91% -90.94% -59.75% -72.50% -92.44% -16.18% -11.27% -21.26% (x264, svtav1) 2 -34.50% -43.31% -67.87% -36.87% -45.73% -70.55% -12.76% -8.79% -16.81% 4 -48.19% -59.95% -83.96% -51.17% -62.82% -86.18% -12.82% -8.27% -15.86% 6 -58.61% -69.18% -90.50% -61.06% -72.60% -92.49% -12.90% -9.38% -17.88% (x264, x265, svtav1) 2 -20.42% -53.20% -78.10% -18.55% -53.63% -78.50% -22.56% -14.57% -27.01% 4 -41.67% -69.15% -90.49% -39.68% -69.49% -90.69% -23.74% -17.81% -32.45% 6 -56.45% -77.61% -94.99% -54.34% -78.32% -95.30% -27.80% -22.62% -40.12% As vJ increases, more representations are eliminated, which reduces the encoding energy (Eenc), storage consumption (S), and storage energy (Esto). HLS bitrate ladder and OPTE representations have high perceptual redundancy compared to JTPS, as JTPS representations are predicted with a perceptual gap of one JND. Hence, encoding energy, storage consump- tion, and storage energy reduction are significantly high with HLS bitrate ladder and OPTE, compared to JTPS. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 14
  • 15. Conclusions Conclusions This paper proposed MCBE, an online energy-efficient multi-codec JND-aware bitrate ladder estimation scheme for adaptive streaming applications. MCBE includes an algorithm to determine an optimized multi-codec encoding bitrate ladder, where redundant representations of new-generation video codecs are eliminated. MCBE also minimizes perceptual redundancy within the representations of each codec based on the JND threshold. Notably, MCBE can be used in conjunction with any bitrate ladder estimation scheme. MCBE on average, yields encoding, storage, and transmission energy savings of 56.45%, 77.61%, and 94.99%, respectively, compared to the state-of-the-art HLS bitrate ladder encoding, for a streaming session with devices supporting AVC, HEVC, and AV1 decoding, considering a JND of six VMAF points. Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 15
  • 16. Q & A Q & A Thank you for your attention! Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de) Reza Farahani (reza.farahani@aau.at) Vignesh V Menon Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Streaming 16