SlideShare a Scribd company logo
1 of 16
Download to read offline
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live
Streaming
Vignesh V Menon1, Hadi Amirpour1, Mohammad Ghanbari1,2, and Christian Timmerer1
1
Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria
2
School of Computer Science and Electronic Engineering, University of Essex, UK
16-19 October 2022
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 1
Outline
1 Introduction
2 Research Problem
3 ETPS
4 Evaluation
5 Conclusions
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 2
Introduction
Introduction
CBR versus VBR
Figure: Constant Bitrate (CBR) versus Variable
Bitrate (VBR) encoding.
In live streaming, Constant Bitrate (CBR) rate-
control mode is used to encode video sequences
at a fixed bitrate ladder. The consistency of
CBR makes it more reliable for time-sensitive
data transport.
In this method, there is no concern about the
bitrate exceeding internet speeds. However, this
method may result in low compression efficiency.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 3
Introduction
Introduction
Variable Bitrate (VBR) encoding
Figure: Two-pass encoding.
VBR encoding schemes are implemented as two-pass encoding, also known as multi-pass
encoding used to retain the best compression efficiency during encoding.
In the first-pass of two-pass encoding, the input data from the video is analyzed (and stored
in a log file). The collected data from the first-pass is used to achieve the best encoding
compression efficiency in the second-pass.
Two-pass encoding is usually controlled by the average bitrate setting, bitrate range setting
(minimal and maximal allowed bitrate), or target video file size.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 4
Introduction
Introduction
Constrained Variable Bitrate (cVBR) encoding
Figure: cVBR encoding.
A ”rate factor” first-pass to identify the optimized CRF to achieve the target bitrate.
In the second-pass, the segment is encoded with the selected optimized CRF with the
maximum bitrate and maximum buffer window constraints.
The desired target bitrate is achieved with maximum compression efficiency, and min-
imum quality fluctuation.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 5
Research Problem
Research Problem
In two-pass encoding, the encoder processes all segments for the stream twice.
Due to the latency introduced by the first-pass to achieve the optimal encoding parameters
for the second-pass, two-pass encoding is not used in live streaming applications.
This paper targets a two-pass encoding scheme optimized for live streaming applications. The
target average bitrate (bT ) is aimed to be achieved uniformly throughout the streaming session
with the highest possible compression efficiency.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 6
ETPS
ETPS
Figure: ETPS architecture.
The first-pass predicts the optimized CRF to achieve the target bitrate.
In the second-pass, the segment is encoded with the selected, optimized CRF with the
maximum bitrate and maximum buffer window constraints.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 7
ETPS Phase 1: Video Complexity Feature Extraction
ETPS
Phase 1: Video Complexity Feature Extraction
Compute texture energy per block
A DCT-based energy function is used to determine the block-wise feature of each frame
defined as:
Hk =
w−1
X
i=0
w−1
X
j=0
e|( ij
wh
)2−1|
|DCT(i, j)| (1)
where wxw is the size of the block, and DCT(i, j) is the (i, j)th DCT component when
i + j > 0, and 0 otherwise.
The energy values of blocks in a frame is averaged to determine the energy per frame1,.2
E =
C−1
X
k=0
Hp,k
C · w2
(2)
1
Michael King et al. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on Multimedia and Expo. 2007,
pp. 1647–1650. doi: 10.1109/ICME.2007.4284983.
2
Vignesh V Menon et al. “Efficient Content-Adaptive Feature-Based Shot Detection for HTTP Adaptive Streaming”. In: 2021 IEEE International Conference
on Image Processing (ICIP). 2021, pp. 2174–2178. doi: 10.1109/ICIP42928.2021.9506092.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 8
ETPS Phase 1: Video Complexity Feature Extraction
ETPS
Phase 1: Video Complexity Feature Extraction
hp: SAD of the block level energy values of frame p to that of the previous frame p − 1.
hp =
C−1
X
k=0
| Hp,k, Hp−1,k |
C · w2
(3)
where C denotes the number of blocks in frame p.
Latency
Speed of feature extraction = 370fps for UHD video with 8 CPU threads and x86 SIMD
optimization
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 9
ETPS Phase 2: CRF Prediction
ETPS
Phase 2: CRF Prediction
Hidden Layer
E R4
Hidden Layer
E R4
Input Layer
E R3
Output Layer
E R1
E
h
log(b)
ĉ
A shallow neural network is trained for each resolu-
tion r and framerate f , which determines the opti-
mized CRF (ĉ) based on E, h, and log(b) for every
video segment.
The input vector passed to the network is [E, h, log(bT )].
The target bitrate input (bT ) is passed in logarithmic scale to the network to reduce internal
covariate shift.3
Rectified Linear Unit (ReLU)4 is used as the activation function and Adam5 is used as
optimizer with a learning rate of 10−4.
3
Sergey Ioffe and Christian Szegedy. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”. In: Proceedings of the
32nd International Conference on Machine Learning. Vol. 37. Lille, France: PMLR, 2015, pp. 448–456.
4
Xavier Glorot, Antoine Bordes, and Y. Bengio. “Deep Sparse Rectifier Neural Networks”. In: vol. 15. Jan. 2010.
5
Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR
2015, San Diego, CA, USA, May 7-9, 2015. 2015.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 10
Evaluation Test Methodology
Test Methodology
Table: Encoder Settings.
Encoder x265 v3.5
Preset ultrafast
Segment length 4 seconds
Resolution Ultra HD (2160p)
Framerate 30 fps
Target bitrates 8, 12, 17, and 20 Mbps
Keras6 is used as the machine learning framework.
E and h features are extracted from the video segments using VCA7 run in eight CPU
threads
6
Nikhil Ketkar. “Introduction to Keras”. In: Oct. 2017, pp. 95–109. isbn: 978-1-4842-2765-7. doi: 10.1007/978-1-4842-2766-4_7.
7
Vignesh V Menon et al. “VCA: Video Complexity Analyzer”. In: 13th ACM Multimedia Systems Conference (MMSys ’22). 2022. isbn: 9781450392839. doi:
10.1145/3524273.3532896. url: https://doi.org/10.1145/3524273.3532896.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 11
Evaluation Test Methodology
Test Methodology
Criteria:
Root Mean Squared Error (RMSE) of cG and the predicted CRF (ĉ), i.e., || cG − ĉ ||2.
The first-pass time, i.e., latency in streaming introduced due to the time taken to extract
E and h, and the inference time of the neural network to predict optimized CRF.
Bjøntegaard delta rates8 BDRP and BDRV to measure the average increase in bitrate of
the representations compared with that of the reference encoding scheme to maintain the
same PSNR and VMAF, respectively.
∆T: the overall difference in encoding time of ETPS with the reference encoding scheme.
The results are compared with CBR encoding and two-pass average bitrate encoding schemes9.
8
G. Bjontegaard. “Calculation of average PSNR differences between RD-curves”. In: VCEG-M33 (2001).
9
https://x265.readthedocs.io/en/master/cli.html#quality-rate-control-and-rate-distortion-options, last access: Jun 30, 2022.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 12
Evaluation Experimental Results
Experimental Results
Table: Results of ETPS against Constant Bitrate (CBR) encoding and two-pass average bitrate encoding
schemes.
Constant bitrate (CBR) Two-pass average bitrate
Dataset Video E h || cG − ĉ ||2 τp (in ms) BDRP BDRV ∆T BDRP BDRV ∆T
MCML10 Basketball 15.75 9.69 0.86 8 -17.80% -14.36% 0.43% -0.48% 0.50% -45.15%
MCML Bunny 23.03 4.89 1.58 7 -9.92% -9.09% 0.78% 0.04% -1.14% -43.88%
MCML Crowd 33.11 7.03 1.58 7 -7.63% -1.47% 0.56% 0.75% 1.05% -43.34%
MCML Dolls 19.91 12.22 0.86 8 -5.73% -4.05% 1.01% 1.21% 1.97% -45.26%
MCML Flowers 12.01 10.47 0.50 7 -8.44% -7.12% -0.59% 0.38% 0.49% -43.90%
MCML Park 26.07 27.34 0.71 9 -5.16% -2.62% 1.10% 0.16% 0.87% -44.27%
SJTU11 BundNightScape 54.90 11.62 1.93 8 -14.63% -13.51% -1.62% -1.11% -1.27% -40.91%
SJTU CampfireParty 51.50 42.38 1.58 8 -4.66% -3.66% -1.72% -0.51% -0.35% -43.04%
SJTU Runners 104.30 17.44 1.32 7 -8.14% -2.44% -1.34% -0.77% -1.08% -42.69%
SJTU TallBuildings 94.35 7.70 1.65 8 -17.30% -12.54% -1.02% 0.89% 0.59% -43.37%
SJTU TrafficAndBuilding 60.54 11.55 1.87 8 -15.41% -9.91% -0.43% 0.06% -2.42% -43.96%
SJTU TrafficFlow 58.93 8.61 1.32 8 -15.66% -1.17% 0.74% 1.19% 1.43% -43.86%
SJTU TreeShade 80.19 15.83 0.50 8 -11.82% -6.85% -0.79% 0.52% 0.59% -44.75%
SJTU Wood 114.26 8.83 0.71 8 -10.14% -21.58% 0.52% 1.34% 1.88% -44.50%
Average 1.02 8 -10.89% -8.60% 0.65% 0.26% 0.38% -43.78%
10
Manri Cheon and Jong-Seok Lee. “Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience”. In: IEEE
Transactions on Circuits and Systems for Video Technology 28.7 (2018), pp. 1467–1480. doi: 10.1109/TCSVT.2017.2683504.
11
L. Song et al. “The SJTU 4K Video Sequence Dataset”. In: Fifth International Workshop on Quality of Multimedia Experience (QoMEX2013) (July 2013).
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 13
Evaluation Experimental Results
Experimental Results
MCML sequences are sequentially encoded using ETPS at 8, 12, 17, and 20 Mbps, and the
achieved average bitrate is recorded.
The average target bitrate is consistently achieved irrespective of the spatial and temporal
complexities of the segments.
0 4 8 12 16 20 24 28 32
Time (in seconds)
8
10
12
14
16
18
20
Average
bitrate
(in
Mbps)
8 Mbps
12 Mbps
17 Mbps
20 Mbps
Figure: The average bitrate in a streaming session where MCML sequences are encoded using ETPS and
are streamed at intervals of four seconds for target average bitrates 8, 12, 17, 20 Mbps.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 14
Conclusions
Conclusions
This paper proposed ETPS, a low-latency two-pass constrained variable bitrate (cVBR)
encoding scheme for live streaming applications.
ETPS includes a prediction algorithm for optimized CRF at a given target bitrate, resolution,
and framerate of video segments.
DCT-energy-based features are used to determine segments’ spatial and temporal complex-
ity.
Live streaming using ETPS has similar compression efficiency as that of the two-pass average
bitrate encoding scheme, but with a significant encoding time-saving of 43.78%.
Compared to CBR encoding, ETPS yields bitrate saving of 10.89% and 8.60% to maintain
the same PSNR and VMAF, respectively, with an additional encoding time of 0.65%.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 15
Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 16

More Related Content

Similar to ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf

International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
IJERD Editor
 

Similar to ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf (20)

OPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdfOPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdf
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video StreamingPerceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
 
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
 
CAPS_Presentation.pdf
CAPS_Presentation.pdfCAPS_Presentation.pdf
CAPS_Presentation.pdf
 
Efficient bitrate ladder construction for live video streaming
Efficient bitrate ladder construction for live video streamingEfficient bitrate ladder construction for live video streaming
Efficient bitrate ladder construction for live video streaming
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
 
INCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVCINCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVC
 
Machine Learning approaches at video compression
Machine Learning approaches at video compression Machine Learning approaches at video compression
Machine Learning approaches at video compression
 
Green_VCA_presentation.pdf
Green_VCA_presentation.pdfGreen_VCA_presentation.pdf
Green_VCA_presentation.pdf
 
Performance Evaluation of Percent Root Mean Square Difference for ECG Signals...
Performance Evaluation of Percent Root Mean Square Difference for ECG Signals...Performance Evaluation of Percent Root Mean Square Difference for ECG Signals...
Performance Evaluation of Percent Root Mean Square Difference for ECG Signals...
 
40120140504006
4012014050400640120140504006
40120140504006
 
International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
 
Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC Streaming
 
Data detection with a progressive parallel ici canceller in mimo ofdm
Data detection with a progressive parallel ici canceller in mimo ofdmData detection with a progressive parallel ici canceller in mimo ofdm
Data detection with a progressive parallel ici canceller in mimo ofdm
 
Rate Distortion Performance for Joint Source Channel Coding of JPEG image Ove...
Rate Distortion Performance for Joint Source Channel Coding of JPEG image Ove...Rate Distortion Performance for Joint Source Channel Coding of JPEG image Ove...
Rate Distortion Performance for Joint Source Channel Coding of JPEG image Ove...
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
 
Evaluation and Analysis of Rate Control Methods for H.264/AVC and MPEG-4 Vide...
Evaluation and Analysis of Rate Control Methods for H.264/AVC and MPEG-4 Vide...Evaluation and Analysis of Rate Control Methods for H.264/AVC and MPEG-4 Vide...
Evaluation and Analysis of Rate Control Methods for H.264/AVC and MPEG-4 Vide...
 
H43044145
H43044145H43044145
H43044145
 
Optimal coding unit decision for early termination in high efficiency video c...
Optimal coding unit decision for early termination in high efficiency video c...Optimal coding unit decision for early termination in high efficiency video c...
Optimal coding unit decision for early termination in high efficiency video c...
 

More from Vignesh V Menon

Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Vignesh V Menon
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
Vignesh V Menon
 

More from Vignesh V Menon (10)

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
 

Recently uploaded

SPLICE Working Group: Reusable Code Examples
SPLICE Working Group:Reusable Code ExamplesSPLICE Working Group:Reusable Code Examples
SPLICE Working Group: Reusable Code Examples
Peter Brusilovsky
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
中 央社
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
中 央社
 

Recently uploaded (20)

male presentation...pdf.................
male presentation...pdf.................male presentation...pdf.................
male presentation...pdf.................
 
8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital Management8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital Management
 
Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"
 
How To Create Editable Tree View in Odoo 17
How To Create Editable Tree View in Odoo 17How To Create Editable Tree View in Odoo 17
How To Create Editable Tree View in Odoo 17
 
Including Mental Health Support in Project Delivery, 14 May.pdf
Including Mental Health Support in Project Delivery, 14 May.pdfIncluding Mental Health Support in Project Delivery, 14 May.pdf
Including Mental Health Support in Project Delivery, 14 May.pdf
 
How to Manage Closest Location in Odoo 17 Inventory
How to Manage Closest Location in Odoo 17 InventoryHow to Manage Closest Location in Odoo 17 Inventory
How to Manage Closest Location in Odoo 17 Inventory
 
Championnat de France de Tennis de table/
Championnat de France de Tennis de table/Championnat de France de Tennis de table/
Championnat de France de Tennis de table/
 
An overview of the various scriptures in Hinduism
An overview of the various scriptures in HinduismAn overview of the various scriptures in Hinduism
An overview of the various scriptures in Hinduism
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
 
Improved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio AppImproved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio App
 
An Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge AppAn Overview of the Odoo 17 Knowledge App
An Overview of the Odoo 17 Knowledge App
 
PSYPACT- Practicing Over State Lines May 2024.pptx
PSYPACT- Practicing Over State Lines May 2024.pptxPSYPACT- Practicing Over State Lines May 2024.pptx
PSYPACT- Practicing Over State Lines May 2024.pptx
 
philosophy and it's principles based on the life
philosophy and it's principles based on the lifephilosophy and it's principles based on the life
philosophy and it's principles based on the life
 
Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...Andreas Schleicher presents at the launch of What does child empowerment mean...
Andreas Schleicher presents at the launch of What does child empowerment mean...
 
Major project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategiesMajor project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategies
 
demyelinated disorder: multiple sclerosis.pptx
demyelinated disorder: multiple sclerosis.pptxdemyelinated disorder: multiple sclerosis.pptx
demyelinated disorder: multiple sclerosis.pptx
 
SPLICE Working Group: Reusable Code Examples
SPLICE Working Group:Reusable Code ExamplesSPLICE Working Group:Reusable Code Examples
SPLICE Working Group: Reusable Code Examples
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
 
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
 

ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf

  • 1. ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming Vignesh V Menon1, Hadi Amirpour1, Mohammad Ghanbari1,2, and Christian Timmerer1 1 Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria 2 School of Computer Science and Electronic Engineering, University of Essex, UK 16-19 October 2022 Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 1
  • 2. Outline 1 Introduction 2 Research Problem 3 ETPS 4 Evaluation 5 Conclusions Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 2
  • 3. Introduction Introduction CBR versus VBR Figure: Constant Bitrate (CBR) versus Variable Bitrate (VBR) encoding. In live streaming, Constant Bitrate (CBR) rate- control mode is used to encode video sequences at a fixed bitrate ladder. The consistency of CBR makes it more reliable for time-sensitive data transport. In this method, there is no concern about the bitrate exceeding internet speeds. However, this method may result in low compression efficiency. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 3
  • 4. Introduction Introduction Variable Bitrate (VBR) encoding Figure: Two-pass encoding. VBR encoding schemes are implemented as two-pass encoding, also known as multi-pass encoding used to retain the best compression efficiency during encoding. In the first-pass of two-pass encoding, the input data from the video is analyzed (and stored in a log file). The collected data from the first-pass is used to achieve the best encoding compression efficiency in the second-pass. Two-pass encoding is usually controlled by the average bitrate setting, bitrate range setting (minimal and maximal allowed bitrate), or target video file size. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 4
  • 5. Introduction Introduction Constrained Variable Bitrate (cVBR) encoding Figure: cVBR encoding. A ”rate factor” first-pass to identify the optimized CRF to achieve the target bitrate. In the second-pass, the segment is encoded with the selected optimized CRF with the maximum bitrate and maximum buffer window constraints. The desired target bitrate is achieved with maximum compression efficiency, and min- imum quality fluctuation. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 5
  • 6. Research Problem Research Problem In two-pass encoding, the encoder processes all segments for the stream twice. Due to the latency introduced by the first-pass to achieve the optimal encoding parameters for the second-pass, two-pass encoding is not used in live streaming applications. This paper targets a two-pass encoding scheme optimized for live streaming applications. The target average bitrate (bT ) is aimed to be achieved uniformly throughout the streaming session with the highest possible compression efficiency. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 6
  • 7. ETPS ETPS Figure: ETPS architecture. The first-pass predicts the optimized CRF to achieve the target bitrate. In the second-pass, the segment is encoded with the selected, optimized CRF with the maximum bitrate and maximum buffer window constraints. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 7
  • 8. ETPS Phase 1: Video Complexity Feature Extraction ETPS Phase 1: Video Complexity Feature Extraction Compute texture energy per block A DCT-based energy function is used to determine the block-wise feature of each frame defined as: Hk = w−1 X i=0 w−1 X j=0 e|( ij wh )2−1| |DCT(i, j)| (1) where wxw is the size of the block, and DCT(i, j) is the (i, j)th DCT component when i + j > 0, and 0 otherwise. The energy values of blocks in a frame is averaged to determine the energy per frame1,.2 E = C−1 X k=0 Hp,k C · w2 (2) 1 Michael King et al. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on Multimedia and Expo. 2007, pp. 1647–1650. doi: 10.1109/ICME.2007.4284983. 2 Vignesh V Menon et al. “Efficient Content-Adaptive Feature-Based Shot Detection for HTTP Adaptive Streaming”. In: 2021 IEEE International Conference on Image Processing (ICIP). 2021, pp. 2174–2178. doi: 10.1109/ICIP42928.2021.9506092. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 8
  • 9. ETPS Phase 1: Video Complexity Feature Extraction ETPS Phase 1: Video Complexity Feature Extraction hp: SAD of the block level energy values of frame p to that of the previous frame p − 1. hp = C−1 X k=0 | Hp,k, Hp−1,k | C · w2 (3) where C denotes the number of blocks in frame p. Latency Speed of feature extraction = 370fps for UHD video with 8 CPU threads and x86 SIMD optimization Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 9
  • 10. ETPS Phase 2: CRF Prediction ETPS Phase 2: CRF Prediction Hidden Layer E R4 Hidden Layer E R4 Input Layer E R3 Output Layer E R1 E h log(b) ĉ A shallow neural network is trained for each resolu- tion r and framerate f , which determines the opti- mized CRF (ĉ) based on E, h, and log(b) for every video segment. The input vector passed to the network is [E, h, log(bT )]. The target bitrate input (bT ) is passed in logarithmic scale to the network to reduce internal covariate shift.3 Rectified Linear Unit (ReLU)4 is used as the activation function and Adam5 is used as optimizer with a learning rate of 10−4. 3 Sergey Ioffe and Christian Szegedy. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”. In: Proceedings of the 32nd International Conference on Machine Learning. Vol. 37. Lille, France: PMLR, 2015, pp. 448–456. 4 Xavier Glorot, Antoine Bordes, and Y. Bengio. “Deep Sparse Rectifier Neural Networks”. In: vol. 15. Jan. 2010. 5 Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015. 2015. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 10
  • 11. Evaluation Test Methodology Test Methodology Table: Encoder Settings. Encoder x265 v3.5 Preset ultrafast Segment length 4 seconds Resolution Ultra HD (2160p) Framerate 30 fps Target bitrates 8, 12, 17, and 20 Mbps Keras6 is used as the machine learning framework. E and h features are extracted from the video segments using VCA7 run in eight CPU threads 6 Nikhil Ketkar. “Introduction to Keras”. In: Oct. 2017, pp. 95–109. isbn: 978-1-4842-2765-7. doi: 10.1007/978-1-4842-2766-4_7. 7 Vignesh V Menon et al. “VCA: Video Complexity Analyzer”. In: 13th ACM Multimedia Systems Conference (MMSys ’22). 2022. isbn: 9781450392839. doi: 10.1145/3524273.3532896. url: https://doi.org/10.1145/3524273.3532896. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 11
  • 12. Evaluation Test Methodology Test Methodology Criteria: Root Mean Squared Error (RMSE) of cG and the predicted CRF (ĉ), i.e., || cG − ĉ ||2. The first-pass time, i.e., latency in streaming introduced due to the time taken to extract E and h, and the inference time of the neural network to predict optimized CRF. Bjøntegaard delta rates8 BDRP and BDRV to measure the average increase in bitrate of the representations compared with that of the reference encoding scheme to maintain the same PSNR and VMAF, respectively. ∆T: the overall difference in encoding time of ETPS with the reference encoding scheme. The results are compared with CBR encoding and two-pass average bitrate encoding schemes9. 8 G. Bjontegaard. “Calculation of average PSNR differences between RD-curves”. In: VCEG-M33 (2001). 9 https://x265.readthedocs.io/en/master/cli.html#quality-rate-control-and-rate-distortion-options, last access: Jun 30, 2022. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 12
  • 13. Evaluation Experimental Results Experimental Results Table: Results of ETPS against Constant Bitrate (CBR) encoding and two-pass average bitrate encoding schemes. Constant bitrate (CBR) Two-pass average bitrate Dataset Video E h || cG − ĉ ||2 τp (in ms) BDRP BDRV ∆T BDRP BDRV ∆T MCML10 Basketball 15.75 9.69 0.86 8 -17.80% -14.36% 0.43% -0.48% 0.50% -45.15% MCML Bunny 23.03 4.89 1.58 7 -9.92% -9.09% 0.78% 0.04% -1.14% -43.88% MCML Crowd 33.11 7.03 1.58 7 -7.63% -1.47% 0.56% 0.75% 1.05% -43.34% MCML Dolls 19.91 12.22 0.86 8 -5.73% -4.05% 1.01% 1.21% 1.97% -45.26% MCML Flowers 12.01 10.47 0.50 7 -8.44% -7.12% -0.59% 0.38% 0.49% -43.90% MCML Park 26.07 27.34 0.71 9 -5.16% -2.62% 1.10% 0.16% 0.87% -44.27% SJTU11 BundNightScape 54.90 11.62 1.93 8 -14.63% -13.51% -1.62% -1.11% -1.27% -40.91% SJTU CampfireParty 51.50 42.38 1.58 8 -4.66% -3.66% -1.72% -0.51% -0.35% -43.04% SJTU Runners 104.30 17.44 1.32 7 -8.14% -2.44% -1.34% -0.77% -1.08% -42.69% SJTU TallBuildings 94.35 7.70 1.65 8 -17.30% -12.54% -1.02% 0.89% 0.59% -43.37% SJTU TrafficAndBuilding 60.54 11.55 1.87 8 -15.41% -9.91% -0.43% 0.06% -2.42% -43.96% SJTU TrafficFlow 58.93 8.61 1.32 8 -15.66% -1.17% 0.74% 1.19% 1.43% -43.86% SJTU TreeShade 80.19 15.83 0.50 8 -11.82% -6.85% -0.79% 0.52% 0.59% -44.75% SJTU Wood 114.26 8.83 0.71 8 -10.14% -21.58% 0.52% 1.34% 1.88% -44.50% Average 1.02 8 -10.89% -8.60% 0.65% 0.26% 0.38% -43.78% 10 Manri Cheon and Jong-Seok Lee. “Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience”. In: IEEE Transactions on Circuits and Systems for Video Technology 28.7 (2018), pp. 1467–1480. doi: 10.1109/TCSVT.2017.2683504. 11 L. Song et al. “The SJTU 4K Video Sequence Dataset”. In: Fifth International Workshop on Quality of Multimedia Experience (QoMEX2013) (July 2013). Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 13
  • 14. Evaluation Experimental Results Experimental Results MCML sequences are sequentially encoded using ETPS at 8, 12, 17, and 20 Mbps, and the achieved average bitrate is recorded. The average target bitrate is consistently achieved irrespective of the spatial and temporal complexities of the segments. 0 4 8 12 16 20 24 28 32 Time (in seconds) 8 10 12 14 16 18 20 Average bitrate (in Mbps) 8 Mbps 12 Mbps 17 Mbps 20 Mbps Figure: The average bitrate in a streaming session where MCML sequences are encoded using ETPS and are streamed at intervals of four seconds for target average bitrates 8, 12, 17, 20 Mbps. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 14
  • 15. Conclusions Conclusions This paper proposed ETPS, a low-latency two-pass constrained variable bitrate (cVBR) encoding scheme for live streaming applications. ETPS includes a prediction algorithm for optimized CRF at a given target bitrate, resolution, and framerate of video segments. DCT-energy-based features are used to determine segments’ spatial and temporal complex- ity. Live streaming using ETPS has similar compression efficiency as that of the two-pass average bitrate encoding scheme, but with a significant encoding time-saving of 43.78%. Compared to CBR encoding, ETPS yields bitrate saving of 10.89% and 8.60% to maintain the same PSNR and VMAF, respectively, with an additional encoding time of 0.65%. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 15
  • 16. Q & A Q & A Thank you for your attention! Vignesh V Menon (vignesh.menon@aau.at) Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 16