In two-pass encoding, also known as multi-pass encoding, the input video content is analyzed in the first-pass to help the second-pass encoding utilize better encoding decisions and improve overall compression efficiency. In live streaming applications, a single-pass encoding scheme is mainly used to avoid the additional first-pass encoding run-time to analyze the complexity of every video content. This paper introduces an Efficient low-latency Two-Pass encoding Scheme (ETPS) for live video streaming applications. In this scheme, Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for every video segment are extracted in the first-pass to predict each target bitrate’s optimal constant rate factor (CRF) for the second-pass constrained variable bitrate (cVBR) encoding. Experimental results show that, on average, ETPS compared to a traditional two-pass average bitrate encoding scheme yields encoding time savings of 43.78% without any noticeable drop in compression efficiency. Additionally, compared to a single-pass constant bitrate (CBR) encoding, it yields bitrate savings of 10.89% and 8.60% to maintain the same PSNR and VMAF, respectively.

1. ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live
Streaming
Vignesh V Menon1, Hadi Amirpour1, Mohammad Ghanbari1,2, and Christian Timmerer1
1
Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria
2
School of Computer Science and Electronic Engineering, University of Essex, UK
16-19 October 2022
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 1

2. Outline
1 Introduction
2 Research Problem
3 ETPS
4 Evaluation
5 Conclusions
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 2

3. Introduction
Introduction
CBR versus VBR
Figure: Constant Bitrate (CBR) versus Variable
Bitrate (VBR) encoding.
In live streaming, Constant Bitrate (CBR) rate-
control mode is used to encode video sequences
at a fixed bitrate ladder. The consistency of
CBR makes it more reliable for time-sensitive
data transport.
In this method, there is no concern about the
bitrate exceeding internet speeds. However, this
method may result in low compression efficiency.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 3

4. Introduction
Introduction
Variable Bitrate (VBR) encoding
Figure: Two-pass encoding.
VBR encoding schemes are implemented as two-pass encoding, also known as multi-pass
encoding used to retain the best compression efficiency during encoding.
In the first-pass of two-pass encoding, the input data from the video is analyzed (and stored
in a log file). The collected data from the first-pass is used to achieve the best encoding
compression efficiency in the second-pass.
Two-pass encoding is usually controlled by the average bitrate setting, bitrate range setting
(minimal and maximal allowed bitrate), or target video file size.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 4

5. Introduction
Introduction
Constrained Variable Bitrate (cVBR) encoding
Figure: cVBR encoding.
A ”rate factor” first-pass to identify the optimized CRF to achieve the target bitrate.
In the second-pass, the segment is encoded with the selected optimized CRF with the
maximum bitrate and maximum buffer window constraints.
The desired target bitrate is achieved with maximum compression efficiency, and min-
imum quality fluctuation.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 5

6. Research Problem
Research Problem
In two-pass encoding, the encoder processes all segments for the stream twice.
Due to the latency introduced by the first-pass to achieve the optimal encoding parameters
for the second-pass, two-pass encoding is not used in live streaming applications.
This paper targets a two-pass encoding scheme optimized for live streaming applications. The
target average bitrate (bT ) is aimed to be achieved uniformly throughout the streaming session
with the highest possible compression efficiency.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 6

7. ETPS
ETPS
Figure: ETPS architecture.
The first-pass predicts the optimized CRF to achieve the target bitrate.
In the second-pass, the segment is encoded with the selected, optimized CRF with the
maximum bitrate and maximum buffer window constraints.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 7

8. ETPS Phase 1: Video Complexity Feature Extraction
ETPS
Phase 1: Video Complexity Feature Extraction
Compute texture energy per block
A DCT-based energy function is used to determine the block-wise feature of each frame
defined as:
Hk =
w−1
X
i=0
w−1
X
j=0
e|( ij
wh
)2−1|
|DCT(i, j)| (1)
where wxw is the size of the block, and DCT(i, j) is the (i, j)th DCT component when
i + j > 0, and 0 otherwise.
The energy values of blocks in a frame is averaged to determine the energy per frame1,.2
E =
C−1
X
k=0
Hp,k
C · w2
(2)
1
Michael King et al. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on Multimedia and Expo. 2007,
pp. 1647–1650. doi: 10.1109/ICME.2007.4284983.
2
Vignesh V Menon et al. “Efficient Content-Adaptive Feature-Based Shot Detection for HTTP Adaptive Streaming”. In: 2021 IEEE International Conference
on Image Processing (ICIP). 2021, pp. 2174–2178. doi: 10.1109/ICIP42928.2021.9506092.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 8

9. ETPS Phase 1: Video Complexity Feature Extraction
ETPS
Phase 1: Video Complexity Feature Extraction
hp: SAD of the block level energy values of frame p to that of the previous frame p − 1.
hp =
C−1
X
k=0
| Hp,k, Hp−1,k |
C · w2
(3)
where C denotes the number of blocks in frame p.
Latency
Speed of feature extraction = 370fps for UHD video with 8 CPU threads and x86 SIMD
optimization
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 9

10. ETPS Phase 2: CRF Prediction
ETPS
Phase 2: CRF Prediction
Hidden Layer
E R4
Hidden Layer
E R4
Input Layer
E R3
Output Layer
E R1
E
h
log(b)
ĉ
A shallow neural network is trained for each resolu-
tion r and framerate f , which determines the opti-
mized CRF (ĉ) based on E, h, and log(b) for every
video segment.
The input vector passed to the network is [E, h, log(bT )].
The target bitrate input (bT ) is passed in logarithmic scale to the network to reduce internal
covariate shift.3
Rectified Linear Unit (ReLU)4 is used as the activation function and Adam5 is used as
optimizer with a learning rate of 10−4.
3
Sergey Ioffe and Christian Szegedy. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”. In: Proceedings of the
32nd International Conference on Machine Learning. Vol. 37. Lille, France: PMLR, 2015, pp. 448–456.
4
Xavier Glorot, Antoine Bordes, and Y. Bengio. “Deep Sparse Rectifier Neural Networks”. In: vol. 15. Jan. 2010.
5
Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR
2015, San Diego, CA, USA, May 7-9, 2015. 2015.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 10

11. Evaluation Test Methodology
Test Methodology
Table: Encoder Settings.
Encoder x265 v3.5
Preset ultrafast
Segment length 4 seconds
Resolution Ultra HD (2160p)
Framerate 30 fps
Target bitrates 8, 12, 17, and 20 Mbps
Keras6 is used as the machine learning framework.
E and h features are extracted from the video segments using VCA7 run in eight CPU
threads
6
Nikhil Ketkar. “Introduction to Keras”. In: Oct. 2017, pp. 95–109. isbn: 978-1-4842-2765-7. doi: 10.1007/978-1-4842-2766-4_7.
7
Vignesh V Menon et al. “VCA: Video Complexity Analyzer”. In: 13th ACM Multimedia Systems Conference (MMSys ’22). 2022. isbn: 9781450392839. doi:
10.1145/3524273.3532896. url: https://doi.org/10.1145/3524273.3532896.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 11

12. Evaluation Test Methodology
Test Methodology
Criteria:
Root Mean Squared Error (RMSE) of cG and the predicted CRF (ĉ), i.e., || cG − ĉ ||2.
The first-pass time, i.e., latency in streaming introduced due to the time taken to extract
E and h, and the inference time of the neural network to predict optimized CRF.
Bjøntegaard delta rates8 BDRP and BDRV to measure the average increase in bitrate of
the representations compared with that of the reference encoding scheme to maintain the
same PSNR and VMAF, respectively.
∆T: the overall difference in encoding time of ETPS with the reference encoding scheme.
The results are compared with CBR encoding and two-pass average bitrate encoding schemes9.
8
G. Bjontegaard. “Calculation of average PSNR differences between RD-curves”. In: VCEG-M33 (2001).
9
https://x265.readthedocs.io/en/master/cli.html#quality-rate-control-and-rate-distortion-options, last access: Jun 30, 2022.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 12

13. Evaluation Experimental Results
Experimental Results
Table: Results of ETPS against Constant Bitrate (CBR) encoding and two-pass average bitrate encoding
schemes.
Constant bitrate (CBR) Two-pass average bitrate
Dataset Video E h || cG − ĉ ||2 τp (in ms) BDRP BDRV ∆T BDRP BDRV ∆T
MCML10 Basketball 15.75 9.69 0.86 8 -17.80% -14.36% 0.43% -0.48% 0.50% -45.15%
MCML Bunny 23.03 4.89 1.58 7 -9.92% -9.09% 0.78% 0.04% -1.14% -43.88%
MCML Crowd 33.11 7.03 1.58 7 -7.63% -1.47% 0.56% 0.75% 1.05% -43.34%
MCML Dolls 19.91 12.22 0.86 8 -5.73% -4.05% 1.01% 1.21% 1.97% -45.26%
MCML Flowers 12.01 10.47 0.50 7 -8.44% -7.12% -0.59% 0.38% 0.49% -43.90%
MCML Park 26.07 27.34 0.71 9 -5.16% -2.62% 1.10% 0.16% 0.87% -44.27%
SJTU11 BundNightScape 54.90 11.62 1.93 8 -14.63% -13.51% -1.62% -1.11% -1.27% -40.91%
SJTU CampfireParty 51.50 42.38 1.58 8 -4.66% -3.66% -1.72% -0.51% -0.35% -43.04%
SJTU Runners 104.30 17.44 1.32 7 -8.14% -2.44% -1.34% -0.77% -1.08% -42.69%
SJTU TallBuildings 94.35 7.70 1.65 8 -17.30% -12.54% -1.02% 0.89% 0.59% -43.37%
SJTU TrafficAndBuilding 60.54 11.55 1.87 8 -15.41% -9.91% -0.43% 0.06% -2.42% -43.96%
SJTU TrafficFlow 58.93 8.61 1.32 8 -15.66% -1.17% 0.74% 1.19% 1.43% -43.86%
SJTU TreeShade 80.19 15.83 0.50 8 -11.82% -6.85% -0.79% 0.52% 0.59% -44.75%
SJTU Wood 114.26 8.83 0.71 8 -10.14% -21.58% 0.52% 1.34% 1.88% -44.50%
Average 1.02 8 -10.89% -8.60% 0.65% 0.26% 0.38% -43.78%
10
Manri Cheon and Jong-Seok Lee. “Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience”. In: IEEE
Transactions on Circuits and Systems for Video Technology 28.7 (2018), pp. 1467–1480. doi: 10.1109/TCSVT.2017.2683504.
11
L. Song et al. “The SJTU 4K Video Sequence Dataset”. In: Fifth International Workshop on Quality of Multimedia Experience (QoMEX2013) (July 2013).
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 13

14. Evaluation Experimental Results
Experimental Results
MCML sequences are sequentially encoded using ETPS at 8, 12, 17, and 20 Mbps, and the
achieved average bitrate is recorded.
The average target bitrate is consistently achieved irrespective of the spatial and temporal
complexities of the segments.
0 4 8 12 16 20 24 28 32
Time (in seconds)
8
10
12
14
16
18
20
Average
bitrate
(in
Mbps)
8 Mbps
12 Mbps
17 Mbps
20 Mbps
Figure: The average bitrate in a streaming session where MCML sequences are encoded using ETPS and
are streamed at intervals of four seconds for target average bitrates 8, 12, 17, 20 Mbps.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 14

15. Conclusions
Conclusions
This paper proposed ETPS, a low-latency two-pass constrained variable bitrate (cVBR)
encoding scheme for live streaming applications.
ETPS includes a prediction algorithm for optimized CRF at a given target bitrate, resolution,
and framerate of video segments.
DCT-energy-based features are used to determine segments’ spatial and temporal complex-
ity.
Live streaming using ETPS has similar compression efficiency as that of the two-pass average
bitrate encoding scheme, but with a significant encoding time-saving of 43.78%.
Compared to CBR encoding, ETPS yields bitrate saving of 10.89% and 8.60% to maintain
the same PSNR and VMAF, respectively, with an additional encoding time of 0.65%.
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 15

16. Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 16