Successfully reported this slideshow.
Your SlideShare is downloading. ×

ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf

Ad

ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live
Streaming
Vignesh V Menon1, Hadi Amirpour1, Mohammad Ghanbari1,...

Ad

Outline
1 Introduction
2 Research Problem
3 ETPS
4 Evaluation
5 Conclusions
Vignesh V Menon ETPS: Efficient Two-pass Encod...

Ad

Introduction
Introduction
CBR versus VBR
Figure: Constant Bitrate (CBR) versus Variable
Bitrate (VBR) encoding.
In live st...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Check these out next

1 of 16 Ad
1 of 16 Ad

ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf

Download to read offline

In two-pass encoding, also known as multi-pass encoding, the input video content is analyzed in the first-pass to help the second-pass encoding utilize better encoding decisions and improve overall compression efficiency. In live streaming applications, a single-pass encoding scheme is mainly used to avoid the additional first-pass encoding run-time to analyze the complexity of every video content. This paper introduces an Efficient low-latency Two-Pass encoding Scheme (ETPS) for live video streaming applications. In this scheme, Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for every video segment are extracted in the first-pass to predict each target bitrate’s optimal constant rate factor (CRF) for the second-pass
constrained variable bitrate (cVBR) encoding. Experimental results show that, on average, ETPS compared to a traditional two-pass average bitrate encoding scheme yields encoding time savings of 43.78% without any noticeable drop in compression efficiency. Additionally, compared to a single-pass constant bitrate (CBR) encoding, it yields bitrate savings of 10.89% and 8.60% to
maintain the same PSNR and VMAF, respectively.

In two-pass encoding, also known as multi-pass encoding, the input video content is analyzed in the first-pass to help the second-pass encoding utilize better encoding decisions and improve overall compression efficiency. In live streaming applications, a single-pass encoding scheme is mainly used to avoid the additional first-pass encoding run-time to analyze the complexity of every video content. This paper introduces an Efficient low-latency Two-Pass encoding Scheme (ETPS) for live video streaming applications. In this scheme, Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for every video segment are extracted in the first-pass to predict each target bitrate’s optimal constant rate factor (CRF) for the second-pass
constrained variable bitrate (cVBR) encoding. Experimental results show that, on average, ETPS compared to a traditional two-pass average bitrate encoding scheme yields encoding time savings of 43.78% without any noticeable drop in compression efficiency. Additionally, compared to a single-pass constant bitrate (CBR) encoding, it yields bitrate savings of 10.89% and 8.60% to
maintain the same PSNR and VMAF, respectively.

Advertisement
Advertisement

ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf

  1. 1. ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming Vignesh V Menon1, Hadi Amirpour1, Mohammad Ghanbari1,2, and Christian Timmerer1 1 Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria 2 School of Computer Science and Electronic Engineering, University of Essex, UK 16-19 October 2022 Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 1
  2. 2. Outline 1 Introduction 2 Research Problem 3 ETPS 4 Evaluation 5 Conclusions Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 2
  3. 3. Introduction Introduction CBR versus VBR Figure: Constant Bitrate (CBR) versus Variable Bitrate (VBR) encoding. In live streaming, Constant Bitrate (CBR) rate- control mode is used to encode video sequences at a fixed bitrate ladder. The consistency of CBR makes it more reliable for time-sensitive data transport. In this method, there is no concern about the bitrate exceeding internet speeds. However, this method may result in low compression efficiency. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 3
  4. 4. Introduction Introduction Variable Bitrate (VBR) encoding Figure: Two-pass encoding. VBR encoding schemes are implemented as two-pass encoding, also known as multi-pass encoding used to retain the best compression efficiency during encoding. In the first-pass of two-pass encoding, the input data from the video is analyzed (and stored in a log file). The collected data from the first-pass is used to achieve the best encoding compression efficiency in the second-pass. Two-pass encoding is usually controlled by the average bitrate setting, bitrate range setting (minimal and maximal allowed bitrate), or target video file size. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 4
  5. 5. Introduction Introduction Constrained Variable Bitrate (cVBR) encoding Figure: cVBR encoding. A ”rate factor” first-pass to identify the optimized CRF to achieve the target bitrate. In the second-pass, the segment is encoded with the selected optimized CRF with the maximum bitrate and maximum buffer window constraints. The desired target bitrate is achieved with maximum compression efficiency, and min- imum quality fluctuation. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 5
  6. 6. Research Problem Research Problem In two-pass encoding, the encoder processes all segments for the stream twice. Due to the latency introduced by the first-pass to achieve the optimal encoding parameters for the second-pass, two-pass encoding is not used in live streaming applications. This paper targets a two-pass encoding scheme optimized for live streaming applications. The target average bitrate (bT ) is aimed to be achieved uniformly throughout the streaming session with the highest possible compression efficiency. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 6
  7. 7. ETPS ETPS Figure: ETPS architecture. The first-pass predicts the optimized CRF to achieve the target bitrate. In the second-pass, the segment is encoded with the selected, optimized CRF with the maximum bitrate and maximum buffer window constraints. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 7
  8. 8. ETPS Phase 1: Video Complexity Feature Extraction ETPS Phase 1: Video Complexity Feature Extraction Compute texture energy per block A DCT-based energy function is used to determine the block-wise feature of each frame defined as: Hk = w−1 X i=0 w−1 X j=0 e|( ij wh )2−1| |DCT(i, j)| (1) where wxw is the size of the block, and DCT(i, j) is the (i, j)th DCT component when i + j > 0, and 0 otherwise. The energy values of blocks in a frame is averaged to determine the energy per frame1,.2 E = C−1 X k=0 Hp,k C · w2 (2) 1 Michael King et al. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on Multimedia and Expo. 2007, pp. 1647–1650. doi: 10.1109/ICME.2007.4284983. 2 Vignesh V Menon et al. “Efficient Content-Adaptive Feature-Based Shot Detection for HTTP Adaptive Streaming”. In: 2021 IEEE International Conference on Image Processing (ICIP). 2021, pp. 2174–2178. doi: 10.1109/ICIP42928.2021.9506092. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 8
  9. 9. ETPS Phase 1: Video Complexity Feature Extraction ETPS Phase 1: Video Complexity Feature Extraction hp: SAD of the block level energy values of frame p to that of the previous frame p − 1. hp = C−1 X k=0 | Hp,k, Hp−1,k | C · w2 (3) where C denotes the number of blocks in frame p. Latency Speed of feature extraction = 370fps for UHD video with 8 CPU threads and x86 SIMD optimization Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 9
  10. 10. ETPS Phase 2: CRF Prediction ETPS Phase 2: CRF Prediction Hidden Layer E R4 Hidden Layer E R4 Input Layer E R3 Output Layer E R1 E h log(b) ĉ A shallow neural network is trained for each resolu- tion r and framerate f , which determines the opti- mized CRF (ĉ) based on E, h, and log(b) for every video segment. The input vector passed to the network is [E, h, log(bT )]. The target bitrate input (bT ) is passed in logarithmic scale to the network to reduce internal covariate shift.3 Rectified Linear Unit (ReLU)4 is used as the activation function and Adam5 is used as optimizer with a learning rate of 10−4. 3 Sergey Ioffe and Christian Szegedy. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”. In: Proceedings of the 32nd International Conference on Machine Learning. Vol. 37. Lille, France: PMLR, 2015, pp. 448–456. 4 Xavier Glorot, Antoine Bordes, and Y. Bengio. “Deep Sparse Rectifier Neural Networks”. In: vol. 15. Jan. 2010. 5 Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015. 2015. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 10
  11. 11. Evaluation Test Methodology Test Methodology Table: Encoder Settings. Encoder x265 v3.5 Preset ultrafast Segment length 4 seconds Resolution Ultra HD (2160p) Framerate 30 fps Target bitrates 8, 12, 17, and 20 Mbps Keras6 is used as the machine learning framework. E and h features are extracted from the video segments using VCA7 run in eight CPU threads 6 Nikhil Ketkar. “Introduction to Keras”. In: Oct. 2017, pp. 95–109. isbn: 978-1-4842-2765-7. doi: 10.1007/978-1-4842-2766-4_7. 7 Vignesh V Menon et al. “VCA: Video Complexity Analyzer”. In: 13th ACM Multimedia Systems Conference (MMSys ’22). 2022. isbn: 9781450392839. doi: 10.1145/3524273.3532896. url: https://doi.org/10.1145/3524273.3532896. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 11
  12. 12. Evaluation Test Methodology Test Methodology Criteria: Root Mean Squared Error (RMSE) of cG and the predicted CRF (ĉ), i.e., || cG − ĉ ||2. The first-pass time, i.e., latency in streaming introduced due to the time taken to extract E and h, and the inference time of the neural network to predict optimized CRF. Bjøntegaard delta rates8 BDRP and BDRV to measure the average increase in bitrate of the representations compared with that of the reference encoding scheme to maintain the same PSNR and VMAF, respectively. ∆T: the overall difference in encoding time of ETPS with the reference encoding scheme. The results are compared with CBR encoding and two-pass average bitrate encoding schemes9. 8 G. Bjontegaard. “Calculation of average PSNR differences between RD-curves”. In: VCEG-M33 (2001). 9 https://x265.readthedocs.io/en/master/cli.html#quality-rate-control-and-rate-distortion-options, last access: Jun 30, 2022. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 12
  13. 13. Evaluation Experimental Results Experimental Results Table: Results of ETPS against Constant Bitrate (CBR) encoding and two-pass average bitrate encoding schemes. Constant bitrate (CBR) Two-pass average bitrate Dataset Video E h || cG − ĉ ||2 τp (in ms) BDRP BDRV ∆T BDRP BDRV ∆T MCML10 Basketball 15.75 9.69 0.86 8 -17.80% -14.36% 0.43% -0.48% 0.50% -45.15% MCML Bunny 23.03 4.89 1.58 7 -9.92% -9.09% 0.78% 0.04% -1.14% -43.88% MCML Crowd 33.11 7.03 1.58 7 -7.63% -1.47% 0.56% 0.75% 1.05% -43.34% MCML Dolls 19.91 12.22 0.86 8 -5.73% -4.05% 1.01% 1.21% 1.97% -45.26% MCML Flowers 12.01 10.47 0.50 7 -8.44% -7.12% -0.59% 0.38% 0.49% -43.90% MCML Park 26.07 27.34 0.71 9 -5.16% -2.62% 1.10% 0.16% 0.87% -44.27% SJTU11 BundNightScape 54.90 11.62 1.93 8 -14.63% -13.51% -1.62% -1.11% -1.27% -40.91% SJTU CampfireParty 51.50 42.38 1.58 8 -4.66% -3.66% -1.72% -0.51% -0.35% -43.04% SJTU Runners 104.30 17.44 1.32 7 -8.14% -2.44% -1.34% -0.77% -1.08% -42.69% SJTU TallBuildings 94.35 7.70 1.65 8 -17.30% -12.54% -1.02% 0.89% 0.59% -43.37% SJTU TrafficAndBuilding 60.54 11.55 1.87 8 -15.41% -9.91% -0.43% 0.06% -2.42% -43.96% SJTU TrafficFlow 58.93 8.61 1.32 8 -15.66% -1.17% 0.74% 1.19% 1.43% -43.86% SJTU TreeShade 80.19 15.83 0.50 8 -11.82% -6.85% -0.79% 0.52% 0.59% -44.75% SJTU Wood 114.26 8.83 0.71 8 -10.14% -21.58% 0.52% 1.34% 1.88% -44.50% Average 1.02 8 -10.89% -8.60% 0.65% 0.26% 0.38% -43.78% 10 Manri Cheon and Jong-Seok Lee. “Subjective and Objective Quality Assessment of Compressed 4K UHD Videos for Immersive Experience”. In: IEEE Transactions on Circuits and Systems for Video Technology 28.7 (2018), pp. 1467–1480. doi: 10.1109/TCSVT.2017.2683504. 11 L. Song et al. “The SJTU 4K Video Sequence Dataset”. In: Fifth International Workshop on Quality of Multimedia Experience (QoMEX2013) (July 2013). Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 13
  14. 14. Evaluation Experimental Results Experimental Results MCML sequences are sequentially encoded using ETPS at 8, 12, 17, and 20 Mbps, and the achieved average bitrate is recorded. The average target bitrate is consistently achieved irrespective of the spatial and temporal complexities of the segments. 0 4 8 12 16 20 24 28 32 Time (in seconds) 8 10 12 14 16 18 20 Average bitrate (in Mbps) 8 Mbps 12 Mbps 17 Mbps 20 Mbps Figure: The average bitrate in a streaming session where MCML sequences are encoded using ETPS and are streamed at intervals of four seconds for target average bitrates 8, 12, 17, 20 Mbps. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 14
  15. 15. Conclusions Conclusions This paper proposed ETPS, a low-latency two-pass constrained variable bitrate (cVBR) encoding scheme for live streaming applications. ETPS includes a prediction algorithm for optimized CRF at a given target bitrate, resolution, and framerate of video segments. DCT-energy-based features are used to determine segments’ spatial and temporal complex- ity. Live streaming using ETPS has similar compression efficiency as that of the two-pass average bitrate encoding scheme, but with a significant encoding time-saving of 43.78%. Compared to CBR encoding, ETPS yields bitrate saving of 10.89% and 8.60% to maintain the same PSNR and VMAF, respectively, with an additional encoding time of 0.65%. Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 15
  16. 16. Q & A Q & A Thank you for your attention! Vignesh V Menon (vignesh.menon@aau.at) Vignesh V Menon ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming 16

×