This document presents an analysis of video complexity features and optimizations for efficient video encoding. It introduces the Video Complexity Analyzer (VCA) library, which extracts low-complexity DCT-based features like texture energy and luminescence to predict optimal encoding parameters. Evaluations show these features correlate well with ground truths like bitrate. Optimizations to VCA like multi-threading, SIMD, and low-pass DCT improve its energy efficiency, providing up to a 97% reduction compared to other methods. In conclusion, VCA is an open-source video complexity analyzer that encoders can use to make encoding decisions in an energy efficient manner.

1. Green Video Complexity Analysis for Efficient Encoding in
Adaptive Video Streaming
Vignesh V Menon1, Christian Feldmann2, Klaus Schoeffmann1, Mohammad Ghanbari3,
Christian Timmerer1
1
Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria
2
Bitmovin, Klagenfurt, Austria
3
School of Computer Science and Electronic Engineering, University of Essex, UK
7 – 10 June 2023
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 1

2. Outline
1 Background
2 SITI
3 Video Complexity Features
4 Accuracy Analysis
5 Performance Optimizations
6 Conclusions
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 2

3. Background
Background
Why VCA?
The optimal encoding parameters depend on the video content complexity.
There is a need to extract content features that can represent the video content complexity
to predict the optimal encoding parameters for that video content.
Encoder
Analysis stats
Encoded bitstream
Input video
Video complexity
analysis
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 3

4. SITI
SITI1
Spatial Information (SI) indicates the maximum spatial detail in a video.
SI = max{std[Sobel(F(x, y, t))]} (1)
where F(x, y, t) represents each video frame.
Temporal Information (TI) is the maximum temporal variation between two successive
frames.
D(x, y, t) = F(x, y, t)) − F(x, y, t − 1)). (2)
TI = max{std[D(x, y, t)]} (3)
1
https://github.com/Telecommunication-Telemedia-Assessment/SITI, last access: Apr 30, 2023.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 4

5. Video Complexity Features Texture Energy
Texture Energy
Compute texture energy per block
A DCT-based energy function is used to determine the block-wise feature of each frame
defined as:
HY ,p,k =
w−1
X
i=0
w−1
X
j=0
e|( ij
wh
)2−1|
|DCT(i, j)| (4)
where wxw is the size of the block, and DCT(i, j) is the (i, j)th DCT component when
i + j > 0, and 0 otherwise.
The energy values of blocks in a frame are averaged to determine the energy per frame.2
EY =
C−1
X
k=0
HY ,p,k
C · w2
(5)
2
Michael King, Zinovi Tauber, and Ze-Nian Li. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on
Multimedia and Expo. 2007, pp. 1647–1650. doi: 10.1109/ICME.2007.4284983.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 5

6. Video Complexity Features Texture energy gradient
Texture energy gradient
hp: SAD of the block level energy values of frame p to that of the previous frame p − 1.
h =
C−1
X
k=0
| HY ,p,k, HY ,p−1,k |
C · w2
(6)
where C denotes the number of blocks in frame p.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 6

7. Video Complexity Features Luminescence
Luminescence
The luminescence of non-overlapping blocks k of each frame p is defined as:
LY ,k =
p
DCT(0, 0) (7)
where DCT(0, 0) is the DC component in the DCT calculation. The block-wise luminescence
is averaged per frame denoted as LY as shown below.
LY =
K−1
X
k=0
LY ,p,k
C · w2
(8)
where C denotes the number of blocks in frame p.
Chroma features
VCA also determines chroma texture energy EU and EV (for U and V planes), and the chromi-
nance LU and LV (for U and V planes).
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 7

8. Video Complexity Features Example heatmap
Video Complexity Features
(a) Original video frame (b) Heatmap of L (c) Heatmap of E
(d) Heatmap of h
Figure: Example heatmap of Luminescence (L), spatial texture (E) and temporal activity (h) features of
the 2nd
frame of CoverSong 1080P 0a86 video of Youtube UGC dataset extracted using VCA.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 8

9. Accuracy Analysis
Accuracy Analysis
Correlation of spatial complexity features with the ground truth
Bitrate in All Intra configuration3 is considered as the ground truth of the spatial complexity.
Figure: PCC between SI and EY , respectively, with bitrate in All Intra configuration with medium preset
of x265 encoder with ultrafast preset of x265 encoder for the VCD dataset.4
3
Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018.
4
Hadi Amirpour et al. “VCD: Video Complexity Dataset”. In: Proceedings of the 13th ACM Multimedia Systems Conference. MMSys ’22. Athlone, Ireland:
Association for Computing Machinery, 2022, 234–239. isbn: 9781450392839. doi: 10.1145/3524273.3532892. url:
https://doi.org/10.1145/3524273.3532892.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 9

10. Accuracy Analysis
Accuracy Analysis
Correlation of complexity features with bitrate in the Low Delay P picture (LDP) configuration
Figure: PCC between SI, EY , TI and h with bitrate in the Low Delay P picture configuration with
ultrafast preset of x265 encoder for the VCD dataset.
EY and h strongly correlate with the encoding bitrate.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 10

11. Performance Optimizations
Performance Optimizations
Multi-threading optimizations
Creates multiple threads within a VCA execution instance, which executes independently
but concurrently, sharing process resources.
Independent threads carry out DCT-energy computation per block.
x86 SIMD optimizations
SIMD optimization5 of DCT functions implemented as intrinsic and assembly codes for
x86 architecture.
5
Praveen Kumar Tiwari et al. “Accelerating x265 with intel® advanced vector extensions 512”. In: White Paper on the Intel Developers Page (2018).
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 11

12. Performance Optimizations
Performance Optimizations
Low-pass DCT optimization
Unlike SI, the EY feature exhibits better correlation across resolutions.
In VCA v2.0, the complexity features are evaluated on the video spatially downsampled by
a factor of two.
(a) (b)
Figure: PCC between the spatial complexity features (a) SI and (b) EY across multiple resolutions for
the VCD dataset.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 12

13. Performance Optimizations
Performance Optimizations
Processing time results
Figure: Processing time of video content complexity analysis methods.
m1: without any performance optimizations
m2: with SIMD
m3: with SIMD and low-pass DCT
m4: with SIMD , low-pass DCT and multi-threading (2 threads)
m5: with SIMD , low-pass DCT and multi-threading (4 threads)
m6: with SIMD , low-pass DCT and multi-threading (8 threads)
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 13

14. Performance Optimizations
Performance Optimizations
Energy consumption results
Figure: Energy consumption of video content complexity analysis methods.
m1: without any performance optimizations
m2: with SIMD
m3: with SIMD and low-pass DCT
m4: with SIMD , low-pass DCT and multi-threading (2 threads)
m5: with SIMD , low-pass DCT and multi-threading (4 threads)
m6: with SIMD , low-pass DCT and multi-threading (8 threads)
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 14

15. Conclusions
Conclusions
VCA is an open-source video complexity analyzer library published under the GNU GPLv3
license.
Low-complexity DCT-based energy features are extracted using VCA v2.0, which encoders
use to derive decisions like bitrate-ladder, frame-type, block-partitioning, and much more.
Multi-threading, x86 SIMD, and low-pass DCT optimizations improve the energy efficiency
of the VCA implementation.
Compared to the state-of-the-art SITI implementation of video complexity analysis, VCA
v2.0 yields a better estimation of video complexity, with an energy consumption reduction
of 97.06%.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 15

16. Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
VCA
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 16