SlideShare a Scribd company logo
Green Video Complexity Analysis for Efficient Encoding in
Adaptive Video Streaming
Vignesh V Menon1, Christian Feldmann2, Klaus Schoeffmann1, Mohammad Ghanbari3,
Christian Timmerer1
1
Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria
2
Bitmovin, Klagenfurt, Austria
3
School of Computer Science and Electronic Engineering, University of Essex, UK
7 – 10 June 2023
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 1
Outline
1 Background
2 SITI
3 Video Complexity Features
4 Accuracy Analysis
5 Performance Optimizations
6 Conclusions
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 2
Background
Background
Why VCA?
The optimal encoding parameters depend on the video content complexity.
There is a need to extract content features that can represent the video content complexity
to predict the optimal encoding parameters for that video content.
Encoder
Analysis stats
Encoded bitstream
Input video
Video complexity
analysis
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 3
SITI
SITI1
Spatial Information (SI) indicates the maximum spatial detail in a video.
SI = max{std[Sobel(F(x, y, t))]} (1)
where F(x, y, t) represents each video frame.
Temporal Information (TI) is the maximum temporal variation between two successive
frames.
D(x, y, t) = F(x, y, t)) − F(x, y, t − 1)). (2)
TI = max{std[D(x, y, t)]} (3)
1
https://github.com/Telecommunication-Telemedia-Assessment/SITI, last access: Apr 30, 2023.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 4
Video Complexity Features Texture Energy
Texture Energy
Compute texture energy per block
A DCT-based energy function is used to determine the block-wise feature of each frame
defined as:
HY ,p,k =
w−1
X
i=0
w−1
X
j=0
e|( ij
wh
)2−1|
|DCT(i, j)| (4)
where wxw is the size of the block, and DCT(i, j) is the (i, j)th DCT component when
i + j > 0, and 0 otherwise.
The energy values of blocks in a frame are averaged to determine the energy per frame.2
EY =
C−1
X
k=0
HY ,p,k
C · w2
(5)
2
Michael King, Zinovi Tauber, and Ze-Nian Li. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on
Multimedia and Expo. 2007, pp. 1647–1650. doi: 10.1109/ICME.2007.4284983.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 5
Video Complexity Features Texture energy gradient
Texture energy gradient
hp: SAD of the block level energy values of frame p to that of the previous frame p − 1.
h =
C−1
X
k=0
| HY ,p,k, HY ,p−1,k |
C · w2
(6)
where C denotes the number of blocks in frame p.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 6
Video Complexity Features Luminescence
Luminescence
The luminescence of non-overlapping blocks k of each frame p is defined as:
LY ,k =
p
DCT(0, 0) (7)
where DCT(0, 0) is the DC component in the DCT calculation. The block-wise luminescence
is averaged per frame denoted as LY as shown below.
LY =
K−1
X
k=0
LY ,p,k
C · w2
(8)
where C denotes the number of blocks in frame p.
Chroma features
VCA also determines chroma texture energy EU and EV (for U and V planes), and the chromi-
nance LU and LV (for U and V planes).
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 7
Video Complexity Features Example heatmap
Video Complexity Features
(a) Original video frame (b) Heatmap of L (c) Heatmap of E
(d) Heatmap of h
Figure: Example heatmap of Luminescence (L), spatial texture (E) and temporal activity (h) features of
the 2nd
frame of CoverSong 1080P 0a86 video of Youtube UGC dataset extracted using VCA.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 8
Accuracy Analysis
Accuracy Analysis
Correlation of spatial complexity features with the ground truth
Bitrate in All Intra configuration3 is considered as the ground truth of the spatial complexity.
Figure: PCC between SI and EY , respectively, with bitrate in All Intra configuration with medium preset
of x265 encoder with ultrafast preset of x265 encoder for the VCD dataset.4
3
Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018.
4
Hadi Amirpour et al. “VCD: Video Complexity Dataset”. In: Proceedings of the 13th ACM Multimedia Systems Conference. MMSys ’22. Athlone, Ireland:
Association for Computing Machinery, 2022, 234–239. isbn: 9781450392839. doi: 10.1145/3524273.3532892. url:
https://doi.org/10.1145/3524273.3532892.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 9
Accuracy Analysis
Accuracy Analysis
Correlation of complexity features with bitrate in the Low Delay P picture (LDP) configuration
Figure: PCC between SI, EY , TI and h with bitrate in the Low Delay P picture configuration with
ultrafast preset of x265 encoder for the VCD dataset.
EY and h strongly correlate with the encoding bitrate.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 10
Performance Optimizations
Performance Optimizations
Multi-threading optimizations
Creates multiple threads within a VCA execution instance, which executes independently
but concurrently, sharing process resources.
Independent threads carry out DCT-energy computation per block.
x86 SIMD optimizations
SIMD optimization5 of DCT functions implemented as intrinsic and assembly codes for
x86 architecture.
5
Praveen Kumar Tiwari et al. “Accelerating x265 with intel® advanced vector extensions 512”. In: White Paper on the Intel Developers Page (2018).
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 11
Performance Optimizations
Performance Optimizations
Low-pass DCT optimization
Unlike SI, the EY feature exhibits better correlation across resolutions.
In VCA v2.0, the complexity features are evaluated on the video spatially downsampled by
a factor of two.
(a) (b)
Figure: PCC between the spatial complexity features (a) SI and (b) EY across multiple resolutions for
the VCD dataset.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 12
Performance Optimizations
Performance Optimizations
Processing time results
Figure: Processing time of video content complexity analysis methods.
m1: without any performance optimizations
m2: with SIMD
m3: with SIMD and low-pass DCT
m4: with SIMD , low-pass DCT and multi-threading (2 threads)
m5: with SIMD , low-pass DCT and multi-threading (4 threads)
m6: with SIMD , low-pass DCT and multi-threading (8 threads)
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 13
Performance Optimizations
Performance Optimizations
Energy consumption results
Figure: Energy consumption of video content complexity analysis methods.
m1: without any performance optimizations
m2: with SIMD
m3: with SIMD and low-pass DCT
m4: with SIMD , low-pass DCT and multi-threading (2 threads)
m5: with SIMD , low-pass DCT and multi-threading (4 threads)
m6: with SIMD , low-pass DCT and multi-threading (8 threads)
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 14
Conclusions
Conclusions
VCA is an open-source video complexity analyzer library published under the GNU GPLv3
license.
Low-complexity DCT-based energy features are extracted using VCA v2.0, which encoders
use to derive decisions like bitrate-ladder, frame-type, block-partitioning, and much more.
Multi-threading, x86 SIMD, and low-pass DCT optimizations improve the energy efficiency
of the VCA implementation.
Compared to the state-of-the-art SITI implementation of video complexity analysis, VCA
v2.0 yields a better estimation of video complexity, with an energy consumption reduction
of 97.06%.
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 15
Q & A
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
VCA
Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 16

More Related Content

Similar to Green_VCA_presentation.pdf

OPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdfOPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdf
Vignesh V Menon
 
Video complexity analyzer (VCA) for streaming applications
 Video complexity analyzer (VCA) for streaming applications Video complexity analyzer (VCA) for streaming applications
Video complexity analyzer (VCA) for streaming applications
Alpen-Adria-Universität
 
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Vignesh V Menon
 
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdfETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
Vignesh V Menon
 
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live StreamingETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
Alpen-Adria-Universität
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
Vignesh V Menon
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video StreamingPerceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
Alpen-Adria-Universität
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Vignesh V Menon
 
CAPS_Presentation.pdf
CAPS_Presentation.pdfCAPS_Presentation.pdf
CAPS_Presentation.pdf
Vignesh V Menon
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Ijripublishers Ijri
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Ijripublishers Ijri
 
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instancesVEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
Alpen-Adria-Universität
 
Low complexity video coding for sensor network
Low complexity video coding for sensor networkLow complexity video coding for sensor network
Low complexity video coding for sensor network
eSAT Journals
 
Low complexity video coding for sensor network
Low complexity video coding for sensor networkLow complexity video coding for sensor network
Low complexity video coding for sensor network
eSAT Publishing House
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
Vignesh V Menon
 
INCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVCINCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVC
Alpen-Adria-Universität
 
International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)
IJERD Editor
 
Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC Streaming
Vignesh V Menon
 
C0161018
C0161018C0161018
C0161018
IOSR Journals
 
C0161018
C0161018C0161018
C0161018
IOSR Journals
 

Similar to Green_VCA_presentation.pdf (20)

OPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdfOPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdf
 
Video complexity analyzer (VCA) for streaming applications
 Video complexity analyzer (VCA) for streaming applications Video complexity analyzer (VCA) for streaming applications
Video complexity analyzer (VCA) for streaming applications
 
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
 
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdfETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
 
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live StreamingETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video StreamingPerceptually-aware Per-title Encoding for Adaptive Video Streaming
Perceptually-aware Per-title Encoding for Adaptive Video Streaming
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
 
CAPS_Presentation.pdf
CAPS_Presentation.pdfCAPS_Presentation.pdf
CAPS_Presentation.pdf
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instancesVEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
VEED: Video Encoding Energy and CO2 Emissions Dataset for AWS EC2 instances
 
Low complexity video coding for sensor network
Low complexity video coding for sensor networkLow complexity video coding for sensor network
Low complexity video coding for sensor network
 
Low complexity video coding for sensor network
Low complexity video coding for sensor networkLow complexity video coding for sensor network
Low complexity video coding for sensor network
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
 
INCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVCINCEPT: Intra CU Depth Prediction for HEVC
INCEPT: Intra CU Depth Prediction for HEVC
 
International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)
 
Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC Streaming
 
C0161018
C0161018C0161018
C0161018
 
C0161018
C0161018C0161018
C0161018
 

More from Vignesh V Menon

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Vignesh V Menon
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Vignesh V Menon
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
Vignesh V Menon
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
Vignesh V Menon
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
Vignesh V Menon
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
Vignesh V Menon
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
Vignesh V Menon
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Vignesh V Menon
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
Vignesh V Menon
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
Vignesh V Menon
 

More from Vignesh V Menon (10)

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementat...
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
 

Recently uploaded

How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
Celine George
 
How To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-marketHow To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-market
Sikandar Ali
 
View Inheritance in Odoo 17 - Odoo 17 Slides
View Inheritance in Odoo 17 - Odoo 17  SlidesView Inheritance in Odoo 17 - Odoo 17  Slides
View Inheritance in Odoo 17 - Odoo 17 Slides
Celine George
 
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
ALBERTHISOLER1
 
2 Post harvest Physiology of Horticulture produce.pptx
2 Post harvest Physiology of Horticulture  produce.pptx2 Post harvest Physiology of Horticulture  produce.pptx
2 Post harvest Physiology of Horticulture produce.pptx
UmeshTimilsina1
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Alvaro Barbosa
 
C# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdfC# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdf
Scholarhat
 
MVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHatMVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHat
Scholarhat
 
Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17
Celine George
 
10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf
10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf
10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf
SSRCreations
 
E-learning Odoo 17 New features - Odoo 17 Slides
E-learning Odoo 17  New features - Odoo 17 SlidesE-learning Odoo 17  New features - Odoo 17 Slides
E-learning Odoo 17 New features - Odoo 17 Slides
Celine George
 
Imagination in Computer Science Research
Imagination in Computer Science ResearchImagination in Computer Science Research
Imagination in Computer Science Research
Abhik Roychoudhury
 
Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.
DrRavindrakshirsagar1
 
Demonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 SlidesDemonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 Slides
Celine George
 
Introduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal UseIntroduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal Use
Excellence Foundation for South Sudan
 
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdfPRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
nservice241
 
How to Manage Line Discount in Odoo 17 POS
How to Manage Line Discount in Odoo 17 POSHow to Manage Line Discount in Odoo 17 POS
How to Manage Line Discount in Odoo 17 POS
Celine George
 
Odoo 17 Events - Attendees List Scanning
Odoo 17 Events - Attendees List ScanningOdoo 17 Events - Attendees List Scanning
Odoo 17 Events - Attendees List Scanning
Celine George
 
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
Nguyen Thanh Tu Collection
 
RDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEWRDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEW
Murugan Solaiyappan
 

Recently uploaded (20)

How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17How to Manage Shipping Connectors & Shipping Methods in Odoo 17
How to Manage Shipping Connectors & Shipping Methods in Odoo 17
 
How To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-marketHow To Sell Hamster Kombat Coin In Pre-market
How To Sell Hamster Kombat Coin In Pre-market
 
View Inheritance in Odoo 17 - Odoo 17 Slides
View Inheritance in Odoo 17 - Odoo 17  SlidesView Inheritance in Odoo 17 - Odoo 17  Slides
View Inheritance in Odoo 17 - Odoo 17 Slides
 
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
Brigada Eskwela 2024 PowerPoint Update for SY 2024-2025
 
2 Post harvest Physiology of Horticulture produce.pptx
2 Post harvest Physiology of Horticulture  produce.pptx2 Post harvest Physiology of Horticulture  produce.pptx
2 Post harvest Physiology of Horticulture produce.pptx
 
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
Benchmarking Sustainability: Neurosciences and AI Tech Research in Macau - Ke...
 
C# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdfC# Interview Questions PDF By ScholarHat.pdf
C# Interview Questions PDF By ScholarHat.pdf
 
MVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHatMVC Interview Questions PDF By ScholarHat
MVC Interview Questions PDF By ScholarHat
 
Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17Mail Server Configuration Using App passwords in Odoo 17
Mail Server Configuration Using App passwords in Odoo 17
 
10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf
10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf
10th Social Studies Enrichment Material (Abhyasa Deepika) EM.pdf
 
E-learning Odoo 17 New features - Odoo 17 Slides
E-learning Odoo 17  New features - Odoo 17 SlidesE-learning Odoo 17  New features - Odoo 17 Slides
E-learning Odoo 17 New features - Odoo 17 Slides
 
Imagination in Computer Science Research
Imagination in Computer Science ResearchImagination in Computer Science Research
Imagination in Computer Science Research
 
Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.Genetics Teaching Plan: Dr.Kshirsagar R.V.
Genetics Teaching Plan: Dr.Kshirsagar R.V.
 
Demonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 SlidesDemonstration module in Odoo 17 - Odoo 17 Slides
Demonstration module in Odoo 17 - Odoo 17 Slides
 
Introduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal UseIntroduction to Google Productivity Tools for Office and Personal Use
Introduction to Google Productivity Tools for Office and Personal Use
 
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdfPRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
PRESS RELEASE - UNIVERSITY OF GHANA, JULY 16, 2024.pdf
 
How to Manage Line Discount in Odoo 17 POS
How to Manage Line Discount in Odoo 17 POSHow to Manage Line Discount in Odoo 17 POS
How to Manage Line Discount in Odoo 17 POS
 
Odoo 17 Events - Attendees List Scanning
Odoo 17 Events - Attendees List ScanningOdoo 17 Events - Attendees List Scanning
Odoo 17 Events - Attendees List Scanning
 
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
BÀI TẬP BỔ TRỢ 4 KỸ NĂNG TIẾNG ANH LỚP 12 - GLOBAL SUCCESS - FORM MỚI 2025 - ...
 
RDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEWRDBMS Lecture Notes Unit4 chapter12 VIEW
RDBMS Lecture Notes Unit4 chapter12 VIEW
 

Green_VCA_presentation.pdf

  • 1. Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming Vignesh V Menon1, Christian Feldmann2, Klaus Schoeffmann1, Mohammad Ghanbari3, Christian Timmerer1 1 Christian Doppler Laboratory ATHENA, Alpen-Adria-Universität, Klagenfurt, Austria 2 Bitmovin, Klagenfurt, Austria 3 School of Computer Science and Electronic Engineering, University of Essex, UK 7 – 10 June 2023 Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 1
  • 2. Outline 1 Background 2 SITI 3 Video Complexity Features 4 Accuracy Analysis 5 Performance Optimizations 6 Conclusions Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 2
  • 3. Background Background Why VCA? The optimal encoding parameters depend on the video content complexity. There is a need to extract content features that can represent the video content complexity to predict the optimal encoding parameters for that video content. Encoder Analysis stats Encoded bitstream Input video Video complexity analysis Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 3
  • 4. SITI SITI1 Spatial Information (SI) indicates the maximum spatial detail in a video. SI = max{std[Sobel(F(x, y, t))]} (1) where F(x, y, t) represents each video frame. Temporal Information (TI) is the maximum temporal variation between two successive frames. D(x, y, t) = F(x, y, t)) − F(x, y, t − 1)). (2) TI = max{std[D(x, y, t)]} (3) 1 https://github.com/Telecommunication-Telemedia-Assessment/SITI, last access: Apr 30, 2023. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 4
  • 5. Video Complexity Features Texture Energy Texture Energy Compute texture energy per block A DCT-based energy function is used to determine the block-wise feature of each frame defined as: HY ,p,k = w−1 X i=0 w−1 X j=0 e|( ij wh )2−1| |DCT(i, j)| (4) where wxw is the size of the block, and DCT(i, j) is the (i, j)th DCT component when i + j > 0, and 0 otherwise. The energy values of blocks in a frame are averaged to determine the energy per frame.2 EY = C−1 X k=0 HY ,p,k C · w2 (5) 2 Michael King, Zinovi Tauber, and Ze-Nian Li. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on Multimedia and Expo. 2007, pp. 1647–1650. doi: 10.1109/ICME.2007.4284983. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 5
  • 6. Video Complexity Features Texture energy gradient Texture energy gradient hp: SAD of the block level energy values of frame p to that of the previous frame p − 1. h = C−1 X k=0 | HY ,p,k, HY ,p−1,k | C · w2 (6) where C denotes the number of blocks in frame p. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 6
  • 7. Video Complexity Features Luminescence Luminescence The luminescence of non-overlapping blocks k of each frame p is defined as: LY ,k = p DCT(0, 0) (7) where DCT(0, 0) is the DC component in the DCT calculation. The block-wise luminescence is averaged per frame denoted as LY as shown below. LY = K−1 X k=0 LY ,p,k C · w2 (8) where C denotes the number of blocks in frame p. Chroma features VCA also determines chroma texture energy EU and EV (for U and V planes), and the chromi- nance LU and LV (for U and V planes). Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 7
  • 8. Video Complexity Features Example heatmap Video Complexity Features (a) Original video frame (b) Heatmap of L (c) Heatmap of E (d) Heatmap of h Figure: Example heatmap of Luminescence (L), spatial texture (E) and temporal activity (h) features of the 2nd frame of CoverSong 1080P 0a86 video of Youtube UGC dataset extracted using VCA. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 8
  • 9. Accuracy Analysis Accuracy Analysis Correlation of spatial complexity features with the ground truth Bitrate in All Intra configuration3 is considered as the ground truth of the spatial complexity. Figure: PCC between SI and EY , respectively, with bitrate in All Intra configuration with medium preset of x265 encoder with ultrafast preset of x265 encoder for the VCD dataset.4 3 Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018. 4 Hadi Amirpour et al. “VCD: Video Complexity Dataset”. In: Proceedings of the 13th ACM Multimedia Systems Conference. MMSys ’22. Athlone, Ireland: Association for Computing Machinery, 2022, 234–239. isbn: 9781450392839. doi: 10.1145/3524273.3532892. url: https://doi.org/10.1145/3524273.3532892. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 9
  • 10. Accuracy Analysis Accuracy Analysis Correlation of complexity features with bitrate in the Low Delay P picture (LDP) configuration Figure: PCC between SI, EY , TI and h with bitrate in the Low Delay P picture configuration with ultrafast preset of x265 encoder for the VCD dataset. EY and h strongly correlate with the encoding bitrate. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 10
  • 11. Performance Optimizations Performance Optimizations Multi-threading optimizations Creates multiple threads within a VCA execution instance, which executes independently but concurrently, sharing process resources. Independent threads carry out DCT-energy computation per block. x86 SIMD optimizations SIMD optimization5 of DCT functions implemented as intrinsic and assembly codes for x86 architecture. 5 Praveen Kumar Tiwari et al. “Accelerating x265 with intel® advanced vector extensions 512”. In: White Paper on the Intel Developers Page (2018). Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 11
  • 12. Performance Optimizations Performance Optimizations Low-pass DCT optimization Unlike SI, the EY feature exhibits better correlation across resolutions. In VCA v2.0, the complexity features are evaluated on the video spatially downsampled by a factor of two. (a) (b) Figure: PCC between the spatial complexity features (a) SI and (b) EY across multiple resolutions for the VCD dataset. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 12
  • 13. Performance Optimizations Performance Optimizations Processing time results Figure: Processing time of video content complexity analysis methods. m1: without any performance optimizations m2: with SIMD m3: with SIMD and low-pass DCT m4: with SIMD , low-pass DCT and multi-threading (2 threads) m5: with SIMD , low-pass DCT and multi-threading (4 threads) m6: with SIMD , low-pass DCT and multi-threading (8 threads) Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 13
  • 14. Performance Optimizations Performance Optimizations Energy consumption results Figure: Energy consumption of video content complexity analysis methods. m1: without any performance optimizations m2: with SIMD m3: with SIMD and low-pass DCT m4: with SIMD , low-pass DCT and multi-threading (2 threads) m5: with SIMD , low-pass DCT and multi-threading (4 threads) m6: with SIMD , low-pass DCT and multi-threading (8 threads) Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 14
  • 15. Conclusions Conclusions VCA is an open-source video complexity analyzer library published under the GNU GPLv3 license. Low-complexity DCT-based energy features are extracted using VCA v2.0, which encoders use to derive decisions like bitrate-ladder, frame-type, block-partitioning, and much more. Multi-threading, x86 SIMD, and low-pass DCT optimizations improve the energy efficiency of the VCA implementation. Compared to the state-of-the-art SITI implementation of video complexity analysis, VCA v2.0 yields a better estimation of video complexity, with an energy consumption reduction of 97.06%. Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 15
  • 16. Q & A Q & A Thank you for your attention! Vignesh V Menon (vignesh.menon@aau.at) VCA Vignesh V Menon Green Video Complexity Analysis for Efficient Encoding in Adaptive Video Streaming 16