The document presents CODA, a content-aware frame dropping algorithm for high frame-rate video streaming. CODA extracts spatial and temporal features from video frames and uses these features to predict an optimized variable framerate for different target bitrates. It was evaluated on several test videos and was shown to improve compression efficiency and reduce encoding time compared to encoding at the original fixed framerate. CODA could help improve performance of high frame-rate video streaming applications. The authors conclude it requires fewer bits than default encoding to maintain quality and reduces overall encoding time.

1. CODA: Content-aware Frame Dropping Algorithm for High
Frame-rate Video Streaming
Vignesh V Menon, Hadi Amirpour, Mohammad Ghanbari, Christian Timmerer
Christian Doppler Laboratory ATHENA, Institute of Information Technology (ITEC), University of Klagenfurt, Austria
Data Compression Conference (DCC)
24 March 2022
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 1

2. Outline
1 Introduction
2 CODA
3 Evaluation
4 Conclusions and Future Directions
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 2

3. Introduction
Introduction
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 3

4. Introduction
Introduction
High Framerate (HFR) videos
High Framerate (HFR) video streaming enhances the viewing experience and improves
visual clarity.1
However, it may lead to an increase of both encoding time complexity and compression
artifacts, particularly at lower bitrates.
1
ITU-R BT.2020-2. “Parameter values for ultra-high definition television systems for production and international programme exchange”. In: 2015.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 4

5. Introduction
Introduction
0.2 0.5 1.2 4.5 16.8
Bitrate (in Mbps)
40
50
60
70
80
90
VMAF
120fps
60fps
30fps
24fps
(a) HoneyBee
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
30
40
50
60
70
80
VMAF
120fps
60fps
30fps
24fps
(b) Lips
Figure: Rate-Distortion (RD) curves of UHD encodings of (a) HoneyBee and (b) Lips sequences for
multiple framerates.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 5

6. Introduction
Introduction
Variable Framerate (VFR) coding
Input
Video
. Temporal
Downsampling
Framerate Selection
Encoder Decoder
Temporal Up-
sampling
Display
f
d ∈ D
ˆ
f = f ∗ (1 − d) ˆ
f
f =
ˆ
f
1−d
Figure: Block diagram of a variable framerate (VFR) coding scheme2
in the context of video encoding.
f and ˆ
f denote the original framerate of the video and the framerate at which the video is encoded. d
represents the frame dropping factor.
2
G. Herrou et al. “Quality-driven Variable Frame-Rate for Green Video Coding in Broadcast Applications”. In: IEEE Transactions on Circuits and Systems for
Video Technology (2020), pp. 1–1. doi: 10.1109/TCSVT.2020.3046881.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 6

7. CODA
CODA
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 7

8. CODA Phase 1: Feature Extraction
CODA
Phase 1: Feature Extraction
Compute texture energy per block
A DCT-based energy function is used to determine the block-wise feature of each frame
defined as:
Hk =
w
X
i=1
h
X
j=1
e|( ij
wh
)2−1|
|DCT(i − 1, j − 1)| (1)
where w and h are the width and height of the block, and DCT(i, j) is the (i, j)th DCT
component when i + j > 2, and 0 otherwise.
The energy values of blocks in a frame is averaged to determine the energy per frame.3
3
Michael King, Zinovi Tauber, and Ze-Nian Li. “A New Energy Function for Segmentation and Compression”. In: 2007 IEEE International Conference on
Multimedia and Expo. 2007, pp. 1647–1650. doi: 10.1109/ICME.2007.4284983.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 8

9. CODA Phase 1: Feature Extraction
Proposed Algorithm
Phase 1: Feature Extraction
hk: SAD of the block level energy values of frame k to that of the previous frame k − 1.
hk =
SAD(Hk(i) − Hk−1(i))
M
(2)
where M denotes the number of CTUs in frame k.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 9

10. CODA Phase 2: Framerate prediction
CODA
Phase 2: Framerate prediction
Inputs:
E, h : spatial and temporal complexities
r : video resolution
fmax : original framerate
D : set of all frame drop factors ˜
d
B : set of all target bitrates b (in kbps)
Output: ˆ
f (b) ∀ b ∈ B
Step 1: Determine ˆ
d(b).
ˆ
d(b) = d0e−
βMA(r,fmax )·h·b
E
Step 2: The predicted optimized framerate for the video is given by:
ˆ
f (b) = fmax · (1 − ˆ
d(b))
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 10

11. Evaluation
Evaluation
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 11

12. Evaluation
Evaluation
E and h values are extracted using VCA open-source software.4
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
20
40
60
80
100
120
Frame-rate
Ground truth (fG)
Predicted (f)
(a)
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
20
40
60
80
100
120
Frame-rate
Ground truth (fG)
Predicted (f)
(b)
Figure: Optimized framerate prediction results of Beauty (a) and ShakeNDry (b) sequences. Please
note that, depending on the content, the optimized framerate in various bitrates are different.
4
V. V Menon, C. Feldmann, and H. Amirpour. VCA. Version 1.0.0. Available: https://github.com/cd-athena/VCA. Feb. 2022.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 12

13. Evaluation
Evaluation
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
30
40
50
60
70
VMAF
Original (120fps)
CODA (VFR)
(a)
0.2 0.5 1.2 4.5 16.8
Bit-rate (in Mbps)
40
50
60
70
VMAF
Original (120fps)
CODA (VFR)
(b)
Figure: Rate-Distortion (RD) results of Beauty (a) and ShakeNDry (b) sequences for the default
encoding and CODA-based VFR encoding.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 13

14. Conclusions and Future Directions
Conclusions and Future Directions
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 14

15. Conclusions and Future Directions
Conclusions
Presented a content-aware frame dropping algorithm for video streaming applications, es-
pecially for HFR videos.
Predicts the optimized framerate for a set of bitrates defined in a bitrate ladder and res-
olution for every video, which helps in improving the overall performance of HFR video
streaming in terms of encoding time and compression efficiency.
UHD encoding using the proposed algorithm requires 15.87% fewer bits to maintain the
same PSNR and 18.20% fewer bits to maintain the same VMAF as compared to the original
framerate encoding. An overall encoding time reduction of 21.82% is also observed.
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 15

16. Conclusions and Future Directions
Q & A
Thank you for your attention!
Vignesh V Menon (vignesh.menon@aau.at)
Hadi Amirpour (hadi.amirpourazarian@aau.at)
Mohammad Ghanbari (ghan@essex.ac.uk)
Christian Timmerer (christian.timmerer@aau.at)
Vignesh V Menon CODA: Content-aware Frame Dropping Algorithm for High Frame-rate Video Streaming 16