The document presents 'Decodra,' a framerate optimization method for adaptive video streaming that balances perceptual quality and energy efficiency through variable framerate techniques. It emphasizes the need for dynamic framerate adjustment based on bandwidth and device capabilities while achieving significant reductions in decoding energy consumption. Key contributions include improved visual quality, especially under resource constraints, and a better user experience across various platforms.

1. Energy-Quality-aware Variable Framerate Pareto-Front
for Adaptive Video Streaming
—
Prajit T Rajendran¹, Samira Afzal², Vignesh V Menon³, Christian Timmerer²
¹Universite Paris-Saclay, CEA, List, F-91120, Palaiseau, France
²Christian Doppler Laboratory ATHENA, Institute of Information Technology (ITEC), Alpen-Adria-Universit¨at, Klagenfurt, Austria
³Video Communication and Applications Dept., Fraunhofer HHI, Germany

2. MHV’24
Background on Adaptive Video Streaming
8-11 Dec 2024 © Fraunhofer
Slide 2
● HTTP Adaptive Streaming (HAS) has become the standard for delivering video
content over various internet speeds and devices [1, 2].
● Key Components:
○ Segmented Video Content: Video is encoded at multiple quality levels and
split into segments (e.g., 2-10 seconds each).
○ Manifest Files (MPD/HLS Manifest): Provides clients with information about
available video bitrates, resolutions, and segment locations.
○ Adaptive Bitrate (ABR) Streaming: Enables real-time switching between
different video qualities based on bandwidth, device capability, and buffer
state.
[1] I. Sodagar, “The MPEG-DASH Standard for Multimedia Streaming Over the Internet,” IEEE MultiMedia, vol. 18, no. 4, pp. 62–67, 2011.
[2] A. Bentaleb, B. Taani, A. C. Begen, C. Timmerer, and R. Zimmermann, “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP,” in IEEE Communications Surveys Tutorials, vol. 21, no. 1, 2019, pp.
562–585.

3. MHV’24
Framerate in Adaptive Streaming
8-11 Dec 2024 © Fraunhofer
Slide 3
● Perceptual Quality: Framerate directly impacts how smooth and lifelike
video playback appears.
● Typical Framerate Standards: Videos are often streamed at standardized
framerates such as 24 fps (film), 30 fps (TV), and 60 fps (gaming and
online content). Emerging standards such as UHDTV can even support
up to 120 fps [1, 2, 3, 4].
[1] M. G. Armstrong et al., “High Frame-Rate Television,” in SMPTE Motion Imaging Journal, vol. 118. SMPTE, Oct. 2009, pp. 54–59.
[2] J. Wu et al., “Enabling Adaptive High-Frame-Rate Video Streaming in Mobile Cloud Gaming Applications,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 12, 2015, pp. 1988–2001.
[3] C. Hwung, “Video Frame Rates: 30fps vs 60fps vs 120fps vs 240fps,” 2023, Link: https://ceciliadigiarty.medium.com/video-frame-rates-30fps-vs-60fps-vs-120fps-vs-240fps-22c94845fce4.
[4] T. Tamasi,”Why is a high frame rate important for e-sports?,” 2019, Link: https://www.nvidia.com/de-de/geforce/news/what-is-fps-and-how-it-helps-you-win-games/
Figure: The general consensus is that a video with a higher frame
rate will offer a smoother playback experience [3].
Figure: Higher framerates minimize distracting artifacts [4].
Importance of Framerate

4. MHV’24
Framerate in Adaptive Streaming
8-11 Dec 2024 © Fraunhofer
Slide 4
● Decoding Complexity: Higher framerates significantly increase computational
complexity during video decoding [1].
● Bandwidth Requirements: Higher framerates require higher bitrates to
maintain visual quality, increasing the data load over the network.
● Challenges with High Framerate Streaming
○ Resource-Constrained Devices: Fixed high-framerate streaming can
lead to playback issues such as stuttering, frame drops, and rapid
battery drain.
○ Energy Consumption vs. Quality: Optimizing the trade-off between
smooth video quality (high framerate) and energy efficiency is crucial
for improving user experience without overburdening the device’s
resources.
● Motivation for Framerate Optimization
○ A dynamic approach to adaptively control framerate based on available
bandwidth, device capabilities, and content characteristics is needed to
balance perceptual quality, decoding complexity, and energy efficiency.
[1] C. Herglotz, A. Heindel, and A. Kaup, “Decoding-Energy-Rate-Distortion Optimization for Video Coding,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 1, p. 171–182, Jan. 2019.
[2] A. Stergiou and R. Poppe, “AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling,” in IEEE Transactions on Image Processing, 2023.
Impact of Framerate on Streaming
Figure: Rate-VMAF and rate-decoding energy for multiple framerates
for Sequence 0286 of Inter-4K dataset [2] encoded using x265
encoder at veryslow preset for Apple HLS ladder.
(a) Sequence 0257
(b) Sequence 0286

5. MHV’24
Related Work
8-11 Dec 2024 © Fraunhofer
Slide 5
● Fixed Framerate Methods: Traditional approaches involve encoding
at fixed framerates (e.g., 24, 30, or 60 fps), which can result in
energy inefficiencies, particularly for mobile and resource-constrained
devices.
● VFR Techniques
○ Threshold-based Adaptation: Techniques that adjust framerate
based on predefined thresholds for motion-related features.
○ Machine Learning-Based Adaptation: Recent methods have
used support vector regression, decision trees, or random forest
classifiers to determine critical framerates that maintain
perceived video quality [1,2,3,4].
● Limitations
○ These approaches neglect decoding complexity considerations,
leading to high computational demands and energy
consumption on devices like smartphones.
[1] Q. Huang et al., "Perceptual Quality Driven Frame-Rate Selection (PQD-FRS) for High-Frame-Rate Video," in IEEE Transactions on Broadcasting, vol. 62, no. 3, pp. 640-653, Sept. 2016, doi: 10.1109/TBC.2016.2570022.
[2] A. V. Katsenou, D. Ma and D. R. Bull, "Perceptually-Aligned Frame Rate Selection Using Spatio-Temporal Features," 2018 Picture Coding Symposium (PCS), San Francisco, CA, USA, 2018, pp. 288-292, doi: 10.1109/PCS.2018.8456274.
[3] G. Herrou et al., “Quality-driven Variable Frame-Rate for Green Video Coding in Broadcast Applications,” in IEEE Transactions on Circuits and Systems for Video Technology, 2020, pp. 1–1.
[4] J. Wu, C. Yuen, N. -M. Cheung, J. Chen and C. W. Chen, "Enabling Adaptive High-Frame-Rate Video Streaming in Mobile Cloud Gaming Applications," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 12,
pp. 1988-2001, Dec. 2015, doi: 10.1109/TCSVT.2015.2441412.
Framerate Adaptation Approaches
Figure: Method in [2].
Figure: Method in [3].

6. MHV’24
Key Concept - Pareto Front
8-11 Dec 2024 © Fraunhofer
Slide 6
● Introduction to Pareto Front
○ A Pareto front is a concept from multi-objective optimization used to
identify solutions that offer the best trade-offs between multiple
competing objectives.
○ In the context of video streaming, the two primary competing objectives
are video quality and decoding energy consumption.
● Pareto Efficiency
○ A solution is Pareto efficient if there is no way to improve one objective
without degrading the other.
○ In the case of video streaming, the goal is to optimize both perceptual
quality and energy efficiency, resulting in a Pareto-optimal set of
framerates that offer the best balance.

7. MHV’24
DECODRA Optimization Problem Formulation
8-11 Dec 2024 © Fraunhofer
Slide 7
● Optimization Objective
○ (i) Maximize quality and
○ (ii) Minimize energy.
● Objective Function
○ Mathematical Formulation
○
○ where v(b,r,f) is the quality metric (e.g., VMAF) for a given bitrate (b), resolution (r), and framerate (f). vmax is the
maximum achievable quality for a given segment. vJ is the quality degradation threshold that determines how
much quality loss is acceptable.
● Constraint
○ Quality Maintenance: The quality of a segment must not fall below a certain threshold:
○

8. MHV’24
Implementation Pipeline Overview
8-11 Dec 2024 © Fraunhofer
Slide 8
1. Spatial Downsampling
○ Reduce video resolution to lower bitrate requirements using bicubic
interpolation.
2. Temporal Downsampling
○ Adjust framerate by selectively dropping frames to lower
computational complexity.
3. Encoding
○ Encode the video segments using multiple
bitrate-resolution-framerate combinations.
4. Temporal & Spatial Upsampling
○ Restore the video to the original framerate and resolution for
playback.
5. Quality and Energy Measurement
○ Use metrics such as VMAF and RAPL to evaluate visual quality and
decoding energy consumption.
6. Pareto-Front Estimation
○ Estimate the optimal framerate that balances energy efficiency with
acceptable quality.
Figure: DECODRA implementation.

9. MHV’24
Pareto-Front Estimation
8-11 Dec 2024 © Fraunhofer
Slide 9
● Pareto-Front Estimation
○ Goal: Identify the optimal trade-off between video quality
and energy consumption.
○ Method: Use the Pareto-front approach to determine the
ideal framerate that maintains high quality while minimizing
decoding energy.
○ Outcome: Establish a balanced energy-quality trade-off to
guide framerate selection dynamically.
● Optimal Framerate Selection
○ Adaptation Logic
■ If quality degradation is acceptable, select lower
framerates to reduce energy usage.
■ If quality must be maximized, choose a higher
framerate to enhance visual perception.

10. MHV’24
Evaluation Setup
8-11 Dec 2024 © Fraunhofer
Slide 10
● Dataset
○ Inter-4K Dataset: Contains diverse video content with various
spatiotemporal complexities, ensuring robust evaluation across
different types of scenes.
● Experimental Parameters
○ Bitrate-Resolution Combinations: Multiple representations
encoded with varying bitrates and resolutions, following the
HLS authoring specification [1].
○ Framerates Evaluated: Standard framerates (e.g., 24, 30, 48,
60 fps) [2] to compare DECODRA’s dynamic approach against
fixed framerate settings.
○ Quality Degradation Thresholds (vJ): Tested at thresholds of 2,
4, and 6 VMAF points to evaluate the trade-offs between
energy and visual quality [3, 4].
[1] Apple Inc., “HLS Authoring Specification for Apple Devices.” [Online]. Available: https://developer.apple.com/documentation/http live streaming/hls authoring specification for apple
devices
[2] W. Woodhall, “Audio production and postproduction.” Jones & Bartlett Learning, 2010, ch. 5, pp. 93–94.
[3] D. Yuan et al., “Visual JND: A Perceptual Measurement in Video Coding,” in IEEE Access, vol. 7, 2019, pp. 29 014–29 022.
[4] J. Zhu et al., “A Framework to Map VMAF with the Probability of Just Noticeable Difference between Video Encoding Recipes,” in 2022 IEEE 14th Image, Video, and
Multidimensional Signal Processing Workshop (IVMSP), 2022, pp. 1–5.

11. MHV’24
Evaluation Metrics
8-11 Dec 2024 © Fraunhofer
Slide 11
● Quality Metrics
○ Video Multi-Method Assessment Fusion (VMAF)
■ Measures perceptual quality considering human vision factors.
○ Peak Signal-to-Noise Ratio (PSNR)
■ Evaluates reconstruction quality compared to the original video.
● Energy Metrics
○ Decoding Energy Consumption
■ Running Average Power Limit (RAPL): Measures power usage during video decoding.
■ CodeCarbon Tool: Evaluates carbon footprint and energy consumption of different framerate encodings [1].
● BD-Rate Metrics
○ BD-PSNR and BD-VMAF [2]
■ Evaluate average improvements in quality for a given bitrate compared to a reference encoding scheme.
○ Energy Reduction [3]
■ Measure relative energy savings achieved by DECODRA compared to fixed framerate methods.
[1] BCG-GAMMA and MILA, “CodeCarbon.” [Online]. Available: https://codecarbon.io/
[2] HSTP-VID-WPOM, “Working practices using objective metrics for evaluation of video coding efficiency experiments,” in International Telecommunication Union, 2020.
[Online]. Available: http://handle.itu.int/11.1002/pub/8160e8da-en
[3] C. Herglotz, A. Heindel, and A. Kaup, “Decoding-Energy-Rate-Distortion Optimization for Video Coding,” IEEE Transactions on Circuits and Systems for Video Technology,
vol. 29, no. 1, p. 171–182, Jan. 2019.

12. MHV’24
Experimental Results - Quality vs. Energy Trade-offs
8-11 Dec 2024 © Fraunhofer
Slide 12
● Quality vs. Bitrate
○ VMAF Increase: DECODRA shows up to 5.14-point
improvement in VMAF compared to 60 fps fixed
framerate encoding at the same bitrate.
○ Adaptive Framerate: Balances between visual quality
and energy use, dynamically selecting lower
framerates for challenging network conditions without
significant quality loss.
● Energy Savings
○ Decoding Energy Reduction: DECODRA achieves up
to 13.27% reduction in decoding energy, especially at
lower bitrates, compared to default high-framerate
methods.
○ Higher Efficiency for Lower Quality Thresholds: The
higher the quality threshold (vJ = 6), the more energy
is saved, highlighting DECODRA’s effectiveness in
minimizing resource use for similar quality.
Figure: Average VMAF decrease and decoding energy reduction observed for each
representation compared to HQ for the Inter-4K dataset.

13. MHV’24
Detailed Analysis - Energy Efficiency
8-11 Dec 2024 © Fraunhofer
Slide 13
● Energy Savings Across Bitrates
○ Higher Savings at Low Bitrates
■ DECODRA achieves up to 32.42% energy reduction at
lower bitrates by reducing framerate without
significantly compromising quality.
○ Adaptive Energy Efficiency
■ As bitrate increases, DECODRA gradually increases
framerate to maintain quality, still achieving better
efficiency compared to fixed framerate methods.
● Comparison with Default Methods
○ Fixed vs. Adaptive Framerate
■ Fixed high framerate methods (e.g., 60 fps) consume
significantly more energy.
○ DECODRA’s dynamic adaptation allows it to maintain
energy efficiency at all bitrates.
● Impact on Mobile Devices
○ Prolonged Battery Life
○ Optimal Energy-Quality Balance
Average coding performance compared to the Default encoding.

14. MHV’24
Detailed Analysis - Visual Quality
8-11 Dec 2024 © Fraunhofer
Slide 14
● Maintaining High Quality
○ VMAF and PSNR Metrics
■ DECODRA achieves a VMAF increase of up to 5.14
points compared to fixed framerate encoding.
■ Maintains high PSNR values, showing consistent
quality improvement across various bitrates.
● Framerate Adaptation Benefits
○ Lower Framerate at Low Bitrate
■ At lower bitrates, DECODRA intelligently selects
lower framerates to reduce artifacts, resulting in
improved perceptual quality compared to fixed high
framerate.
○ Quality Threshold Control (vJ)
■ Adjusting quality thresholds allows DECODRA to
balance between perceptual quality and energy
consumption, with minimal perceived loss even at
higher energy savings.

15. MHV’24
Conclusions
8-11 Dec 2024 © Fraunhofer
Slide 15
● We presented DECODRA, a decoding complexity-aware framerate optimization approach for adaptive video
streaming.
● DECODRA leverages a Variable Framerate Pareto-Front (VFR-PF) to dynamically adjust framerate based on
bitrate and resolution.
● Key Contributions
○ Energy Efficiency: DECODRA significantly reduces decoding energy consumption, making it particularly
beneficial for mobile and resource-constrained devices.
○ Maintaining Quality: Despite reducing energy use, DECODRA preserves visual quality, maintaining high
VMAF and PSNR scores compared to traditional fixed framerate approaches.
○ Dynamic Adaptation: DECODRA successfully adapts framerate based on network conditions and device
capabilities, achieving a balance between smooth motion and energy savings.
● Impact: The proposed method ensures that end users receive the best possible experience with minimal impact on
device resources, thereby enhancing streaming quality across different platforms.

16. Thank you for your attention
— ● Prajit T. Rajendran (prajit.rajendran@ieee.org)
● Samira Afzal (samira.afzal@aau.at)
● Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
● Christian Timmerer (christian.timmerer@aau.at)