SlideShare a Scribd company logo
1 of 15
Download to read offline
Gain of Grain: A Film Grain Handling Toolchain
for VVC based Open Implementations
—
Vignesh V Menon, Postdoctoral Researcher, Fraunhofer HHI
Feb. 12, 2024, Mile High Video (MHV‘24)
MHV’24
Introduction
12.02.2024 © Fraunhofer
Slide 2
Film grain in video coding
• Film grain, an inherent characteristic of analog film, contributes to the unique visual aesthetics and cinematic experience in movies [1].
• The emergence of the Versatile Video Coding (VVC) [2, 3] standard brings new opportunities and challenges in efficiently representing film
grain, aiming to preserve its artistic value while ensuring compatibility with modern video compression techniques.
• Compression artifacts like blockiness are introduced when grainy video content is encoded at low bitrates. Film grain presents a challenge in
video coding due to its random and non-uniform nature, which can amplify compression artifacts if not adequately handled during
encoding.
[1] Inseong Hwang et al. “Enhanced Film Grain Noise Removal for High Fidelity Video Coding”. In: 2013 International Conference on Information Science and Cloud Computing Companion. 2013, pp. 668–674. doi: 10.1109/ISCC-
C.2013.69.
[2] Benjamin Bross et al. “Overview of the Versatile Video Coding (VVC) Standard and its Applications”. In: IEEE Transactions on Circuits and Systems for Video Technology. Vol. 31. 10. 2021, pp. 3736–3764. doi:
10.1109/TCSVT.2021.3101953
[3] Adam Wieckowski et al. “A Complete End to End Open Source Toolchain for the Versatile Video Coding (VVC) Standard”. In: Proceedings of the 29th ACM International Conference on Multimedia. New York, NY, USA:
Association for Computing Machinery, 2021, 3795–3798. isbn: 9781450386517. doi: 10.1145/3474085.3478320.
MHV’24
Introduction
12.02.2024 © Fraunhofer
Slide 3
Film grain in video coding
Figure: Illustration of compression artifacts introduced due to film grain when encoded using VVenC encoder
at low bitrates.
▪ Film grain is observed to amplify compression artifacts if
not adequately handled during encoding.
▪ Integrating film grain handling into VVC requires
adherence to standardized encoding practices while
accommodating unique film grain characteristics [4].
[4] Zoubida Ameur, Wassim Hamidouche, Edouard François, Miloš Radosavljević, Daniel Ménard, and Claire-Hélène Demarty, “Deep-Based Film Grain Removal and Synthesis,” IEEE Transactions on Image Processing, vol. 32, pp. 5046–
5059, 2022.
MHV’24
State-of-the-art toolchain
12.02.2024 © Fraunhofer
Slide 4
Components:
▪ Denoising: reducing or removing noise from the video, including film grain.
▪ Encoding: compressing the video data for efficient storage and transmission.
▪ Decoding: reconstructing the video data from the compressed format.
▪ Grain synthesis: generating or enhancing film grain in the video.
MHV’24
Proposed toolchain
12.02.2024 © Fraunhofer
Slide 5
MHV’24
Proposed toolchain
12.02.2024 © Fraunhofer
Slide 6
MCTF in VVenC:
▪ VVenC already employs a denoising stage [5], based on a framework proposed
initially in [6]– motion compensated temporal (pre-)filtering.
▪ The filter performs a blockwise motion search for each filtered frame in
neighboring frames to remove the noise.
▪ Using up to eight predictors, a weighted average of the current frame block and
its predictors is generated and used for further encoding.
▪ Improved search strategies, reference number reduction, and flexible block size
were introduced, improving the filter runtime and operation [5].
Denoising
[5] Adam Wieckowski, Tobias Hinz, Christian R. Helmrich, Benjamin Bross, and Detlev Marpe, “An optimized temporal filter implementation for practical applications,” in 2022 Picture Coding Symposium (PCS), 2022, pp. 247–251.
[6] Jack Enhorn, Rickard Sjöberg, and Per Wennersten, “A Temporal Pre-Filter For Video Coding Based On Bilateral Filtering,” in 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 1161–1165.
MHV’24
Proposed toolchain
12.02.2024 © Fraunhofer
Slide 7
▪ The framework for film grain handling based on frequency filtering and
parameterization of grain is implemented in VVenC (to be released soon).
▪ The film grain pattern is characterized using a horizontal high cut-off
frequency and a vertical high cut-off frequency obtained in the discrete cosine
transform domain.
▪ The film grain pattern is scaled to the appropriate level using a stepwise
scaling function that considers the underlying image’s characteristics.
▪ Afterward, the film grain pattern is blended into the image using additive
blending [7].
Film grain estimation
[7] M. Sean, Y. Peng, H. Walt, P. Fangjun, L. Taoran, C. Tao, M. Radosavljević, E. François, G. Vijayakumar, P. Kaustubh et al., “Fixed-point grain blending process for film grain characteristics sei message,” in Joint Video Experts Team
(JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, document JVET-W0095, 2021.
MHV’24
Proposed toolchain
12.02.2024 © Fraunhofer
Slide 8
▪ To convey the grain parameters to the decoder, the encoder embeds it as
supplemental enhancement information (SEI) in the bitstream [8], as the "Film
Grain SEI.“
▪ Film Grain SEI inherits the same syntax and semantics of the AVC film grain
SEI message [9].
▪ Since we implement a frequency filtering model for film grain estimation,
film_grain_model_id is set to 0.
▪ Additive blending is used when blending_mode_id is set to 0.
▪ Since our implementation analyses film grain for only the luma channel,
comp_model_present_flag[0] is set to 1.
▪ FGC SEI message is inserted at each frame, which is indicated by setting the
film_grain_characteristics_persistence_flag to 0. It also means the FGC SEI
message only applies to the current decoded frame.
Film Grain SEI
[8] “RDD 5:2006 - SMPTE Registered Disclosure Doc - Film Grain Technology —Specifications for H.264 | MPEG-4 AVC Bitstreams,” RDD 5:2006, pp. 1–18, 2006.
[9] Vijayakumar Gayathri Ramakrishna, Kaustubh Shripad Patankar, and Mukund Srinivasan, “Cloud-Based Workflow for AVC Film Grain Synthesis,” ser. MHV’23. New York, NY, USA: Association for Computing Machinery, 2023, p.
66–71.
MHV’24
Experimental setup
12.02.2024 © Fraunhofer
Slide 9
Comparisons:
▪ We run experiments on an AMD EYPC 7502P processor (32 cores),
where we run each VVenC v1.10 instance using four CPU threads,
with two-pass rate control, enabling adaptive quantization.
▪ All sequences are down-scaled to 1920x1080 8bit.
▪ We consider the Default toolchain as the benchmark, where :
▪ Encode the input video using VVenC with MCTF disabled.
▪ Decode the resulting bitstream using the VTM decoder.
▪ FGA and FGS are disabled in this toolchain.
Table: Experimental parameters used to evaluate the proposed film grain handling
toolchain.
MHV’24
Quality assessment metrics
12.02.2024 © Fraunhofer
Slide 10
Observations:
▪ Traditional metrics like PSNR and SSIM are not suitable for evaluating
the perceptual quality of film grain coding owing to their lack of
texture sensitivity.
▪ PSNR and SSIM are sensitive to noise, such that they penalize the
addition of synthesized film grain.
▪ VMAF [10], while more advanced, is not trained to evaluate the
perceptual quality of VVC-coded videos [11]. Figure: RD curves using the Default toolchain, MCTF (without FGS), and the
proposed toolchain for IntoTree test sequence encoded at faster preset.
[10] Zhi Li, Christos Bampis, Julie Novak, Anne Aaron, Kyle Swanson, Anush Moorthy, and Jan De Cock, “VMAF: The journey continues,” in Netflix Technology Blog, vol. 25, 2018.
[11] Christian R. Helmrich, Benjamin Bross, Jonathan Pfaff, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand, “Information on and analysis of the VVC encoders in the SDR UHD verification test,” in WG 05 MPEG Joint Video
Coding Team(s) with ITU-T SG 16, document JVET-T0103, Oct. 2020.
Take aways:
▪ Given these limitations, specialized metrics focusing on texture enhancement, perception of controlled noise, and overall film-like appearance
would be more appropriate for evaluating film grain coding, subject to future work.
▪ Metrics that include human perception aspects and consider texture fidelity alongside noise would offer a better assessment of the quality
enhancements film grain brings to video content.
MHV’24
Subjective quality assessment
12.02.2024 © Fraunhofer
Slide 11
▪ Original ▪ No MCTF (250 kbps)
▪ With the proposed toolchain (250 kbps)
▪ MCTF (250 kbps)
MHV’24
Subjective quality assessment
12.02.2024 © Fraunhofer
Slide 12
▪ Original ▪ MCTF (3 Mbps) ▪ With the proposed toolchain (3 Mbps)
MHV’24
Runtime complexity
12.02.2024 © Fraunhofer
Slide 13
Encoding and decoding
Encoding time:
▪ FGA contributes to the increased relative duration required for encoding
as the preset progresses towards faster configuration.
▪ The overall encoding time using the proposed toolchain reduces up to
11.59 % using slower preset.
Table: Runtime complexity of the proposed toolchain compared to
the default toolchain with the same preset.
[13] SVT. “The SVT High Definition Multi Format Test Set”. In: Feb. 2006. url: https://tech.ebu.ch/docs/hdtv/svt-multiformat-conditions-v10.pdf
[14] Li Song et al. “The SJTU 4K video sequence dataset”. In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX). 2013, pp. 34–35. doi: 10.1109/QoMEX.2013.6603201. url:
https://doi.org/10.1109/QoMEX.2013.6603201.
[15] Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018. url: https://www.researchgate.net/publication/326506581
Decoding time:
▪ The decoding time has increased due to the supplementary
computational load of FGS.
▪ The effect on decoding time is intertwined with the encoding presets;
as encoding presets progress towards slower configurations, the
proportional increase in decoding time tends to diminish.
MHV’24
Conclusions and future directions
12.02.2024 © Fraunhofer
Slide 14
Conclusions
▪ We presented an overview of the film grain handling toolchain for VVC-based open implementation using the VVenC encoder and VTM
decoder.
▪ The experimental results show that the proposed toolchain improves the subjective quality of the grainy video content encoded at multiple
bitrates, compared to default toolchain, considering VVenC encoding.
▪ Using slower preset, the proposed toolchain reduces the encoding duration by up to 11.59% while the decoding time increases by up to
15.32%.
▪ These outcomes underscore the inherent trade-offs in optimizing the film grain handling toolchain, emphasizing the need for a balanced
approach that prioritizes encoding efficiency and video quality.
Future directions
▪ Denoising and film grain estimation shall be tuned for various encoding presets in VVenC.
▪ More sophisticated models shall be investigated to represent film grain accurately in the digital domain. This could involve advanced statistical
models and/or machine learning approaches to capture the intricate characteristics of film grain.
▪ Develop quality assessment metrics that better capture the human perception of film grain.
Thank you for your attention
— ▪ Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
▪ Adam Wieckowski (adam.wieckowski@hhi.fraunhofer.de)
▪ Jens Brandenburg (jens.Brandenburg@hhi.Fraunhofer.de)
▪ Benjamin Bross (benjamin.bross@hhi.fraunhofer.de)
▪ Thomas Schierl (thomas.schierl@hhi.fraunhofer.de)
▪ Detlev Marpe (detlev.marpe@hhi.fraunhofer.de)

More Related Content

Similar to Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations

Video saliency-recognition by applying custom spatio temporal fusion technique
Video saliency-recognition by applying custom spatio temporal fusion techniqueVideo saliency-recognition by applying custom spatio temporal fusion technique
Video saliency-recognition by applying custom spatio temporal fusion technique
IAESIJAI
 
Paper id 36201508
Paper id 36201508Paper id 36201508
Paper id 36201508
IJRAT
 

Similar to Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations (20)

Online Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC StreamingOnline Bitrate ladder prediction for Adaptive VVC Streaming
Online Bitrate ladder prediction for Adaptive VVC Streaming
 
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
 
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
 
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
 
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
COMPARISON OF CINEPAK, INTEL, MICROSOFT VIDEO AND INDEO CODEC FOR VIDEO COMPR...
 
Comparison of Cinepak, Intel, Microsoft Video and Indeo Codec for Video Compr...
Comparison of Cinepak, Intel, Microsoft Video and Indeo Codec for Video Compr...Comparison of Cinepak, Intel, Microsoft Video and Indeo Codec for Video Compr...
Comparison of Cinepak, Intel, Microsoft Video and Indeo Codec for Video Compr...
 
18 17 jan17 13470 rakesh ahuja revised-version(edit)
18 17 jan17 13470 rakesh ahuja revised-version(edit)18 17 jan17 13470 rakesh ahuja revised-version(edit)
18 17 jan17 13470 rakesh ahuja revised-version(edit)
 
A Hybrid DWT-SVD Method for Digital Video Watermarking Using Random Frame Sel...
A Hybrid DWT-SVD Method for Digital Video Watermarking Using Random Frame Sel...A Hybrid DWT-SVD Method for Digital Video Watermarking Using Random Frame Sel...
A Hybrid DWT-SVD Method for Digital Video Watermarking Using Random Frame Sel...
 
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
Energy-efficient Adaptive Video Streaming with Latency-Aware Dynamic Resoluti...
 
Video copy detection using segmentation method and
Video copy detection using segmentation method andVideo copy detection using segmentation method and
Video copy detection using segmentation method and
 
Video saliency-recognition by applying custom spatio temporal fusion technique
Video saliency-recognition by applying custom spatio temporal fusion techniqueVideo saliency-recognition by applying custom spatio temporal fusion technique
Video saliency-recognition by applying custom spatio temporal fusion technique
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
IRJET- A Reliable and Robust Video Watermarking Scheme Over Cloud
IRJET- A Reliable and Robust Video Watermarking Scheme Over CloudIRJET- A Reliable and Robust Video Watermarking Scheme Over Cloud
IRJET- A Reliable and Robust Video Watermarking Scheme Over Cloud
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
Jiri ece-01-03 adaptive temporal averaging and frame prediction based surveil...
 
Robust Video Watermarking Scheme Based on Intra-Coding Process in MPEG-2 Style
Robust Video Watermarking Scheme Based on Intra-Coding Process in MPEG-2 Style Robust Video Watermarking Scheme Based on Intra-Coding Process in MPEG-2 Style
Robust Video Watermarking Scheme Based on Intra-Coding Process in MPEG-2 Style
 
International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)International Journal of Engineering Research and Development (IJERD)
International Journal of Engineering Research and Development (IJERD)
 
Paper id 36201508
Paper id 36201508Paper id 36201508
Paper id 36201508
 
06 13sept 8313 9997-2-ed an adaptive (edit lafi)
06 13sept 8313 9997-2-ed an adaptive (edit lafi)06 13sept 8313 9997-2-ed an adaptive (edit lafi)
06 13sept 8313 9997-2-ed an adaptive (edit lafi)
 
2019-06-14:3 - Reti neurali e compressione video
2019-06-14:3 - Reti neurali e compressione video2019-06-14:3 - Reti neurali e compressione video
2019-06-14:3 - Reti neurali e compressione video
 

More from Vignesh V Menon

Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Vignesh V Menon
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Vignesh V Menon
 
TQPM.pdf
TQPM.pdfTQPM.pdf
TQPM.pdf
Vignesh V Menon
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
Vignesh V Menon
 

More from Vignesh V Menon (20)

Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
Optimal Quality and Efficiency in Adaptive Live Streaming with JND-Aware Low ...
 
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdfContent_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
Content_adaptive_video_coding_for_HTTP_Adaptive_Streaming.pdf
 
VCIP_MCBE_presentation.pdf
VCIP_MCBE_presentation.pdfVCIP_MCBE_presentation.pdf
VCIP_MCBE_presentation.pdf
 
Green Variable framerate encoding for Adaptive Live Streaming
Green Variable framerate encoding  for Adaptive Live StreamingGreen Variable framerate encoding  for Adaptive Live Streaming
Green Variable framerate encoding for Adaptive Live Streaming
 
JASLA_presentation.pdf
JASLA_presentation.pdfJASLA_presentation.pdf
JASLA_presentation.pdf
 
Green_VCA_presentation.pdf
Green_VCA_presentation.pdfGreen_VCA_presentation.pdf
Green_VCA_presentation.pdf
 
TQPM.pdf
TQPM.pdfTQPM.pdf
TQPM.pdf
 
CAPS_Presentation.pdf
CAPS_Presentation.pdfCAPS_Presentation.pdf
CAPS_Presentation.pdf
 
Doctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdfDoctoral Symposium presentation.pdf
Doctoral Symposium presentation.pdf
 
LiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdfLiveVBR presentation at VQEG NORM.pdf
LiveVBR presentation at VQEG NORM.pdf
 
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdfETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
ETPS_Efficient_Two_pass_Encoding_Scheme_for_Adaptive_Streaming.pdf
 
OPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdf
OPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdfOPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdf
OPSE_Online Per-Scene Encoding for Adaptive HTTP Live Streaming.pdf
 
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdfPerceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
Perceptually-aware Per-title Encoding for Adaptive Video Streaming.pdf
 
Research@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdfResearch@Lunch_Presentation.pdf
Research@Lunch_Presentation.pdf
 
OPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdfOPTE: Online Per-title Encoding for Live Video Streaming.pdf
OPTE: Online Per-title Encoding for Live Video Streaming.pdf
 
Video Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdfVideo Complexity Dataset (VCD).pdf
Video Complexity Dataset (VCD).pdf
 
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive StreamingLive-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
Live-PSTR: Live Per-Title Encoding for Ultra HD Adaptive Streaming
 
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVCIEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
IEEE MMSP'21: INCEPT: Intra CU Depth Prediction for HEVC
 
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming...
 
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
IEEE ICIP'22:Efficient Content-Adaptive Feature-based Shot Detection for HTTP...
 

Recently uploaded

The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
heathfieldcps1
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
ashishpaul799
 

Recently uploaded (20)

Post Exam Fun(da) Intra UEM General Quiz - Finals.pdf
Post Exam Fun(da) Intra UEM General Quiz - Finals.pdfPost Exam Fun(da) Intra UEM General Quiz - Finals.pdf
Post Exam Fun(da) Intra UEM General Quiz - Finals.pdf
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024
 
How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17How to the fix Attribute Error in odoo 17
How to the fix Attribute Error in odoo 17
 
size separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticssize separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceutics
 
The Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. HenryThe Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. Henry
 
An Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptxAn Overview of the Odoo 17 Discuss App.pptx
An Overview of the Odoo 17 Discuss App.pptx
 
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfDanh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
 
philosophy and it's principles based on the life
philosophy and it's principles based on the lifephilosophy and it's principles based on the life
philosophy and it's principles based on the life
 
Open Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPointOpen Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPoint
 
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
Basic Civil Engineering notes on Transportation Engineering, Modes of Transpo...
 
“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
REPRODUCTIVE TOXICITY STUDIE OF MALE AND FEMALEpptx
REPRODUCTIVE TOXICITY  STUDIE OF MALE AND FEMALEpptxREPRODUCTIVE TOXICITY  STUDIE OF MALE AND FEMALEpptx
REPRODUCTIVE TOXICITY STUDIE OF MALE AND FEMALEpptx
 
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdfTelling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
 
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdfPost Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
 
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxMatatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 
The Benefits and Challenges of Open Educational Resources
The Benefits and Challenges of Open Educational ResourcesThe Benefits and Challenges of Open Educational Resources
The Benefits and Challenges of Open Educational Resources
 

Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations

  • 1. Gain of Grain: A Film Grain Handling Toolchain for VVC based Open Implementations — Vignesh V Menon, Postdoctoral Researcher, Fraunhofer HHI Feb. 12, 2024, Mile High Video (MHV‘24)
  • 2. MHV’24 Introduction 12.02.2024 © Fraunhofer Slide 2 Film grain in video coding • Film grain, an inherent characteristic of analog film, contributes to the unique visual aesthetics and cinematic experience in movies [1]. • The emergence of the Versatile Video Coding (VVC) [2, 3] standard brings new opportunities and challenges in efficiently representing film grain, aiming to preserve its artistic value while ensuring compatibility with modern video compression techniques. • Compression artifacts like blockiness are introduced when grainy video content is encoded at low bitrates. Film grain presents a challenge in video coding due to its random and non-uniform nature, which can amplify compression artifacts if not adequately handled during encoding. [1] Inseong Hwang et al. “Enhanced Film Grain Noise Removal for High Fidelity Video Coding”. In: 2013 International Conference on Information Science and Cloud Computing Companion. 2013, pp. 668–674. doi: 10.1109/ISCC- C.2013.69. [2] Benjamin Bross et al. “Overview of the Versatile Video Coding (VVC) Standard and its Applications”. In: IEEE Transactions on Circuits and Systems for Video Technology. Vol. 31. 10. 2021, pp. 3736–3764. doi: 10.1109/TCSVT.2021.3101953 [3] Adam Wieckowski et al. “A Complete End to End Open Source Toolchain for the Versatile Video Coding (VVC) Standard”. In: Proceedings of the 29th ACM International Conference on Multimedia. New York, NY, USA: Association for Computing Machinery, 2021, 3795–3798. isbn: 9781450386517. doi: 10.1145/3474085.3478320.
  • 3. MHV’24 Introduction 12.02.2024 © Fraunhofer Slide 3 Film grain in video coding Figure: Illustration of compression artifacts introduced due to film grain when encoded using VVenC encoder at low bitrates. ▪ Film grain is observed to amplify compression artifacts if not adequately handled during encoding. ▪ Integrating film grain handling into VVC requires adherence to standardized encoding practices while accommodating unique film grain characteristics [4]. [4] Zoubida Ameur, Wassim Hamidouche, Edouard François, Miloš Radosavljević, Daniel Ménard, and Claire-Hélène Demarty, “Deep-Based Film Grain Removal and Synthesis,” IEEE Transactions on Image Processing, vol. 32, pp. 5046– 5059, 2022.
  • 4. MHV’24 State-of-the-art toolchain 12.02.2024 © Fraunhofer Slide 4 Components: ▪ Denoising: reducing or removing noise from the video, including film grain. ▪ Encoding: compressing the video data for efficient storage and transmission. ▪ Decoding: reconstructing the video data from the compressed format. ▪ Grain synthesis: generating or enhancing film grain in the video.
  • 6. MHV’24 Proposed toolchain 12.02.2024 © Fraunhofer Slide 6 MCTF in VVenC: ▪ VVenC already employs a denoising stage [5], based on a framework proposed initially in [6]– motion compensated temporal (pre-)filtering. ▪ The filter performs a blockwise motion search for each filtered frame in neighboring frames to remove the noise. ▪ Using up to eight predictors, a weighted average of the current frame block and its predictors is generated and used for further encoding. ▪ Improved search strategies, reference number reduction, and flexible block size were introduced, improving the filter runtime and operation [5]. Denoising [5] Adam Wieckowski, Tobias Hinz, Christian R. Helmrich, Benjamin Bross, and Detlev Marpe, “An optimized temporal filter implementation for practical applications,” in 2022 Picture Coding Symposium (PCS), 2022, pp. 247–251. [6] Jack Enhorn, Rickard Sjöberg, and Per Wennersten, “A Temporal Pre-Filter For Video Coding Based On Bilateral Filtering,” in 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 1161–1165.
  • 7. MHV’24 Proposed toolchain 12.02.2024 © Fraunhofer Slide 7 ▪ The framework for film grain handling based on frequency filtering and parameterization of grain is implemented in VVenC (to be released soon). ▪ The film grain pattern is characterized using a horizontal high cut-off frequency and a vertical high cut-off frequency obtained in the discrete cosine transform domain. ▪ The film grain pattern is scaled to the appropriate level using a stepwise scaling function that considers the underlying image’s characteristics. ▪ Afterward, the film grain pattern is blended into the image using additive blending [7]. Film grain estimation [7] M. Sean, Y. Peng, H. Walt, P. Fangjun, L. Taoran, C. Tao, M. Radosavljević, E. François, G. Vijayakumar, P. Kaustubh et al., “Fixed-point grain blending process for film grain characteristics sei message,” in Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, document JVET-W0095, 2021.
  • 8. MHV’24 Proposed toolchain 12.02.2024 © Fraunhofer Slide 8 ▪ To convey the grain parameters to the decoder, the encoder embeds it as supplemental enhancement information (SEI) in the bitstream [8], as the "Film Grain SEI.“ ▪ Film Grain SEI inherits the same syntax and semantics of the AVC film grain SEI message [9]. ▪ Since we implement a frequency filtering model for film grain estimation, film_grain_model_id is set to 0. ▪ Additive blending is used when blending_mode_id is set to 0. ▪ Since our implementation analyses film grain for only the luma channel, comp_model_present_flag[0] is set to 1. ▪ FGC SEI message is inserted at each frame, which is indicated by setting the film_grain_characteristics_persistence_flag to 0. It also means the FGC SEI message only applies to the current decoded frame. Film Grain SEI [8] “RDD 5:2006 - SMPTE Registered Disclosure Doc - Film Grain Technology —Specifications for H.264 | MPEG-4 AVC Bitstreams,” RDD 5:2006, pp. 1–18, 2006. [9] Vijayakumar Gayathri Ramakrishna, Kaustubh Shripad Patankar, and Mukund Srinivasan, “Cloud-Based Workflow for AVC Film Grain Synthesis,” ser. MHV’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 66–71.
  • 9. MHV’24 Experimental setup 12.02.2024 © Fraunhofer Slide 9 Comparisons: ▪ We run experiments on an AMD EYPC 7502P processor (32 cores), where we run each VVenC v1.10 instance using four CPU threads, with two-pass rate control, enabling adaptive quantization. ▪ All sequences are down-scaled to 1920x1080 8bit. ▪ We consider the Default toolchain as the benchmark, where : ▪ Encode the input video using VVenC with MCTF disabled. ▪ Decode the resulting bitstream using the VTM decoder. ▪ FGA and FGS are disabled in this toolchain. Table: Experimental parameters used to evaluate the proposed film grain handling toolchain.
  • 10. MHV’24 Quality assessment metrics 12.02.2024 © Fraunhofer Slide 10 Observations: ▪ Traditional metrics like PSNR and SSIM are not suitable for evaluating the perceptual quality of film grain coding owing to their lack of texture sensitivity. ▪ PSNR and SSIM are sensitive to noise, such that they penalize the addition of synthesized film grain. ▪ VMAF [10], while more advanced, is not trained to evaluate the perceptual quality of VVC-coded videos [11]. Figure: RD curves using the Default toolchain, MCTF (without FGS), and the proposed toolchain for IntoTree test sequence encoded at faster preset. [10] Zhi Li, Christos Bampis, Julie Novak, Anne Aaron, Kyle Swanson, Anush Moorthy, and Jan De Cock, “VMAF: The journey continues,” in Netflix Technology Blog, vol. 25, 2018. [11] Christian R. Helmrich, Benjamin Bross, Jonathan Pfaff, Heiko Schwarz, Detlev Marpe, and Thomas Wiegand, “Information on and analysis of the VVC encoders in the SDR UHD verification test,” in WG 05 MPEG Joint Video Coding Team(s) with ITU-T SG 16, document JVET-T0103, Oct. 2020. Take aways: ▪ Given these limitations, specialized metrics focusing on texture enhancement, perception of controlled noise, and overall film-like appearance would be more appropriate for evaluating film grain coding, subject to future work. ▪ Metrics that include human perception aspects and consider texture fidelity alongside noise would offer a better assessment of the quality enhancements film grain brings to video content.
  • 11. MHV’24 Subjective quality assessment 12.02.2024 © Fraunhofer Slide 11 ▪ Original ▪ No MCTF (250 kbps) ▪ With the proposed toolchain (250 kbps) ▪ MCTF (250 kbps)
  • 12. MHV’24 Subjective quality assessment 12.02.2024 © Fraunhofer Slide 12 ▪ Original ▪ MCTF (3 Mbps) ▪ With the proposed toolchain (3 Mbps)
  • 13. MHV’24 Runtime complexity 12.02.2024 © Fraunhofer Slide 13 Encoding and decoding Encoding time: ▪ FGA contributes to the increased relative duration required for encoding as the preset progresses towards faster configuration. ▪ The overall encoding time using the proposed toolchain reduces up to 11.59 % using slower preset. Table: Runtime complexity of the proposed toolchain compared to the default toolchain with the same preset. [13] SVT. “The SVT High Definition Multi Format Test Set”. In: Feb. 2006. url: https://tech.ebu.ch/docs/hdtv/svt-multiformat-conditions-v10.pdf [14] Li Song et al. “The SJTU 4K video sequence dataset”. In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX). 2013, pp. 34–35. doi: 10.1109/QoMEX.2013.6603201. url: https://doi.org/10.1109/QoMEX.2013.6603201. [15] Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018. url: https://www.researchgate.net/publication/326506581 Decoding time: ▪ The decoding time has increased due to the supplementary computational load of FGS. ▪ The effect on decoding time is intertwined with the encoding presets; as encoding presets progress towards slower configurations, the proportional increase in decoding time tends to diminish.
  • 14. MHV’24 Conclusions and future directions 12.02.2024 © Fraunhofer Slide 14 Conclusions ▪ We presented an overview of the film grain handling toolchain for VVC-based open implementation using the VVenC encoder and VTM decoder. ▪ The experimental results show that the proposed toolchain improves the subjective quality of the grainy video content encoded at multiple bitrates, compared to default toolchain, considering VVenC encoding. ▪ Using slower preset, the proposed toolchain reduces the encoding duration by up to 11.59% while the decoding time increases by up to 15.32%. ▪ These outcomes underscore the inherent trade-offs in optimizing the film grain handling toolchain, emphasizing the need for a balanced approach that prioritizes encoding efficiency and video quality. Future directions ▪ Denoising and film grain estimation shall be tuned for various encoding presets in VVenC. ▪ More sophisticated models shall be investigated to represent film grain accurately in the digital domain. This could involve advanced statistical models and/or machine learning approaches to capture the intricate characteristics of film grain. ▪ Develop quality assessment metrics that better capture the human perception of film grain.
  • 15. Thank you for your attention — ▪ Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de) ▪ Adam Wieckowski (adam.wieckowski@hhi.fraunhofer.de) ▪ Jens Brandenburg (jens.Brandenburg@hhi.Fraunhofer.de) ▪ Benjamin Bross (benjamin.bross@hhi.fraunhofer.de) ▪ Thomas Schierl (thomas.schierl@hhi.fraunhofer.de) ▪ Detlev Marpe (detlev.marpe@hhi.fraunhofer.de)