Film Grain Coding for Versatile Video Coding Systems: Techniques, Challenges, and Applications

Film Grain Coding for Versatile Video Coding Systems:
Techniques, Challenges, and Applications
—
Vignesh V Menon, Postdoctoral Researcher, Fraunhofer HHI, Berlin, Germany
Philippe de Lagrange, Senior Engineer, InterDigital, Cesson-Sevigne, France
Dec 08, 2024, IEEE Visual Communications and Image Processing (IEEE VCIP 2024)

Biography
Dr. Vignesh V Menon
● Postdoctoral Researcher, Fraunhofer HHI, Germany
● PhD Informatics- University of Klagenfurt, Austria
● MSc Information and Network Engineering- KTH
Royal Institute of Technology, Sweden
● 8+ years expertise in video coding
Philippe de Lagrange
● Senior researcher engineer, InterDigital R&D France
● Active contributor to MPEG (JVET and AG5)
● 20+ years in the video field (decoders, encoders,
standardization). Previously in radiocommunications
and satellite positioning
● Both hardware/firmware and algorithmic expertise
● Engineering degree from INSA Rennes

MHV’24
Agenda
Slide 3
1. Introduction to Film Grain in Video Coding
2. Fundamentals of Film Grain Modeling
3. Film Grain Handling in VVC
4. Implementation examples, Software tools, Standardization activities
© Fraunhofer © Interdigital

Introduction to Film Grain in Video Coding
—
Understanding its History, Importance, and Challenges

MHV’24
Introduction
08.12.2024 © Fraunhofer
Slide 5
Motivation
Source: Grain Ghost -- Frames2 -- Uncut Gems with Darius Khondji, ASC, AFC.
● Uncut Gems by the Safdie brothers with cinematography by Darius Khondji, ASC, AFC.
● Background visual elements bring emotion and content to the scene, even when they are not
consciously noticed by the audience.
Darius Kondji:
“I remember that I pushed the film negative one or two stops to bring out the natural grain.”

MHV’24
Introduction
08.12.2024 © Fraunhofer © Interdigital
Slide 6
Motivation
Source: Grain Ghost -- Frames2 -- Uncut Gems with Darius Khondji, ASC, AFC

Introduction
—
Film Stock & Film Grain

MHV’24
Introduction- What is film stock?
Slide 8
Definition
● Analog medium that is used for recording motion pictures or animation.
● Recorded on by a movie camera, developed, edited, and projected onto a
screen using a movie projector.
● Strip or sheet of transparent plastic film base coated on one side with a
gelatin emulsion containing microscopically small light-sensitive silver halide
crystals.
● The sizes and other characteristics of the crystals determine the sensitivity,
contrast and resolution of the film.
K. Keller et al. "Photography" in Ullmann's Encyclopedia of Industrial Chemistry, 2005, Wiley-VCH, Weinheim. doi:10.1002/14356007.a20_001

MHV’24
Introduction- What is film stock?
Slide 9
Physical characteristics
In color stock film, three sandwiched layers of color-sensitive
monochrome halides are reinterpreted at processing time into genuine
color layers. However, despite the use of color in these illustrations,
each layer is essentially monochrome, with color added at the
developing stage, and targeted to each assigned color strata.
Emulsion layers
https://blog.metaphysic.ai/the-challenge-of-simulating-grain-in-film-stocks-of-the-past/

MHV’24
Introduction- What is film grain?
Slide 10
Definition
● Visible silver crystals in a film negative's emulsion.
● These light-sensitive silver halides change into pure
metallic silver when exposed to light, which is how an
image is captured on film.
○ Grain is an inherent part of a film image.
● The higher a film stock's ISO is, the bigger the silver
crystals are.
○ A higher speed film (ISO 400 and above) will have
more visible grain, while a slower speed film (ISO 100
or less) will have a finer grain.
https://richardphotolab.com/blogs/post/film-grain-and-pixelation
Grain structure in Kodak TMAX 400.

MHV’24
Introduction
Slide 12
Historical Examples of Film Grain
● Autochrome
○ Early color photography used a normal
black-and-white emulsion covered by a
multi-colored dust layer.
○ Featured a rich texture.
● Iconic Film Stocks
○ Evolution from coarse nitrate films to fine
acetate and polyester films.
24x24 and 2x2mm
crops from a
1916 autochrome
glass plate.
A. Samama,
RMN-GP (France)
https://www.afcinema.com/Au-revoir-la-haut.html

MHV’24
Introduction
Slide 13
Historical Examples of Film Grain In the mid-1970s, director Martin Scorsese
used 35mm Eastman Color Negative 100T
5254/7254 film stock for 'Taxi Driver',
enabling live shooting on the night-time
streets of New York.
M. Anderson, ‘The Challenge of Simulating Grain in Film Stocks of the Past,’ Link: https://blog.metaphysic.ai/the-challenge-of-simulating-grain-in-film-stocks-of-the-past/
● Adds authenticity and a timeless
aesthetic to content.
● Evokes nostalgia, realism, and specific
moods in cinematography.
Damien Van Der Cruyssen (colorist):
“Film grain is a background that is not usually noticed
by audiences because it's made of small elements,
and because it changes from frame to frame. Grain's
mobility and texture give life and visual complexity to
the footage. I believe that grain's continuous subtle
modulation of the background draws you into the
image and keeps you engaged.”

MHV’24
Introduction
Slide 14
Historical Examples of Film Grain
In the case of James Cameron's 'Aliens' (1986), among the grainiest mainstream movies of the 1980s,
the film was shot on one of the last batches of Kodak's large-halide Eastman 400T 5295 Neg. Film and
Eastman Color Negative 400T 5294 Film.
M. Anderson, ‘The Challenge of Simulating Grain in Film Stocks of the Past,’ Link: https://blog.metaphysic.ai/the-challenge-of-simulating-grain-in-film-stocks-of-the-past/

Introduction
—
Artistic Importance

MHV’24
Introduction
Slide 16
Artistic Importance of Film Grain
● Enhancing Visual Storytelling
○ Grain creates an emotional connection
by evoking nostalgia or intensity.
○ Used creatively in genres like:
■ Noir: Heavy grain for a gritty,
dramatic tone.
■ Drama: Subtle grain for naturalism
and emotional resonance.

MHV’24
Introduction
Slide 17

MHV’24
Introduction
Slide 18
● Movies with notable grain, or purposefully shot on
film:
○ Oppenheimer
○ The Lord of the rings
○ Blade runner
○ The Revenant
○ The Hateful Eight
○ Moonlight
○ The Big Lebowski
○ Alien
○ Iron Man
○ Dune
https://www.kodak.com/en/motion/blog/

MHV’24
Introduction
Slide 19
Calibration
● Modern Applications
○ Filmmakers often try to use actual film or add
synthetic grain in digital workflows to recreate
analog aesthetics.
○ Very often not all the movie is shot on film;
some scenes are not for different reasons
(aerials, low light, ...).
○ Post-production grain is generally used to
harmonize the whole film – "calibration".
○ They do the same with color to harmonize
everything across different capture devices and
lighting conditions.
Thumbnails from Netflix Meridian test movie
G. Fossati, ‘From Grain to Pixel: The Archival Life of Film in Transition’, Third Revised Edition, Amsterdam
University Press, 2018.

MHV’24
Introduction
Slide 20
Technical Importance of Film Grain
● Realism and Texture
○ Adds a layer of textural detail to smooth digital
frames, mimicking the imperfections of film.
○ Prevents overly clean visuals that feel artificial.
● Preserving Artistic Intent in Restorations
○ Grain preservation ensures remastered classics
retain their original look and feel. Damien Van Der Cruyssen (colorist):
“Most movies shooting digitally want to add
grain. In fact, there’s only one digital feature
I've graded that didn’t add grain”
Darius Khondji:
“We added grain to the digital footage to
match the film footage. At first we tried adding
digital grain, but it wasn't as nice as the actual
film grain. So Damien, my colorist, suggested
adding real film grain to the digital footage,
and that worked.”

MHV’24
Introduction- Why Film Grain Matters
Slide 21
Psychological Impact of Film Grain
● Viewer Perception
○ Film grain subconsciously signals authenticity and effort.
○ Grain patterns influence emotional engagement:
■ Subtle grain: Comforting, nostalgic.
■ Heavy grain: Dramatic, intense.
● Cinematic Language
○ Grain often reflects the tone and setting of a film:
■ E.g., gritty urban landscapes or dreamlike period
dramas.
Damien Van Der Cruyssen (colorist):
“Darius and the Safdies wanted a
bolder look than in other scenes. I
strengthened the grain to play with the
emotions of the couple's fight. And I
added contrast to the grain. They liked
the roughness of this image.”

MHV’24
Introduction- Historical Context
Slide 22
Transition to Digital
● Challenges in Early Digital Encoding
○ Analog-to-digital conversion initially struggled to preserve the randomness and texture of film
grain.
○ Encoding grain required high bitrates, making compression inefficient.
● Modern Solutions
○ Synthesis of grain during playback reduces the need for high bitrates.
○ Tools in codecs like AVC and VVC introduced efficient grain modeling and synthesis.

Introduction
—
Challenges in Encoding Film Grain

MHV’24
Introduction
Slide 24
High Encoding Costs
● Film grain is random and high-frequency, making it hard to compress efficiently.
● Conventional encoding methods require high bitrates to preserve grain details, increasing storage and
transmission costs.
Rate-Distortion plots of the reference HEVC and VVC encoding for various sequences.
Grain
Video
Video
Video
Grain
Grain
DinnerScene2 OldTownCross MeridianIntoCar

MHV’24
Introduction
Slide 25
Encoding removes film grain
● Default video encoding does not focus on grain
preservation.
First frame of OldTownCross sequence encoded at 2.4 Mbps (faster preset).
Temporal and Spatial Inconsistencies
● Temporal instability can occur when grain
patterns vary frame-to-frame, causing flickering.
● Compression algorithms struggle to predict or
encode the randomness of grain accurately.

MHV’24
Introduction
Slide 27
Film Grain Modeling & Synthesis
● Grain Synthesis
○ Add film grain metadata in the encoded bitstream.
○ Efficient grain generation at playback, avoiding encoding overhead.
● Efficiency
○ Significant bitrate savings (~90%) compared to encoding grain directly.
Source video
Encode Decode Add grain
Output video
Bitstream
with metadata
Metadata = model parameters
Model parameters

MHV’24
Introduction
Slide 28
Film Grain handling in VVC
● VVC-based open implementations to handle film grain efficiently.
Source video
Output video
Bitstream
with metadata
Metadata = model parameters
Model parameters

MHV’24
Introduction
Slide 29
Hardware Models for Grain Synthesis (VFGS)
● VFGS (Versatile Film Grain Synthesis)
○ Efficient hardware model implemented for real-time grain synthesis.
○ Processes film grain metadata to generate grain dynamically during playback.
● Key Features
○ Low power consumption.
○ High processing speed ensures smooth playback.
○ Compatible with devices ranging from mobile phones to large-screen displays.
● Advantages
○ Eliminates the need for high-bitrate grain encoding.
○ Ensures grain consistency across diverse hardware setups.

MHV’24
Introduction
Slide 30
Industry Actions and Standardization
● Standardization Efforts
○ 3GPP: Testing grain synthesis for mobile streaming.
○ DVB: Technology evaluation, market survey
○ JVET: Refining grain synthesis techniques in VVC.
● Cross-Industry Collaboration
○ Ensuring consistent grain reproduction across devices and platforms.

Fundamentals of Film Grain Modeling
—

MHV’24
Slide 32
Introduction, summary of problem statement
● Film grain…
● Conveys an artistic intent
○ tool for storytelling (style)
○ many moviemakers use film when they can, or
mimic it in post-production
● But is lost in compression
○ we can use tricks, but limited
○ recent codecs remove even more
○ strong use cases for home cinema at reduced
bitrate
Video
Grain

MHV’24
Slide 33
Solution: statistical coding (film grain synthesis)
Source video
Grain model fitting
Output video
Bitstream
with metadata
Metadata = model
parameters
Model parameters
Encode grain using a statistical model, instead of coding each pixel
● Convey model parameters
● Recreate grain after video decoding, using the model and parameters

MHV’24
Slide 34
Source video
Grain model fitting
Output video
Bitstream
with metadata
Metadata = model
parameters
Model parameters
● There are end-to-end learned methods, but we will focus on the MPEG normative approach
○ FGC (Film Grain Characteristics) SEI message, applicable to AVC, HEVC and VVC
● InterDigital is the inventor of this technology (taking over Technicolor)
See the FGC SEI message specification in ITU-T H.274
See also ISO/IEC TR 23002-9 for an in-depth technical description, different possibilities and implementation considerations

MHV’24
Slide 35
Original Compressed Compressed + grain synthesis

MHV’24
Slide 36
Film grain synthesis: key qualities for each step
Source video
Grain model fitting
Output video
Bitstream
with metadata
Metadata = model
parameters
Model parameters
Synthesis
o Doable in hardware (display pipe), low
cost, interoperability
o Good rendering quality (looks like film)

MHV’24
Slide 37
Source video
Grain model fitting
Output video
Bitstream
with metadata
Metadata = model
parameters
Model parameters
Model
o Capability to capture
real grain behaviour

MHV’24
Slide 38
Source video
Grain model fitting
Output video
Bitstream
with metadata
Metadata = model
parameters
Model parameters
Analysis
o Find accurate model parameters
o Fit source grain, or what encoder removes

MHV’24
Slide 39
Film grain synthesis: similarity metrics
● Fidelity of synthetic film grain to original is usually assessed by visual inspection, because it is an
artistic matter and pixel-based metrics (PSNR, SSIM, VMAF, etc) do not help in this case.
● A dedicated similarity metric could be helpful. Potential candidates are:
○ LPIPS (Learned Perceptual Image Patch Similarity) from R. Zang et al.
○ Proprietary solutions, like the Film Grain Similarity (FGS) Metric (also a learned metric) being
developed by IMAX.
○ Conventional methods involving multi-scale, orientation and frequency analysis, like SFGA
(subband based film grain assessment) proposed by Disney.
LPIPS: https://arxiv.org/pdf/1801.03924
IMAX: IBC2024 Tech Papers: "Quantitative assessment of film grain similarity: an objective model"
SFGA: JVET-AF0209, JVET-AG0295

MHV’24
Slide 40
Film grain synthesis: harmonization and artefact hiding
● Film grain synthesis has to additional benefit of hiding compression imperfection and potential
irregularities when the encoder attempts to encode part of the grain.
Compressed frame #0
some grain is lost
Compressed frame #16
grain is badly impaired
Compressed frame #16 + synthesis
much closer to original

MHV’24
Slide 41
Statistical model, metadata
Focus on film grain statistical model (FGC SEI message)
● Describing the grain texture
● And how it varies in a picture (dark / light)
● Examples

MHV’24
Slide 42
Statistical model, metadata
● Model has to describe grain texture
(strength, coarseness, …)
● And how it varies in a picture (dark vs
light areas)
crops of the
same picture
different sources

MHV’24
Slide 43
Statistical model, metadata - grain texture
FGC SEI has two possible modes to describe grain texture: FF (frequency filtering) or AR (auto-regressive).
Each uses up to 6 parameters, with different meaning for each mode, and default values when using less.
● FF mode gain, Fh_high, Fv_high, Fh_low, Fv_low, xcc
● AR mode gain, c1, xcc, c2, A, c3
See also: ITU-T H.274 (specification),
ISO/IEC TR 23002-9 (technical report)
1
Ac1
c1
c3
Ac2 Ac2
A²c3
AR filter kernel
Gaussian noise
Grain texture
AR filter
Inverse DCT
Cutoff frequencies

MHV’24
Slide 44
original (film scan) synthesis, 20% too weak synthesis, 20% too strong
Gain:
impact on grain
amplitude

MHV’24
Slide 45
original (film scan) synthesis, grain too coarse
Frequency cutoff:
impact on grain size

MHV’24
Slide 46
Statistical model, metadata - grain variation
Grain varies (both strength and size) depending
on light received (= sample values, or “intensity”)
FGC SEI message defines intensity intervals,
and grain parameters for each interval.

MHV’24
Slide 47
Sky grain applied to the whole picture: wrong True grain (source - denoised)

MHV’24
Slide 48
Original Compressed + grain synthesis
Compressed

MHV’24
Slide 49
Original Synthesis

MHV’24
Slide 50
Statistical model, metadata - examples
Compressed

MHV’24
Slide 51
Compressed

MHV’24
Slide 52
Compressed

MHV’24
Slide 53
Compressed

MHV’24
Slide 54
Original Wrong parameters 2
Wrong parameters 1

MHV’24
Slide 55
Compressed

MHV’24
Slide 56
Compressed

MHV’24
Slide 57
Compressed

MHV’24
Slide 58
Summary
● Film grain is often valued by movie artists as part of the narrative
● But it is hard to preserve at practical bitrates. This question is arising even more today, with
○ Compression gains of recent standards (e.g. VVC) coming at the cost of noise and grain removal
○ Strong use cases of streaming at low bitrate for home cinema
● Film grain synthesis comes to the rescue = statistical coding of the grain
○ Normative model (FGC SEI message), interoperable
○ Good quality of grain characterization
○ Mature and available for widely used standards (AVC, HEVC, VVC)
○ Practical synthesis implementation is possible and demonstrated
○ Correct model parameters (e.g. grain analysis) are also key

Film Grain Handling in VVC
—

—
Overview of VVC

MHV’24
Slide 62
Overview of VVC
Typical VVC encoder [2].
● Developed by Joint Video Experts Team (JVET).
● Improved Compression Efficiency: Achieves up to
50% bitrate savings compared to HEVC [1],
enabling higher-quality video at lower data rates.
● Enhanced Versatility: Optimized for a broad
range of content, from standard-definition video to
8K ultra-high definition (UHD) and immersive
media like 360° videos.
● Support for Emerging Applications: Designed for
adaptive streaming, AR/VR, and live
broadcasting.
[1] G. J. Sullivan et al., “Overview of the high efficiency video coding (HEVC) standard,” in IEEE Transactions on circuits and systems for video technology, vol. 22, no. 12. IEEE, Dec. 2012, pp. 1649–1668.
[2] B. Bross et al., "Overview of the Versatile Video Coding (VVC) Standard and its Applications," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736-3764, Oct. 2021, doi:
10.1109/TCSVT.2021.3101953.

MHV’24
Slide 63
Comparison of VVC with AVC and HEVC
● Compression Gains: VVC delivers 2x the compression
efficiency of AVC [1] and significantly outperforms HEVC
[2] in high-resolution scenarios.
● Flexible Block Partitioning: Introduces quadtree plus
multi-type tree (QTMT) structures, allowing better
adaptation to video content [4].
● Advanced Coding Tools: Includes affine motion
compensation, larger transform sizes, and better
intra-prediction modes.
● Adaptation to Hardware: Designed to leverage modern
parallel processing and hardware accelerators for
real-time performance.
[1] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560–576, 2003.
[2] G. J. Sullivan et al., “Overview of the high efficiency video coding (HEVC) standard,” in IEEE Transactions on circuits and systems for video technology, vol. 22, no. 12. IEEE, Dec. 2012, pp. 1649–1668.
[3] Uhrina, Miroslav, Lukas Sevcik, Juraj Bienik, and Lenka Smatanova. 2024. "Performance Comparison of VVC, AV1, HEVC, and AVC for High Resolutions" Electronics 13, no. 5: 953.
https://doi.org/10.3390/electronics13050953
[4] A. Wieckowski, J. Ma, H. Schwarz, D. Marpe and T. Wiegand, "Fast Partitioning Decision Strategies for The Upcoming Versatile Video Coding (VVC) Standard," 2019 IEEE International Conference on Image
Processing (ICIP), Taipei, Taiwan, 2019, pp. 4130-4134, doi: 10.1109/ICIP.2019.8803533.
Table: Averaged BD-BR savings depending on codec and resolution [3].
Table: Averaged BD-PSNR savings depending on codec and resolution [3].

MHV’24
Slide 64
Overview of the VVenC and VVdeC Implementations
● VVenC [1] is an open-source implementation of
the VVC encoder, and VVdeC [2] is the
corresponding decoder.
● These tools provide researchers and industry
professionals with the opportunity to explore the
capabilities of VVC, including its handling of film
grain.
● VVenC analyzes the film grain during encoding
and generates the necessary FGS SEI messages,
while VVdeC reconstructs the grain during
playback, ensuring that the final output maintains
the desired visual aesthetic.
[1] A. Wieckowski, J. Brandenburg, T. Hinz, C. Bartnik, V. George, G. Hege, C. Helmrich, A. Henkel, C. Lehmann, C. Stoffers, I. Zupancic, B. Bross, and D. Marpe, “Vvenc: An Open And Optimized
VVC Encoder Implementation,” in 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Jul. 2021, pp. 1–2.
[2] A. Wieckowski, G. Hege, C. Bartnik, C. Lehmann, C. Stoffers, B. Bross, and D. Marpe, “Towards A Live Software Decoder Implementation For The Upcoming Versatile Video Coding (VVC) Codec,”
in Proc. IEEE International Conference on Image Processing (ICIP), pp. 3124–3128.

MHV’24
Slide 65
What is Film Grain Synthesis in VVC?
● Why Film Grain Synthesis?
○ Compression Efficiency: Directly storing grain
increases the bitrate. FGS stores only grain
parameters, making compression more efficient.
○ Artistic Intent: Film grain contributes to the aesthetics
of cinematic content, enhancing visual texture and
mood.
Grain
Video

—
SEI for FGS

MHV’24
Slide 67
Supplemental Enhancement Information (SEI) message
● Role of Film Grain Characteristics (FGC) SEI message
○ Compact Representation: Grain parameters such as
intensity, size, and distribution are encoded in the
VSEI.
○ Decoder Flexibility: VSEI enables playback devices
to synthesize grain with minimal computational
overhead.
○ Backward Compatibility: Devices without FGS
support can still decode the base video without
errors, ignoring the VSEI.
[1] International Telecommunication Union, "H.274: Versatile supplemental enhancement information messages for coded video bitstreams," Sep. 2023. [Online]. Available: https://www.itu.int/rec/T-REC-H.274
[2] MPEG video technologies, "Part 7: Versatile supplemental enhancement information messages for coded video bitstreams," in ISO/IEC DIS 23002-7. [Online]. Available:
https://www.iso.org/standard/87644.html
If there is no SEI, FGS is skipped.

MHV’24
Slide 68
How VVC Handles Film Grain
● Grain Analysis and Modeling (Encoding)
○ The encoder analyzes the video to identify and model film grain characteristics such as intensity, size,
and distribution.
○ These characteristics are represented as compact parameters.
● Parameter Storage in FGC SEI message
○ Grain parameters are stored in the FGC SEI message, ensuring minimal bitrate overhead.
○ These parameters describe how to recreate the grain pattern during decoding.
● Grain Synthesis (Decoding)
○ The decoder reads the grain parameters from the FGC SEI message.
○ Film grain is dynamically reconstructed and added to the decoded video.

—
Grain Modeling in VVenC

MHV’24
Slide 70
Grain Modeling during Encoding (1/5)
Purpose
● Analyze and model film grain patterns for accurate synthesis during
decoding.
Key Objectives
● Identify flat, low-complexity regions in the video for grain modeling.
● Estimate grain parameters like scaling factors and cutoff frequencies.
● Ensure accurate reproduction of grain texture with minimal overhead.
Film grain modeling during encoding
implemented in VVenC.
V. V. Menon, A. Wieckowski, J. Brandenburg, B. Bross, T. Schierl, and D. Marpe. 2024. Gain of Grain: A Film Grain Handling Toolchain for VVC-based Open Implementations. In Proceedings of the
3rd Mile-High Video Conference (MHV '24). Association for Computing Machinery, New York, NY, USA, 47–53. https://doi.org/10.1145/3638036.3640805

MHV’24
Slide 71
Mask Generation for Region Selection
Objective: Identify flat and uniform areas suitable for grain analysis.
Steps:
● Edge Detection: Apply the Canny edge detector to detect high-frequency
regions.
● Low-Intensity Suppression: Mask out low-intensity areas using thresholding.
● Morphological Operations: Refine the mask using dilation and erosion to
ensure clean boundaries.
● Subsampling and Combination: Process the frame at multiple resolutions
(full, half, quarter) and combine results.

MHV’24
Slide 72
Grain Parameter Estimation
Objective: Estimate scaling factors and cutoff frequencies
for film grain synthesis.
Steps:
● Calculate the difference between the original and
filtered frames to estimate grain texture.
● Analyze 64×64 blocks for flat, uniform regions without
edges.
● Compute mean and variance for these blocks to model
intensity-based grain patterns.
● Perform polynomial regression to estimate scaling
factors for intensity intervals.

MHV’24
Slide 73
Adaptive Cutoff Frequency Estimation
Purpose: Determine the horizontal and vertical cutoff frequencies for
film grain.
Method:
● Use adaptive sampling to refine significant rows and columns in
the DCT matrix.
● Focus on bins with high mean squared DCT values.
● Iterate to improve frequency estimates for efficient modeling.

MHV’24
Slide 74
Final Scaling Factors and SEI Integration
● Final Scaling Factors
○ Use Lloyd-Max quantization to derive final scaling factors.
○ Merge small intervals to reduce redundancy and optimize
SEI representation.
● SEI Encoding
○ Store intensity intervals and corresponding scaling factors
in SEI.
○ Ensure parameters are compatible with 8-bit range for
decoding.

MHV’24
Slide 75
Storing Grain Parameters
● To convey the grain parameters to the decoder, the encoder
embeds it as SEI in the bitstream [1][2], as the "Film Grain SEI.“
● Film Grain SEI inherits the same syntax and semantics of the AVC
film grain SEI message [3].
● Since we implement a frequency filtering model for film grain
estimation, film_grain_model_id is set to 0.
● Additive blending is used when blending_mode_id is set to 0.
● Since our implementation analyses film grain for only the luma
channel, comp_model_present_flag[0] is set to 1.
● FGC SEI message is inserted at each GOP interval, which is
indicated by setting the film_grain_characteristics_persistence_flag
to 1. It also means the FGC SEI message applies till the next SEI is
signalled.
[1] International Telecommunication Union, "H.274: Versatile supplemental enhancement information messages for coded video bitstreams," Sep. 2023. [Online]. Available: https://www.itu.int/rec/T-REC-H.274
[2] MPEG video technologies, "Part 7: Versatile supplemental enhancement information messages for coded video bitstreams," in ISO/IEC DIS 23002-7. [Online]. Available: https://www.iso.org/standard/87644.html
[3] Vijayakumar Gayathri Ramakrishna, Kaustubh Shripad Patankar, and Mukund Srinivasan, “Cloud-Based Workflow for AVC Film Grain Synthesis,” ser. MHV’23. New York, NY, USA: Association for Computing Machinery,
2023, p. 66–71.

—
Grain Synthesis in VVdeC

MHV’24
Slide 77
Decoder-Side Film Grain Synthesis
● On the decoder side, VVdeC uses the FGC SEI
messages to synthesize the film grain during
playback.
● This approach helps achieve a realistic grain effect
that matches the original video.
● The synthesis process ensures that the grain is
added back in a consistent and natural manner,
preserving the artistic intent of the original content
without significantly increasing the bitrate.
M. Radosavljevic, E. François, E. Reinhard, W. Hamidouche, T. Amestoy, "Implementation of film-grain technology within VVC," Proc. SPIE 11842, Applications of Digital Image Processing XLIV,
118420D (1 August 2021); https://doi.org/10.1117/12.2594084

MHV’24
Slide 78
End-to-End Workflow for Film Grain in VVC

—
Experimental Results

MHV’24
Slide 80
Quality assessment metrics
Observations
● Traditional metrics like PSNR and SSIM are not suitable for evaluating the
perceptual quality of film grain coding owing to their lack of context sensitivity.
● PSNR and SSIM are sensitive to noise, such that they penalize the addition
of synthesized film grain.
● VMAF, while more advanced, is not trained to evaluate the perceptual quality
of VVC-coded videos.
Take aways
● Given these limitations, specialized metrics focusing on texture enhancement, perception of controlled noise,
and overall film-like appearance would be more appropriate for evaluating film grain coding, subject to future
work.
● Metrics that include human perception aspects and consider texture fidelity alongside noise would offer a better
assessment of the quality enhancements film grain brings to video content.

MHV’24
Slide 81
Quality assessment
▪ Original ▪ Default (3 Mbps) ▪ With the proposed toolchain (3 Mbps)

MHV’24
Slide 82
Quality assessment
▪ Original
▪ With the proposed toolchain (250 kbps)
▪ Default (250 kbps)

MHV’24
Slide 83
Quality assessment- Masking Compression Artifacts

MHV’24
Slide 84
Encoding and Decoding Runtime Complexity
Encoding Speed
● FGA contributes to the negligible increased relative duration
required for encoding.
Decoding Speed
● On average, VVdeC (FGS) is approximately 60 times faster than
VTM (FGS) (single thread) and can handle real-time decoding.
● The optimization of FGS within the VVdeC decoder is a work in
progress and remains a focus for future improvements.
Table: Decoding speeds (in fps) of VTM and VVdeC.
Table: Encoding runtime increase with FGA.

MHV’24
Slide 85
Industry Feedback
Content Creators
● Priority: Preserving the artistic look and feel of original
content.
● Feedback: High-quality grain synthesis is essential for
cinematic productions and archival content.
● Challenge: Concerns about grain inconsistencies or loss of
fidelity during playback.
Fraunhofer HHI team at NAB Show in Las Vegas, 2024.
Feedback cycle flow diagram.

MHV’24
Slide 86
Industry Feedback
Distributors and Streaming Platforms
● Priority: Reducing bitrate and ensuring smooth playback across diverse networks.
● Feedback: Appreciate the minimal overhead of VSEI metadata but emphasize the need for robust grain
synthesis for varied content.
● Challenge: Balancing compression efficiency and perceptual quality for large-scale streaming.
Consumer Electronics (CE) Device Makers
● Priority: Efficient decoding with minimal hardware overhead.
● Feedback: Grain synthesis should not increase computational complexity significantly.
● Challenge: Ensuring grain synthesis aligns with real-time decoding requirements, especially on
resource-constrained devices.

MHV’24
Slide 87
Key Takeaways
● Compression Efficiency
○ Film grain synthesis in VVC achieves significant bitrate savings by encoding grain as metadata
rather than pixel data.
● Artistic Fidelity
○ Preserves the visual aesthetics and mood intended by filmmakers.
○ Seamlessly reconstructs grain during playback, ensuring alignment with creative intent.
● Standardized Integration
○ Film Grain SEI in VVC ensures compatibility across devices, maintaining high-quality output with
minimal overhead.
● Industry Adoption
○ Well-suited for streaming platforms and CE devices, balancing playback quality and network
efficiency.
● Forward Looking
○ Future developments in metrics and synthesis techniques will further enhance grain fidelity and
compression trade-offs.

Implementation examples,
Software tools,
Standardization activities
—

MHV’24
Software tools & standardization activities
Slide 89
Hardware-friendly synthesis: VFGS
● Grain pattern(s) + randomization and local adaptation
● Multi-model: generic hardware interface, pixel-based process suitable for display pipeline (no line buffer)
● Standalone, or integrated in VVdeC
https://github.com/InterDigitalInc/VersatileFilmGrain
Firmware
Hardware
Model parameters
(e.g. FGC SEI)
Pixel stream (in) Pixel stream (out)
Gain
LUT
Pat-
terns
Pat-
terns
Pat-
terns
Pat-
terns
Pattern
LUT
vfgs_fw.c / .h
vfgs_hw.c / .h

MHV’24
Slide 90
Hardware-friendly synthesis: VFGS
See ISO/IEC TR 23002-9 for more detail

MHV’24
Slide 91
Hardware-friendly synthesis: VFGS – the trick
Switching pattern pixel-wise !
Works because:
● the different patterns are
correlated with each other
● the random offset is kept
constant for a significant area
(e.g. 16x16)
>> picks the same place in a related
pattern; change is still smooth.
Same gaussian noise
Different filters to make
different patterns
Same
random
window

MHV’24
Slide 92
Hardware-friendly synthesis: VFGS – frequency shaping
Gaussian noise
DCT mask DCT coefficients
Grain pattern
(inverse DCT)

MHV’24
Slide 93
VvdecWebPlayer - demo
● VVC player embedded in web page
● Live decode and grain synthesis
● Uses VVdeC with VFGS, WebASM
https://github.com/fraunhoferhhi/vvdecWebPlayer

MHV’24
Slide 94
GUI tool for FGC - demo
https://github.com/InterDigitalInc/VersatileFilmGrain
double-click to
split interval
drag to adjust
focus area
right-click to
temporary disable
the light part
original
grain synthesis

MHV’24
Slide 95
NN-based grain analysis (preliminary)
● InterDigital is currently experimenting with NN-based film grain analysis, to
generate FGC SEI model parameters directly from the source image.
Original NN-based
Manual tuning

MHV’24
Slide 96
NN-base grain analysis (preliminary)
● Network infers intervals, frequency cutoffs, and amplitude
Original NN-based
Manual tuning

MHV’24
Slide 97
Integration in Post-Production Tools (one day maybe)
● InterDigital / Technicolor has made a film grain plugin (for Nuke, etc), that has been
used for several movies (e.g. 2017 “Au Revoir Là-Haut” where the director wanted to
replicate the look of autochrome photographs).
● We may consider making a similar tool that would also output FGC SEI parameters
https://www.afcinema.com/Au-revoir-la-haut.html

MHV’24
Slide 98
MPEG activity
● Production of a technical report (ISO/IEC TR 23002-9), published in July 2024.
ITU twin text expected soon (ITU-T H Suppl. 2?).
○ General introduction, use cases, metadata formats, grain synthesis, grain analysis,
implementations & tools
○ Edition 2 just started.
● Software tools
○ Grain synthesis in VVC and HEVC reference software
■ Also VFGS standalone in dedicated repository, with GUI tool
○ Grain analysis in VVC and HEVC reference software
○ FGC SEI insertion in existing bitstream
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM
https://vcgit.hhi.fraunhofer.de/jvet-ahg-fgt
https://vcgit.hhi.fraunhofer.de/jvet/HM

MHV’24
Slide 99
MPEG activity
● Visual tests to assess:
○ Rate savings when replacing native grain encoding with synthesis (high quality,
just noticeable difference) [done]
○ Rate savings / quality improvement for a broader range of qualities [done]
○ Grain fidelity [todo]
○ Using a variety of video test content with different grain characteristics. Encoding
with HEVC and VVC.

MHV’24
Slide 100
MPEG activity - visual test material
JVET-AI0138, JVET-AI2022

MHV’24
Software tools & standardization
Slide 101
MPEG activity - visual test results
JVET-AI0337, JVET-AJ0374 - preliminary

MHV’24
● VFGS is a grain synthesis hardware model, aiming at low cost and good quality, compatible with both
FF and AR modes of FGC SEI and other metadata formats.
● A software version has been embedded in VVdeC and optimized for real-time
● VvdeCWebPlayer shows that the technology can be used today with VVC.
● Software support also exists in MPEG/JVET reference software, with further improvements expected
● Other experiments and future plans like NN-based grain analysis
● MPEG/JVET activity include
○ Technical report
○ Visual testing, proving strong benefits (ref SW encoding for HEVC and VVC, VFGS for grain
synthesis). Report expect soon.
Slide 102
Recap and Takeaway points

MHV’24
Future Scope
Slide 103
● Enhanced Metrics Development
○ Create perceptual metrics tailored to assess film grain fidelity.
○ Incorporate human perception elements for better quality evaluation.
● Wider Industry Adoption
○ Standardize VSEI usage across all VVC applications to avoid Quality of Experience (QoE)
inconsistencies.
○ Collaborate with streaming platforms, content creators, and CE manufacturers for seamless
integration.
● Exploration of HDR and High-Resolution Content
○ Adapt grain synthesis algorithms for HDR and 8K+ content.
○ Maintain artistic fidelity under demanding resolution and contrast scenarios.
● Future Research Directions
○ Develop machine learning approaches for automatic grain analysis and synthesis.

Thank you for your attention
— ▪ Vignesh V Menon (vignesh.menon@hhi.fraunhofer.de)
▪ Philippe de Lagrange (Philippe.DeLagrange@InterDigital.com)

Film Grain Coding for Versatile Video Coding Systems: Techniques, Challenges, and Applications

More Related Content

Similar to Film Grain Coding for Versatile Video Coding Systems: Techniques, Challenges, and Applications

More from Vignesh V Menon

Recently uploaded

Film Grain Coding for Versatile Video Coding Systems: Techniques, Challenges, and Applications