As presented by Reto Kromer and Jerôme Martinez at the 2019 FIAT/IFTA World Conference in Dubrovnik.

1. Reversibly Normalise Film Scans & Optimize Storage
The RAWcooked projectThe RAWcooked project
Data is never RAW.
Data is always cooked in a way or another.
Jérôme Martinez
FIAT/IFTA, October 2019

2. Open source software company focused on digital media
analysis. We work (different levels of involvement) on:
MediaInfo
Convenient unified display of the most relevant technical and tag data for video and audio
files
MediaConch
Implementation checker, policy checker, & reporter
QCTools
Helps users analyze and understand their digitized video files through use of audiovisual
analytics and filtering
BWF MetaEdit, AVI MetaEdit, MOV MetaEdit
Embedding, validating, and exporting of metadata
DV Analyzer
Checking presence of technical errors in DV captures
MediaArea

3. Raw A/V filesRaw A/V files
Huge size
(4K+ is there, can be 100 MB/frame, several TB per hour)
1 file per video frame (thousands of files in a directory)
Not playable as is by several players (VLC...)
So many DPX or TIFF format flavors
(interoperability issues)

4. FFV1FFV1
Lossless video compression format
Open source, patent free
Adopted by several archives
Being standardized (IETF)
Frames are divided by slices, with checksums

5. CompressionCompression
Example with 1 second at 24 fps 10-bit HD film on a 6-core
(12-thread) Skylake-X CPU:
24 DPX files (or in ZIP/TAR uncompressed): 189 MB
1 compressed ZIP file: 175 MB in 10 seconds
1 compressed LZMA2 file: 154 MB in 30 seconds
1 FFV1/MKV Intra 16-slice file: 105 MB in 1.5 seconds

6. Tests on SD 10-bit contentTests on SD 10-bit content
Sponsored by VIAA
On several TB of MXF/JP2k/PCM content
Different sources and kinds of material
E5-2698V3 (16 cores+HT, 2.3-3.6 GHz)

7. Tests on SD 10-bit contentTests on SD 10-bit content
Results vary depending on the content of files
From 0.4x to 2.4x (average 0.7x) real time/thread
(encoding/decoding)
From 0.7x to 16x (average 3.5x) the speed of JP2k
(FFmpeg)
Be er compression ratio by 2% compared to JP2k
(9% with additional FLAC audio compression)
h ps://viaa.be/files/a achments/.669/VIAA_Preservation_R

8. AdvantagesAdvantages
Open source and free (FFmpeg)
Package is 1.5-3x smaller than DPX/TIFF
Cheksum by "Cluster" (usually 1 second) at container
level
Cheksum by "Slice" (you choose how many per frame)
at video level
Files are natively playable by several players (VLC,
FFplay...)

9. Disadvantages of FFV1Disadvantages of FFV1
alonealone
You lose some metadata
(DPX/TIFF header: scan software, some colorimetry
info, film type, DPX time code, shu er angle, gamma...)
Complicated command line
But...

10. RAWcookedRAWcooked
Easy: just a short command line
"rawcooked YourDirectoryName"
Store DPX/TIFF headers/footers in a specific Matroska
a achment
Store other sidecar files as Matroksa a achments
Output is a single Matroska/FFV1/FLAC file
Encoding is reversible (bit-by-bit to original files)
"rawcooked YourMatroskaFileName.mkv"

11. Easy check of integrityEasy check of integrity
Check if the file is healthy
"rawcooked --check YourMatroskaFileName.mkv"
Check if DPX headers are conform to specs
"rawcooked --conch YourMatroskaFileName.mkv"
Add error correction codes while encoding the file
e.g. with overhead of 1.5%, you can lose 4 blocks every
252 blocks without losing any content
"rawcooked --ecc YourDirectoryName"
Fix the corrupted file
"rawcooked --fix YourMatroskaFileName.mkv"

12. Use caseUse case
Archive asks a digitilization to their supplier
Classic workflow with the scanner
+ "rawcooked --all YourDirectoryName"
Transport... (2x less file sizes, less costly)
Archive receives content & checks the integrity
(file health, DPX conformance...)
"rawcooked --check YourMatroskaFileName.mkv"
Archive can visually check the content with
e.g. VLC Media Player
Storage (cost divided by 2 due to compression)
Revert to exact original DPX if someone needs it
"rawcooked YourMatroskaFileName.mkv"

13. Supported input formatsSupported input formats
DPX/Raw: 8/10/12/16 bit, RGB/RGBA
TIFF/Raw: 16 bit, RGB
WAV/PCM: 16/24 bit, 1/2/6 channel, 44/48/96 kHz
AIFF/PCM: 16/24 bit, 1/2/6 channel, 44/48/96 kHz
Based on files from our sponsors
More formats or format flavors on request

14. Our sponsorsOur sponsors

15. Our sponsorsOur sponsors
AV Preservation by reto.ch (main sponsor)
National Audiovisual Centre Luxembourg (CNA)
National Library of Norway
Irish Film Institute (IFI)
Northwest University Library
National Library of Wales
Walter J. Brown Media Archives
The MediaPreserve
British Film Institute

16. Financial sustainabilityFinancial sustainability
Open source code provided without lock to sponsors
Deliveries on our website are with a lock
DPX 8/10 bit RGB & WAV 2ch 48kHz flavors are usable
by default
We provide a key for other format flavors and features
(temporary key possible)
1000 € for first flavor/feature
+ 500 € per additional flavor/feature
To be compared with storage cost saving
(storage cost divided by 2)

17. Potential improvementsPotential improvements
Speed improvement (GPU/SSE/AVX)?
Support of reels?
CFA/Bayer/RGGB support?
Graphical interface?
More input formats?

18. Stay in touchStay in touch
MediaArea: ,h ps://mediaarea.net @MediaArea_net
RAWcooked: h ps://MediaArea.net/RAWcooked
Jérôme Martinez: jerome@mediaarea.net
Slides: h ps://mediaarea.net/Events
License (except images): CC BY

19. Reto Kromer • AV Preservation by reto.ch
Champions of Value and Trust:
AV Archives in the all-media world
FIAT/IFTA, Dubrovnik, Croatia
22–25 October 2019
The missing piece of software

20. Reading
Reto Kromer: Matroska and FFV1: One File
Format for Film and Video Archiving?,
in «Journal of Film Preservation», n. 96 (April
2017), FIAF, Brussels, Belgium, p. 41–45
➔ https://retokromer.ch/publications/
JFP_96.html

21. Compression
• chroma subsampling
• uncompressed
• lossless compression
• lossy compression
• compressed-born

22. Chroma Subsampling
• 4:4:4
• 4:2:2
• 4:2:0 / 4:1:1

23. Uncompressed
+ data simpler to process
+ software runs faster
– bigger files
– slower writing, transmission and reading
Examples: TIFF, DPX, OpenEXR

24. Lossless Compression
+ smaller files
+ faster writing, transmission and reading
– data processing complexer
– software runs slower
Examples: JPEG 2000, FFV1

25. Lossy Compression
• optimised for postproduction
Examples: ProRes 422, DNxHD, ProRes 4444,
DNxHR
• optimised for access
Examples: H.264 (AVC), H.265 (HEVC),
AV1

26. Born-Compressed
• optimised for both image acquisition and
postproduction
Examples: CineForm RAW, ProRes RAW

27. Bayer

28. container:
• folder
• TAR
• ZIP
• MXF
• Matroska
codec:
• TIFF
• DPX
• JPEG 2000
• FFV1
• OpenEXR
• CineForm RAW
• ProRes RAW

29. avantages disavantages
TIFF
DPX
OpenEXR
data easier
to process bigger files
JPEG 2000
FFV1
smaller files data complexer
to process

30. RAWcooked
• encoding into Matroska (.mkv) using FFV1
video codec and FLAC audio codec
• all metadata preserved
• decoding with bit-by-bit reversibility
• possibility to embed sidecar files (e.g.
MD5, LUT, XML)
• compatibility with media players

31. AV Preservation by reto.ch
chemin du Suchet 5
1024 Écublens
Switzerland
Web: reto.ch
Twitter: @retoch
Email: info@reto.ch