The next generation JPEG standards

The next generation JPEG standards
Touradj Ebrahimi
JPEG Convenor
21 June 2018 www.jpeg.org 1

What is JPEG?
www.jpeg.org 2
• Joint Photographic
Experts Group
– ISO/IEC
– ITU-T
• Informally known as
JPEG
– WG1 in official
communications
21 June 2018

JPEG Family of Standards
www.jpeg.org 321 June 2018

JPEG ecosystem revolutionized digital
photography
www.jpeg.org 4
Source: KPCB Internet Trends 2016 (June 2016).
1995-96 Technology and Engineering
Emmy award (together with MPEG-2)
21 June 2018

First JPEG standard 25th
anniversary
21 June 2018
Three celebrations
– Torino 76th WG1 meeting
– Macau 77th WG1 meeting
– La Jolla 79th WG1 meeting
www.jpeg.org 5

JPEG (ISO/IEC 10918)

JPEG Reference Software
21 June 2018
• Call for proposals issued in Torino 76th
WG1 meeting
• Responses received in Macau 77th WG1
meeting
• Working Draft issued in Rio 78th WG1
meeting
• CD issued in La Jolla 79th WG1 meeting
• Joint specification with ITU-T
www.jpeg.org 7

JPEG 2000
2015 Technology and Engineering
Emmy award (JPEG 2000 interoperability)

JPEG 2000 framework
www.jpeg.org 9
Part 1/13
Core Codec
Part 2
Extensions
Part 10
3D Extensions
Part 9
JPIP
Part 3
MJPEG 2000
Part 6
JPM
Image Codec
Tools
File Format
Part 8
JPSEC
Part 11
JPWL
Part 14
JPXML
E2E Toolset
Extra Functionality
Codec Tools
Part 4
Compliance Testing
Part 5
Reference Software
Part 12
ISO Base Media
21 June 2018

JPEG XR (ISO/IEC 29199)
Complexity
Performance
JPEG
JPEG 2000
JPEG XR

21 June 2018
Part Title WD PDAM DAM FDAM AMD
2-AMD1(2012) Media type specification - 15/02 15/06 16/02 17/10
2-AMD2 (2012) Additional colour signal type identifiers 18/02 18/04 18/07 - 19/01
2-AMD3 (2012) Support for JPEG XR coding in the ISO/IEC 23008-12 file
format
- 18/07 18/10 - 19/07
4-AMD1 Additional JPEG XR Conformance Test Streams - 13/01 14/04 - 14/10
5-AMD1 Extension of the reference software: Support for the
Boxed-based File format
- 14/07 14/10 15/02 15/06
Part Title WD CD/
PDTR/
PDAM
DIS/
DAM
FDIS/DTR/
FDAM
IS/
TR/
AMD
1 System architecture 07/11 09/10 - 10/03 11/07
2 Image coding specification - 07/11 08/07 09/01 09/06
3 Motion JPEG XR 08/11 09/01 09/10 10/03 10/10
4 Conformance testing 08/03 08/07 09/04 09/10 10/07
5 Reference software 08/03 08/07 09/04 09/10 10/07
www.jpeg.org 12

JPEG 2000 (ISO/IEC 15444)

High Throughput JPEG 2000 (ISO/IEC 15444-
15)

High Throughput JPEG 2000
21 June 2018
27-31 March 2017 Meeting 75. Draft CfP ready for publication.
17-21 July 2017 Meeting 76. Review feedback from Draft CfP and develop Final CfP for publication.
1 September 2017 Test Bench source code, Anchor Block Code Library and Reference Block Coder Library made available.
October 1 2017 Registration of intent to Meeting 77, and indicate their intent to provide a complete submission and
contribute to the standardization process.
23-27 October 2017 Meeting 77. Review of registrations of intents. Finalize set of Test Images.
January 2018 Meeting 78. Review proposed modifications to the Test Bench.
2 February 2018 Test Bench frozen
1 March 2018 Submission by proponents to Meeting 79.
April 2018 Meeting 79. Evaluation of submissions. Produce WD
October 2018 Meeting 81. CD
January 2019 Meeting 82. DIS
June 2019 Meeting 83. IS
www.jpeg.org 15

Block coder
under test
Measuring Throughput
Block Coder Library
JPEG 2000 Part 1
Reference
Implementation
Encoder
Test Image
Quantized
Sample
Set
Encoding
Function
Decoding
Function
Test Bench
Codestream
Output
Samples
Open source software running on
a reference Intel platform

Decoder Evaluations

Encoder Evaluations

Bitrate increase

JPEG vs JPEG 2000 vs JPEG XR

JPEG XT backward compatible
compression
www.jpeg.org 21
• A JPEG legacy compatible HDR image compression standard
21 June 2018

JPEG XT design principles
• Two-layer coding, with base layer a legacy JPEG coded LDR and enhancement
layer with additional features
– IDR and HDR coding
– Lossless coding
– Alpha channel coding
– …
• Enhancement layer uses as much as possible JPEG Legacy coding tools

JPEG XT profiles

JPEG XT (ISO/IEC 18477)

JPEG XT potential extensions
• Structured image editing
• Animated JPEG
• …

Compression game in the last 3
decades
Increase complexity
Get better compression

Result of this compression game …
Increasingly complex systems
…
But we seem to be happy about
it and continue on this trend
exclusively!
Henrique Edouardo Vittorio Carlos (265 pounds)

JPEG XS Light weight / Low Latency Image
Coding
• Transparent quality
• Low complexity
• Low latency
• Modest compression

JPEG XS Light weight / Low Latency
• Several applications identified:
– Broadcast applications
– Live-production
– Digital Cinema
– Industrial vision
– Professional imaging systems
– Consumer TV
– Mobile video
– Camera array based recordings
– Ultra high frame rate capture
– Medical Imaging
– Video Surveillance and security
– Wireless set-top boxes
– Low-cost visual sensors in Internet of Things (IoT)
– Self-driving cars
– Drones
– VR/AR/MR

JPEG XS (ISO/IEC 21122)

JPEG XS coding
21 June 2018
image
DC Offset, scaling,
clamping
Multiple Component
Transformation
Discrete Wavelet
Transform
Rate
prediction
& control
Quantization
Entropy coding
Significance
coding
Bitplane
count coding
Data coding Sign coding
Packing and codestream generation codestream
www.jpeg.org 33

Objective evaluations: single
generation
• JPEG XS outperforms VC-2 &
DSC
• JPEG XS outperforms JPEG
• J2K Broadcast significantly better
21 June 2018
• JPEG XS and J2K tile-based
allocation very close
• Compared to Oct16:
• Apr17: +0.45 dB
• Apr17_2V: +0.8 dB
www.jpeg.org 34

JPEG Systems (ISO/IEC 19566)

JPEG Systems – Stitched images for 360
Degree
EOI
SOI
APP1 (Exif)
JPEG-1
decoder
EOI
APP11
New Features
(eg. Privacy,
Residual,
Multi-Images)
JPEG codestream
with optional features
Image Data
XML
<?xml version="1.0"?>
JPEG new
feature
decoder
Metadata
Stitching parameters
Image 1
….
Image N

21 June 2018
SOI
APP1 (Exif)
EOI
APP11
(protected
metadata)
JPEG-1 decoder
JPEG Privacy &
Security
decoder
APP1 (Exif)
APP1 (Exif)
JPEG compatible
codestream with
data protection
Image data
APP11
(protected
image data)
Image Data
APP11
(protected
metadata)
Image data
APP11
(protected
image data)
APP3 (JPSearch)
APP3 (JPSearch)
APP3 (JPSearch)
JPEG Systems – Stitched images for Privacy &
Security
www.jpeg.org 37

JPEG Systems – JUMBF Structure
11023103813x1803
c10c120mu3920cz3
4zcn34tz30tzcn304z
t30vncz3409czn30cz
309z30zn1
e.g. encrypted for
privacy and security
e.g. unstitched image
parts for
JPEG 360
Type of Metadata or Application
XML Metadata

JPEG Systems
Part Title WD
PDTR/
CD
PDTR-
2/DIS
FDIS TR/IS
1 JPEG Systems – File format and File structure 14/07 15/02 15/06 - 15/10
2 JPEG Systems - Transport mechanisms and packaging 15/02 16/02 - 16/10
3 JPEG Systems - Feature list and Boxtype ID’s 15/06 16/06 cancelled - cancelled
4 JPEG Systems - Privacy, Security and IPR features 16/10 18/07 18/10 - 19/07
5 JPEG Systems - JPEG Universal Metadata Box Format
(JUMBF)
18/01 18/04 18/07 19/04
6 JPEG Systems - JPEG 360 18/04 18/04 18/07 - 19/04
www.jpeg.org 3921 June 2018 www.jpeg.org 39

Should pictures be flat?

3D again?
• Isn’t 3D dead?

3D has been a hit for a
while…but...
JPS
MPO

What happened to 3D?
• It has not delivered the quality of
experience users expect!

The real world around us is 3D!
“The most perfect photograph
currently shows only one
aspect of reality; it reduces to a
unique image fixed on a plane,
as a drawing or a painting
would be traced by hand.”
“Can we ask photography to
render all the richness that the
direct view of an object offers
?” Gabriel Lippmann (1845 – 1921)
Lippmann is remembered as the
inventor of a method for reproducing
colors by photography, based on the
interference phenomenon.
1908 Nobel Prize in Physics

Capturing reality

Rendering/synthetizing reality

JPEG PLENO
JPEG PLENO
targets a standard
framework for the
representation and
exchange of new imaging
modalities such as light-
field,
point-cloud and
holographic imaging.

Plenoptic representation of visual
information
• 7D function
– spatial position (x,y,z)
– viewing direction (q,f)
– wavelength (l)
– time (t)
Pf(x,y,z,q,f,l,t)

JPEG Pleno end-to-end processing
chain

JPEG Pleno work plan and schedule
www.jpeg.org 50
Meeting Status Date
74thJPEG meeting Final CfP for light field coding 17/01
76th JPEG meeting Evaluation of submissions 17/07
78th JPEG meeting WD1 18/01
79th JPEG meeting WD2 18/04
80thJPEG meeting Validations 18/07
81st JPEG meeting CD 18/10
83rd JPEG meeting DIS 19/04
85th JPEG meeting IS 19/10
21 June 2018

Light field capture

Light field display

JPEG Pleno (ISO/IEC 21794)

JPEG Pleno Design Principles
• A framework but not a super format
• One or limited number of representation models
• Where needed, backward compatible with legacy JPEG standards
• Investigate quality evaluation methodologies
Panorama
360/VR
Spatial photo
Point cloud
photo
Light field
photo
Holograhy
Depth photo

JPEG Pleno E2E Architecture
Coding Tools + System Layer

JPEG Pleno Light Fields Codec
architecture
Reference View
Decoder
Prediction
Residual
Encoding
Depth/Disparity
Encoding
Metadata
Signalling
Reference
Views
Intermediate
Views
Scene Analysis &
PartitioningLight Field
Reference View
Encoding
ENCODED
LIGHT FIELD
Codestream
Depth/Disparity
Decoder
Metadata Parser
Calculation of
Prediction
Coefficients
Calculation of
Prediction
Residuals
Calculation of
Predicted IV
Entropy Coding
of Prediction
Coefficients
Random access is an
important requirement!

Advanced Image Coding (AIC)
www.jpeg.org 57
• Advanced Image Coding
– Part 1: Guidelines for codec evaluation
– Part 2: Evaluation procedure for assessing visually lossless coding
– Part 2 AMD1: Evaluation of high dynamic range content
– Part 2 AMD2: Evaluation of image sequences
• Call for information issued in February 2015 to receive information on next generation
still image compression with superior compression efficiency, as well as other useful
features needed in future multimedia applications
• PCS 2015 Feature Event - Evaluation of current and future image compression
technologies
• ICIP 2016 Image Compression Grand Challenge - Evaluation of innovative ideas for
image compression technologies when compared to existing standards.
21 June 2018

PCS 2015 image compression grand
challenge
21 June 2018 58www.jpeg.org
• ACR-HR: Absolute Category Rating with Hidden Reference
• 5-level discrete scale from poor to bad to fair to good to excellent
• Randomization of presentation order
• Expert viewing methodology based on ITU-R BT.1663
• 6 codecs tested for their subjective quality
– 6(codecs)X6(images)X4(bit rates)+6(originals)=150 stimuli
• 27 experts selected from QoMEX2015 participants
– 3 sessions of 50 stimuli (circa 15 min per session)
– 9 scores per stimuli
• Short training for bad, fair and excellent quality illustrations
• Display: Apple MacBook Pro Retina 15in
Stimulus
1
Vote
1
Stimulus
50
Vote
50
time
Training

PCS 2015 image compression grand
challenge
59
21 June 2018
www.jpeg.org
woman bike

ICIP 2016 Image Compression Grand
Challenge (GC)• Contents: 7 (1 training + 6 test):
– Resolutions - 800x1152 or 800x1280 depending on content
– Subjective evaluations on cropped versions to fit display
– Objective metrics performed on the cropped versions
• Stimuli:
– Original images
– Compressed/decompressed images with 10 codecs
• JPEG (default)
• JPEG (PSNR)
• JPEG (visual)
• JPEG 2000 (PSNR)
• JPEG 2000 (visual)
• JPEG XR (444)
• JPEG XR (420)
• HEVC (SCC ext.)
• Daala
• WebP
• 8 bit rates for objective metrics:
– 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75 and 2 bpp
• 4 bit rates for subjective evaluations:
– 0.25, 0.5, 0.75 and 1 bpp or 0.75, 1, 1.25 and 1.5 bpp depending on content
training
60
bike cafe honolulu
p08 p26 woman

ICIP2016 GC subjective evaluation
methodology• Subjective evaluation methodology based on ITU-T P.910
• ACR-HR: Absolute Category Rating with Hidden Reference
• Randomization of presentation order
• 5-level discrete scale: bad, poor, fair, good, excellent
• 10 codecs tested for their subjective quality
– 10(codecs) x 6(images) x 4(bit rates) + 6(originals) = 246
stimuli
• 21 naïve subjects participated in VUB, UBI and EPFL labs
• Each subject completed 3 sessions of 80 stimuli (circa 15
min per session, 30 min break)
• Short training for bad, fair and excellent quality illustrations
• Display: Apple MacBook Pro Retina 15 inch or equivalent
• Typical office environment
61
Stimulus
1
Vote
1
Stimulus
80
Vote
80
time
Training
21 June 2018 www.jpeg.org

ICIP2016 GC objective metrics
• PSNR
– Widely used quality metric in image processing community.
– Performed for both Y channel and RGB.
• SSIM: Structural Similarity Index
– Mean of similarity between an image under test and its reference based on structural information.
• MSSIM: Multiscale Structural Similarity Index
– Multiscale version of SSIM.
• FSIM: Feature Similarity Index
– Based on SSIM.
– Adds a comparison of low-level feature sets between the reference and the distorted images.
– analyzes the high phase congruency extracting highly informative features and the gradient magnitude, to encode the
contrast information.
– This analysis is complementary and reflects different aspects of the HVS in assessing the local quality of an image.
– Performed for both Y and C channels.

ICIP2016 GC objective metrics
• HDR-VDP2.2: High Dynamic Range Visible Difference Predictor
– Calibrated metric developed for HDR images
– Considers a light-adaptive contrast sensitivity function, as the ranges of light adaptation can vary substantially.
– Includes a specific model of the point spread function (PSF) of the eye optics, as human optical lens flare can be very strong in high contrast
HDR content.
– The front-end amplitude non-linearity is based on integration of the Weber-Fechner law.
– Takes into account the angular resolution.
– Uses a multi-scale decomposition.
– A neural noise block is defined to calculate per-pixel probabilities maps of visibility and the predicted quality metric.
• CIEDE2000: Color difference metric
– Includes weighting factors for lightness, chroma, and hue (like the CIE1976 L*a*b* perceptual space).
– Also includes factors to handle the relationship between chroma and hue.
• VIF: Visual Information Fidelity
– Analyses the natural scene statistics.
– Uses an image degradation model and the HVS model.
– Based on the quantification of the Shannon information present in both the reference and the distorted images.

ICIP2016 GC: SSIM results

ICIP2016 GC subjective evaluation
results
65

The next generation JPEG standards

Image compression is hot...
CLIC
workshop

Standardize a new image coding format that:
● Offers state-of-the-art compression efficiency
● Offers support for low-end and high-end imaging applications
● Has the potential to replace JPEG (ISO/IEC 10918)
JPEG XL

● This wish has been fulfilled for previous JPEG standards
● Royalty-free (baseline) is highly encouraged to accommodate widespread
adoption of a new image format
Royalty-free wish

● Compression efficiency
○ Objective quality assessment: evaluation framework available
○ Subjective quality assessment (different test labs)
○ Anchors: JPEG, WebP, JPEG 2000, HEVC
● Test material
○ Different image classes
○ SDR and HDR/WCG
Call for Proposals: evaluation criteria

Call for Proposals: timeline
19/10
IS18/04
CfP
launched
19/04
DIS
19/01
CD
18/10
- Evaluation of responses
- First WD
18/09
Submission of
responses

• 80th ISO/IEC JTC1/SC29/WG1 (JPEG)
Meeting
o Location: Berlin, Germany
o July 7, 2018 - July 13, 2018
• 81st ISO/IEC JTC1/SC29/WG1 (JPEG)
Meeting
o Location: Vancouver, Canada
o October 13, 2018 - October 19, 2018
• 82nd ISO/IEC JTC1/SC29/WG1 (JPEG)
Meeting
o Location: Lisanon, Portugal
o January 19, 2019 - January 25, 2019
Next WG1 meetings

More information
Prof. Touradj Ebrahimi
JPEG Convenor
École Polytechnique Fédérale
de Lausanne (EPFL)
Touradj.Ebrahimi@epfl.ch
www.jpeg.org/contact.html

The next generation JPEG standards

The next generation JPEG standards

Recommended

More Related Content

What's hot

What's hot(20)

Similar to The next generation JPEG standards

Similar to The next generation JPEG standards(20)

More from Touradj Ebrahimi

More from Touradj Ebrahimi(20)

Recently uploaded

Recently uploaded(20)

The next generation JPEG standards

Editor's Notes