Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The next generation JPEG standards

1,030 views

Published on

Slides of a talk I gave in June 2018 at Google, giving an overview of various JPEG standardisation activities in compression and a short introductory with past projects.

Published in: Technology
  • Tired of being scammed? Take advantage of a program that, actually makes you money!  https://tinyurl.com/y4urott2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Thanks. Great summary of the next standards and projects of the JPEG Group!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

The next generation JPEG standards

  1. 1. The next generation JPEG standards Touradj Ebrahimi JPEG Convenor 21 June 2018 www.jpeg.org 1
  2. 2. What is JPEG? www.jpeg.org 2 • Joint Photographic Experts Group – ISO/IEC – ITU-T • Informally known as JPEG – WG1 in official communications 21 June 2018
  3. 3. JPEG Family of Standards www.jpeg.org 321 June 2018
  4. 4. JPEG ecosystem revolutionized digital photography www.jpeg.org 4 Source: KPCB Internet Trends 2016 (June 2016). 1995-96 Technology and Engineering Emmy award (together with MPEG-2) 21 June 2018
  5. 5. First JPEG standard 25th anniversary 21 June 2018 Three celebrations – Torino 76th WG1 meeting – Macau 77th WG1 meeting – La Jolla 79th WG1 meeting www.jpeg.org 5
  6. 6. JPEG (ISO/IEC 10918) 21 June 2018 www.jpeg.org 6
  7. 7. JPEG Reference Software 21 June 2018 • Call for proposals issued in Torino 76th WG1 meeting • Responses received in Macau 77th WG1 meeting • Working Draft issued in Rio 78th WG1 meeting • CD issued in La Jolla 79th WG1 meeting • Joint specification with ITU-T www.jpeg.org 7
  8. 8. JPEG 2000 2015 Technology and Engineering Emmy award (JPEG 2000 interoperability) 21 June 2018 www.jpeg.org 8
  9. 9. JPEG 2000 framework www.jpeg.org 9 Part 1/13 Core Codec Part 2 Extensions Part 10 3D Extensions Part 9 JPIP Part 3 MJPEG 2000 Part 6 JPM Image Codec Tools File Format Part 8 JPSEC Part 11 JPWL Part 14 JPXML E2E Toolset Extra Functionality Codec Tools Part 4 Compliance Testing Part 5 Reference Software Part 12 ISO Base Media 21 June 2018
  10. 10. JPEG XR (ISO/IEC 29199) Complexity Performance JPEG JPEG 2000 JPEG XR 21 June 2018 www.jpeg.org 10
  11. 11. JPEG XR (ISO/IEC 29199) 21 June 2018 www.jpeg.org 11
  12. 12. JPEG XR (ISO/IEC 29199) 21 June 2018 Part Title WD PDAM DAM FDAM AMD 2-AMD1(2012) Media type specification - 15/02 15/06 16/02 17/10 2-AMD2 (2012) Additional colour signal type identifiers 18/02 18/04 18/07 - 19/01 2-AMD3 (2012) Support for JPEG XR coding in the ISO/IEC 23008-12 file format - 18/07 18/10 - 19/07 4-AMD1 Additional JPEG XR Conformance Test Streams - 13/01 14/04 - 14/10 5-AMD1 Extension of the reference software: Support for the Boxed-based File format - 14/07 14/10 15/02 15/06 Part Title WD CD/ PDTR/ PDAM DIS/ DAM FDIS/DTR/ FDAM IS/ TR/ AMD 1 System architecture 07/11 09/10 - 10/03 11/07 2 Image coding specification - 07/11 08/07 09/01 09/06 3 Motion JPEG XR 08/11 09/01 09/10 10/03 10/10 4 Conformance testing 08/03 08/07 09/04 09/10 10/07 5 Reference software 08/03 08/07 09/04 09/10 10/07 www.jpeg.org 12
  13. 13. JPEG 2000 (ISO/IEC 15444) 21 June 2018 www.jpeg.org 13
  14. 14. High Throughput JPEG 2000 (ISO/IEC 15444- 15) 21 June 2018 www.jpeg.org 14
  15. 15. High Throughput JPEG 2000 21 June 2018 27-31 March 2017 Meeting 75. Draft CfP ready for publication. 17-21 July 2017 Meeting 76. Review feedback from Draft CfP and develop Final CfP for publication. 1 September 2017 Test Bench source code, Anchor Block Code Library and Reference Block Coder Library made available. October 1 2017 Registration of intent to Meeting 77, and indicate their intent to provide a complete submission and contribute to the standardization process. 23-27 October 2017 Meeting 77. Review of registrations of intents. Finalize set of Test Images. January 2018 Meeting 78. Review proposed modifications to the Test Bench. 2 February 2018 Test Bench frozen 1 March 2018 Submission by proponents to Meeting 79. April 2018 Meeting 79. Evaluation of submissions. Produce WD October 2018 Meeting 81. CD January 2019 Meeting 82. DIS June 2019 Meeting 83. IS www.jpeg.org 15
  16. 16. Block coder under test Measuring Throughput Block Coder Library JPEG 2000 Part 1 Reference Implementation Encoder Test Image Quantized Sample Set Encoding Function Decoding Function Test Bench Codestream Output Samples Open source software running on a reference Intel platform 21 June 2018 www.jpeg.org 16
  17. 17. Decoder Evaluations 21 June 2018 www.jpeg.org 17
  18. 18. Encoder Evaluations 21 June 2018 www.jpeg.org 18
  19. 19. Bitrate increase 21 June 2018 www.jpeg.org 19
  20. 20. JPEG vs JPEG 2000 vs JPEG XR 21 June 2018 www.jpeg.org 20
  21. 21. JPEG XT backward compatible compression www.jpeg.org 21 • A JPEG legacy compatible HDR image compression standard 21 June 2018
  22. 22. JPEG XT design principles • Two-layer coding, with base layer a legacy JPEG coded LDR and enhancement layer with additional features – IDR and HDR coding – Lossless coding – Alpha channel coding – … • Enhancement layer uses as much as possible JPEG Legacy coding tools www.jpeg.org 2221 June 2018
  23. 23. JPEG XT profiles 21 June 2018 www.jpeg.org 23
  24. 24. JPEG XT (ISO/IEC 18477) 21 June 2018 www.jpeg.org 24
  25. 25. JPEG XT (ISO/IEC 18477) 21 June 2018 www.jpeg.org 25
  26. 26. JPEG XT potential extensions • Structured image editing • Animated JPEG • … www.jpeg.org 2621 June 2018
  27. 27. Compression game in the last 3 decades 21 June 2018 www.jpeg.org 27 Increase complexity Get better compression
  28. 28. Result of this compression game … 21 June 2018 www.jpeg.org 28 Increasingly complex systems … But we seem to be happy about it and continue on this trend exclusively! Henrique Edouardo Vittorio Carlos (265 pounds)
  29. 29. Result of this compression game … 21 June 2018 www.jpeg.org 29 Increasingly complex systems … But we seem to be happy about it and continue on this trend exclusively! Henrique Edouardo Vittorio Carlos (265 pounds)
  30. 30. JPEG XS Light weight / Low Latency Image Coding • Transparent quality • Low complexity • Low latency • Modest compression 21 June 2018 www.jpeg.org 30
  31. 31. JPEG XS Light weight / Low Latency • Several applications identified: – Broadcast applications – Live-production – Digital Cinema – Industrial vision – Professional imaging systems – Consumer TV – Mobile video – Camera array based recordings – Ultra high frame rate capture – Medical Imaging – Video Surveillance and security – Wireless set-top boxes – Low-cost visual sensors in Internet of Things (IoT) – Self-driving cars – Drones – VR/AR/MR 21 June 2018 www.jpeg.org 31
  32. 32. JPEG XS (ISO/IEC 21122) 21 June 2018 www.jpeg.org 32
  33. 33. JPEG XS coding 21 June 2018 image DC Offset, scaling, clamping Multiple Component Transformation Discrete Wavelet Transform Rate prediction & control Quantization Entropy coding Significance coding Bitplane count coding Data coding Sign coding Packing and codestream generation codestream www.jpeg.org 33
  34. 34. Objective evaluations: single generation • JPEG XS outperforms VC-2 & DSC • JPEG XS outperforms JPEG • J2K Broadcast significantly better 21 June 2018 • JPEG XS and J2K tile-based allocation very close • Compared to Oct16: • Apr17: +0.45 dB • Apr17_2V: +0.8 dB www.jpeg.org 34
  35. 35. JPEG Systems (ISO/IEC 19566) 21 June 2018 www.jpeg.org 35
  36. 36. JPEG Systems – Stitched images for 360 Degree EOI SOI APP1 (Exif) JPEG-1 decoder EOI APP11 New Features (eg. Privacy, Residual, Multi-Images) JPEG codestream with optional features Image Data XML <?xml version="1.0"?> JPEG new feature decoder Metadata Stitching parameters Image 1 …. Image N 21 June 2018 www.jpeg.org 36
  37. 37. 21 June 2018 SOI APP1 (Exif) EOI APP11 (protected metadata) JPEG-1 decoder JPEG Privacy & Security decoder APP1 (Exif) APP1 (Exif) JPEG compatible codestream with data protection Image data APP11 (protected image data) Image Data APP11 (protected metadata) Image data APP11 (protected image data) APP3 (JPSearch) APP3 (JPSearch) APP3 (JPSearch) JPEG Systems – Stitched images for Privacy & Security www.jpeg.org 37
  38. 38. JPEG Systems – JUMBF Structure 11023103813x1803 c10c120mu3920cz3 4zcn34tz30tzcn304z t30vncz3409czn30cz 309z30zn1 e.g. encrypted for privacy and security e.g. unstitched image parts for JPEG 360 Type of Metadata or Application XML Metadata 21 June 2018 www.jpeg.org 38
  39. 39. JPEG Systems Part Title WD PDTR/ CD PDTR- 2/DIS FDIS TR/IS 1 JPEG Systems – File format and File structure 14/07 15/02 15/06 - 15/10 2 JPEG Systems - Transport mechanisms and packaging 15/02 16/02 - 16/10 3 JPEG Systems - Feature list and Boxtype ID’s 15/06 16/06 cancelled - cancelled 4 JPEG Systems - Privacy, Security and IPR features 16/10 18/07 18/10 - 19/07 5 JPEG Systems - JPEG Universal Metadata Box Format (JUMBF) 18/01 18/04 18/07 19/04 6 JPEG Systems - JPEG 360 18/04 18/04 18/07 - 19/04 www.jpeg.org 3921 June 2018 www.jpeg.org 39
  40. 40. Should pictures be flat? 21 June 2018 www.jpeg.org 40
  41. 41. 3D again? • Isn’t 3D dead? 21 June 2018 www.jpeg.org 41
  42. 42. 3D has been a hit for a while…but... 21 June 2018 www.jpeg.org 42 JPS MPO
  43. 43. What happened to 3D? • It has not delivered the quality of experience users expect! 21 June 2018 www.jpeg.org 43
  44. 44. The real world around us is 3D! “The most perfect photograph currently shows only one aspect of reality; it reduces to a unique image fixed on a plane, as a drawing or a painting would be traced by hand.” “Can we ask photography to render all the richness that the direct view of an object offers ?” Gabriel Lippmann (1845 – 1921) Lippmann is remembered as the inventor of a method for reproducing colors by photography, based on the interference phenomenon. 1908 Nobel Prize in Physics 21 June 2018 www.jpeg.org 44
  45. 45. Capturing reality 21 June 2018 www.jpeg.org 45
  46. 46. Rendering/synthetizing reality 21 June 2018 www.jpeg.org 46
  47. 47. JPEG PLENO JPEG PLENO targets a standard framework for the representation and exchange of new imaging modalities such as light- field, point-cloud and holographic imaging. 21 June 2018 www.jpeg.org 47
  48. 48. Plenoptic representation of visual information • 7D function – spatial position (x,y,z) – viewing direction (q,f) – wavelength (l) – time (t) 21 June 2018 www.jpeg.org 48 Pf(x,y,z,q,f,l,t)
  49. 49. JPEG Pleno end-to-end processing chain www.jpeg.org 4921 June 2018
  50. 50. JPEG Pleno work plan and schedule www.jpeg.org 50 Meeting Status Date 74thJPEG meeting Final CfP for light field coding 17/01 76th JPEG meeting Evaluation of submissions 17/07 78th JPEG meeting WD1 18/01 79th JPEG meeting WD2 18/04 80thJPEG meeting Validations 18/07 81st JPEG meeting CD 18/10 83rd JPEG meeting DIS 19/04 85th JPEG meeting IS 19/10 21 June 2018
  51. 51. Light field capture 21 June 2018 www.jpeg.org 51
  52. 52. Light field display 21 June 2018 www.jpeg.org 52
  53. 53. JPEG Pleno (ISO/IEC 21794) 21 June 2018 www.jpeg.org 53
  54. 54. JPEG Pleno Design Principles • A framework but not a super format • One or limited number of representation models • Where needed, backward compatible with legacy JPEG standards • Investigate quality evaluation methodologies 21 June 2018 www.jpeg.org 54 Panorama 360/VR Spatial photo Point cloud photo Light field photo Holograhy Depth photo
  55. 55. JPEG Pleno E2E Architecture 21 June 2018 www.jpeg.org 55 Coding Tools + System Layer
  56. 56. JPEG Pleno Light Fields Codec architecture 21 June 2018 www.jpeg.org 56 Reference View Decoder Prediction Residual Encoding Depth/Disparity Encoding Metadata Signalling Reference Views Intermediate Views Scene Analysis & PartitioningLight Field Reference View Encoding ENCODED LIGHT FIELD Codestream Depth/Disparity Decoder Metadata Parser Calculation of Prediction Coefficients Calculation of Prediction Residuals Calculation of Predicted IV Entropy Coding of Prediction Coefficients Random access is an important requirement!
  57. 57. Advanced Image Coding (AIC) www.jpeg.org 57 • Advanced Image Coding – Part 1: Guidelines for codec evaluation – Part 2: Evaluation procedure for assessing visually lossless coding – Part 2 AMD1: Evaluation of high dynamic range content – Part 2 AMD2: Evaluation of image sequences • Call for information issued in February 2015 to receive information on next generation still image compression with superior compression efficiency, as well as other useful features needed in future multimedia applications • PCS 2015 Feature Event - Evaluation of current and future image compression technologies • ICIP 2016 Image Compression Grand Challenge - Evaluation of innovative ideas for image compression technologies when compared to existing standards. 21 June 2018
  58. 58. PCS 2015 image compression grand challenge 21 June 2018 58www.jpeg.org • ACR-HR: Absolute Category Rating with Hidden Reference • 5-level discrete scale from poor to bad to fair to good to excellent • Randomization of presentation order • Expert viewing methodology based on ITU-R BT.1663 • 6 codecs tested for their subjective quality – 6(codecs)X6(images)X4(bit rates)+6(originals)=150 stimuli • 27 experts selected from QoMEX2015 participants – 3 sessions of 50 stimuli (circa 15 min per session) – 9 scores per stimuli • Short training for bad, fair and excellent quality illustrations • Display: Apple MacBook Pro Retina 15in Stimulus 1 Vote 1 Stimulus 50 Vote 50 time Training
  59. 59. PCS 2015 image compression grand challenge 59 21 June 2018 www.jpeg.org woman bike
  60. 60. ICIP 2016 Image Compression Grand Challenge (GC)• Contents: 7 (1 training + 6 test): – Resolutions - 800x1152 or 800x1280 depending on content – Subjective evaluations on cropped versions to fit display – Objective metrics performed on the cropped versions • Stimuli: – Original images – Compressed/decompressed images with 10 codecs • JPEG (default) • JPEG (PSNR) • JPEG (visual) • JPEG 2000 (PSNR) • JPEG 2000 (visual) • JPEG XR (444) • JPEG XR (420) • HEVC (SCC ext.) • Daala • WebP • 8 bit rates for objective metrics: – 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75 and 2 bpp • 4 bit rates for subjective evaluations: – 0.25, 0.5, 0.75 and 1 bpp or 0.75, 1, 1.25 and 1.5 bpp depending on content training 60 bike cafe honolulu p08 p26 woman 21 June 2018 www.jpeg.org 60
  61. 61. ICIP2016 GC subjective evaluation methodology• Subjective evaluation methodology based on ITU-T P.910 • ACR-HR: Absolute Category Rating with Hidden Reference • Randomization of presentation order • 5-level discrete scale: bad, poor, fair, good, excellent • 10 codecs tested for their subjective quality – 10(codecs) x 6(images) x 4(bit rates) + 6(originals) = 246 stimuli • 21 naïve subjects participated in VUB, UBI and EPFL labs • Each subject completed 3 sessions of 80 stimuli (circa 15 min per session, 30 min break) • Short training for bad, fair and excellent quality illustrations • Display: Apple MacBook Pro Retina 15 inch or equivalent • Typical office environment 61 Stimulus 1 Vote 1 Stimulus 80 Vote 80 time Training 21 June 2018 www.jpeg.org
  62. 62. ICIP2016 GC objective metrics • PSNR – Widely used quality metric in image processing community. – Performed for both Y channel and RGB. • SSIM: Structural Similarity Index – Mean of similarity between an image under test and its reference based on structural information. • MSSIM: Multiscale Structural Similarity Index – Multiscale version of SSIM. • FSIM: Feature Similarity Index – Based on SSIM. – Adds a comparison of low-level feature sets between the reference and the distorted images. – analyzes the high phase congruency extracting highly informative features and the gradient magnitude, to encode the contrast information. – This analysis is complementary and reflects different aspects of the HVS in assessing the local quality of an image. – Performed for both Y and C channels. 21 June 2018 www.jpeg.org 62
  63. 63. ICIP2016 GC objective metrics • HDR-VDP2.2: High Dynamic Range Visible Difference Predictor – Calibrated metric developed for HDR images – Considers a light-adaptive contrast sensitivity function, as the ranges of light adaptation can vary substantially. – Includes a specific model of the point spread function (PSF) of the eye optics, as human optical lens flare can be very strong in high contrast HDR content. – The front-end amplitude non-linearity is based on integration of the Weber-Fechner law. – Takes into account the angular resolution. – Uses a multi-scale decomposition. – A neural noise block is defined to calculate per-pixel probabilities maps of visibility and the predicted quality metric. • CIEDE2000: Color difference metric – Includes weighting factors for lightness, chroma, and hue (like the CIE1976 L*a*b* perceptual space). – Also includes factors to handle the relationship between chroma and hue. • VIF: Visual Information Fidelity – Analyses the natural scene statistics. – Uses an image degradation model and the HVS model. – Based on the quantification of the Shannon information present in both the reference and the distorted images. 21 June 2018 www.jpeg.org 63
  64. 64. ICIP2016 GC: SSIM results 6421 June 2018 www.jpeg.org
  65. 65. ICIP2016 GC subjective evaluation results 21 June 2018 www.jpeg.org 65
  66. 66. Image compression is hot... CLIC workshop 17 June 2018 www.jpeg.org
  67. 67. Standardize a new image coding format that: ● Offers state-of-the-art compression efficiency ● Offers support for low-end and high-end imaging applications ● Has the potential to replace JPEG (ISO/IEC 10918) JPEG XL 17 June 2018 www.jpeg.org
  68. 68. ● This wish has been fulfilled for previous JPEG standards ● Royalty-free (baseline) is highly encouraged to accommodate widespread adoption of a new image format Royalty-free wish 17 June 2018 www.jpeg.org
  69. 69. ● Compression efficiency ○ Objective quality assessment: evaluation framework available ○ Subjective quality assessment (different test labs) ○ Anchors: JPEG, WebP, JPEG 2000, HEVC ● Test material ○ Different image classes ○ SDR and HDR/WCG Call for Proposals: evaluation criteria 17 June 2018 www.jpeg.org
  70. 70. Call for Proposals: timeline 19/10 IS18/04 CfP launched 19/04 DIS 19/01 CD 18/10 - Evaluation of responses - First WD 18/09 Submission of responses 17 June 2018 www.jpeg.org
  71. 71. Better together
  72. 72. • 80th ISO/IEC JTC1/SC29/WG1 (JPEG) Meeting o Location: Berlin, Germany o July 7, 2018 - July 13, 2018 • 81st ISO/IEC JTC1/SC29/WG1 (JPEG) Meeting o Location: Vancouver, Canada o October 13, 2018 - October 19, 2018 • 82nd ISO/IEC JTC1/SC29/WG1 (JPEG) Meeting o Location: Lisanon, Portugal o January 19, 2019 - January 25, 2019 Next WG1 meetings
  73. 73. More information Prof. Touradj Ebrahimi JPEG Convenor École Polytechnique Fédérale de Lausanne (EPFL) Touradj.Ebrahimi@epfl.ch www.jpeg.org/contact.html 21 June 2018 www.jpeg.org 74

×