What’s new in MPEG?

1. What’s new in MPEG? Webinar | July 21, 2020 | 10:00 UTC and 21:00 UTC Jörn Ostermann MPEG Convenor Versatile Video Coding Video-based Point Cloud Compression MPEG 3DAudio MPEG Roadmap Carriage of VVC and EVC MPEG Immersive Video Further Information: https://bit.ly/mpeg131 Bart Kroon MPEG Video Marius Preda MPEG 3DG Young-Kwon Lim MPEG Systems Gary Sullivan JVET Jens-Rainer Ohm JVET Schuyler Quackenbush MPEG Audio MPEG Web Site: https://mpeg-standards.com/meetings/mpeg-131/

2. Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 Finalization of Versatile Video Coding Webinar, 21 July 2020 Gary Sullivan and Jens-Rainer Ohm JVET Co-chairs

3. Documents approved in recent meeting • Versatile Video Coding (JVET-S2001) – Twin text: ITU-T H.266 | ISO/IEC 23090-3 – Description of bitstream syntax and semantics, processes for core decoding and high-level syntax as necessary for decoding • Versatile SEI messages for coded video bitstreams (JVET-S2007) – Twin text: ITU-T H.274 | ISO/IEC 23002-7 – Independent SEI messages and VUI, specification not needed for core decoding process, could be used with VVC or other video standards • Test Model 10 of Versatile Video Coding (VTM 10) (JVET-S2002) – Encoder and algorithm description – Has corresponding software implementation • Draft 4 of VVC conformance testing (JVET-S2008) • VVC verification test plan (v3) (JVET-S2009)

4. VTM9 compared to HEVC-HM, "common test conditions" (CTC) Random Access is most important in storage, streaming, broadcast • UHD average >40% (PSNR) – both luma and chroma • Reasonable complexity tradeoff Random Access Over HM-16.20 Y U V EncT DecT Class A1 −38.74% −37.19% −44.34% 884% 186% Class A2 −43.13% −39.74% −38.35% 999% 199% Class B −34.74% −46.77% −44.61% 935% 189% Class C −29.90% −30.58% −32.56% 1212% 199% Class E Overall −35.93% −39.13% −40.09% 1004% 193% Class D −27.64% −26.48% −26.11% 1326% 194% Class F −41.55% −44.78% −46.09% 689% 163% Performance of VVC (PSNR)

5. Visual Subjective Performance of VVC • Test with non-expert viewers, sequences not included in CTC (from preparation of verification test) • Notable: Visual results seem to be better for VVC than when measured by PSNR (from JVET-S0246)

6. Versatility of VVC Video Applications • Designed for a wide variety of types of video • Camera captured, computer-generated, and mixed content – Screen sharing – Adaptive streaming – Game streaming – Video with scrolling text, etc. • Standard and high dynamic range (emphasis on 10 bit video) • Various colour formats, including 4:4:4 and wide gamut • 360° video with various projection map types • Multiview video (including depth maps) • MPEG’s video-based point cloud compression • Lossless coding support

7. Special Features with High-Level Syntax • Flexible access mechanisms, including localized access using “subpictures” • Extraction and merging at bitstream level • Special boundary handling for gradual refresh and 360° video • Layered coding, including low-complex scalability operation • Nested temporal sublayering • Predictive reference picture resampling • Wavefront parallel processing similar to HEVC, with less CTU row delay • General constraints information: Mechanism to identify tool usage at high level

8. Overview of coding tools • Partititioning: Multi-type tree (Quad/binary/ternary) • Intra prediction using – more directional modes (incl. wide angles), DC and planar – sample smoothing with various adaptation methods (position dependent) – inheritance of chroma modes and chroma sample prediction from luma – multi-line prediction, matrix weighted prediction • Inter prediction using advanced MV coding, affine models, sub-block and geometric/diagonal partitioning, decoder side motion refinement (three tools named DMVR, BDOF, PROF) • Combined intra/inter prediction • Switchable primary and secondary transforms • New adaptive loop filter based on local classification, in-loop amplitude mapping stage, additional elements in deblocking • Quantization with log step size switching (& trellis-based dependent quantization) • Context-adaptive arithmetic coding with various improvements • Support for screen content (intra block copy, palette mode, transform skip) and lossless and near-lossless coding

9. • Document archives (publicly accessible, >10k docs) – http://phenix.int-evry.fr/jvet – http://wftp3.itu.int/av-arch/jvet-site – http://phenix.int-evry.fr/jct – http://wftp3.itu.int/av-arch/jctvc-site • Software for VVC-VTM, HEVC-HM, and 360° Video (publicly accessible): – https://jvet.hhi.fraunhofer.de/ – https://hevc.hhi.fraunhofer.de/ – https://jvet.hhi.fraunhofer.de/svn/svn_360Lib/ Obtain documents and software

10. ARL audio research labs MPEG-H 3D Audio Baseline Profile Schuyler Quackenbush MPEG Audio Chair 1

11. ARL audio research labs MPEG-H 3D Audio - Introduction • MPEG-H 3D Audio standard was finalized in 2015, specifying the Low Complexity Profile • The Low Complexity Profile enables delivery of: – Channels and Objects – Higher-Order Ambisonics (HOA). • Audio Objects are a key component in enabling advanced personalization options in broadcast applications – Dialog enhancement – Language selection 2

12. ARL audio research labs MPEG-H 3D Audio – New Profile • In July 2019, industry requested a new profile dedicated to broadcast, streaming and streaming immersive music applications. • In July 2020, WG11 (MPEG) announces the completion of Amendment 2 on 3D Audio which specifies the new Baseline Profile addressing this industry request. 3

13. ARL audio research labs MPEG-H 3D Audio Baseline Profile • Tailored for broadcast, streaming, and high-quality immersive music delivery, the Baseline profile: 4 Baseline Profile Advanced Coding: Channels and Objects Loudness Control and DRC Rendering and Downimx Rich Metadata Set Personalization and Interactivity Accessibility and Dialog Enhancement Seamless Configuration Changes Sample Accurate Ad-insertion and Splicing … Low Complexity Profile HOA LPD DRC – Dynamic Range Control LPD – Linear Prediction Domain – Supports Channels and Objects. – Is a subset of the Low Complexity profile. – Supports all advanced broadcast and streaming features

14. ARL audio research labs MPEG-H 3D Audio Baseline Profile • In addition, the Baseline Profile: – Enables the use of up to 24 audio objects in Level 3 for high quality immersive music delivery. – Can be signaled in a backwards compatible fashion, such that Baseline Profile bitstreams will be decoded by all MPEG-H enabled devices that support either one of the two profiles 5

15. ARL audio research labs 3D Audio Baseline Profile Verification Test Report • Reports on the results of five subjective listening tests assessing the performance of the 3D Audio Baseline Profile. • Covers a wide range of bit rates and immersive audio use cases • The tests were conducted in nine different test sites: – Dolby, ETRI, Fraunhofer IIS, Gaudio, NHK, Nokia, Orange, Qualcomm and Sony • With a total of: – 341 listeners – 1,144,592 subjective scores 6 Public Document

16. ARL audio research labs 3D Audio Baseline Profile Verification Test Report • Three Tests achieve "Excellent" quality on the MUSHRA scale: – Test 2: 11.1 or 7.1 channels at 512 kb/s to 256 kb/s rate – Test 3: 7.1, 5.1 and 2.0 channels at 256 kb/s to 48 kb/s rate – Test 4: Content as Test 2, but binauralized for headphones at 384 kb/s • Two Tests achieve "ITU-R High-Quality Emission" quality – Test 1 "Ultra-HD Broadcast": 22.2 channels at 768 kbs – Test 5 "High-Quality Immersive Music Delivery": 24 audio objects coded at 1.5 Mb/s, presented as 11.1 (7.1 + 4H) loudspeakers 7

17. ARL audio research labs 360 Reality Audio Music Service • 360 Reality Audio music can be enjoyed by consumers using: – Tidal, – Deezer, – Nugs.net, – Amazon Music HD and – Sony Select (China). 8https://www.sony.com/electronics/360-reality-audio https://www.amazon.com/music/unlimited/why-hd?ref=dmm_LP_WHYHD

18. Point Cloud Compression in MPEG MPEG 131st Press Release, ISO/IEC FDIS 23090-5 Visual Volumetric Video-based Coding and Video-based Point Cloud Compression July 2020 Institut Polytechnique de Paris, FRANCE Marius PREDA MPEG 3D Graphics Chair

19. Point Cloud A set of 3D points • not ordered, • without relations between them Each point is defined by • (X, Y, Z) • (R, G, B) or (Y, U, V) • eflec ance, an a enc ,

20. Point Clouds

21. Sport viewing with point clouds 360° backgroun d 3D objects 1-3 Gbps per object

22. Point Cloud 800,000 points -> 1 000 Mbps (uncompressed) Compression is required in order to make PC useful

23. Very sparse occupancy of the 3D space - (usually) the objects are represented by their surface and not by volumes - In 2D a pixel has 8 neighbors, in 3D - 26 and many of them are transparent Point Cloud Compression basic principles

24. 2014 2015 2016 2017 2018 2019 2020 MPEG initiated the work on PCC G-PCC 10/2020 V-PCC 07/2020 First Committee Draft issued in October 2018 In April 2017 MPEG issued a Call for Proposals 9 technology leading companies responded and MPEG evaluated them in October 2017 0 5 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Patch generation Packing Geometry image generation Texture image generation Occupancy map compression Image padding Compressed bitstream Input point cloud frame Occupancy map Auxiliary patch-info compression Patchinfo Texture images Geometry images Padded geometry images Padded texture images Compressed geometry video Compressed Texture video multiplexer Compressed occupancy map Compressed auxiliary patch information Reconstructed geometry imagesSmoothing Video Compression Smoothed geometry V-PCC Video- based PCC G-PCC Geometry- based PCC Point Cloud Compression in MPEG

25. Video-based Point Cloud Compression Main ideas: (1) a point coordinate is encoded as a distance with respect to a particular plane inspired from he displacemen mapping in Graphics Pixel intensity Vertex Height

26. Video-based Point Cloud Compression Main ideas: (2) the color (or any attribute) associated to a 3D vertex is encoded in a 2D texture inspired from he e re mapping in Graphics Vertex color Pixel color

27. Video-based Point Cloud Compression Projecting all the points on a single plane would result to several 3D points having the same 2D projection - > several depth values should be stored per pixel

28. Video-based Point Cloud Compression Projecting per patch is preferred: - A set of points (patch) in a small neighborhood is projected on the same plane - The set of projection planes is very limited - 6 faces of the cube - 4 additional diagonal planes

29. Video-based Point Cloud Compression Encoding the 3D point clouds as a set of 2D patches Geometry Color (Attributes)

30. Video-based Point Cloud Compression Encoding the 3D point clouds as a set of 2D patches - For enforcing lossless, the missed points are encoded separately =

31. Video-based Point Cloud Compression Encoding the 3D point clouds as a set of 2D videos: depth, color and occupancy maps MPEG is very good in video coding! Problem solved

32. Video-based Point Cloud Compression Encoding 3D point clouds as a set of 2D videos: color, depth and occupancy map 100,000 points @ 30fps 360 Mbps (uncompressed) 1 Mbps (MPEG PCC 2020) 7 Mbps 4.4 Mbps

33. Video-based Point Cloud Compression V-PCC implementations publicly available Integrated real-time decoder and renderer source code is also available for Android, Windows & Linux www.mpeg-pcc.org

34. Video-based Point Cloud Compression Beyond V-PCC ISO/IEC FDIS 23090-5 Visual Volumetric Video-based Coding and Video- based Point Cloud Compression Visual Volumetric Video-based Coding is an MPEG framework for 3D to 2D projection based coding technologies - used by V-PCC - used by MIV (MPEG Immersive Video) - to be used for future projects (Dynamic Mesh Coding)

35. 30 organizations 90 authors MPEG PCC contributors

36. MPEG Immersive Video (MIV) ISO/IEC 23090-12 Bart Kroon bart.kroon@philips.com

37. MPEG Immersive Video (MIV) ISO/IEC 23090-12 • Schedule for MIV: • MPEG 131 – July 2020 – CD • MPEG 133 – Jan. 2021 – DIS • MPEG 135 – July 2021 – FDIS • Schedule for V3C and V-PCC (2nd ed.): • MPEG 132 – Oct. 2020 – CD • MPEG 133 – Jan. 2021 – DIS • MPEG 135 – July 2021 – FDIS • The MIV committee draft references FDIS 23090-5 (1st ed.) Video-based Visual Volumetric Coding (V3C) Video-based Point Cloud Coding (V-PCC) MPEG Immersive Video (MIV)

38. Example encoder source 15 views (photorealistic): • 4K × 2K • 360° equirectangular projection (ERP) • Geometry (=depth range) • Texture attribute (YCbCr)

39. Example encoder source 16 physical cameras: • 2K × 1K • Perspective projection • Geometry (=depth range) • Texture attribute (YCbCr) 210 mm 210 mm

40. MIV codec model Multi-view video • Geometry (G) • Texture attribute (T) • View parameters MIV encoder MIV decoder/ renderer V3C bitstream Reconstruction Viewing space Viewport video • (Geometry) • Texture attribute Original T G Atlas Complete (basic) view Patches from additional views

41. Bitstream structure V3C unit stream V3C parameter set V3C unit V3C unit V3C unit V3C unit Sub bitstream V3C unit Access unit Access unit Access unit… V3C unit Access unit Access unit Access unit… Sub bitstreams: • Common atlas data (has view parameters) • Multiple atlases: • Geometry video data • Attribute video data • Atlas data (has patch parameters)

42. Test model – Encoder Attribute video data Geometry video data Parameters Camera data Format Bitstream (V3C sample stream with MIV extensions) Source views View parameters Geometry video data Attribute video data Pack patches Into atlases Geometry video data (raw) Attribute video data (raw) Encode video sub bitstreams (HEVC) Atlas data Parameter set View parameters list Bitstream (one file) Multiplex Automatic parameter selection (geometry quality, basic/additional views, atlas frame sizes) SEI messagesSEI messages Prune views (Flag redundant pixels) (Simplified)

43. Test model – Decoder/renderer Filter out blocks Color code Core processes Filter viewport Decoded access unit (all conformance points) Patch culling Pruned view reconstruction View synthesis Inpainting Viewing space handling Viewport Viewport parameters Geometry upscaling (Simplified)

44. Discussion • Flexible standard for multiview video with depth: • Video codec agnostic (e.g. HEVC, VVC, …) • MIV Main uses a subset of V3C • Extensible with more V3C features (multiple attributes, occlusion video data, SEI messages, etc.) • MIV-specific extensions (coding per group of views, auxiliary patches, object-based coding, etc.) • Please participate: • Test model: https://gitlab.com/mpeg-i-visual/tmiv • Test material (14 sequences) available on request

45. Carriage of VV and EVC in MPEG Systems Youngkwon Lim Chair of MPEG Systems young.L@Samsung.com

46. 2 What is carriage? Video Coding Standard ISO/IEC 13818-1 MPEG-2 Systems Delivery over MPEG-2 TS ISO/IEC 14496-15 NAL File Format Storage and Delivery over ISOBMFF ISO/IEC 14496-12 ISO Base Media File Format ISO/IEC 23008-12 Image file format Storage and Delivery over ISOBMFF as a image or image sequence ISO/IEC 23000-19 Common Media Application Format Brands definition for CMAF Segments with a specific video codec ISO/IEC 23009-1 Media Presentation Description and Segment Formats MPEG-DASH extension for a specific video codec ISO/IEC 23000-22 Multi Image Application Format Brands definition for Image File Format with a specific video codec ISO/IEC 23008-1 MPEG Media Transport Delivery over MMT

47. 3 13818-1 MPEG-2 Systems ISO/IEC 13818-1:2019 AMD 2 Carriage of VVC in MPEG-2 TS • Current Stage : DAM • ETA for Final Stage : 2021/04 • Features • VVC data alignment with PES packets • VVC video descriptor and VVC HRD descriptor • Constraints on transport of VVC bitstream • T-STD extension for single layer VVC and layered temporal video subsets ISO/IEC 13818-1:2019 AMD 3 Carriage of EVC in MPEG-2 TS and update of the MPEG-H 3D Audio descriptor • Current Stage : CDAM • ETA for Final Stage : 2021/07 • Features • EVC data alignment with PES packets • descriptors carrying metadata for EVC elementary streams • constraints for the transport of EVC elementary streams • the T-STD buffer model for EVC elementary streams

48. 4 14496-15 NAL File Format • Current Stage : DAM • ETA for Final Stage : 2021/04 • VVC related features • definition of sample, sub-sample, sync sample, decoder configuration record and etc. • storage format for single-layer VVC (ISO/IEC 23090-3) video streams • storage of multiple layers in one track or each layer/sub-layer in its own track • storage format for VVC bitstreams with more than one layer. • EVC related features • definition of sample, sub-sample, sync sample, decoder configuration record and etc. • storage format for single-layer EVC video streams ISO/IEC 14496-15:2019 AMD 2 Carriage of VVC and EVC in ISOBMFF

49. 5 23008-12 Image file format • Current Stage : CDAM • ETA for Final Stage : 2021/07 • VVC related features • definition of image item, sub-sample item, VVC operating points information property, subpicture items and etc. • definition of VVC image sequences • definition of VVC-specific brands, vvic for image and image collections and vvcc for image sequence • EVC related features • definition of image item, sub-sample item and etc. • Definition of EVC-specific brands • evbi and evbs for EVC baseline profile image and image sequence, respectively • evmi and evms for EVC main profile image and image sequence, respectively ISO/IEC 23008-12:2017 AMD 3 Support for VVC, EVC, slideshows and other improvements

50. 6 23008-1 MPEG Media Transport • Current Stage : CDAM • ETA for Final Stage : 2021/07 • Features • Use of CMAF track constraints for MPU ISO/IEC 23008-1 3rd edition AMD 2 Carriage of EVC in MMT

51. Thank you! Questions?

53. 2020 2021 2022 2023 2024 Jan 2025Jan 2019 129 133 137 141 145 149130 131 132 134 135 136 138 139 140 142 143 144 146 147 148125 126 127 128MPEG meeting: VVC Extensions (Machine Learning) Media Coding MIV v.2 6 DoF Audio Scene Description v.2Scene Description VersatileVideo Coding MPEG ImmersiveVideo (MIV) Neural Network Compression for Multimedia EssentialVideo Coding (MPEG-5) Low Complexity EnhancementVideo PCC Systems Support Genome Annotation Compression Geometry PCC v.2Geometry Point Cloud Compression (G-PCC) CMAF v.2 Colour Support in Open Font Format Partial File Format v.2 Dynamic Mesh Compression OMAF v.3OMAF v.2 Video Decoding Interface Beyond Media Network-Based Media Processing NBMP v.2 VCM v.2Video Coding for Machines Systems andTools Genome Compression Genome Compression v.2 VisualVolumetricVideo-Based Coding (V3C) NNC for Multimedia v.2

What’s new in MPEG?

What’s new in MPEG?

Recommended

More Related Content

What's hot

What's hot(20)

Similar to What’s new in MPEG?

Similar to What’s new in MPEG?(20)

More from Alpen-Adria-Universität

More from Alpen-Adria-Universität(20)

Recently uploaded

Recently uploaded(20)

What’s new in MPEG?