SlideShare a Scribd company logo
Versatile Video Coding – Video Compression beyond
HEVC: Coding Tools for SDR and 360°Video
AAU Klagenfurt, May 14th, 2018
Mathias Wien
Institut für Nachrichtentechnik
RWTH Aachen University
wien@ient.rwth-aachen.de
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
2
• Introduction
 Standardization development and process
• Versatile Video Coding Development
 Joint Call for Proposals Outcome
• Coding Tools
 Versatile Video Coding Test Model
• Tools proposed by RWTH
 Geometric Partitioning
 360° Tools
• Summary and Outlook
 Next steps
Outline
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
3
INTRODUCTION
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
4
Video coding standardization organisations
• ISO/IEC MPEG = “Moving Picture Experts Group”
(ISO/IEC JTC 1/SC 29/WG 11 = International Standardization Organization and International Electrotechnical
Commission, Joint Technical Committee 1, Subcommittee 29, Working Group 11)
• ITU-T VCEG = “Video Coding Experts Group”
(ITU-T SG16/Q6 = International Telecommunications Union – Telecommunications Standardization Sector
[United Nations Organization, formerly CCITT], Study Group 16, Working Party 3, Question 6)
• JVT = “Joint Video Team” collaborative team of MPEG & VCEG, responsible for developing Advanced Video
Coding (AVC) (discontinued in 2009), documents and software publicly available
• JCT-VC = “Joint Collaborative Team on Video Coding” team of MPEG & VCEG , responsible for
developing High Efficiency Video Coding (HEVC) (established January 2010), documents and software
publicly available
• JVET = “Joint Video Exploration Team” exploring potential for new technology beyond HEVC (established
Oct. 2015) – renamed to “Joint Video Experts Team” responsible for developing Versatile Video Coding
(VVC) from April 2018, documents and software publicly available
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
5
History of international video coding standardization
H.263/+/++
(1995-2000+)
MPEG-4
Visual
(1998-2001+)
MPEG-1
(1993)
ISO/IECITU-T
H.120
(1984-1988)
H.261
(1990+)
H.262 / 13818-2
(1994/95-1998+)
H.264 / 14496-10
AVC
(2003-2008+)
H.265 / 23008-2
HEVC
(2013-2016+)
Videotelephony
Computer
SD HD 4K UHD
(Advanced Video Coding) (High Efficiency
Video Coding)
(MPEG-2)
H.26x / 23090-3
VVC
(2020-...)
8K, 360, ...
(Versatile Video Coding)
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
6
The scope of video standardization
• Only Specifications of the Bitstream, Syntax, and Decoder are standardized:
• Permits optimization beyond the obvious
• Permits complexity reduction for implementability
• Provides no guarantees of quality
Pre-Processing Encoding
Source
Destination
Post-Processing
& Error Recovery
Decoding
Scope of Standard
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
7
Hybrid Coding Concept
Basis of every standard since H.261
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
8
Performance history of standard generations
0 100 200 300
28
30
32
34
36
38
40
bit rate (kbit/s)
PSNR
(dB)
Foreman
10 Hz, QCIF
100 frames
HEVC
AVC H.262/MPEG-2 H.261H.263 +
MPEG-4 Visual
JPEG
35
Bit-rate Reduction: 50%
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
9
• Video is continually increasing by resolution
 HD existing, UHD (4Kx2K, 8Kx4K) appearing
 Mobile services going towards HD/UHD
 Stereo, multi-view, 360° video
• Devices available to record and display ultra-high resolutions
 Becoming affordable for home and mobile consumers
• Video has multiple dimensions to grow the data rate
 Frame resolution, Temporal resolution
 Color resolution, bit depth
 Multi-view
 Visible distortion still an issue with existing networks
• Necessary video data rate grows faster than feasible network transport capacities
 Better video compression (50% rate of current HEVC) needed, even after availability of 5G
Motivation for improved video compression
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
10
VVC DEVELOPMENT
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
11
• Exploration activity in the JVET
starting Oct/2015
 Investigation of tools integrated on top of HEVC:
Joint Exploration Model (JEM) software
 Larger block sizes / transforms, improved intra / inter
prediction tools, decoder-side derivation / refinement
methods, adaptive loop filters, …(*)
Objective gains >25% measurable on test set
 But: Evaluation software only, significant increase of
encoder run times
 Joint Call for Evidence (issued Mar/2017, evaluated Jul/2017):
Significant compression gains asserted
 Joint Call for Proposals (issued Oct/2017, evaluated Apr/2018):
Kick-off for Versatile Video Coding (VVC)
Steps towards next generation standard – Versatile Video Coding (VVC)
Figure from: JVET AHG report: Tool evaluation (AHG1) [JVET-H0001](*) Details on JEM coding tools in VCAS lecture
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
12
• VVC should be applicable for many types of data
 SDR and HDR up to extremly high resolutions
 All kind of camera generated content
 Computer generated content
 Non-camera video modalities e.g. medical data
 360°, lightfield, depth, and volumetric video
• VVC should support flexible random and localized access
 Low delay, random access, trick modes
 Error resilience, video buffer, system layer interface
 Possible support for scalability and multi-view
Steps towards next generation standard – Versatile Video Coding (VVC)
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
13
• Document JVET-H1002
• Test categories
 Standard dynamic range (SDR): 5 UHD and 5 HD sequences
 High dynamic range (HDR): 3 HLG and 5 PQ sequences
 360° video (360): 5 sequences in ERP format
• Constraint sets
 Constraint set 1 (C1): Random access configuration
 Max 1.1s random access intervals, structural delay max 16 pictures
 Constraint set 2 (C2): Low delay configuration only evaluated for SDR HD sequences
 No picture reordering between input and output
• Encoding constraints
 No pre-processing, post-processing only within the coding loop
 Static quantizer setting with one-time change to meet target bitrate
 Relevant optimization methods to be reported
Joint Call for Proposals (CfP) on Video Compression with Capability beyond HEVC
UHD = Ultra High Definition, HD = High Definition, HLG = Hybrid Log Gamma, PQ = Perceptive Quantization (ITU-T BT2020), ERP = Equirectangular Projection
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
14
• SDR-A: 3840×2160
• SDR-B: 1920×1080
• HDR (PQ HD, HLG 4K)
• 360 Video (8K, 6K)
VVC CfP Test Sequences
FoodMarket4 60p CatRobot1 60p DaylightRoad2 60p ParkRunning3 50p Campfire 30p
BasketballDrive 50p Cactus 50p BQTerrace 60p RitualDance 60p MarketPlace 60p
Market3 HD50p Hurdles HD50p Starting HD50p ShowGirls2 HD25p Cosmos1 HD24p
DayStreet 60p PeopleInShop... SunsetBeach 60p
ChairliftRide 30p KiteFlite 30p Harbor 30p Trolley 30p Balboa 60p
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
15
• Category-specific submissions (total 46):
 SDR: 22 submissions (8 of which are registered only in this category)
 HDR: 12 submissions
 360°: 12 submissions (2 of which are registered only in this category)
For all categories: HEVC anchors (HM) and JEM anchors
• Proposals described in input documents JVET-J0011...JVET-J0033
 Participation of 32 institutions
• Evaluation: Double stimulus test
 Rate points: lowest rate was typically less than "fair" quality for HEVC, but still possible to code
 Three ways of judging benefit:
 Mean MOS over all test cases (28x4 test points: 23x4 C1, 5x4 C2 )
 Count cases where a proposal was visually better/worse than JEM
 Count cases where a proposal was visually better than HEVC (HEVC at higher rate point)
• Reports: Input subjective test [JVET-J0080], output CfP results [JVET-J1003]
VVC CfP Responses
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
16
• Objective performance: best performers report
>40% bit rate reduction compared to HEVC,
>10% compared to JEM (for SDR case)
 2 proposals used some degree of subjective optimization
 1 proposal used large-segment multipass encoding
 Similar ranges for HDR and 360°
 Obviously, proposals with more elements show better performance
 Nevertheless, some proposals show similar performance as JEM with significant complexity/run time
reduction vs. JEM
• Subjective tests generally show similar (or even better) tendency
 Benefit over HEVC very clear
 Benefit over JEM visible at various points
Performance
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
17
• JVET-J1003:
Report of subjective
evaluation contains
28 plots as shown,
one per sequence
• Count significant
cases of positive/
negative benefit
with non-overlapping
confidence interval
against JEM
Performance
HM
JEM
Rate1...4
Proposals ranked by MOS (per rate point)
+1 credit
-1 credit
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
18
• "Mean" and "significance-count"
method suggested at least 7
proposals that were obviously
better than JEM
Performance SDR
Pxx 10
Pxx 8
Pxx 8
Pxx 6
Pxx 6
Pxx 6
Pxx 6
Pnn 3
Pnn 3
Pnn 2
Pnn 2
Pnn 1
Pnn 1
JEM 0
Pnn 0
Pnn -1
Pnn -1
Pnn -1
Pnn -2
Pnn -2
Pnn -2
Pnn -3
Pnn -4
HM -36
Pxx 6,53
Pxx 6,46
Pxx 6,41
Pxx 6,37
Pxx 6,33
Pxx 6,33
Pxx 6,26
Pnn 6,23
Pnn 6,17
Pnn 6,15
Pnn 6,13
Pnn 6,11
Pnn 6,04
Pnn 6,04
Pnn 6,03
Pnn 6,03
Pnn 6,01
JEM 6,01
Pnn 6,00
Pnn 5,96
Pnn 5,94
Pnn 5,88
Pnn 5,86
HM 4,57
Mean MOS Significance vs. JEM
60 ... +60
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
19
• Similar
tendency
in HDR
and 360°
categories
• Mostly same
coding tools
as in SDR
provide good
benefit
Performance HDR / 360°
Mean MOS Signif. vs. JEM
Pxx 6,04
Pxx 6,00
Pxx 5,94
Pxx 5,93
Pxx 5,86
Pnn 5,85
Pnn 5,80
Pnn 5,67
JEM 5,62
Pnn 5,60
Pnn 5,59
Pnn 5,45
Pnn 5,11
HM 4,14
Pxx 7
Pxx 3
Pxx 2
Pxx 2
Pxx 2
Pnn 1
Pnn 1
JEM 0
Pnn 0
Pnn 0
Pnn -1
Pnn -1
Pnn -6
HM -20
32 ... +32
Mean MOS Signif. vs. JEM
Pxx 6,20
Pxx 6,19
Pxx 6,06
Pxx 6,03
Pxx 5,99
Pxx 5,96
Pxx 5,86
Pnn 5,69
Pnn 5,67
Pnn 5,51
Pnn 5,45
JEM 5,11
HM 3,79
Pnn 3,45
Pxx 9
Pxx 9
Pxx 8
Pnn 7
Pxx 7
Pxx 6
Pxx 5
Pxx 4
Pnn 2
Pnn 1
Pnn 1
JEM 0
HM -9
Pnn -12
20 ... +20HDR 360°
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
20
• Comparison of proposals to HEVC at higher rate points
 Subjective quality of best performing proposals always equal or even better (about 1/3 of cases) than HEVC
at next higher rate point, over all categories (with approx. 40% less rate)
 Subjective quality of best performing proposals always equal or even better (about 1/5 of cases) than HEVC
at second next higher rate point, in SDR-UHD category (with approx. 65% less rate)
• Highest rate point HEVC may be close to transparent quality in many cases, difficult to become better
• Though not always the same proposal performing best at a given rate point, it can be anticipated that merits of
different proposals could be combined
 50% (or more) bit rate reduction with same quality will probably be achievable
Performance compared to HEVC
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
21
CODING TOOLS
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
22
• In terms of large architecture: Most proposals similar, no deviation from
hybrid coding mainstream
• Most improvements from further refinements of well-known building
blocks of HEVC and JEM
 Partititioning: Quad/binary, augmented by ternary tree and finer
 Intra prediction using
 directional modes, DC and planar
 sample smoothing with various adaptation
 inheritance of chroma modes and chroma sample prediction from luma
 Inter prediction: advanced motion vector prediction, affine models, sub-
block partitioning, switchable primary transforms, mostly DCT/DST variants
 Secondary transforms targeting specific cases of prediction residual
characteristics
 Adaptive loop filter based on local classification, some new variants
 Quantization / context-adaptive arithmetic coding
CfP analysis: What was proposed?
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
23
• New elements (some come with high complexity):
 Decoder side estimation for mode/MV derivation and sample prediction both in intra and inter coding (JEM)
 Finer partitioning: Asymmetric, geometric
 Neural networks for prediction, loop filtering, upsampling, (encoder control)
 Additional elements using template matching
 Intra block copy / current picture referencing
 Additional non-linear, de-noising and statistics-based loop filters
 Additional linear and non-linear elements in prediction
• HDR specific:
 New adaptive reshaping and quantization, also in-loop
 HDR-specific modifications of existing tools, e.g. deblocking
• 360-video specific:
 Variants of projection formats, geometry-corrected face boundary padding
 Modification and disabling of existing tools at face boundaries
CfP analysis: What was proposed?
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
24
• VVC Working Draft 1 / Test Model 1 (VTM1): basic approach
• VTM Block structure
 Unified tree (coding block unites prediction and transform)
 CTU size 128x128, rectangular blocks (dyadic sizes),
smallest luma size 4x4
 Maximum transform size 64x64
• VTM: Some removed elements of HEVC:
 Mode dependent transform (DST-VII), mode dependent scan
 Strong intra smoothing
 Sign data hiding in transform coding
 Unnecessary high-level syntax (e.g. VPS)
 Tiles and wavefront
 Quantization weighting
VVC Test Model and Benchmark Set
• Benchmark Set defined in addition to
VTM, including the following well-known
JEM tools:
• 65 intra prediction modes
• Coefficient coding
• AMT + 4x4 NSST
• Affine motion
• GALF
• Subblock merge candidate (ATMVP)
• Adaptive motion vector precision
• Decoder motion vector refinement
• LM Chroma mode
Purpose: testing benefit of technology
against better performing set
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
25
• Prediction block partitioning of a 2N×2N CB
• Transform block partitioning of a CB
 Quadtree partitioning of CB → Residual Quad Tree (RQT)
 Transform size 4×4 to 32×32
 TB size 4×4 to 64×64
 PB boundaries inside TBs allowed
HEVC: Prediction Blocks (PBs) and Transform Blocks (TBs)
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
27
• Simple ternary-tree split was used in several proposals, can be alternated
with binary split
• Further proposed variants of partitioning included
 Asymmetric binary split modes
 Diagonal and geometric (wedge-shaped) split modes
Block Partitioning: Quadtree – Ternary Tree – Binary Tree
Example:
(source: JVET-J1002)
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
28
TOOLS PROPOSED BY RWTH
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
29
• Motivation: Towards object-oriented coding
 Follow object boundaries more closely
 Less coding artifacts where it matters
• Prediction, transform and coding driven by actual object
shape under RD-constraint
 Inter- and intra-predicted segments for handling of
disocclusions
 Overlapped wedge based filtering at partition boundary
 Shape-adaptive DCT for spatially localized transform
coding
RWTH Proposal: Geometric Partitioning (GEO)
Source: M. Bläser, J. Sauer, and M. Wien, “Description of SDR and 360o video coding technology proposal
by RWTH Aachen University,” Doc. JVET-J0023, Joint Video Experts Team of ITU-T VCEG and ISO/IEC MPEG, San Diego, USA, 10th meeting, Apr. 2018
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
30
• GEO available for all block sizes ≥ 8×8 luma samples
• Partitioning is represented by two coordinate points 𝑃0 and 𝑃1 on the block boundary
• Prediction of two coordinate points 𝑃0 and 𝑃1 from 16 pre-defined templates (scaled for non-square blocks)
 Alternative: Spatial or temporal prediction
 Refinement: block size dependent offset
• Integration with AMVP, MERGE, FRUC
(no AFFINE (yet))
GEO: Partitioning Coding and Prediction
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
31
• No transform-tree in JEM 7.0  localization of residual error for larger blocks required
• ΔSA-DCT adapted from MPEG 4 software for blocks up to 128×128
• Currently floating point implementation – integer transform targeted
• SA-DCT signaled as additional transform choice next to full block DCT ( 4 total GEO transform modes)
• Coding of transform coefficients (TSBs, significance flags) with regard to shape
GEO: Shape-Adaptive DCT for Geometric Partitions
Segment with high prediction error
Segment with low prediction error
Example of 64×32 residual block
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
32
Results for GEO
JEM 7.0 JEM 7.0 + GEO
• Visual improvements at object boundaries
 Sharper contours
 Less staircase-effect
 More background details
• Objective gains (BD-rate savings)
 Against HEVC: ~33% on C1, ~25% on C2
 Against JEM: ~0.8% for both, C1 and C2
JEM 7.0
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
33
Results for GEO
JEM 7.0 JEM 7.0 + GEO
• Visual improvements at object boundaries
 Sharper contours
 Less staircase-effect
 More background details
• Objective gains (BD-rate savings)
 Against HEVC: ~33% on C1, ~25% on C2
 Against JEM: ~0.8% for both, C1 and C2
JEM 7.0 + GEO
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
34
• Motivation: Special characteristics of 360 content
 360° symmetry not exploited by current codecs
 Motion across face boundaries possible
 Geometric distortions
 Motion compensation suboptimal
 Not correctly treated by loop filters
 Here shown for cube, but similar problems for
other coding formats
• Proposal: Doing things correctly that “broke” for
360°content
 Face extension for motion estimation and
compensation
 Loop filtering over continuous boundaries
according to 3D arrangement
RWTH Proposal: 360°Coding Tools
Source: M. Bläser, J. Sauer, and M. Wien, “Description of SDR and 360o video coding technology proposal by RWTH Aachen University,”
Doc. JVET-J0023, Joint Video Experts Team of ITU-T VCEG and ISO/IEC MPEG, San Diego, USA, 10th meeting, Apr. 2018
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
35
360°coding tools - Face extension for cube projection (EAC/CMP/ACP)
𝑯 𝐵2𝐴 =
0 0 𝑓2
0 𝑓 0
−1 0 0
𝑓 =
face width
2
• Approach can be transferred to other coding formats
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
36
• Reference samples of blocks at face
boundaries changed
 Original: Samples from top or left block are
used
 Modified: Samples are chosen according to
3D cube geometry
• Approach can be transferred to other coding
formats
360°coding tools - Corrected deblocking filter (DBF)
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
37
• Objective gains (BD-rate savings)
 Against HEVC anchor: ~31%
E2E WS-PSNR
 Agains JEM (same projection
format): ~1.6%
 Gains higher for sequences with
high motion
Results for 360°coding tools
JEM deblocking Proposed deblocking
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
38
SUMMARY AND OUTLOOK
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
39
• Report of Results from the Call for Proposals on Video Compression with Capability beyond HEVC
[JVET-J1003]
 Documentation of results per sequence, marking HM and JEM anchors, not identifying individual proponents
 Assessment of qualitative (and as far as possible quantitative) benefit of submitted technology compared to
anchors
• Working Draft 1 of Versatile Video Coding [JVET-J1001]
 "Reduced" HEVC plus quad/binary/ternary tree structure
• Test Model 1 of Versatile Video Coding (VTM 1) [JVET-J1002]
 Corresponding encoder and algorithm description
Documents issued after CfP Results
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
40
• CE1: Partitioning
• CE2: In-loop filters
• CE3: Intra prediction and mode coding
• CE4: Inter prediction and MV coding
• CE5: Arithmetic coding engine
• CE6: Transforms and transform signalling
• CE7: Quantization and coefficient coding
• CE8: Current picture referencing
• CE9: Decoder side MV derivation
• CE10: Combined and multi-hypothesis prediction
• CE11: Composite reference pictures
• CE12: Mapping for HDR content
• CE13: Projection formats
Core Experiments defined by JVET
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
41
• Call for Proposals demonstrated availability of significant compression benefit
 HEVC out-performed by virtually all proposals
 Subjective results suggest initial rate savings of 40+% over HEVC at starting point
• Versatile Video Coding (VVC): First Working Draft and Test Model defined
 Reduced initial tool set
 Step-by-step integration of tools
 Evaluation of concurring variants of tools
 Consideration of algorithmic complexity
 Further fast progress expected, goal: finalization 2020
Summary and Outlook
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video |
Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018
42
• Document archives (publicly accessible)
 JVET / VVC:
 http://phenix.it-sudparis.eu/jvet
 http://ftp3.itu.ch/av-arch/jvet-site
 JCT-VC / HEVC:
 http://phenix.it-sudparis.eu/jct
 http://ftp3.itu.ch/av-arch/jctvc-site
• Software for HEVC, JEM, and 360 Video (publicly accessible):
 https://jvet.hhi.fraunhofer.de/svn/svn_VVCSoftware_VTM
 https://jvet.hhi.fraunhofer.de/svn/svn_VVCSoftware_BMS
 https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/
 https://jvet.hhi.fraunhofer.de/svn/svn_360Lib/
 https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/
Further Information

More Related Content

What's hot

Video Compression, Part 2-Section 1, Video Coding Concepts
Video Compression, Part 2-Section 1, Video Coding Concepts Video Compression, Part 2-Section 1, Video Coding Concepts
Video Compression, Part 2-Section 1, Video Coding Concepts
Dr. Mohieddin Moradi
 
H.264 vs HEVC
H.264 vs HEVCH.264 vs HEVC
H.264 vs HEVC
Marcin Walendowski
 
Video Compression, Part 3-Section 2, Some Standard Video Codecs
Video Compression, Part 3-Section 2, Some Standard Video CodecsVideo Compression, Part 3-Section 2, Some Standard Video Codecs
Video Compression, Part 3-Section 2, Some Standard Video Codecs
Dr. Mohieddin Moradi
 
Introduction to HEVC
Introduction to HEVCIntroduction to HEVC
Introduction to HEVC
Yoss Cohen
 
Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2
Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2
Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2
Dr. Mohieddin Moradi
 
HEVC VIDEO CODEC By Vinayagam Mariappan
HEVC VIDEO CODEC By Vinayagam MariappanHEVC VIDEO CODEC By Vinayagam Mariappan
HEVC VIDEO CODEC By Vinayagam Mariappan
Vinayagam Mariappan
 
H.264 video standard
H.264 video standardH.264 video standard
H.264 video standard
Sajan Sahu
 
HEVC intra coding
HEVC intra codingHEVC intra coding
HEVC intra coding
Manohar Kuse
 
Digital video
Digital videoDigital video
Digital video
Muhammad King
 
Introduction to Video Compression Techniques - Anurag Jain
Introduction to Video Compression Techniques - Anurag JainIntroduction to Video Compression Techniques - Anurag Jain
Introduction to Video Compression Techniques - Anurag Jain
Videoguy
 
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdfTutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
ssuserc5a4dd
 
Mp3
Mp3Mp3
H263.ppt
H263.pptH263.ppt
H263.ppt
Videoguy
 
Unit iv
Unit ivUnit iv
Unit iv
swapnasalil
 
A short history of video coding
A short history of video codingA short history of video coding
A short history of video coding
Iain Richardson
 
HDR and WCG Video Broadcasting Considerations
HDR and WCG Video Broadcasting ConsiderationsHDR and WCG Video Broadcasting Considerations
HDR and WCG Video Broadcasting Considerations
Dr. Mohieddin Moradi
 
Latest Technologies in Production & Broadcasting
Latest  Technologies in Production & BroadcastingLatest  Technologies in Production & Broadcasting
Latest Technologies in Production & Broadcasting
Dr. Mohieddin Moradi
 
High Efficiency Video Codec
High Efficiency Video CodecHigh Efficiency Video Codec
High Efficiency Video Codec
Tejus Adiga M
 
Video Compression Standards - History & Introduction
Video Compression Standards - History & IntroductionVideo Compression Standards - History & Introduction
Video Compression Standards - History & Introduction
Champ Yen
 
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
Mathias Wien
 

What's hot (20)

Video Compression, Part 2-Section 1, Video Coding Concepts
Video Compression, Part 2-Section 1, Video Coding Concepts Video Compression, Part 2-Section 1, Video Coding Concepts
Video Compression, Part 2-Section 1, Video Coding Concepts
 
H.264 vs HEVC
H.264 vs HEVCH.264 vs HEVC
H.264 vs HEVC
 
Video Compression, Part 3-Section 2, Some Standard Video Codecs
Video Compression, Part 3-Section 2, Some Standard Video CodecsVideo Compression, Part 3-Section 2, Some Standard Video Codecs
Video Compression, Part 3-Section 2, Some Standard Video Codecs
 
Introduction to HEVC
Introduction to HEVCIntroduction to HEVC
Introduction to HEVC
 
Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2
Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2
Serial Digital Interface (SDI), From SD-SDI to 24G-SDI, Part 2
 
HEVC VIDEO CODEC By Vinayagam Mariappan
HEVC VIDEO CODEC By Vinayagam MariappanHEVC VIDEO CODEC By Vinayagam Mariappan
HEVC VIDEO CODEC By Vinayagam Mariappan
 
H.264 video standard
H.264 video standardH.264 video standard
H.264 video standard
 
HEVC intra coding
HEVC intra codingHEVC intra coding
HEVC intra coding
 
Digital video
Digital videoDigital video
Digital video
 
Introduction to Video Compression Techniques - Anurag Jain
Introduction to Video Compression Techniques - Anurag JainIntroduction to Video Compression Techniques - Anurag Jain
Introduction to Video Compression Techniques - Anurag Jain
 
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdfTutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
 
Mp3
Mp3Mp3
Mp3
 
H263.ppt
H263.pptH263.ppt
H263.ppt
 
Unit iv
Unit ivUnit iv
Unit iv
 
A short history of video coding
A short history of video codingA short history of video coding
A short history of video coding
 
HDR and WCG Video Broadcasting Considerations
HDR and WCG Video Broadcasting ConsiderationsHDR and WCG Video Broadcasting Considerations
HDR and WCG Video Broadcasting Considerations
 
Latest Technologies in Production & Broadcasting
Latest  Technologies in Production & BroadcastingLatest  Technologies in Production & Broadcasting
Latest Technologies in Production & Broadcasting
 
High Efficiency Video Codec
High Efficiency Video CodecHigh Efficiency Video Codec
High Efficiency Video Codec
 
Video Compression Standards - History & Introduction
Video Compression Standards - History & IntroductionVideo Compression Standards - History & Introduction
Video Compression Standards - History & Introduction
 
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
 

Similar to Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360° Video

Versatile Video Coding: Compression Tools for UHD and 360° Video
Versatile Video Coding: Compression Tools for UHD and 360° VideoVersatile Video Coding: Compression Tools for UHD and 360° Video
Versatile Video Coding: Compression Tools for UHD and 360° Video
Mathias Wien
 
VVC tutorial at VCIP 2020 together with Benjamin Bross
VVC tutorial at VCIP 2020 together with Benjamin BrossVVC tutorial at VCIP 2020 together with Benjamin Bross
VVC tutorial at VCIP 2020 together with Benjamin Bross
Mathias Wien
 
Video Standards - Key Insights
Video Standards - Key InsightsVideo Standards - Key Insights
Video Standards - Key Insights
Chetan Manchanda
 
Trends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding StandardizationTrends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding Standardization
Mathias Wien
 
What’s new in MPEG?
What’s new in MPEG?What’s new in MPEG?
What’s new in MPEG?
Alpen-Adria-Universität
 
Ultra HD Forum NAB 2016 Press Conference
Ultra HD Forum NAB 2016 Press ConferenceUltra HD Forum NAB 2016 Press Conference
Ultra HD Forum NAB 2016 Press Conference
Benjamin SCHWARZ
 
09a video compstream_intro_trd_23-nov-2005v0_2
09a video compstream_intro_trd_23-nov-2005v0_209a video compstream_intro_trd_23-nov-2005v0_2
09a video compstream_intro_trd_23-nov-2005v0_2
Pptblog Pptblogcom
 
Standardisation In Media Formats
Standardisation In Media FormatsStandardisation In Media Formats
Standardisation In Media Formats
FITT
 
Dynamics of Video
Dynamics of Video Dynamics of Video
Dynamics of Video
IMTC
 
The H.264/AVC Advanced Video Coding Standard: Overview and ...
The H.264/AVC Advanced Video Coding Standard: Overview and ...The H.264/AVC Advanced Video Coding Standard: Overview and ...
The H.264/AVC Advanced Video Coding Standard: Overview and ...
Videoguy
 
Aquila Broadcast Premium Video Compression
Aquila Broadcast Premium Video CompressionAquila Broadcast Premium Video Compression
Aquila Broadcast Premium Video Compression
Champion InfoMetrics Pvt Ltd.
 
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global DeliveryMaking Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Akamai Developers & Admins
 
The H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC StandardThe H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC Standard
IMTC
 
R&S. UH. Practical aspects of the implementation
R&S. UH. Practical aspects of the implementationR&S. UH. Practical aspects of the implementation
R&S. UH. Practical aspects of the implementation
Sergii Pedorenko
 
MPEG Immersive Media
MPEG Immersive MediaMPEG Immersive Media
MPEG Immersive Media
ITU
 
1-MIV-tutorial-part-1.pdf
1-MIV-tutorial-part-1.pdf1-MIV-tutorial-part-1.pdf
1-MIV-tutorial-part-1.pdf
JunZhao68
 
Immersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to HolographyImmersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to Holography
Alpen-Adria-Universität
 
JVT-D023
JVT-D023JVT-D023
JVT-D023
Lowell Winger
 
MPEG4 codec for Access Grid
MPEG4 codec for Access GridMPEG4 codec for Access Grid
MPEG4 codec for Access Grid
Videoguy
 
MPEG4 codec for Access Grid
MPEG4 codec for Access GridMPEG4 codec for Access Grid
MPEG4 codec for Access Grid
Videoguy
 

Similar to Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360° Video (20)

Versatile Video Coding: Compression Tools for UHD and 360° Video
Versatile Video Coding: Compression Tools for UHD and 360° VideoVersatile Video Coding: Compression Tools for UHD and 360° Video
Versatile Video Coding: Compression Tools for UHD and 360° Video
 
VVC tutorial at VCIP 2020 together with Benjamin Bross
VVC tutorial at VCIP 2020 together with Benjamin BrossVVC tutorial at VCIP 2020 together with Benjamin Bross
VVC tutorial at VCIP 2020 together with Benjamin Bross
 
Video Standards - Key Insights
Video Standards - Key InsightsVideo Standards - Key Insights
Video Standards - Key Insights
 
Trends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding StandardizationTrends and Recent Developments in Video Coding Standardization
Trends and Recent Developments in Video Coding Standardization
 
What’s new in MPEG?
What’s new in MPEG?What’s new in MPEG?
What’s new in MPEG?
 
Ultra HD Forum NAB 2016 Press Conference
Ultra HD Forum NAB 2016 Press ConferenceUltra HD Forum NAB 2016 Press Conference
Ultra HD Forum NAB 2016 Press Conference
 
09a video compstream_intro_trd_23-nov-2005v0_2
09a video compstream_intro_trd_23-nov-2005v0_209a video compstream_intro_trd_23-nov-2005v0_2
09a video compstream_intro_trd_23-nov-2005v0_2
 
Standardisation In Media Formats
Standardisation In Media FormatsStandardisation In Media Formats
Standardisation In Media Formats
 
Dynamics of Video
Dynamics of Video Dynamics of Video
Dynamics of Video
 
The H.264/AVC Advanced Video Coding Standard: Overview and ...
The H.264/AVC Advanced Video Coding Standard: Overview and ...The H.264/AVC Advanced Video Coding Standard: Overview and ...
The H.264/AVC Advanced Video Coding Standard: Overview and ...
 
Aquila Broadcast Premium Video Compression
Aquila Broadcast Premium Video CompressionAquila Broadcast Premium Video Compression
Aquila Broadcast Premium Video Compression
 
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global DeliveryMaking Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
 
The H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC StandardThe H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC Standard
 
R&S. UH. Practical aspects of the implementation
R&S. UH. Practical aspects of the implementationR&S. UH. Practical aspects of the implementation
R&S. UH. Practical aspects of the implementation
 
MPEG Immersive Media
MPEG Immersive MediaMPEG Immersive Media
MPEG Immersive Media
 
1-MIV-tutorial-part-1.pdf
1-MIV-tutorial-part-1.pdf1-MIV-tutorial-part-1.pdf
1-MIV-tutorial-part-1.pdf
 
Immersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to HolographyImmersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to Holography
 
JVT-D023
JVT-D023JVT-D023
JVT-D023
 
MPEG4 codec for Access Grid
MPEG4 codec for Access GridMPEG4 codec for Access Grid
MPEG4 codec for Access Grid
 
MPEG4 codec for Access Grid
MPEG4 codec for Access GridMPEG4 codec for Access Grid
MPEG4 codec for Access Grid
 

More from Förderverein Technische Fakultät

Supervisory control of business processes
Supervisory control of business processesSupervisory control of business processes
Supervisory control of business processes
Förderverein Technische Fakultät
 
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
Förderverein Technische Fakultät
 
A Game of Chess is Like a Swordfight.pdf
A Game of Chess is Like a Swordfight.pdfA Game of Chess is Like a Swordfight.pdf
A Game of Chess is Like a Swordfight.pdf
Förderverein Technische Fakultät
 
From Mind to Meta.pdf
From Mind to Meta.pdfFrom Mind to Meta.pdf
From Mind to Meta.pdf
Förderverein Technische Fakultät
 
Miniatures Design for Tabletop Games.pdf
Miniatures Design for Tabletop Games.pdfMiniatures Design for Tabletop Games.pdf
Miniatures Design for Tabletop Games.pdf
Förderverein Technische Fakultät
 
Distributed Systems in the Post-Moore Era.pptx
Distributed Systems in the Post-Moore Era.pptxDistributed Systems in the Post-Moore Era.pptx
Distributed Systems in the Post-Moore Era.pptx
Förderverein Technische Fakultät
 
Don't Treat the Symptom, Find the Cause!.pptx
Don't Treat the Symptom, Find the Cause!.pptxDon't Treat the Symptom, Find the Cause!.pptx
Don't Treat the Symptom, Find the Cause!.pptx
Förderverein Technische Fakultät
 
Engineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdfEngineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdf
Förderverein Technische Fakultät
 
The Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdfThe Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdf
Förderverein Technische Fakultät
 
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Förderverein Technische Fakultät
 
Towards a data driven identification of teaching patterns.pdf
Towards a data driven identification of teaching patterns.pdfTowards a data driven identification of teaching patterns.pdf
Towards a data driven identification of teaching patterns.pdf
Förderverein Technische Fakultät
 
Förderverein Technische Fakultät.pptx
Förderverein Technische Fakultät.pptxFörderverein Technische Fakultät.pptx
Förderverein Technische Fakultät.pptx
Förderverein Technische Fakultät
 
The Computing Continuum.pdf
The Computing Continuum.pdfThe Computing Continuum.pdf
The Computing Continuum.pdf
Förderverein Technische Fakultät
 
East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...
Förderverein Technische Fakultät
 
Machine Learning in Finance via Randomization
Machine Learning in Finance via RandomizationMachine Learning in Finance via Randomization
Machine Learning in Finance via Randomization
Förderverein Technische Fakultät
 
IT does not stop
IT does not stopIT does not stop
Advances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksAdvances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial Networks
Förderverein Technische Fakultät
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
Förderverein Technische Fakultät
 
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdfIndustriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Förderverein Technische Fakultät
 
Introduction to 5G from radio perspective
Introduction to 5G from radio perspectiveIntroduction to 5G from radio perspective
Introduction to 5G from radio perspective
Förderverein Technische Fakultät
 

More from Förderverein Technische Fakultät (20)

Supervisory control of business processes
Supervisory control of business processesSupervisory control of business processes
Supervisory control of business processes
 
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
 
A Game of Chess is Like a Swordfight.pdf
A Game of Chess is Like a Swordfight.pdfA Game of Chess is Like a Swordfight.pdf
A Game of Chess is Like a Swordfight.pdf
 
From Mind to Meta.pdf
From Mind to Meta.pdfFrom Mind to Meta.pdf
From Mind to Meta.pdf
 
Miniatures Design for Tabletop Games.pdf
Miniatures Design for Tabletop Games.pdfMiniatures Design for Tabletop Games.pdf
Miniatures Design for Tabletop Games.pdf
 
Distributed Systems in the Post-Moore Era.pptx
Distributed Systems in the Post-Moore Era.pptxDistributed Systems in the Post-Moore Era.pptx
Distributed Systems in the Post-Moore Era.pptx
 
Don't Treat the Symptom, Find the Cause!.pptx
Don't Treat the Symptom, Find the Cause!.pptxDon't Treat the Symptom, Find the Cause!.pptx
Don't Treat the Symptom, Find the Cause!.pptx
 
Engineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdfEngineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdf
 
The Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdfThe Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdf
 
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
 
Towards a data driven identification of teaching patterns.pdf
Towards a data driven identification of teaching patterns.pdfTowards a data driven identification of teaching patterns.pdf
Towards a data driven identification of teaching patterns.pdf
 
Förderverein Technische Fakultät.pptx
Förderverein Technische Fakultät.pptxFörderverein Technische Fakultät.pptx
Förderverein Technische Fakultät.pptx
 
The Computing Continuum.pdf
The Computing Continuum.pdfThe Computing Continuum.pdf
The Computing Continuum.pdf
 
East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...
 
Machine Learning in Finance via Randomization
Machine Learning in Finance via RandomizationMachine Learning in Finance via Randomization
Machine Learning in Finance via Randomization
 
IT does not stop
IT does not stopIT does not stop
IT does not stop
 
Advances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksAdvances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial Networks
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdfIndustriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
 
Introduction to 5G from radio perspective
Introduction to 5G from radio perspectiveIntroduction to 5G from radio perspective
Introduction to 5G from radio perspective
 

Recently uploaded

Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
Aftab Hussain
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Zilliz
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 

Recently uploaded (20)

Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 

Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360° Video

  • 1. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video AAU Klagenfurt, May 14th, 2018 Mathias Wien Institut für Nachrichtentechnik RWTH Aachen University wien@ient.rwth-aachen.de
  • 2. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 2 • Introduction  Standardization development and process • Versatile Video Coding Development  Joint Call for Proposals Outcome • Coding Tools  Versatile Video Coding Test Model • Tools proposed by RWTH  Geometric Partitioning  360° Tools • Summary and Outlook  Next steps Outline
  • 3. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 3 INTRODUCTION
  • 4. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 4 Video coding standardization organisations • ISO/IEC MPEG = “Moving Picture Experts Group” (ISO/IEC JTC 1/SC 29/WG 11 = International Standardization Organization and International Electrotechnical Commission, Joint Technical Committee 1, Subcommittee 29, Working Group 11) • ITU-T VCEG = “Video Coding Experts Group” (ITU-T SG16/Q6 = International Telecommunications Union – Telecommunications Standardization Sector [United Nations Organization, formerly CCITT], Study Group 16, Working Party 3, Question 6) • JVT = “Joint Video Team” collaborative team of MPEG & VCEG, responsible for developing Advanced Video Coding (AVC) (discontinued in 2009), documents and software publicly available • JCT-VC = “Joint Collaborative Team on Video Coding” team of MPEG & VCEG , responsible for developing High Efficiency Video Coding (HEVC) (established January 2010), documents and software publicly available • JVET = “Joint Video Exploration Team” exploring potential for new technology beyond HEVC (established Oct. 2015) – renamed to “Joint Video Experts Team” responsible for developing Versatile Video Coding (VVC) from April 2018, documents and software publicly available
  • 5. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 5 History of international video coding standardization H.263/+/++ (1995-2000+) MPEG-4 Visual (1998-2001+) MPEG-1 (1993) ISO/IECITU-T H.120 (1984-1988) H.261 (1990+) H.262 / 13818-2 (1994/95-1998+) H.264 / 14496-10 AVC (2003-2008+) H.265 / 23008-2 HEVC (2013-2016+) Videotelephony Computer SD HD 4K UHD (Advanced Video Coding) (High Efficiency Video Coding) (MPEG-2) H.26x / 23090-3 VVC (2020-...) 8K, 360, ... (Versatile Video Coding)
  • 6. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 6 The scope of video standardization • Only Specifications of the Bitstream, Syntax, and Decoder are standardized: • Permits optimization beyond the obvious • Permits complexity reduction for implementability • Provides no guarantees of quality Pre-Processing Encoding Source Destination Post-Processing & Error Recovery Decoding Scope of Standard
  • 7. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 7 Hybrid Coding Concept Basis of every standard since H.261
  • 8. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 8 Performance history of standard generations 0 100 200 300 28 30 32 34 36 38 40 bit rate (kbit/s) PSNR (dB) Foreman 10 Hz, QCIF 100 frames HEVC AVC H.262/MPEG-2 H.261H.263 + MPEG-4 Visual JPEG 35 Bit-rate Reduction: 50%
  • 9. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 9 • Video is continually increasing by resolution  HD existing, UHD (4Kx2K, 8Kx4K) appearing  Mobile services going towards HD/UHD  Stereo, multi-view, 360° video • Devices available to record and display ultra-high resolutions  Becoming affordable for home and mobile consumers • Video has multiple dimensions to grow the data rate  Frame resolution, Temporal resolution  Color resolution, bit depth  Multi-view  Visible distortion still an issue with existing networks • Necessary video data rate grows faster than feasible network transport capacities  Better video compression (50% rate of current HEVC) needed, even after availability of 5G Motivation for improved video compression
  • 10. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 10 VVC DEVELOPMENT
  • 11. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 11 • Exploration activity in the JVET starting Oct/2015  Investigation of tools integrated on top of HEVC: Joint Exploration Model (JEM) software  Larger block sizes / transforms, improved intra / inter prediction tools, decoder-side derivation / refinement methods, adaptive loop filters, …(*) Objective gains >25% measurable on test set  But: Evaluation software only, significant increase of encoder run times  Joint Call for Evidence (issued Mar/2017, evaluated Jul/2017): Significant compression gains asserted  Joint Call for Proposals (issued Oct/2017, evaluated Apr/2018): Kick-off for Versatile Video Coding (VVC) Steps towards next generation standard – Versatile Video Coding (VVC) Figure from: JVET AHG report: Tool evaluation (AHG1) [JVET-H0001](*) Details on JEM coding tools in VCAS lecture
  • 12. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 12 • VVC should be applicable for many types of data  SDR and HDR up to extremly high resolutions  All kind of camera generated content  Computer generated content  Non-camera video modalities e.g. medical data  360°, lightfield, depth, and volumetric video • VVC should support flexible random and localized access  Low delay, random access, trick modes  Error resilience, video buffer, system layer interface  Possible support for scalability and multi-view Steps towards next generation standard – Versatile Video Coding (VVC)
  • 13. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 13 • Document JVET-H1002 • Test categories  Standard dynamic range (SDR): 5 UHD and 5 HD sequences  High dynamic range (HDR): 3 HLG and 5 PQ sequences  360° video (360): 5 sequences in ERP format • Constraint sets  Constraint set 1 (C1): Random access configuration  Max 1.1s random access intervals, structural delay max 16 pictures  Constraint set 2 (C2): Low delay configuration only evaluated for SDR HD sequences  No picture reordering between input and output • Encoding constraints  No pre-processing, post-processing only within the coding loop  Static quantizer setting with one-time change to meet target bitrate  Relevant optimization methods to be reported Joint Call for Proposals (CfP) on Video Compression with Capability beyond HEVC UHD = Ultra High Definition, HD = High Definition, HLG = Hybrid Log Gamma, PQ = Perceptive Quantization (ITU-T BT2020), ERP = Equirectangular Projection
  • 14. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 14 • SDR-A: 3840×2160 • SDR-B: 1920×1080 • HDR (PQ HD, HLG 4K) • 360 Video (8K, 6K) VVC CfP Test Sequences FoodMarket4 60p CatRobot1 60p DaylightRoad2 60p ParkRunning3 50p Campfire 30p BasketballDrive 50p Cactus 50p BQTerrace 60p RitualDance 60p MarketPlace 60p Market3 HD50p Hurdles HD50p Starting HD50p ShowGirls2 HD25p Cosmos1 HD24p DayStreet 60p PeopleInShop... SunsetBeach 60p ChairliftRide 30p KiteFlite 30p Harbor 30p Trolley 30p Balboa 60p
  • 15. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 15 • Category-specific submissions (total 46):  SDR: 22 submissions (8 of which are registered only in this category)  HDR: 12 submissions  360°: 12 submissions (2 of which are registered only in this category) For all categories: HEVC anchors (HM) and JEM anchors • Proposals described in input documents JVET-J0011...JVET-J0033  Participation of 32 institutions • Evaluation: Double stimulus test  Rate points: lowest rate was typically less than "fair" quality for HEVC, but still possible to code  Three ways of judging benefit:  Mean MOS over all test cases (28x4 test points: 23x4 C1, 5x4 C2 )  Count cases where a proposal was visually better/worse than JEM  Count cases where a proposal was visually better than HEVC (HEVC at higher rate point) • Reports: Input subjective test [JVET-J0080], output CfP results [JVET-J1003] VVC CfP Responses
  • 16. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 16 • Objective performance: best performers report >40% bit rate reduction compared to HEVC, >10% compared to JEM (for SDR case)  2 proposals used some degree of subjective optimization  1 proposal used large-segment multipass encoding  Similar ranges for HDR and 360°  Obviously, proposals with more elements show better performance  Nevertheless, some proposals show similar performance as JEM with significant complexity/run time reduction vs. JEM • Subjective tests generally show similar (or even better) tendency  Benefit over HEVC very clear  Benefit over JEM visible at various points Performance
  • 17. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 17 • JVET-J1003: Report of subjective evaluation contains 28 plots as shown, one per sequence • Count significant cases of positive/ negative benefit with non-overlapping confidence interval against JEM Performance HM JEM Rate1...4 Proposals ranked by MOS (per rate point) +1 credit -1 credit
  • 18. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 18 • "Mean" and "significance-count" method suggested at least 7 proposals that were obviously better than JEM Performance SDR Pxx 10 Pxx 8 Pxx 8 Pxx 6 Pxx 6 Pxx 6 Pxx 6 Pnn 3 Pnn 3 Pnn 2 Pnn 2 Pnn 1 Pnn 1 JEM 0 Pnn 0 Pnn -1 Pnn -1 Pnn -1 Pnn -2 Pnn -2 Pnn -2 Pnn -3 Pnn -4 HM -36 Pxx 6,53 Pxx 6,46 Pxx 6,41 Pxx 6,37 Pxx 6,33 Pxx 6,33 Pxx 6,26 Pnn 6,23 Pnn 6,17 Pnn 6,15 Pnn 6,13 Pnn 6,11 Pnn 6,04 Pnn 6,04 Pnn 6,03 Pnn 6,03 Pnn 6,01 JEM 6,01 Pnn 6,00 Pnn 5,96 Pnn 5,94 Pnn 5,88 Pnn 5,86 HM 4,57 Mean MOS Significance vs. JEM 60 ... +60
  • 19. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 19 • Similar tendency in HDR and 360° categories • Mostly same coding tools as in SDR provide good benefit Performance HDR / 360° Mean MOS Signif. vs. JEM Pxx 6,04 Pxx 6,00 Pxx 5,94 Pxx 5,93 Pxx 5,86 Pnn 5,85 Pnn 5,80 Pnn 5,67 JEM 5,62 Pnn 5,60 Pnn 5,59 Pnn 5,45 Pnn 5,11 HM 4,14 Pxx 7 Pxx 3 Pxx 2 Pxx 2 Pxx 2 Pnn 1 Pnn 1 JEM 0 Pnn 0 Pnn 0 Pnn -1 Pnn -1 Pnn -6 HM -20 32 ... +32 Mean MOS Signif. vs. JEM Pxx 6,20 Pxx 6,19 Pxx 6,06 Pxx 6,03 Pxx 5,99 Pxx 5,96 Pxx 5,86 Pnn 5,69 Pnn 5,67 Pnn 5,51 Pnn 5,45 JEM 5,11 HM 3,79 Pnn 3,45 Pxx 9 Pxx 9 Pxx 8 Pnn 7 Pxx 7 Pxx 6 Pxx 5 Pxx 4 Pnn 2 Pnn 1 Pnn 1 JEM 0 HM -9 Pnn -12 20 ... +20HDR 360°
  • 20. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 20 • Comparison of proposals to HEVC at higher rate points  Subjective quality of best performing proposals always equal or even better (about 1/3 of cases) than HEVC at next higher rate point, over all categories (with approx. 40% less rate)  Subjective quality of best performing proposals always equal or even better (about 1/5 of cases) than HEVC at second next higher rate point, in SDR-UHD category (with approx. 65% less rate) • Highest rate point HEVC may be close to transparent quality in many cases, difficult to become better • Though not always the same proposal performing best at a given rate point, it can be anticipated that merits of different proposals could be combined  50% (or more) bit rate reduction with same quality will probably be achievable Performance compared to HEVC
  • 21. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 21 CODING TOOLS
  • 22. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 22 • In terms of large architecture: Most proposals similar, no deviation from hybrid coding mainstream • Most improvements from further refinements of well-known building blocks of HEVC and JEM  Partititioning: Quad/binary, augmented by ternary tree and finer  Intra prediction using  directional modes, DC and planar  sample smoothing with various adaptation  inheritance of chroma modes and chroma sample prediction from luma  Inter prediction: advanced motion vector prediction, affine models, sub- block partitioning, switchable primary transforms, mostly DCT/DST variants  Secondary transforms targeting specific cases of prediction residual characteristics  Adaptive loop filter based on local classification, some new variants  Quantization / context-adaptive arithmetic coding CfP analysis: What was proposed?
  • 23. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 23 • New elements (some come with high complexity):  Decoder side estimation for mode/MV derivation and sample prediction both in intra and inter coding (JEM)  Finer partitioning: Asymmetric, geometric  Neural networks for prediction, loop filtering, upsampling, (encoder control)  Additional elements using template matching  Intra block copy / current picture referencing  Additional non-linear, de-noising and statistics-based loop filters  Additional linear and non-linear elements in prediction • HDR specific:  New adaptive reshaping and quantization, also in-loop  HDR-specific modifications of existing tools, e.g. deblocking • 360-video specific:  Variants of projection formats, geometry-corrected face boundary padding  Modification and disabling of existing tools at face boundaries CfP analysis: What was proposed?
  • 24. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 24 • VVC Working Draft 1 / Test Model 1 (VTM1): basic approach • VTM Block structure  Unified tree (coding block unites prediction and transform)  CTU size 128x128, rectangular blocks (dyadic sizes), smallest luma size 4x4  Maximum transform size 64x64 • VTM: Some removed elements of HEVC:  Mode dependent transform (DST-VII), mode dependent scan  Strong intra smoothing  Sign data hiding in transform coding  Unnecessary high-level syntax (e.g. VPS)  Tiles and wavefront  Quantization weighting VVC Test Model and Benchmark Set • Benchmark Set defined in addition to VTM, including the following well-known JEM tools: • 65 intra prediction modes • Coefficient coding • AMT + 4x4 NSST • Affine motion • GALF • Subblock merge candidate (ATMVP) • Adaptive motion vector precision • Decoder motion vector refinement • LM Chroma mode Purpose: testing benefit of technology against better performing set
  • 25. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 25 • Prediction block partitioning of a 2N×2N CB • Transform block partitioning of a CB  Quadtree partitioning of CB → Residual Quad Tree (RQT)  Transform size 4×4 to 32×32  TB size 4×4 to 64×64  PB boundaries inside TBs allowed HEVC: Prediction Blocks (PBs) and Transform Blocks (TBs)
  • 26. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 27 • Simple ternary-tree split was used in several proposals, can be alternated with binary split • Further proposed variants of partitioning included  Asymmetric binary split modes  Diagonal and geometric (wedge-shaped) split modes Block Partitioning: Quadtree – Ternary Tree – Binary Tree Example: (source: JVET-J1002)
  • 27. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 28 TOOLS PROPOSED BY RWTH
  • 28. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 29 • Motivation: Towards object-oriented coding  Follow object boundaries more closely  Less coding artifacts where it matters • Prediction, transform and coding driven by actual object shape under RD-constraint  Inter- and intra-predicted segments for handling of disocclusions  Overlapped wedge based filtering at partition boundary  Shape-adaptive DCT for spatially localized transform coding RWTH Proposal: Geometric Partitioning (GEO) Source: M. Bläser, J. Sauer, and M. Wien, “Description of SDR and 360o video coding technology proposal by RWTH Aachen University,” Doc. JVET-J0023, Joint Video Experts Team of ITU-T VCEG and ISO/IEC MPEG, San Diego, USA, 10th meeting, Apr. 2018
  • 29. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 30 • GEO available for all block sizes ≥ 8×8 luma samples • Partitioning is represented by two coordinate points 𝑃0 and 𝑃1 on the block boundary • Prediction of two coordinate points 𝑃0 and 𝑃1 from 16 pre-defined templates (scaled for non-square blocks)  Alternative: Spatial or temporal prediction  Refinement: block size dependent offset • Integration with AMVP, MERGE, FRUC (no AFFINE (yet)) GEO: Partitioning Coding and Prediction
  • 30. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 31 • No transform-tree in JEM 7.0  localization of residual error for larger blocks required • ΔSA-DCT adapted from MPEG 4 software for blocks up to 128×128 • Currently floating point implementation – integer transform targeted • SA-DCT signaled as additional transform choice next to full block DCT ( 4 total GEO transform modes) • Coding of transform coefficients (TSBs, significance flags) with regard to shape GEO: Shape-Adaptive DCT for Geometric Partitions Segment with high prediction error Segment with low prediction error Example of 64×32 residual block
  • 31. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 32 Results for GEO JEM 7.0 JEM 7.0 + GEO • Visual improvements at object boundaries  Sharper contours  Less staircase-effect  More background details • Objective gains (BD-rate savings)  Against HEVC: ~33% on C1, ~25% on C2  Against JEM: ~0.8% for both, C1 and C2 JEM 7.0
  • 32. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 33 Results for GEO JEM 7.0 JEM 7.0 + GEO • Visual improvements at object boundaries  Sharper contours  Less staircase-effect  More background details • Objective gains (BD-rate savings)  Against HEVC: ~33% on C1, ~25% on C2  Against JEM: ~0.8% for both, C1 and C2 JEM 7.0 + GEO
  • 33. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 34 • Motivation: Special characteristics of 360 content  360° symmetry not exploited by current codecs  Motion across face boundaries possible  Geometric distortions  Motion compensation suboptimal  Not correctly treated by loop filters  Here shown for cube, but similar problems for other coding formats • Proposal: Doing things correctly that “broke” for 360°content  Face extension for motion estimation and compensation  Loop filtering over continuous boundaries according to 3D arrangement RWTH Proposal: 360°Coding Tools Source: M. Bläser, J. Sauer, and M. Wien, “Description of SDR and 360o video coding technology proposal by RWTH Aachen University,” Doc. JVET-J0023, Joint Video Experts Team of ITU-T VCEG and ISO/IEC MPEG, San Diego, USA, 10th meeting, Apr. 2018
  • 34. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 35 360°coding tools - Face extension for cube projection (EAC/CMP/ACP) 𝑯 𝐵2𝐴 = 0 0 𝑓2 0 𝑓 0 −1 0 0 𝑓 = face width 2 • Approach can be transferred to other coding formats
  • 35. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 36 • Reference samples of blocks at face boundaries changed  Original: Samples from top or left block are used  Modified: Samples are chosen according to 3D cube geometry • Approach can be transferred to other coding formats 360°coding tools - Corrected deblocking filter (DBF)
  • 36. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 37 • Objective gains (BD-rate savings)  Against HEVC anchor: ~31% E2E WS-PSNR  Agains JEM (same projection format): ~1.6%  Gains higher for sequences with high motion Results for 360°coding tools JEM deblocking Proposed deblocking
  • 37. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 38 SUMMARY AND OUTLOOK
  • 38. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 39 • Report of Results from the Call for Proposals on Video Compression with Capability beyond HEVC [JVET-J1003]  Documentation of results per sequence, marking HM and JEM anchors, not identifying individual proponents  Assessment of qualitative (and as far as possible quantitative) benefit of submitted technology compared to anchors • Working Draft 1 of Versatile Video Coding [JVET-J1001]  "Reduced" HEVC plus quad/binary/ternary tree structure • Test Model 1 of Versatile Video Coding (VTM 1) [JVET-J1002]  Corresponding encoder and algorithm description Documents issued after CfP Results
  • 39. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 40 • CE1: Partitioning • CE2: In-loop filters • CE3: Intra prediction and mode coding • CE4: Inter prediction and MV coding • CE5: Arithmetic coding engine • CE6: Transforms and transform signalling • CE7: Quantization and coefficient coding • CE8: Current picture referencing • CE9: Decoder side MV derivation • CE10: Combined and multi-hypothesis prediction • CE11: Composite reference pictures • CE12: Mapping for HDR content • CE13: Projection formats Core Experiments defined by JVET
  • 40. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 41 • Call for Proposals demonstrated availability of significant compression benefit  HEVC out-performed by virtually all proposals  Subjective results suggest initial rate savings of 40+% over HEVC at starting point • Versatile Video Coding (VVC): First Working Draft and Test Model defined  Reduced initial tool set  Step-by-step integration of tools  Evaluation of concurring variants of tools  Consideration of algorithmic complexity  Further fast progress expected, goal: finalization 2020 Summary and Outlook
  • 41. Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR and 360°Video | Mathias Wien | RWTH Aachen University | Institut für Nachrichtentechnik | 14.05.2018 42 • Document archives (publicly accessible)  JVET / VVC:  http://phenix.it-sudparis.eu/jvet  http://ftp3.itu.ch/av-arch/jvet-site  JCT-VC / HEVC:  http://phenix.it-sudparis.eu/jct  http://ftp3.itu.ch/av-arch/jctvc-site • Software for HEVC, JEM, and 360 Video (publicly accessible):  https://jvet.hhi.fraunhofer.de/svn/svn_VVCSoftware_VTM  https://jvet.hhi.fraunhofer.de/svn/svn_VVCSoftware_BMS  https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/  https://jvet.hhi.fraunhofer.de/svn/svn_360Lib/  https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ Further Information