AWS_Re_invent_22_VNova.pdf

© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Accelerating high -performance
video transcoding with LCEVC
SESSION CODE: CMP219
Ian Jefferson (He/Him)
Business Development Americas
V-Nova

V-Nova Confidential
Enhanced Immersive Experience

V-Nova Confidential 4
The only standard designed to enhance any
codec.
MPEG Standard
A step-change in video experience for
existing workflows and devices powered by
SMARTs from V-Nova.
Enhancement
Low-Complexity
How to deliver Photorealistic
6DoF Experiences at Scale?
Focus of this presentation:

V-Nova Confidential
Most of this presentation was previously covered in this
blog post
6
MPEG Standard
SMARTs from V-Nova.
Enhancement
Low-Complexity
https://v-nova-19969824.hs-sites.com/en/xr-and-the-metaverse

© Copyright V-Nova 2022
Premise a: Quality is non-negotiable
7
codec.
SMARTs from V-Nova.
Enhancement
300 million users own at least an avatar in virtual
worlds like Fortnite
Over 10% of OCSE population are hard-core
gamers who play more than 20 hours per week
These people, and their friends, will not accept to
go back to pixelated graphics reminiscent of
1990’s
… so we need high-resolution
volumetric data sets (meshes
and/or point clouds) and latest-
gen rendering at high-res high-fps
performed by powerful discrete
GPUs

Premise b: XR devices must become truly lightweight,
and will not offer enough compute power
8
codec.
MPEG Standard
SMARTs from V-Nova.
… so the only option is to render
somewhere else, via “split computing”
High–density processing for sustainable
green data centres.
Enhancement
Wearable XR devices targeting 1 Watt of power
consumption
Moore’s Law and Koomey’s law flattening,
almost no power gains below 4 nm silicon
Quantum computing or other computing
disruptions not a near-term option
Most people don’t have each a gaming PC or a
gaming console, especially when in mobility

For high-quality volumetric
rendering, a powerful GPU is
necessary
Per each viewer
While hard-core gamers have a
business case for purchasing it, most
casual users (or users in mobility) are
unlikely to buy an additional bulky rig
to set up and maintain
9
Premise c: The rendering rig can either be nearby or in the
cloud, preferably the latter
… so CDNs must evolve to become GDNs (Graphics
Distribution Networks), offering time-shared
rendering services via seamless split-computing
(indeed this is already happening, see NVIDIA
CloudXR or Meta Avalanche)

Challenge I: how can we send stereoscopic 4K 90 fps video
in ultra-low-latency within wi-fi or 5G constraints?
10
Realistically, it’s difficult for available per-
user bandwidth to exceed 30 Mbps
The sudden drops of bandwidth typical of
wireless delivery generate latency jitter
Latency is compensated by means of
forward-looking rendering and last-
millisecond reprojection
Both realistic reprojections and AR require
depth information
How can conventional video codecs (h.264,
HEVC, AV1, VVC) produce good XR quality
below 30 Mbps, plus dealing with sudden
drops of connectivity and leaving space for
depth maps?
… so we need some way to enhance the compression efficiency of
video codecs!

Challenge II: how can we possibly serve
truly photorealistic experiences?
11
codec.
Photorealistic ray-tracing / path-tracing
rendering still requires tens of minutes per
frame on a powerful server
Can we realistically make that real time
(300,000x speedup), for millions of
concurrent users?
Do we really need to re-render everything
once per every viewer, at display frame
rates?
… so we need some way to perform hybrid cloud
rendering!
(rendering once for an entire zone-of-view and
then performing a faster final rendering for just
the point of view)

12
Enter
Low-complexity Multi-layer
Data Coding
Enabling interoperability, wow-factor and scale in XR

Hierarchical
signal representation
“Signal” can include textures, video frames, geometry /
depth, meshes, point clouds, attributes for rendering and
physics engines
The overarching concept is reminiscent of 3D rendering, and
leverages the same hardware acceleration
Final
Image
Smart upscaling
with fewer
residuals
Add
back
residuals
Full-resolution
reconstruction
13
Intermediate
Levels and
Regions of
Interest
independently
transmittable or
decodable

MPEG-5 LCEVC applies these methods to enhance video
encoded with a conventional codec at lower resolution
14
LCEVC enhances any compression
technology in use today or tomorrow
STANDARD
Retrofittable via software
and backward-compatible

LCEVC enhances any video codec
15
Source: SPIE Aug 2021, “Towards much better SVT-AV1 quality-cycles tradeoffs for VOD applications” (Intel, Facebook), V-Nova Convex Hull analysis (1080p, 720p, 540, 432p, 360p, 288p)
PROCESSING POWER REQUIREMENTS
Best
Worst
COMPRESSION
Compression vs Complexity (Meta-Intel methodology, convex-hull)
Lower Higher
• LCEVC significantly improves the
trade-off for any codec, using
less energy to produce better
quality
• LCEVC processing can efficiently
leverage on available
hardware acceleration,
enabling to retrofit LCEVC also
on existing hardware

© Copyright V-Nova 2022 16
Asgards, h.264 25 Mbps
NVENC LCEVC NVENC
… and in so doing solves the First Challenge
Frame 84 Frame 84

Redpill VR, HEVC 25 Mbps
… and in so doing solves the First Challenge
Frame 84 Frame 84
NVENC LCEVC NVENC

When depth-maps/geometry are necessary,
LCEVC also enables MIV HEVC/VVC to operate at higher
resolutions, bit-depths and qualities

19
Rendering Once
for an entire
Volume of ViewPoints
Enabling photorealistic 6DoF video

How we achieved Hollywood CGI’s photorealism –
The PresenZ format
Render once from a Zone of View,
producing a dense point cloud
Compress point-cloud data via efficient
and massively parallel multi-layer coding

https://store.steampowered.com/app/1674620/Construct_VR__The_Volumetric_Movie/
21
This V-Nova PresenZ combination enabled the world’s first
photorealistic 6DoF VR movie, distributed via Steam Store
2022 Lumiere Award Winner
Advanced Imaging Society

Remote compute with LCEVC-enhanced casting
enables to deliver that quality to lightweight devices
AWS: g5x4 large instance
CPU: 16 vCPU, GPU: Nvidia A10G
Wifi router
local
SSD
V-Nova point cloud decompress
(CPU/GPU)
Pre-rendered
point cloud
DATA CENTER USER PREMISES
VR Headset
pos. and rot.
Send tracking
Decompress and display
NVidia CloudXR video
Video.
PresenZ VR rendering according
to VR Headset pos/rot (GPU)
Images to
display on
VR Headset
INTERNET
LCEVC-enabled
NVidia CloudXR
compress and stream
Video to display in VR

Accurate head tracking combined with glass-free
stereoscopy is bringing 6DoF also to flat screens,
providing further use cases for cloud rendering
23

Example Use Cases
Repurpose content
produced for standard
viewing, from existing
3D assets and pipelines
Monetize 20 years
of CG Animations
Mix pre-rendered
photorealistic content with
real-time rendered content,
e.g., for B2B design
reviews, marketing,
performances/events in VR,
etc.
Higher-quality
Metaverse experiences
Photorealistic virtual
showcases at home and
in stores, via standard
and/or auto-
stereoscopic head-
tracking displays
Augmented stores
and e-commerce

V-Nova Confidential
Thank you
v-nova.com

AWS_Re_invent_22_VNova.pdf

Recommended

Recommended

More Related Content

Similar to AWS_Re_invent_22_VNova.pdf

Similar to AWS_Re_invent_22_VNova.pdf (20)

Recently uploaded

Recently uploaded (20)

AWS_Re_invent_22_VNova.pdf