SlideShare a Scribd company logo
End-to-End VR Audio Solution for 

Fully Immersive/Interactive Experience
Henney Oh, Ph. D.
2016 AES Int’l Conference on AVAR. Sep 30 - Oct 1, 2016. @ Los Angeles Convention Center
|
This workshop covers
2
• Backgrounds
• Immersive and interactive
• Binaural rendering
• Formats: object-based audio vs. scene-based audio
• End-to-end workflow for VR audio: codec-based and app-based
• Solutions
• Mixing and mastering with GAUDIO Works
• Sol Player, a reference quality VR/360 audio player (API)
• GAO, a VR audio format for Ambisonics + object + channel
• GAUDIO Craft, a VR/3D audio plug-in for gaming VR
• Hands-on and practices
|
GAUDIO LAB is
3
• Founded by guys from MPEG Audio standardization community
• Proudly, inventors of the binaural rendering technology for MPEG-H
3D Audio (ISO/IEC 23003-8)
• Strong audio engineering forces of 5 Ph.D. and 2 Masters
• Now, focus on providing end-to-end solution for whole VR audio
ecosystem
|
What is VR Audio ?
5
Technology or content that enables
“being there” experience in respect of 

auditory perception
Immersive & Interactive
|
What is immersive ?
6
videolink: https://youtu.be/X1tsJtcXllg
|
What is interactive ?
7
• 3DOF (Yaw-Pitch-Roll)
‣ Head orientation only
‣ 360 Video
‣ Ambisonics
• 6DOF (3DOF + X-Y-Z)
‣ Head orientation + Walk around
‣ Full VR
‣ Object-based audio
DOF (Degree Of Freedom)
Yaw
Roll
Pitch
Yaw
Roll
Pitch
Y
Z X
Listener’s freedom of movements for interactivity in VR space
|
Loudspeaker or Headphone ?
8
• Hard to remove local room presence
(you are “here”)
• Virtual space limited by room size
• 1 scene / room
• No local room artifact 

(block the “here” environment)
• Unlimited virtual space
• 1 scene / person
Vs.
Loudspeaker Headphone (earphone)
|
Headphone. then, “binaural”
9
= Having or relating to two ears
widely used to mean “headphone virtualization”
|
Binaural Rendering
10
HRTF (HRIR)
Listen at anechoic room
BRIR = HRTF + RIR
Listen at real room
Signal processing technology to make sound virtualise in 3D space with
headphone representation using HRTF or BRIR
|
Binaural Rendering
11
Engine
=
Binaural rendering
VR audio Automobile
Core technology to enable VR audio
Compared to the engine in automobile
Core: Core:
|
VR Audio Format
12
• Channel
• Ambisonics
• Object
Concern all the ecosystem from content creation, delivery and consumption
|
An exemplary VR sound scene
13
Guitar
Bell
Bird
Fountain
You
BGM
Reflections
Diffusion
Direct
Diegetic Sound
Non-diegetic Sound
Scene
|
Channel-based
14
• A traditional scene representation
• Pre-rendered in studio to target channel configuration
• Each channel directly maps to each loudspeaker
• More channel, more immersive
|
Channel to Binaural Rendering
15
Set loudspeaker layout as the source positions and binaural rendering
45 degree315 degree
135 degree225 degree
|
Ambisonics (Scene-based)
16
• Loudspeaker-independent sound-field
representation on spherical harmonics
• Captured by B-format microphone
• Easy for 3DOF, 

NOT suitable for 6DOF
Spherical Harmonics
B-format FOA microphone
FOA (First Order Ambisonics)
HOA (Higher Order Ambisonics, 3rd)
|
Ambisonics
17
Non-diegetic stereo (BGM) CANNOT be implemented
|
Ambisonics to Binaural Rendering
18
Ambisonics to Channel Channel to BinauralRotation (3DOF)
Two-step rendering required after rotating sphere to the head orientation
|
Object-based
Object 1
Object 2
Object 3
Object 4
Object 5 (Stereo)
Send every sound component constructing sound-scene to playback-side
Metadata
To Playback
|
Object to Binaural Rendering
20
Object to Binaural
Straightforward and easy. Directly from object position to the ears
Complexity linearly increases as the number of objects increases
|
Comparative benefit of object-based
21
Directly Binaural Rendering
to Headphone
Rotation
(3DOF)
Sound
Sources
(Tracks)
…
Binaural
Rendering
Binaural
Rendering
Channel
Mapping
Channel mapped scene
PlayerProduction
ChannelRendering
to 

Channel
AmbisonicsSound
Sources
(Tracks)
Sound
Sources
(Tracks)
Object
……
…
Rendering
to
Ambisonics
|
VR Audio Format Transformation
22
Channel
Object
Ambisonics
O
O
O
O
|
Mixed format representation
23
Channel
Ambisonics
Object
Possible and recommended to represent sound scene with the
combinations of the 3 formats
|
End-to-end workflow in VR audio
24
Sound acquisition Post-production Transmission Play/Rendering
Cinematic
VR
Gaming
VR
Codec-based distribution
App-based distribution
Player
+
Content + EngineAuthoring tool
(Game Engine)
Sound Source
Player + EngineContent
(format)
Authoring tool
(DAW)
Compatibility
Sound source
|
Codec-based vs. App-based
25
Codec-based App-based
Typically in Cinematic VR Gaming VR
What have to deliver Content only Content + Playback engine
Live streaming capability Yes No
Upgradability Difficult Easy
Content distribution
platform
VR Players App market per OS
Solution for VR Audio
|
GAUDIO Core - the binaural rendering engine
27
• High quality - low complexity
• Pinpoint spatial resolution
• All 3 formats (Object, Ambisonics, Channel)
• Advanced distance rendering
• Source directivity, occulusion, reverberations
• Diegetic & non-diegetic selection per each source
• HRTF personalisation ready
Key Features
|
GAUDIO Works
28
Cinematic VR
content
(360 video + VR audio)
Channel
Object
Ambisonics
VR/360 content creation tool for cinematic VR
Sound acquisition Post-production Transmission Play/Rendering
360 Video
|
GAUDIO Works
29
• Sound powered by GAUDIO Core
• Support all formats for In-Out
• Object, Ambisonics, Channel
• Intuitive user interfaces
• Realtime automation R/W on 360 Video
• 2 independent 3D map-view
• Automated listener’s head orientation
• Easy to export
• Direct export to 360 video (mp4, mov)
• AV lipsyncPro Tools HD
AAX Plug-in
Key Features
|
GAUDIO Works hands-on
30
videolink: https://youtu.be/gB0ZlAzAf94
|
GAUDIO Works: Release
31
• Beta release today (Oct. 1, 2016)
• Download at our website

http://www.gaudiolab.com/
• It’s free! (forever)
|
GAUDIO Sol Player / Sol API
32
Enjoy VR
Sound acquisition Post-production Transmission Play/Rendering
VR/360 audio player / player api
Cinematic VR
content
(360 video + VR audio)
L
R
|
GAUDIO Sol Player / Sol API
33
• Sound powered by GAUDIO Core
• Support any combination of object, Ambisonics, and/or channel
• Low complexity and low latency
• Support multiple platforms
C D E F G
do re mi fa Sol
Why ‘Sol’ ?
Sol Player = G-Player
Key Features
< Reference Sol Player >
|
GAUDIO Sol API
34
OS
Device
Android Mac Windows iOS (coming soon)
Cardboard Gear VR Oculus Rift HTC Vive
Now available in
|
GAUDIO Sol Player Hands-on (Android version)
35
videolink: https://youtu.be/JDr8uKv3hN0
|
Sol Reference Player: Release
36
• It’s also free! (forever)
• Available at
• Android (Google Play) -
already
• Gear VR (Oculus store) -
under certification
• Oculus Rift (in a month)
• HTC Vive (in a month)< Reference Sol Player >
• Stand-alone 360 video with Sol
• Monitoring purpose
• Demo for customer or end-user
|
GAO
37
• Dedicated and optimised format for VR audio content delivery
• Gaudio Ambisonics + Objects
• Interface between Works and Sol
• Deliver all VR audio formats including object, Ambisonics, and channel
• Independent from any compression codecs
Ambisonics Objects
FOA, HOA up to 64 objects
Sound acquisition Post-production Transmission Play/Rendering
|
GAUDIO Works, GAO and Sol
38
Other platforms 

supporting FOA
FOA
Sound acquisition Post-production Transmission Play/Rendering
|
GAUDIO Craft
39
• Sound powered by GAUDIO Core
• Sound can be simulated and monitored with GAUDIO Works
• Easy to migrate from existing projects
• Intuitive interface for developer easily localise the sounds
• Support major platforms:
Sound acquisition Post-production Transmission Play/Rendering
VR/3D game engine plug-in
Unity3D Unreal
Key Features
|
Conclusions
40
• Covered technical backgrounds and solutions in end-to-end VR
audio ecosystem
• Introduced GAUDIO’s proposed solutions for the ecosystem: Works,
Sol, GAO, and Craft
• Please let us know your problems, then, we keep supplying
technologies aiming at ultimate VR audio experience
• Hands-on demo on the booth !
Sound for VR
www.gaudiolab.com | info@gaudiolab.com | 501 475 1974
7083 Hollywood Boulevard, Los Angeles, CA
You’re brilliant, We’re hiring !
(for LA office)
THANK YOU

More Related Content

Viewers also liked

360 Degree and VR Live Streaming Q&A
360 Degree and VR Live Streaming Q&A360 Degree and VR Live Streaming Q&A
360 Degree and VR Live Streaming Q&A
3xscreen
 
Virtual Reality Storytelling - ISOJ Slides from Dan Pacheco
Virtual Reality Storytelling - ISOJ Slides from Dan PachecoVirtual Reality Storytelling - ISOJ Slides from Dan Pacheco
Virtual Reality Storytelling - ISOJ Slides from Dan Pacheco
Dan Pacheco
 
Virtual Reality Journalism
Virtual Reality JournalismVirtual Reality Journalism
Virtual Reality Journalism
Francesco Marconi
 
Virtual Reality Formats - Branded Content
Virtual Reality Formats - Branded Content Virtual Reality Formats - Branded Content
Virtual Reality Formats - Branded Content
Rhiannon Monks
 
VRStorytelling, Class 1, Fall 2016
VRStorytelling, Class 1, Fall 2016VRStorytelling, Class 1, Fall 2016
VRStorytelling, Class 1, Fall 2016
Dan Pacheco
 
Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...
Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...
Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...
Intel® Software
 
Dan Pacheco
Dan PachecoDan Pacheco
Dan Pacheco
Knight Center
 
Guide to Virtual Reality Storytelling
Guide to Virtual Reality StorytellingGuide to Virtual Reality Storytelling
Guide to Virtual Reality Storytelling
Dejan Gajsek
 
VR Journalism Workshop at Journalism Interactive
VR Journalism Workshop at Journalism InteractiveVR Journalism Workshop at Journalism Interactive
VR Journalism Workshop at Journalism Interactive
JournovationSU
 
Vr for education in korea conception and case
Vr for education in korea   conception and caseVr for education in korea   conception and case
Vr for education in korea conception and case
Choi Man Dream
 
2016 Cannes Lions Winners by FM COMMUNICATIONS
2016 Cannes Lions Winners by FM COMMUNICATIONS2016 Cannes Lions Winners by FM COMMUNICATIONS
2016 Cannes Lions Winners by FM COMMUNICATIONS
fmcommunications
 
Aes (advance encryption standard)
Aes (advance encryption standard) Aes (advance encryption standard)
Aes (advance encryption standard) Sina Manavi
 
[Creative report 1] 2016 VR Report
[Creative report 1] 2016 VR Report[Creative report 1] 2016 VR Report
[Creative report 1] 2016 VR Report
fmcommunications
 
Set Your Hook
Set Your HookSet Your Hook
Set Your Hook
Gavin McMahon
 
Virtual Reality
Virtual RealityVirtual Reality
Virtual Reality
Sagar Reddy
 

Viewers also liked (16)

360 Degree and VR Live Streaming Q&A
360 Degree and VR Live Streaming Q&A360 Degree and VR Live Streaming Q&A
360 Degree and VR Live Streaming Q&A
 
Virtual Reality Storytelling - ISOJ Slides from Dan Pacheco
Virtual Reality Storytelling - ISOJ Slides from Dan PachecoVirtual Reality Storytelling - ISOJ Slides from Dan Pacheco
Virtual Reality Storytelling - ISOJ Slides from Dan Pacheco
 
Seung
SeungSeung
Seung
 
Virtual Reality Journalism
Virtual Reality JournalismVirtual Reality Journalism
Virtual Reality Journalism
 
Virtual Reality Formats - Branded Content
Virtual Reality Formats - Branded Content Virtual Reality Formats - Branded Content
Virtual Reality Formats - Branded Content
 
VRStorytelling, Class 1, Fall 2016
VRStorytelling, Class 1, Fall 2016VRStorytelling, Class 1, Fall 2016
VRStorytelling, Class 1, Fall 2016
 
Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...
Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...
Narrative Fiction Storytelling in 360 Stereoscopic Panoramic VR: Old Techniqu...
 
Dan Pacheco
Dan PachecoDan Pacheco
Dan Pacheco
 
Guide to Virtual Reality Storytelling
Guide to Virtual Reality StorytellingGuide to Virtual Reality Storytelling
Guide to Virtual Reality Storytelling
 
VR Journalism Workshop at Journalism Interactive
VR Journalism Workshop at Journalism InteractiveVR Journalism Workshop at Journalism Interactive
VR Journalism Workshop at Journalism Interactive
 
Vr for education in korea conception and case
Vr for education in korea   conception and caseVr for education in korea   conception and case
Vr for education in korea conception and case
 
2016 Cannes Lions Winners by FM COMMUNICATIONS
2016 Cannes Lions Winners by FM COMMUNICATIONS2016 Cannes Lions Winners by FM COMMUNICATIONS
2016 Cannes Lions Winners by FM COMMUNICATIONS
 
Aes (advance encryption standard)
Aes (advance encryption standard) Aes (advance encryption standard)
Aes (advance encryption standard)
 
[Creative report 1] 2016 VR Report
[Creative report 1] 2016 VR Report[Creative report 1] 2016 VR Report
[Creative report 1] 2016 VR Report
 
Set Your Hook
Set Your HookSet Your Hook
Set Your Hook
 
Virtual Reality
Virtual RealityVirtual Reality
Virtual Reality
 

Similar to GAUDIO at AES conference 2016_published ver

The Next-Gen Technologies Driving Immersion
The Next-Gen Technologies Driving ImmersionThe Next-Gen Technologies Driving Immersion
The Next-Gen Technologies Driving Immersion
Qualcomm Research
 
How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...
How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...
How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...
DevGAMM Conference
 
Distributed high-quality image manipulation and review in a virtual collabora...
Distributed high-quality image manipulation and review in a virtual collabora...Distributed high-quality image manipulation and review in a virtual collabora...
Distributed high-quality image manipulation and review in a virtual collabora...
ETCenter
 
Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...
Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...
Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...
Chris Adamson
 
Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...
Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...
Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...
AugmentedWorldExpo
 
Collaborate pro 900_datasheet
Collaborate pro 900_datasheetCollaborate pro 900_datasheet
Collaborate pro 900_datasheet
Dan Kowalske
 
What’s new in MPEG?
What’s new in MPEG?What’s new in MPEG?
What’s new in MPEG?
Alpen-Adria-Universität
 
Phillip Heckinger (Microsoft): Empowering AR & VR with Tech Art
Phillip Heckinger (Microsoft): Empowering AR & VR with Tech ArtPhillip Heckinger (Microsoft): Empowering AR & VR with Tech Art
Phillip Heckinger (Microsoft): Empowering AR & VR with Tech Art
AugmentedWorldExpo
 
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global DeliveryMaking Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Akamai Developers & Admins
 
Making Immersive Virtual Reality Possible in Mobile
Making Immersive Virtual Reality Possible in MobileMaking Immersive Virtual Reality Possible in Mobile
Making Immersive Virtual Reality Possible in Mobile
Qualcomm Research
 
CineGrid @ Holland Festival 2010
CineGrid @ Holland Festival 2010CineGrid @ Holland Festival 2010
CineGrid @ Holland Festival 2010
Frank Kresin
 
Build a vr video player from scratch
Build a vr video player from scratchBuild a vr video player from scratch
Build a vr video player from scratch
Andre Chen
 
Latest Technologies in Production & Broadcasting
Latest  Technologies in Production & BroadcastingLatest  Technologies in Production & Broadcasting
Latest Technologies in Production & Broadcasting
Dr. Mohieddin Moradi
 
Flyers Media Player Arv S9X.pdf
Flyers Media Player Arv S9X.pdfFlyers Media Player Arv S9X.pdf
Flyers Media Player Arv S9X.pdf
Digital Signage Specialist
 
E pillars - Aver Video Conferencing
E pillars - Aver Video ConferencingE pillars - Aver Video Conferencing
E pillars - Aver Video Conferencing
Unnikrishnan P
 
Let's Make Some Noise with Web Audio API
Let's Make Some Noise with Web Audio APILet's Make Some Noise with Web Audio API
Let's Make Some Noise with Web Audio API
GlobalLogic Ukraine
 
The H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC StandardThe H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC Standard
IMTC
 
Motion Graphics and Compositing Video Analysis Worksheet
Motion Graphics and Compositing Video Analysis Worksheet Motion Graphics and Compositing Video Analysis Worksheet
Motion Graphics and Compositing Video Analysis Worksheet Luke Finlay
 
Interactive Voice Con
Interactive Voice ConInteractive Voice Con
Interactive Voice Con
Dru Wynings
 
Wideband Audio Conferencing with Asterisk
Wideband Audio Conferencing with AsteriskWideband Audio Conferencing with Asterisk
Wideband Audio Conferencing with AsteriskSaúl Ibarra Corretgé
 

Similar to GAUDIO at AES conference 2016_published ver (20)

The Next-Gen Technologies Driving Immersion
The Next-Gen Technologies Driving ImmersionThe Next-Gen Technologies Driving Immersion
The Next-Gen Technologies Driving Immersion
 
How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...
How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...
How Audio Objects Improve Spatial Accuracy / Mads Maretty Sønderup (Audiokine...
 
Distributed high-quality image manipulation and review in a virtual collabora...
Distributed high-quality image manipulation and review in a virtual collabora...Distributed high-quality image manipulation and review in a virtual collabora...
Distributed high-quality image manipulation and review in a virtual collabora...
 
Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...
Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...
Fall Premieres: Media Frameworks in iOS 11, macOS 10.13, and tvOS 11 (CocoaCo...
 
Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...
Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...
Christian Sander (Dear Reality): The Future of Sound: Augmented Audio and Vir...
 
Collaborate pro 900_datasheet
Collaborate pro 900_datasheetCollaborate pro 900_datasheet
Collaborate pro 900_datasheet
 
What’s new in MPEG?
What’s new in MPEG?What’s new in MPEG?
What’s new in MPEG?
 
Phillip Heckinger (Microsoft): Empowering AR & VR with Tech Art
Phillip Heckinger (Microsoft): Empowering AR & VR with Tech ArtPhillip Heckinger (Microsoft): Empowering AR & VR with Tech Art
Phillip Heckinger (Microsoft): Empowering AR & VR with Tech Art
 
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global DeliveryMaking Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
Making Virtual Reality Real: 4K VR/AR Encoding and Global Delivery
 
Making Immersive Virtual Reality Possible in Mobile
Making Immersive Virtual Reality Possible in MobileMaking Immersive Virtual Reality Possible in Mobile
Making Immersive Virtual Reality Possible in Mobile
 
CineGrid @ Holland Festival 2010
CineGrid @ Holland Festival 2010CineGrid @ Holland Festival 2010
CineGrid @ Holland Festival 2010
 
Build a vr video player from scratch
Build a vr video player from scratchBuild a vr video player from scratch
Build a vr video player from scratch
 
Latest Technologies in Production & Broadcasting
Latest  Technologies in Production & BroadcastingLatest  Technologies in Production & Broadcasting
Latest Technologies in Production & Broadcasting
 
Flyers Media Player Arv S9X.pdf
Flyers Media Player Arv S9X.pdfFlyers Media Player Arv S9X.pdf
Flyers Media Player Arv S9X.pdf
 
E pillars - Aver Video Conferencing
E pillars - Aver Video ConferencingE pillars - Aver Video Conferencing
E pillars - Aver Video Conferencing
 
Let's Make Some Noise with Web Audio API
Let's Make Some Noise with Web Audio APILet's Make Some Noise with Web Audio API
Let's Make Some Noise with Web Audio API
 
The H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC StandardThe H.265/MPEG-HEVC Standard
The H.265/MPEG-HEVC Standard
 
Motion Graphics and Compositing Video Analysis Worksheet
Motion Graphics and Compositing Video Analysis Worksheet Motion Graphics and Compositing Video Analysis Worksheet
Motion Graphics and Compositing Video Analysis Worksheet
 
Interactive Voice Con
Interactive Voice ConInteractive Voice Con
Interactive Voice Con
 
Wideband Audio Conferencing with Asterisk
Wideband Audio Conferencing with AsteriskWideband Audio Conferencing with Asterisk
Wideband Audio Conferencing with Asterisk
 

Recently uploaded

Why React Native as a Strategic Advantage for Startup Innovation.pdf
Why React Native as a Strategic Advantage for Startup Innovation.pdfWhy React Native as a Strategic Advantage for Startup Innovation.pdf
Why React Native as a Strategic Advantage for Startup Innovation.pdf
ayushiqss
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
Globus
 
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, BetterWebinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
XfilesPro
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Globus
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Explore Modern SharePoint Templates for 2024
Explore Modern SharePoint Templates for 2024Explore Modern SharePoint Templates for 2024
Explore Modern SharePoint Templates for 2024
Sharepoint Designs
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownload
vrstrong314
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Hivelance Technology
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
takuyayamamoto1800
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
Tier1 app
 
Visitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.appVisitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.app
NaapbooksPrivateLimi
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
AMB-Review
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
Paco van Beckhoven
 
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Anthony Dahanne
 
Accelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with PlatformlessAccelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with Platformless
WSO2
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
KrzysztofKkol1
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
Globus
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
Max Andersen
 

Recently uploaded (20)

Why React Native as a Strategic Advantage for Startup Innovation.pdf
Why React Native as a Strategic Advantage for Startup Innovation.pdfWhy React Native as a Strategic Advantage for Startup Innovation.pdf
Why React Native as a Strategic Advantage for Startup Innovation.pdf
 
Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024Globus Compute Introduction - GlobusWorld 2024
Globus Compute Introduction - GlobusWorld 2024
 
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, BetterWebinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
Explore Modern SharePoint Templates for 2024
Explore Modern SharePoint Templates for 2024Explore Modern SharePoint Templates for 2024
Explore Modern SharePoint Templates for 2024
 
top nidhi software solution freedownload
top nidhi software solution freedownloadtop nidhi software solution freedownload
top nidhi software solution freedownload
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
Visitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.appVisitor Management System in India- Vizman.app
Visitor Management System in India- Vizman.app
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
 
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
 
Accelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with PlatformlessAccelerate Enterprise Software Engineering with Platformless
Accelerate Enterprise Software Engineering with Platformless
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
 
Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
 

GAUDIO at AES conference 2016_published ver

  • 1. End-to-End VR Audio Solution for 
 Fully Immersive/Interactive Experience Henney Oh, Ph. D. 2016 AES Int’l Conference on AVAR. Sep 30 - Oct 1, 2016. @ Los Angeles Convention Center
  • 2. | This workshop covers 2 • Backgrounds • Immersive and interactive • Binaural rendering • Formats: object-based audio vs. scene-based audio • End-to-end workflow for VR audio: codec-based and app-based • Solutions • Mixing and mastering with GAUDIO Works • Sol Player, a reference quality VR/360 audio player (API) • GAO, a VR audio format for Ambisonics + object + channel • GAUDIO Craft, a VR/3D audio plug-in for gaming VR • Hands-on and practices
  • 3. | GAUDIO LAB is 3 • Founded by guys from MPEG Audio standardization community • Proudly, inventors of the binaural rendering technology for MPEG-H 3D Audio (ISO/IEC 23003-8) • Strong audio engineering forces of 5 Ph.D. and 2 Masters • Now, focus on providing end-to-end solution for whole VR audio ecosystem
  • 4.
  • 5. | What is VR Audio ? 5 Technology or content that enables “being there” experience in respect of 
 auditory perception Immersive & Interactive
  • 6. | What is immersive ? 6 videolink: https://youtu.be/X1tsJtcXllg
  • 7. | What is interactive ? 7 • 3DOF (Yaw-Pitch-Roll) ‣ Head orientation only ‣ 360 Video ‣ Ambisonics • 6DOF (3DOF + X-Y-Z) ‣ Head orientation + Walk around ‣ Full VR ‣ Object-based audio DOF (Degree Of Freedom) Yaw Roll Pitch Yaw Roll Pitch Y Z X Listener’s freedom of movements for interactivity in VR space
  • 8. | Loudspeaker or Headphone ? 8 • Hard to remove local room presence (you are “here”) • Virtual space limited by room size • 1 scene / room • No local room artifact 
 (block the “here” environment) • Unlimited virtual space • 1 scene / person Vs. Loudspeaker Headphone (earphone)
  • 9. | Headphone. then, “binaural” 9 = Having or relating to two ears widely used to mean “headphone virtualization”
  • 10. | Binaural Rendering 10 HRTF (HRIR) Listen at anechoic room BRIR = HRTF + RIR Listen at real room Signal processing technology to make sound virtualise in 3D space with headphone representation using HRTF or BRIR
  • 11. | Binaural Rendering 11 Engine = Binaural rendering VR audio Automobile Core technology to enable VR audio Compared to the engine in automobile Core: Core:
  • 12. | VR Audio Format 12 • Channel • Ambisonics • Object Concern all the ecosystem from content creation, delivery and consumption
  • 13. | An exemplary VR sound scene 13 Guitar Bell Bird Fountain You BGM Reflections Diffusion Direct Diegetic Sound Non-diegetic Sound Scene
  • 14. | Channel-based 14 • A traditional scene representation • Pre-rendered in studio to target channel configuration • Each channel directly maps to each loudspeaker • More channel, more immersive
  • 15. | Channel to Binaural Rendering 15 Set loudspeaker layout as the source positions and binaural rendering 45 degree315 degree 135 degree225 degree
  • 16. | Ambisonics (Scene-based) 16 • Loudspeaker-independent sound-field representation on spherical harmonics • Captured by B-format microphone • Easy for 3DOF, 
 NOT suitable for 6DOF Spherical Harmonics B-format FOA microphone FOA (First Order Ambisonics) HOA (Higher Order Ambisonics, 3rd)
  • 18. | Ambisonics to Binaural Rendering 18 Ambisonics to Channel Channel to BinauralRotation (3DOF) Two-step rendering required after rotating sphere to the head orientation
  • 19. | Object-based Object 1 Object 2 Object 3 Object 4 Object 5 (Stereo) Send every sound component constructing sound-scene to playback-side Metadata To Playback
  • 20. | Object to Binaural Rendering 20 Object to Binaural Straightforward and easy. Directly from object position to the ears Complexity linearly increases as the number of objects increases
  • 21. | Comparative benefit of object-based 21 Directly Binaural Rendering to Headphone Rotation (3DOF) Sound Sources (Tracks) … Binaural Rendering Binaural Rendering Channel Mapping Channel mapped scene PlayerProduction ChannelRendering to 
 Channel AmbisonicsSound Sources (Tracks) Sound Sources (Tracks) Object …… … Rendering to Ambisonics
  • 22. | VR Audio Format Transformation 22 Channel Object Ambisonics O O O O
  • 23. | Mixed format representation 23 Channel Ambisonics Object Possible and recommended to represent sound scene with the combinations of the 3 formats
  • 24. | End-to-end workflow in VR audio 24 Sound acquisition Post-production Transmission Play/Rendering Cinematic VR Gaming VR Codec-based distribution App-based distribution Player + Content + EngineAuthoring tool (Game Engine) Sound Source Player + EngineContent (format) Authoring tool (DAW) Compatibility Sound source
  • 25. | Codec-based vs. App-based 25 Codec-based App-based Typically in Cinematic VR Gaming VR What have to deliver Content only Content + Playback engine Live streaming capability Yes No Upgradability Difficult Easy Content distribution platform VR Players App market per OS
  • 27. | GAUDIO Core - the binaural rendering engine 27 • High quality - low complexity • Pinpoint spatial resolution • All 3 formats (Object, Ambisonics, Channel) • Advanced distance rendering • Source directivity, occulusion, reverberations • Diegetic & non-diegetic selection per each source • HRTF personalisation ready Key Features
  • 28. | GAUDIO Works 28 Cinematic VR content (360 video + VR audio) Channel Object Ambisonics VR/360 content creation tool for cinematic VR Sound acquisition Post-production Transmission Play/Rendering 360 Video
  • 29. | GAUDIO Works 29 • Sound powered by GAUDIO Core • Support all formats for In-Out • Object, Ambisonics, Channel • Intuitive user interfaces • Realtime automation R/W on 360 Video • 2 independent 3D map-view • Automated listener’s head orientation • Easy to export • Direct export to 360 video (mp4, mov) • AV lipsyncPro Tools HD AAX Plug-in Key Features
  • 30. | GAUDIO Works hands-on 30 videolink: https://youtu.be/gB0ZlAzAf94
  • 31. | GAUDIO Works: Release 31 • Beta release today (Oct. 1, 2016) • Download at our website
 http://www.gaudiolab.com/ • It’s free! (forever)
  • 32. | GAUDIO Sol Player / Sol API 32 Enjoy VR Sound acquisition Post-production Transmission Play/Rendering VR/360 audio player / player api Cinematic VR content (360 video + VR audio) L R
  • 33. | GAUDIO Sol Player / Sol API 33 • Sound powered by GAUDIO Core • Support any combination of object, Ambisonics, and/or channel • Low complexity and low latency • Support multiple platforms C D E F G do re mi fa Sol Why ‘Sol’ ? Sol Player = G-Player Key Features < Reference Sol Player >
  • 34. | GAUDIO Sol API 34 OS Device Android Mac Windows iOS (coming soon) Cardboard Gear VR Oculus Rift HTC Vive Now available in
  • 35. | GAUDIO Sol Player Hands-on (Android version) 35 videolink: https://youtu.be/JDr8uKv3hN0
  • 36. | Sol Reference Player: Release 36 • It’s also free! (forever) • Available at • Android (Google Play) - already • Gear VR (Oculus store) - under certification • Oculus Rift (in a month) • HTC Vive (in a month)< Reference Sol Player > • Stand-alone 360 video with Sol • Monitoring purpose • Demo for customer or end-user
  • 37. | GAO 37 • Dedicated and optimised format for VR audio content delivery • Gaudio Ambisonics + Objects • Interface between Works and Sol • Deliver all VR audio formats including object, Ambisonics, and channel • Independent from any compression codecs Ambisonics Objects FOA, HOA up to 64 objects Sound acquisition Post-production Transmission Play/Rendering
  • 38. | GAUDIO Works, GAO and Sol 38 Other platforms 
 supporting FOA FOA Sound acquisition Post-production Transmission Play/Rendering
  • 39. | GAUDIO Craft 39 • Sound powered by GAUDIO Core • Sound can be simulated and monitored with GAUDIO Works • Easy to migrate from existing projects • Intuitive interface for developer easily localise the sounds • Support major platforms: Sound acquisition Post-production Transmission Play/Rendering VR/3D game engine plug-in Unity3D Unreal Key Features
  • 40. | Conclusions 40 • Covered technical backgrounds and solutions in end-to-end VR audio ecosystem • Introduced GAUDIO’s proposed solutions for the ecosystem: Works, Sol, GAO, and Craft • Please let us know your problems, then, we keep supplying technologies aiming at ultimate VR audio experience • Hands-on demo on the booth !
  • 41. Sound for VR www.gaudiolab.com | info@gaudiolab.com | 501 475 1974 7083 Hollywood Boulevard, Los Angeles, CA You’re brilliant, We’re hiring ! (for LA office) THANK YOU