Metaverse_SGAsia2022.pdf

Metaverse is Coming…
Is the Metaverse Dying Again?
BTS through Metaverse! [Invited Talk @ BrainLink2022, 12/4/22]
Woontack Woo (禹雲澤), Ph.D.
G-School of CT/ UVR Lab., Head
KI-ITC AR Research Center, Director
KAIST, Daejeon 34141, Korea

2012-2022 © Woo, KAIST ARRC & UVR Lab., Daejeon 34141, Korea
Y. Lee, S. Oh, C. Shin, W. Woo, "Recent Trends in Ubiquitous VR," in Proc. of International Symposium on Ubiquitous VR, pp. 33-36, 2008.
Y. Lee, S. Oh, C. Shin, W. Woo, “Ubiquitous Virtual Reality and Its Key Dimension,” In Proc. of IWUVR, pp. 5-8., 2009
Vision-driven UVR Research Since 2001
사회관계망
Real World
Seamless
Augmentation btw
dual worlds
Simulate
…
3D Link
…
Augment CoI
3D Link
U-Content
Virtualized World
Metaverse ?
…
SPACE (3rd skin)
Intimate/Private
Social
General/public
Avatar
lUbiquitous VR, HHC BTS with CaMAR! [Woo01]
•*CaMAR: Context-aware Mediated & Augmented Reality
§3D Link (with IoE) between twin (real & virtual) worlds
§CoI (Context-of-Interest) Augmentation with information/knowledge,
not just 5 senses (sight, sound, haptics, smell, taste, etc.).
§Bidirectional Interaction/Collaboration for O2O/H2H Communications
lUVR vs. Metaverse

lUbiquitous VR, HHC BTS with CaMAR! [Woo01]
§From ‘Smart' of AR to ‘Smile' of Augmented Society!
R&D Foci of UVR Lab.
2001-2022 © Woo, UVR Lab., Korea
Augmented
Reality
• WWB
• UVR
• DT
• Metaverse
Augmented
Human
• Body+
• Brain+
• Social+
Ubiquitous
Computing
• IoT, IoE
• Context-aware
UI
Wearable
Robotics
• Cyborg
• Bionics,
• Implantable,
Virtual Augmentation
Physical Augmentation
Humans
Environment

lToward DT-based Metaverse [UVR Lab’22]
UVR3.0 (21-25)
• Object-aware SLAM for Real-
scale Realistic Digital Twin
TranSpace3.0 (21~27)
• Avatar-mediated Augmented
Collaboration
WISE UI/X (19-26)
• Augmented Human with
Wise Organic UI
A-Metamuse (21-23)
• Authoring for Augmented
Meta-Gallery/Museum
Meta-X (23?-)
• Authoring for Augmented
Lib., Concert, Tour & Game
R&D Projects of UVR Lab.

WISE UI/UX in 2022

Content
Metaverse, Dying Again with XR?
What is Metaverse? Is it Sustainable?
Metaverse 3.0, Does it work for Our Daily Life?
Metaverse 3.0, What Should We Do?
Q&A

lWhat is Metaverse?
§My short definition: UVR, an extended virtual world in smart space!
§Why are we interested in Metaverse? How to make the most of it?
Metaverse is Coming Again?
What do we do in Metaverse?
VS: Virtual Space
DS: Meta Space
DT: Virtualized Space
We are in Physical World!

lOmniverse by Jensen Huang [Nvidia 2020]
Metaverse is Coming (Again)!

lA Honeypot Trap by John Carmack [Oculus 2021]
Metaverse is Bullshit!

lSnow Crash [Neal Stephenson 92]
§Second Life by Linden Lab [03]
§Open Source Metaverse Project [04-08]
•A future internet, a shared and open standards-based virtual space
§Metaverse Roadmap [ASF ’07]
•The nexus of ‘virtually-enhanced physical reality’ and ‘physically persistent
virtual space’
•Nexus of
• Environment
• Humans
• Simulation
• Augmentation
lWhat’s Next?
§Multiverse
§Pataverse
What’s Metaverse? [ASF’07]
Augmented Reality
Virtual World
(Digital Twin)
Mirror World
Life-Logging
Augmented Society

lGame?
§Roblox? Fortnite?
lVR/AR/XR?
§Second Life? Meta Horizon?
lWeb3.0?
§From 2D screen to 3D space? DAO?
lAvatar World?
§Zepeto? vs. D-Human?
lVirtual Assets? Whatelse?
§Weverse? OpenSea?
What’s Metaverse?
https://cmte.ieee.org/futuredirect
ions/2017/09/27/can-we-have-a-
digital-twin/

lCore Properties of Metaverse [Woo21]
§Metaverse: next-generation internet (3D internet with web3)?
•UVR, an Extended Virtual World: avatar-mediated, co-present,
persistent, interoperable extended virtual world (open social platform) with
trusted economic systems through real-virtual nexus
§From <Hyperlink with HTML> to <Hyper-jump with USD> [NVidia’22]
What’s Metaverse?
Real-Virtual Nexus
• Convergence of persistent virt
ual world with the augmented
world
Avatar-mediated World
• Avatar-mediated access, expl
oration, utilization and experi
ence in the virtual world
Authoring and Possession
• Connect, communicate, collab
orate to engage in various dail
y activities without time and s
pace limitations
• Create and Own
Trusted Economic System
• A reliable convergence econo
mic system that enables econ
omic compensation, ownershi
p, and exchange of the results
of various activities
Interoperable Platform
• Interoperable social platform
between various metaverses t
o expand the production, own
ership, and exchange of econo
mic values such as assets and
experiences
Remaining Challenges:
•R-V nexus: 5G -> 6G/Edge/Cloud
•Browser: DT/MV+ Avatar
•In-situ Authoring with NUI
•HTML -> USD & Web2.0 -> Web3
•Context-aware D-SLAM
•Virtual Assets
•Carriable -> Wearable UI/UX

Metaverse is a Bubble!
Metaverse is Nowhere!
Metaverse is Now Here!
Then, is it Sustainable?
Is Metaverse Dying Again?

lThe New D-Gold Rush in Metaverse?
China
2
0
0
3
2
0
2
1
2
0
0
8
2
0
0
9
2
0
1
4
USA
Korea

lIs it just Buzzword?
§on Silicon Velley or Wall Street? in Korea or China?
lThe Metaverse is Bullshit because
§It already exists, and it's called the internet
§Tech Moguls missed the part where cyberpunk is dystopian
§Its promised cross-compatibility doesn't actually work
§No one can actually explain why it's better
•[Wes Felon@ C Gamer, Oct. 29, 2021]
https://outline.com/DUdr8Y
lFrom FB to Meta!
§Then, What will Meta do?
§10K researchers with $10B/Y
§+10K & $10B/Y for next 5 years

lMetaverse3.0: Meta’s Vision (2021)
D-Gold Rush in Metaverse3.0!

lMetaverse3.0 with XR Glasses [Woo’21]
§Avatar + Network + R-V World/Society + Economy + what else?
Metaverse (Novel, Web1.0 & Internet
1.0, 1990s-)
Snow Crash (1992)
Metaverse 1.0 with Internet (Web2.0, 2000s-)
2nd Life (2003)
Metaverse 2.0 with Smart phone (2010s-)
MV Returns (2020)
Metaverse 3.0 with XR
Glasses? (Web3.0, 2020s-)
A-Metaverse (2030?)
Every 15~20 years
or so a new major
platform emerges?
[07] Metaverse
[14] [19]
[03] 2nd Life, HoloLens1 HoloLens2
Digital Twin
[01] Ubiquitous VR
Prosumer
Login with id/pwd
Creasumer
Login with SNS id
CreaOwner
Login with Wallet

lEvolution(進化) of Personal Media
§Powerful, cheaper, lighter, smaller, smarter, more intimate
•Desktop(Type & Click) vs. Mobile (Point & Touch)
§So, What’s next? XR, DT, Metaverse?
•Speech, Gaze & Gesture for Wearable?
http://goo.gl/K6xAIA

lEvolution(進化) of Immersive Content
§UVR Content beyond TS [Woo’06]
Perception (5+ senses)
Interactive
Experience
Audio-visual+Haptic+Senses
Intelligent Interaction
Interaction
Response +Dangerous
+Expensive
+Impossible

lSo, Is Metaverse Dying Again?
lOr Is Metaverse Coming Again?
“The best way to predict the future is to create it!”

Play, culture and art
Work, communication,
collaboration
Learning (Education),
Health, Welfare, Consumption
Energy/
Environment (water, air, etc.)
Mobility
Virtualized Twin City
Real City
@Magic-verse by Magic Leap
lMetaverse Eco-System [Woo20]
§Interworking with heterogeneous XR experience platforms
§O2O Bidirectional Economic Activities
§Data right (sovereignty, privacy, security)
Augmented Society in Metaverse3.0
Infra:
IoT/IoE,
5G/6G,
Edge/Cloud
Platform:
Digital Twin,
VW,
Simulation, AI
Avatar:
Realistic
/CG
Smart
Human
Personal
UI/UX:
Wearable
Interaction
Authoring
Tool:
Creating
Content,
Assets
Economy
System:
NFT+, P2E
Interoper
able
Standard:
Avatar,
Asset,
Platform
N
P
U
D
C
E
S

lAugmented Citizen?
§AR3.0 over 5G/Edge/Cloud will augment on-device processing
§Augmented humans will expand citizen’s body/brain/social abilities!
5G/6G
• High
speed
Networks
• -> tracking
data
• <-
retrieved/
rendered
data
Edge Cloud
• Real-time
response
• Partial
rendering
• analysis
• retrieving
Cloud
• computati
onally
intensive
task
• Less
latency
sensitive
content
Power-efficient,
latency insensitive
on-device rendering
and tracking
Round-Trip Delay (or RTT) of 5G for VR/AR
• Handheld VR/AR: ~200ms
• Head-mounted VR (CG,AI, HCI): ~ 50ms (SKT?)
• Wearable AR (CV, CG, AI, HCI): ~ 10ms (KT?)

lPossible Applications
§Expand human abilities:
body, brain, social relation
§Transform the way we live
and interact with the world
§Revolutionize industries:
productivity, efficiency, safety
DT +
Metaverse +
XR
Game
performance
exhibition
Tourism,
Theme park
Advertisem
ent
Dramas,
Movies
Publication
(Webton,
Books, etc.)
Sports
Education,
Training
Medical
care
Manufacturing
Constructions,
architectures
Cities
And more…

Infra:
IoT/IoE,
5G/6G,
Edge/Cloud
Platform:
Digital Twin,
VW,
Simulation, AI
Avatar:
Realistic CG
Smart
Human
Personal
UI/UX:
Wearable
Interaction
Authoring
Tool:
Creating
Content,
Assets
Economy
System:
NFT+, P2E
Interoper
able
Standard:
Avatar,
Asset,
Platform
B+
B-
C
C
C
D
C
lWhere are We Now? What Should We Do? [Woo20]
§Equal opportunity,
§Fair competition,
§Sustainable ecosystem, etc.
UVR Experience in Metaverse3.0 Era

lDT/MV+XR for A-Learning [KAIST+KISTI‘21]
§Problem-solving, self-directed learning without time-space limitations
(borders/schools/grades/classes)
<Issues: Asymmetries>
•Physical environment
•Experience platform
•Available information
(shared space, view, object
size & representation, etc.)
•Role/experience
<Challenges>
•Avatar reconstruction and
representation
•Avatar-based collaboration
•Authenticity of Virtual
Restoration: Objective,
Structural, Existential
•Minimizing asymmetry
and maximizing social
presence
Object-aware 3D Scene Graph
(Host) Science Museum
IoT, Pad, XR Glasses
Experience, Interaction,
Collaboration in Metaverse
IoT, Pad, XR Glasses
(Guest) Class Room
DT Generation and Management
Building Room Object V-Camera

l3D Modeling: From SLAM to Spatial AI [CVPR20, ICL]
§Interaction with environment
§Spatial <AI vs. IA>
§Cost-effective solutions?
Level Task Technology Papers Outcome
Level 5
[Woo]
Understanding context of user’s
activities in dynamic environment
Space merge & object/activity recognition
(in-situ modeling for unknown)
3D DSG by MIT?
Augmented
collaboration
Level 4
Understanding relationships of
surrounding objects
Object-level dynamic mapping MID-Fusion
Advanced
interaction
Level 3
Understanding what objects are in
static environment
Semantic understanding SemanticFusion Basic interaction
Level 2 Recognizing spaces and walls Dense mapping with loop-closure ElasticFusion
Collision avoidance
/ motion planning
Level 1 Robust localizing 3D position Sparse Visual SLAM (+IMU) OKVIS/ OVKIS2 Position control
https://www.slamcore.com/
Andrew Davison

lSemantic SLAM: Remaining Challenges
§3D Data Acquisition (input point clouds):
•Noisy, unsegmented, occlusion issues, etc.
§3D Reconstruction: scene- vs. object-level
§Dynamic Scene Management
•Diversity of objects: cluttered, occluded, deform, move, etc.
• Cues: geometric primitives, joints, repeats, …
•Active 3D Scene Graph: objects, attributes, relationships, …
lStill, Long Way to go…
§To know (unknown) objects of interest (OoI) in dynamic environment
§To understand multimodal interaction with both hands
§To understand H-2-H communication/collaboration, etc.
§To aware context of scene, object, recourse, user, etc.
§And many more issues…
Fully Human-
designed
algorithm
End to end
learned
algorithm

l3D Avatar
Immature Mature
Realistic
Unrealistic

l3D Avatar for Metaverse, Some Issues?
§CG vs. Realistic 3D Video Avatar:
•CG: Kawaii engineering?
•Video-based: 3D face/body, gesture, clothes, hair/fur, etc.
• Parameter estimation of face/body/hands? (SMPL-X model?)
• Bare-face/body reconstruction with XR glasses?
• Bare-hands interaction: (Self-)occlusion or grabbing objects
• Facial expression? Dynamic gesture? Changing clothes, hairs?
• Collaboration? (Sign) language interpreter?
• And many more issues….
§Possible Solutions? Remaining issues?
•Ill-posed 3D from 2D image(s): ground-truth for learning?
•Controlling avatar with a sense of unity!
• movement, expression, interaction, etc.
•Social co-existence with heterogeneous personal platforms?
•Intelligent (adaptive) CG/Video Avatar?
• Using avatar-space/object information (e.g. collision, contact, etc.)
body shape
capture
Shape
modeling
Skeletal model
ing
Texture (cloth
es) modeling
Posture/expre
ssion/motion
estimation
Transmission
Video Avatar

lWearable XR Experiences!
Sensing
(Video, IMU,
IoT/E, etc.)
Object/Space
Recognition/
Tracking (CV)
Localization of
XR Glasses
(CV)
Augmentation
(CG & InfoViz)
Multimodal
Interaction
(HCI)
Story/Flow
Experience
(UX)
AI is
everywhere!
Edge/Cloud
(DT+AI &
XR Assets)
XR Content
Rendering
DT+XR
Modeling/
Authoring

lWhy is In-situ XR Authoring in Metaverse Important?
§Limitations in authoring (due to a lack of resources to utilize CAD data),
§Challenges creating net-new XR content without 3D files,
§In addition, Bad UX that make content creation laborious
lCapabilities of Authoring Tools
§To render interactive content
§To build the executable services
§To provide spatial experiences
lXR Authoring for Metaverse
§Simpler, easier, faster, cheaper to create!
•Use-case-driven design, lightweight, hybrid, UX
§How hybrid solutions might be able to bridge the gap
•Hybrid: CAD-inspired (Vuforia Studio) vs. lightweight options
•Making AR content in AR: Vuforia Instruct, Microsoft’s Dynamics 365 Guides
§Improvements to the (UX)

lAuthoring Tool Landscape
ARToolKit
Vuforia
ARKit
ARCore
DART
WebXR
Amazon Sumerian
Unity MARS
Unreal
Autodesk
Adobe Substance 3D
Vuforia Instruct
MS MRTK
Clay PPT
Paper *****
Level of Functionality/Fidelity in XR
(Programing)
Skill/Experience
Required

UVR=Metaverse3.0, What should We do?
• Avatar-mediated interaction with virtual objects in real world
• In-situ 3D modeling/manipulation of avatar & objects-aware Env.
• Intuitive authoring and capturing augmented UX
• Evaluation of metaverse (platform and UX), what else?
UVR R&D Direction to prepare for Metaverse3.0
• 3D Dynamic Scene Graphs with Semantic/Context-aware SLAM
• Authoring/capturing UX of space-telling in metaverse
• Wise UI with HQS for wearable XR Glasses
So, Still Remaining Challenges?
• To connect context-aware SLAM, instant NeRF and USD?
• To augment interaction/collaboration in a distance w/ co-presense?
• To embed personalized wise agent with HQS
Vision-driven UVR R&D

lMetaverse3.0, Toward UVR!
§Content is King!
•The fruit is sweet, but…
§Eco-system is Queen!
•Plan, create, distribute, utilize, share
•Content + Platform + Network + Device
• IoT/IoE, Big Data, 5G, DT/MV, AI, AR/VR
§User/UX is Emperor!
•Economic value creation through creative activities!
•Context-aware UX/services
§R&E is God!
•KAIST CT-AR Program in 2020
•KAIST Metaverse Program in 2022!
Metaverse3.0, What Should We Do?

Summary
Metaverse, Dying Again with XR?
What is Metaverse? Is it Sustainable?
Metaverse 3.0, Does it work for Our Daily Life?
Metaverse 3.0, What Should We Do?
Q&A

Metaverse Era, What is Our Choice?
lUtopia or Dystopia? XR or eXtinct?
"Do you want to sell sugared water for the rest of your life?
Or do you want to come with me and change the world?“
- Steve Jobs
“실수도 실패도 좌절도 있을 것입니다. 그래도 포기하지 않고
도전하는 사람이 세상을 바꿉니다!”
- 우운택

lARRC/UVR Lab. Members
§Director: Woontack Woo
§ARRC: 4 Ph.D. & 1 MS Researchers
§UVR Lab.: 12 Ph.D. & 12 MS Students
§1 undergraduate students (URP/Intern)
§1 visiting scholars
§2 administration staff
lInternational Collaborators
§Adjunct Professors @ GSCT (May 1, 2021 ~)
• Mark Billinghurst (UniSA, AU),
• Vincent Lepetit (ENPC ParisTech, FR),
• Hideo Saito (Keio Univ. JP)
§More Collaborators
• A. Argyros, K. Kunze, T. Starner, etc.
KAIST ARRC/UVR Lab. 2022
The Ubiquitous VR research aims at
the development of new computing
paradigms for
”DigiLog Life in Smart Spaces”

lMore Information
§Woontack Woo, Ph.D.
• wwoo@kaist.ac.kr
• http://uvrlab.org
• http://arrc.kaist.ac.kr
lSupported by
§㈜PostMedia, ㈜VirNect, ㈜AST, ㈜MAXST
㈜ExTriple, , ㈜Victoria Production
§과기정통부 (MIST) IITP, NRF, KISTI/NST
§문화부 (MCST) GIST KCTRI/KOCCA
§산업부 (MOTIE) KEA/KIAT
§대전시 DICIA, KAIST
lRelated Events
§KAIST CT-AR since Sep. 2020 and Metaverse from 2022!
“The future is already here. It is just not uniformly distributed”
by William Gibson (SF writer)
Welcome to UVR Era!

Metaverse_SGAsia2022.pdf

More Related Content

Similar to Metaverse_SGAsia2022.pdf

More from Woontack Woo

Recently uploaded

Metaverse_SGAsia2022.pdf