From AR to Augmented {City and Citizen}

XR for Good: Augmented Smartizen?
From AR to A-{City & Citizen}
Woontack Woo (禹雲澤), Ph.D.
KAIST ARRC & UVR Lab
XR4GOOD@ISMAR2019, Oct. 19. 2019
AR/VR/AH as a New Medium

lDirector: Woontack Woo
lARRC/UVR Lab. in 2019
§3 Ph.D. & 5 MS Researchers
§10 Ph.D. & 6 MS Students
§1 visitor & 2 Interns
§1 admin. staff
lCollaborators (2012-)
§T. Kim, V. Lepetit,
A. Argyros, M. Billinghurst,
S. Feiner, D. Schmalstieg,
T. Starner, S. Kim, etc.
KAIST UVR Lab. (2001-)
The Ubiquitous VR research aims at
the development of new computing
paradigms for ”Augmented Citizen
in Augmented City”

lResearch Focus
§From AR to Augmented {City & Smartizen}
KAIST ARRC/UVR Lab. 2019
2012-2019 © Woo, KAIST UVR Lab., Daejeon 34141, Korea
Augmented
Reality
• WW Board
• Metaverse
• UVR
Augmented
Human
• Body+
• Brain+
• Social+
Ubiquitous
Computing
• IoT, IoE
• Context-
aware UI
Wearable
Robotics
• Cyborg
• Bionics,
• Implantable,
Virtual Augmentation
Physical Augmentation
HumanEnvironment

lXR for Good
§We invent technologies to expand human’s abilities
§XR is a technology
§XR is a tool to expand human abilities
lGood for Human?
§Body
§Mind (Brain)
§Society (Social relationship)
Take Home Message

Content
VR/AR comes back! Where are we now?
Ubiquitous VR as AR2.0!
Toward Augmented City
What about Augmented Citizen?
What should we do?

lTrends of Personal Media
§Desktop, Laptop, mobile and many problems…
•Smaller, lighter, cheaper, faster, smarter, more intimate, etc.
lWhat’s Next?
§Wearables: From ‘HC Interface’ to ‘HC Integration’
§Wise Companion: From AI to ‘Assistive Intelligence’
§R-V Convergence: From IoT to ‘Internet of Everything’
Personal New Media

Brief History of AR
1968
HM 3D Display (I. Sutherland
@ Utah)
1992
Augmented Reality (T. Caudell
@ Boing)
1994
WearComp (S. Mann)
Taxonomy (P. Milgram)
1995
NaviCam (J. Rekimoto @ Sony)
1997
A survey of AR (R.Azuma)
MARS (S. Feiner @ Columbia)
Wearable AR (T.Starner)
1998 (@USC)
Tinmith (B. Thomas)
1st IWAR & 1st ISMR
1999 (@ATR)
ARToolkit (H. Kato @ HC)
WorldBoard (J. Spohrer @
IBM)
2000
AR Quake (B. Thomas @ SA)
BARS (S. Julier @ )
1st ISAR
2001 (@GIST)
mCollaAR (G. Reitmayr)
MagicBook (M. Billinghurst)
Ubiquitous VR (UVR Lab.)
2003
Human Pacman (A.D. Cheok)
iLamp (R. Raskar @ MERL)
PDA-AR (D. Wagner @
TUGraz)
2004
Phone-AR (M. Mohring @ BU)
2007
PTAM (G. Klein)
SONY “Eye of Judgement”
DigiLog Book (UVR Lab.)
CAMAR (UVR Lab)
2008
Wikitude (Mobilizy)
DigiLog Miniature (UVR
Lab.)
2009
Layar (SPRXmobile)
Arhrrrr! (GATECH)
SLAM on iPhone (G. Klein)
SONY “EyePet”
2012 (@KAIST)
Trans-Space (UVR Lab)
SONY PS Vita (mAR)

lVR/AR was Hot in 90s!
lVR/AR comes back! Why?
Where is VR/AR going?
AR1.0 (Desktop, 1990s-)
Augment Real
Objects with Virtual
Content
AR2.0 (Mobile, 2000s-)
Augment Real Space
with Content
AR3.0 (Smart phone, 2010s-)
Augment Real Place
with Dynamic
(Social) Content
AR4.0 (Glass, 2020s-)
Augment Human
with Context-aware
(Social) Content

lFive AR/VR Trends in 2019
§AR/VR will combine with 5G,
IoT, AI, edge/cloud computing
§AR/VR will increasingly be
collaborative and social
•Also will be used in training and teaching
§AR increasingly finding its way in cities and vehicles
•Mobile AR hits in advertising, commerce, navigation, etc.
•Augmented hearables with AI
§Affordable AR Glasses come
•Meta/Daqri are gone, Apple comes!
§VR hits the mainstream in
consumer entertainment
Where is VR/AR Now?

lKeynote @ISMAR2019
Where is VR/AR Now?

lGartner Hype Cycle for Emerging Tech 2019
Where are We Now?

lBeyond AR/VR
§World Board, Metaverse, 2nd Life, etc.
§Ubiquitous Virtual Reality
•VR in ubiComp Environment
•Context-aware MAR
UVR as a DigiLog Medium
UVR
Measured
Space
Real
Space
Virtual
Space
SensingModeling
Augmentation
& Interaction
World Board [’99]:
Information in places
by J. C. Spohrer (IBM)

lWhat is Ubiquitous VR? [Woo01]
§3D Link with IoT/IoE btw twin (real & virtual) worlds
§PoI/CoI (Context-of-Interest) Augmentation with Data, not
just sight (sound, haptics, smell, taste, etc.) but information
§Bidirectional Interaction/Collaboration with Intelligence for
O2O/H2H Communications
UVR as a DigiLog Medium
SPACES (3rd skin)
Intimate/Private
Personal
Social
General/public
Real
Space Social Networks
Seamless
Augmentation btw
dual spaces
How
to
Link
Seam
lessly?
CoI
LINK
U-Content
Virtual space

UVR Main Challenges
lRemaining Technical Challenges
Wearable Platform: Smart watch, Glasses, etc.
Tracking with D-SALM: Simultaneous Object-aware Tracking and Mapping
Augmentation: Context-aware Multimodal Augmentation & infoViz
Interaction: Natural Multimodal Interaction and Collaboration
Authoring: UX Authoring, Capturing and Augmented Human
Sharing : Co-presence and Tele-collaboration w/ Organic & Emphatic UI
Social Platform: 3D map-based in-situ Authoring, Storytelling & Standard for eco-system

lSome Definitions
§Augmented Reality (Azuma’98)
•To register interactive 3D virtual objects into a 3D real environment in
real time
§Augmented Space (L. Manovich’02)
•The physical space overlaid with dynamically changing information,
multimedia in form and localized for each user
§Augmented Urban Spaces (A. Aurigi’08)
•To co-develop integrated urban planning practices
through XR (VR/AR/MR) technologies
• To improve stakeholder participation
• To link long-term visions with short-term actions
• To plan (solve) challenges in the Augmented Urban
•Places, Connections, Situations and Augmentation
Augmented World

lAugmented City (Woo’19)
§Digital Twin(DT) performs integrated monitoring, efficient
management, prediction/verification simulation of the city
§A-City is a smart city where citizens manage and solve their
own urban problems through the organic linkage of ICT
technologies such as IoT, 5G, edge/cloud,
AI, DT and VR/AR
•Organic linkage btw real & virtual
•CoI(OoI/PoI)-based 3D augmentation
•R-V bidirectional interaction/collaboration
•AND Participatory problem solving
Augmented World
Augmented
City
Sentie
nt
Open
source
Intellig
ence
Produc
tive
Creativ
e
Recycli
cal
Resilie
nt
Fluid
Reticul
ar
Strateg
ic
M. Carta (2016)

lAugmented City with IoT/E, AI, VR/AR?
§Strolling is scrolling digital information/knowledge!
§Communication/cooperation beyond the limits of space-time
§So, extend the abilities of City, Citizen, Society
Augmented World
Social Media Services
Life: Home
Healthcare, wellness
Work: Office
Collaboration
Play: Public Places
Entertainment
Management
Energy/Water/Air/Mobility
Digital Twin
Physical World

lData Layers and Analysis
§Partial
•A small # of data sources,
e.g., temperature, pressure,
device state
§Clone
•All meaningful & measurable
data sources from an asset
§Augmented
•Data from the connected
asset with derivative data,
correlated data from
federated sources, and/or
intelligence data from
analytics and algorithms.
A-City with Digital Twin
Aggrega
tion
Analysis
Visualiz
ation
Insight
Informa
tion
Manage
ment
Sensing
Transfo
rm
Data
(Environment,
User, Process,
etc.)
Analytics
(Monitoring,
planning,
verification,
management, etc.)
City (Infra,
Robot, Citizen,
etc.)
IoT
(Sensors,
citizens, etc.)

AR Platform
lDigital Twin with Digital Thread
A-City with Digital Twin
Sensing &
Monitoring
Analysis &
Simulation
Planning
Design &
Development
Test &
Verification
Management
& Services
Multimodal User Interface
Data
Human
Sensor
IoT sensor
(Things,
People,
Processes)
DB
Data
Human
Sensor
IoT sensor
(Things,
People,
Processes)
DB
Digital Twin
Authoring
Planning
AR ServicesAR Services
Life
Work
Play
Geo info
Data
Data
Platform
Geo Data
Storage
Information
Wise Geo-AR
Context
Awareness
NUI
Management
USER
Information
Authentication
Open
DB/
Server
AR
Server
Visualization
Verification
Operation
Simulation
MonitoringAggregation
Analysis
Simulation
Visualization
Management
2D/3D
Rendering
City
Management
I/O
Authentication User Profile
D-SLAM4AR
Robot
In-situ
Authoring

lXR Glasses with 5G
AR for A-Smartizen

lXR over 5G?
§5G Edge Cloud will augment on-device processing
AR for A-Smartizen
5G
• -> tracking
data
• <-
retrieved/
rendered
data
Edge Cloud
• Partial
rendering
• analysis
• retrieving
Cloud
• Less
latency
sensitive
content
Power-efficient,
latency insensitive
on-device rendering
and tracking
Round-Trip Delay (or RTT) of 5G for VR/AR
• Handheld VR/AR: ~200ms
• Head-mounted VR (CG,AI, HCI): ~ 50ms (SKT?)
• Wearable AR (CV, CG, AI, HCI): ~ 10ms (KT?)

lAugmented Smartizen
§IoT, 5G, DT, AI, AR/VR Glasses
lUser Interface for A-City
§Strolling is scrolling
§Seeing is selection
§Bilking is clicking
lHow to Personalize UI/UX in A-City?
AR for A-Smartizen

lVR/AR extends Space-Time!
lAI/Robots come!
lWhat about Human in UVR era?
•Idiot, slave of technology? Isolation?
•Homo Deus, Upgrading humans into GODS?
•Or have a right to be a bit lazy and to enjoy social life?
lFrom AR ti A-Huamn!
Augmented Smartizen
http://goo.gl/yJPDtC

lFrom Augmented ‘Reality’ to ‘Citizen’
§Does AR/VR improve the quality of lives?
§What about UVR(or CAMAR)?
§How to make balance?
Augmented Smartizen
Human Technology
Usable
Useful
Efficient
Intended
Inconvenience
Artificial
Unintelligence
Induced
Enervation

lHomo Sapiens
§Human: social animal with body/brain and mind
lPhysical vs. Virtual Augmentation
§Body+
•To act, respond, interact,
move spatio-temporally
§Brain+
•To perceive, recognize, decide,
memorize/retrieve
§Mind/Social+
•To communicate, collaborate
with empathy
Augmented Smartizen
Act
ST Move
Body
Activity
Brain
Activity
Perceive/Cognize Decide
Collaborate
Communicate
Empathic
Social
Activity
Interact

lThen, How to achieve AH?
§Body+ Brain+ and Social+
§First of all,
Augmented Smartizen
There are 3 Things extremely hard:
i)Steel,
ii)A Diamond, and
iii)To know one's self.
- Benjamin Franklin (1750)
“Knowing yourself is the beginning
of all wisdom.”
- Aristotle
“Know thyself.”
- Socrates

lWhy Quantified Self?
§Dilemma of personalized services
Augmented Smartizen
Your data is their data!
Big Data vs. Small Data Privacy vs. Personalization Push vs. Pull Service

lPersonalization with Quantified Self
§A personal agent monitoring users with QS
•Provides a digital window into user’s health status,
feelings, value system, and even social relationship
•Aids user’s memory using context-aware reminders
•Negotiates with other agents on behalf of the user
•Recommends proper options to make decision
§The user develop a symbiotic relationship with the agent
•Compensates for diminishing capabilities in later life of the aging user
§Difficulties in developing the personalized agent
•Protect Privacy => Limited Information => Limited Personalization
• Privacy -> to use interpreted HQS/e-personality instead of raw data
• Information -> to use e-personality from small personal history data
• Personalization -> for agent to pull personalized content/services
Augmented Smartizen

lHolistic QS? (HQS, Woo’14)
§Quantified Index that represents
user’s state for a given time duration
§To support the personalized service
w/o disclosing private information
§Main components: 4 types
•Physical, Emotional, Social, and
Digital behavior state
Augmented Smartizen
References
M. Swan, ”The Quantified Self: Fundamental Disruption in Big Data Science and
Biological Discovery,” Big Data. Vol.2, pp.85-99, Jun. 2013
H.Yoon, Y.Doh, M.Yi, W.Woo,“A Conceptual Framework for Augmented Smart Coach
Based on Quantified Holistic Self,” HCII 2014, pp.498-508, Jun. 2014
J.Lee, E.Kim, J.Yu, J.Kim W. Woo, “Holistic Quantified Self Framework for Augmented
Human” in Proc. HCII DAPI, pp.188-201, 2018
W.Woo, “Augmented Human: Augmented Reality and Beyond”, in Proc. ACM MM
AltMM2018, pp.1, Oct. 2018

lHQS monitors/tracks the User
§Small (Personal) Data but Big ‘Challenge & Opportunity’
§E-personality based on HQS
•Patterns of HQS over time
§Personal agent based on E-personality
•Agent that monitors, negotiates, recommends
Augmented Smartizen
Integratio
n
User profile,
Sensors &
Context
Holistic
Quantified Self
Analysis
E-Personality
Bank
Personalized
Agent
E-PersonalityE-Personality

lPossible Applications of Personal Agent
§Personal AR Guardian
(healthcare, life coach/guard, etc.)
§Sustainable self-management
with game-like competition
§Personal black-box for
protecting self from criminal
Augmented Smartizen

lSocial Augmentation with Personal Agent
§From Quantified Self to Quality of Social Life!
§Experience Sharing: Empathy
§Mutual Respect and Negotiation
§Recommendation for deliberative decision
What’s Next?
Family
Friends
Groups
Communities
Companies
Countries
…
Trusted
Organization
Personalized
Augmentation
UX
Sharing
Sensing and
Monitoring
Context-
aware
Negotiation
Quantified Self

lHow to Solve Social Issues in City?
§Location-based information visualization
§Participation for collective intelligence
§Experience sharing
lPossible Applications
§Living, Work, Play, Shopping, Management in city
•City: crime, air pollution, traffic jam, etc.
•Harbor: air pollution by ships
•River: water pollution by wastewater
What’s Next?

What Should We Do?
lDeploy or Die! Obsolete or Augment?
AR for Good
We invent technologies to expand human’s abilities
AR is a technology
AR is a tool to expand human abilities
Good for Human?
Body, Mind (Brain), Society (Social relationship)

Summary
VR/AR comes back! Where are we now?
Ubiquitous VR as AR2.0!
Toward Augmented City
What about Augmented Smartizen?
What should we do?

lFor more Information?
§Woontack Woo, Ph.D.
•wwoo@kaist.ac.kr
•http://uvr.kaist.ac.kr
•http://arrc.kaist.ac.kr
lSupported by
§KAIST, ㈜ PostMedia, ㈜ VirNect, ㈜ AST
§과기정통부 IITP/NRF, 문화부 KOCCA
“The future is already here. It is just not uniformly distributed”
by William Gibson (SF writer)
Welcome to UVR Lab.!

From AR to Augmented {City and Citizen}

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to From AR to Augmented {City and Citizen}

Similar to From AR to Augmented {City and Citizen} (20)

More from Woontack Woo

More from Woontack Woo (20)

Recently uploaded

Recently uploaded (20)

From AR to Augmented {City and Citizen}