Slides for a 45 minute talk given at MIT for MAS.S61 on 2/23/2023

1. Avi Bar-Zeev, Feb 23, 2023
MIT Media Lab MAS.S61
AR, the Metaverse and Beyond…

2. 30+ Years
HoloLens (2010)
Second Life (2001)
DisneyQuest (1997)
Disney VR (1993) Keyhole (1999) -> Google Earth
Apple XR (2016)
Amazon (2015)

3. Experience Prototyping
What, When, Why
• Designers and free-range programmers need to map the solution space
• Before spending billions of dollars on any products
• XP is about e
ffi
cient exploration (breadth
fi
rst) of the solution space
• Answering key questions aligned to the organization’s priorities
• Is it di
ff
erent from academic explorations?
• Less about inspiration, originality and more about building con
fi
dence
• XP teams are responsible for guiding very expensive decisions

4. Where are we?

5. Tom F
fi
ske
Prof. Bob Stone

6. Licking the Cookie
What in the World Wide World is going on with Metaverse hype?
Underpromise and Overdeliver

7. Source: Avi Bar-Zeev / RealityPrime
The VR / Metaverse / AI Hype Cycle
The Future of Work
The Future of Information
The Next Internet
2021
Facebook
Rebrands
~2024
Chick-
fi
l-A
Chick-averse
1992
Snow Crash
2008
Second Life
2022
NFTs
~2026
Interoperability &
Standards
Metaverse
Cyberspace
Next Name
Virtual Reality
Information
Superhighway
Arti
fi
cial
Intelligence
1960s-1980
Ussing, Gibson
The Future of Everything
Above/Below the Radar

8. South Park’s Gnomes had a really simple idea to make money.
But without understanding “Phase 2,” it can’t work
It’s the Gnomes

9. Speculative Futures

10. Timeline of Common XR Terminology
* The 1968 Sutherland demo is often incorrectly remembered as “The Sword of Damocles” — that was the name of the large overhead mechanical tracker
Copyright 2022 Avi Bar-Zeev, Licensed under Creative Commons CC BY-ND 2.0. https://creativecommons.org/licenses/by-nd/2.0/
Metaverse3,4,5
(various)
Memex
(Bush, 1945)
The Metaverse1
(Stephenson, 1992)
The Metaverse2
(Roadmap, 2006)
The Veldt
(Bradbury, 1950)
The Web web5
The Other Plane
(Vinge, 1981)
Telesphere (HMD) & Sensorama
(Heilig, 1956)
The Ultimate Display*
(Sutherland, 1965)
Virtual Reality
(Lanier, 1987)
XR
(Mann/Wyco
ff
, 1991)
Virtual Reality
(Artaud, 1938)
Mixed Reality1 Spectrum
(MIlgram/Kishino, 1994)
Arti
fi
cial Reality
(Krueger, 1983)
Real-World Metaverse
(Hanke, 2021)
Spectacles
(Weinbaum, 1935)
CAVE
(Cruz-Neira et al, 1992)
Wearable Computing
(Mann, 1980)
Video-See-Through
(Tamura, 1997)
Waldo
(Heinlein, 1942)
Spatial Computing
(Jacobson, 1993)
Cyberspace1
(Gibson, 1982)
1. A vast 3D virtual world
laid out along a linear central
axis, populated by alternate-
identity 3D avatars. Devices
support AR, but no one used
that feature, because reality
was too dystopian.
2. Separated out 4 major components
of The Metaverse future:
a. Virtual Worlds (see Stephenson),
b. Mirror Worlds (see Google Earth)
c. Augmented Reality (see HoloLens)
d. Life-logging (see Gordon Bell)
3. The Next Web, sometimes
with 3D & Avatars, but requiring
decentralized content,
monetization and identity
4. The Fusion of all things
Digital and Physical (a.k.a. AR)
5. Any 3D virtual world
Holographic Telepresence
(various)
Telexistence
(Tachi, 1980s)
The Matrix
(Wachowski, 1999)
The Oasis
(Cline, 2011)
The Internet web3
Web 2.0
2025
2015 2020
2010
2005
1995 2000
1990
1985
1975 1980
1970
1940 1945
1935 1960
1955 1965
Hyperreality
(Baudrillard, 1981)
Information Superhighway
v5
AR Cloud
(Inbar, 2017)
PolySocial Reality
(Applin/Fischer, 2011)
Cyberspace2,3
Cyberspace2,3. The de
fi
nition morphed from 3D virtual worlds to The Internet as is, and “cyber” remains common short-hand for network security.
Virtual Reality
(Broderick, 1982)
Multi-User Dungeon
(Bartle, 1978)
Virtual
Worlds
Stereopsis
Camera
Obscura
Telepresence
(Gunkel, 1970s)
Virtual Environments
(Fisher et al, 1980s)
Synthetic Environments
(Various, 1990s)
Cyberspace
(Ussing/Ho
ff
, 1960s)
Digital Twin
(Hayashi, 1998)
Holodeck
(Roddenberry et al, 1988)
Viewpoint Dependent Imaging
(Fisher, 1981)
1950
The Days of Perky Pat
(Dick, 1963)
Avatar+
(Farmer/Morningstar, 1985)
+ The term is from Hinduism and many hundreds of years old, but this is the
fi
rst known use for agents on some display
Augmented Reality
(Caudell, 1990)
Mirror Worlds
(Gelertner, 1991)

11. A
Framework
for Understanding
XR’s Effects on People

12. Virtual Worlds 101
Typical Human
Communication
“Car”
Mediated
Communication
The Realities
of XR

13. Airplane
Telephone
Television
Computer
electricity
Wheel
(directed energy) (primitive telepathy) (primitive clairvoyance)
(primitive telekinesis) (primitive teleportation) (enhanced intelligence)
(primitive omniscience)
or
(the power to know
every dumb fact in existence)
(primitive omnipotence)
or
(the power to believe
you’re important)
(primitive omnipresence)
or
(the power to keep up with friends you
couldn’t be bothered to call)
Every game-changing technology
can be seen as a human superpower
For each major technology:
Describe it like a human superpower
Find the intrinsic human need
Project it forward into new tech
[XR advances all of these superpowers]

14. Superpowers for Everyone?
Super Unique
We each have deeply personal ideas of what is super
Super Normal
As ‘superpower’ becomes common, we may take it for granted
Super Advantaged
Some people seek to overcome natural limits. Some seek power over others…
Super Skewed
Power inequalities create anxiety, anger, instability, and blowback
Super Bene
fi
cent
Personal tech should enable everyone equitably (not identically)
Superpowers,
coming of age

15. Super Backlash

16. What if…?
Microsoft Teams

17. Super Power Spectrum
Collision
of Intents
Socially
Destructive
Neutral
Joint
Attention
Mutual
Bene
fi
t
Socially
Constructive
Socially
Destabilizing

18. Most Literal XR Superpowers
Virtual Spectrum
Superpower Example
Flying (+clairvoyance) Google Earth VR
Telekenisis Hand Tracking
Teleportation Locomotion UX
Super Strength Manipulation UX
Invisibility Self-Protection
Directed Energy Harassment
leap motion blog
https://www.nature.com/articles/srep09831, @2015
James Liu, Hirav Parekh, Majed Al-Zayer, Eelke Folmer

19. Most Literal XR Superpowers
Augmented Spectrum
Superpower Example
Telepathy Telepresence
Time Control History Sim
Emotional Intel. Sensor+AI
X-Ray Vision Surgery Aid
Night Vision Military IVAS
Mind Control Ad-Tech
Stanford XR and Law

20. Holographic Telepresence (ala Qualcomm)

21. Spatial Entertainment

22. Perceptual Intelligence
aka Situational Awareness
Not Literal

23. The World-Wide-World
AR Cloud
Not Literal
Read/Write World (2011)

24. Augmented Emotional Intelligence
Biometric Feedback Glasses
Pupil dilation
Eye gaze patterns
Blood
fl
ow
Voice analysis
Feeling Anxious, Nervous,
Frustrated, Angry
Colorize lenses,
fl
ashing LEDs
Private feedback

25. Zeitgeist

26. To the Future

27. How did The Web Evolve?
T
W E
F
G
I
There is no map (but knock yourself out if you want to build one)

28. How will The Metaverse Evolve?
There is no map (but knock yourself out if you want to build one)

29. Screenshot from Twitter
Interoperability is Essential

30. Links, References and Portals
Anything new here for XR?
• Browsing across links is like
teleporting from one page to another
• Embedded references can pull
remote content into a page
• iFrames are (theoretically safe)
sandboxed pages-within-pages
(in practice, not so safe)
Linked Worlds
Composite
Worlds
Portals

31. Not XR
XR Consumer XR
Business/Industrial XR
On “The iPhone Moment”
2020 2030
Critical Mass
100%
0%
15-20%
App
Store
2004 2014
Con
fi
dence
Point
2025 ± 36 months

32. Why is AR Special?

33. Is the Metaverse the Sum of all Virtual Worlds?
Native Apps, Zoom, and Talking Face-to-Face use Virtual Worlds
Must everything be part of a globally shared space?
Hell No!

34. AR<->VR Spectrum
More Present Here Present Anywhere Else
vs.

35. AR comes in many Coordinate Systems
Experiences are all relative. Some are relative to the planet
Geo Body Head (HUD)
2:05 pm
Local Object

36. AR should be Additive
But adding too much is subtractive

37. Keiichi Matsuda’s HyperReality, plus/minus

38. Separating “Verses” and “Purses”
• Contextual Discovery Engine —
fi
nds the most relevant content (anonymously)
• Personal AI Shield — keeps our private data safe and defends vs. all exploits
• Holographic Telepresence — brings remote people into your local world
Personal
Data
World
Data
Verses Purses
AR is Mainly Personal
Versional Personal

39. AR is Mainly Personal
If it’s just for you, it’s not The Metaverse
Suggested paths and 1:1 hologram convos are usually visible only to you.
Mapbox Rémi Rousseau

40. AR is Mainly Personal
Replacing Ads with Personal Photos

41. Do you want your most personal data
being addressable in the Metaverse?
Even assuming competent encryption and permissions
Hell No!

42. AI is Mostly About People Too
Augmented Human Intelligence
Model Prompt
Learning

43. AR is ‘Bigger’ than The Metaverse
Not by Volume, but by Ubiquity and UX
Message from
Anne
Message from
Anne
If The Metaverse is the Internet,
and XR hardware+software is the Browser
then AR+AI UX may be the common User Interface

44. Is There Something Beyond The Meta?
Rather than Adding more Fiction, Can we Pierce the Veil?
Understanding Reality Exploring Inner Worlds

45. Enlightenment
Are we Living in
a Simulation?
Finding
Meaning in XR
Remember Plato’s Cave?
Is There Something Beyond The Meta?

46. The Levels of Versality

47. Let’s Define “Levels of Versality”

48. The Adverse (noun) : a "seemingly free” reality funded by advertising [*which you
actually pay for via higher product prices] in which advertisers place bets about your
wants and needs, violating your privacy for pro
fi
t.
The Dejaverse (noun): 3D virtual worlds like we’ve seen since 2001 but with better
graphics. Typically ignores dystopian red
fl
ags because “this time it’s di
ff
erent.”
The Meterverse (noun) : a virtual world, potentially overlapping our real lives, in
which everything we do costs us time, money, or privacy.
To Metaverse (verb) : an act of punditry, in which one takes any cool idea from
the last 30 years and calls it “The Metaverse.”
Some Practical Definitions
credits: Matt Miesnieks, @imaginaryZina

49. The Seven Major Verses
Adverse
(directed for
advertisers)
Dejaverse
(revisited
directions)
Transverse
(directed
across)
Metaverse
or
Meterverse
Exoverse
(directed
outside)
Esoverse
(directed
within)
Holiverse
(directed to
wholeness)
We Are Here
Facebook
time

50. Path to and Beyond Meta
Level
Level 0
Adverse
(directed by advertisers)
Level 1
Dejaverse
(revisited directions)
Level 2
Transverse
(directed across)
Level 3
Meta/Meterverse
(directed beyond)
Level 4
Exoverse
(directed outside)
Level 5
Esoverse
(directed within)
Level 99
Holiverse
(directed to wholeness)
Spatial
Metaphors
2D Pages
Web of Links
Parallax 2.5D
Silo 3D Worlds
Virtual Earths
Connected
Virtual Worlds
Virtual Universe
Real World
AR+IOT
AR Interface for
any Reality
Omnipresence
People &
Avatars
Cursors, Names,
Icons
Cartoon Avatars
Video Avatars
Virtual Actors
Portable ID
OK Telepresence
Full Avatar
Telepresence
Holographic
Telepresence
AI Twins
Async Comms
Omniscience
Immortality
Artifacts &
Interaction
Text, Photos,
Videos
Virtual Clothing &
Accessories
Portable
Inventory
(NFT, DAO)
Mashable Assets
Intuitive Search
AI Creation
Assistants
Holodeck
Post Symbolic
Communication
Omnipotence
Context &
Privacy
Cookies
Web Bugs
Global Trackers
Do Not Track
VPNs
Cloaked IDs
Portable
Permissions
Consensual
Reality
Contextual
Discovery
Engines
Personal
AI Shield
Solipsism
or
Hive Mind
Monetization Targeted Ads
Creator
Economies
Distributed
Ownership
Distributed
Authorship Chains
>50% Creatorship >90% Creatorship
Universal
Creatorship
The Levels of Versality
article

51. The Whateverse is a shared reality that we don't care what it's called, just tell us what
it does or does not do.
The Internet of People keeps the proper focus on people, connected
Please also consider:
Virtual Worlds are various spaces we can virtually inhabit
Presence is the feeling of “being” in a particular place (also not in your head)
Co-Presence is the feeling that other people are also present and truly human
Co-Reality is any web of worlds where we can be present/co-present
Better terms?
Whateverse: Rebecca Evans

52. Conclusions
On XR vs. Metaverse Futures
XR Metaverse
It’s About…
People, Perception,
Communication,
Augmented Humanity
Spaces, Commerce, Analysis,
Network E
ff
ects,
Escapism, Transhumanism
Dependencies… Better Hardware & Algorithms
(lower power, cost, weight)
Better Communication
Better Regulation
More Democracy
Public View… Limited Personal Footprint
(e.g., avatar, virtual clothing)
Anything Addressable
(like the internet)
Privacy is Supported by…
Contextual Discovery Engine
Personal AI Shield
Veri
fi
able Credentials, DIDs,
(and god forbid, blockchain)
Augmented
Human
Fostering
Abundance

53. Q & A

54. Bonus

55. Avi Bar-Zeev
President: xrguild.org
Consulting: www.realityprime.com
The Eyes Have It
Deep Dive into Eye Tracking and Mental Privacy in XR
Stanford Existing Law and Extended Reality Conference
Jan 6th, 2023

56. retina
fovea
periphery
periphery
corneal
bulge
lens
Human Visual Perception
The Human Eye

57. Your Eyes
visual targets
Eye Tracking
Cameras
Scene
Understanding
& Tracking
Cameras
Add Facial Computing
AR
Glasses
Scene
Understanding
& Tracking
Cameras
Eye Tracking
Cameras

58. Bene
fi
ts of Eye Tracking
• Foveated Rendering and Compression
• Avatar Eyes / Emotional Fidelity
• Spatial User Interfaces for AR & VR
• Re-directed Walking for VR
• Marketing Research and Lab-Testing Designs
• Diagnosis of Autism, Multiple Sclerosis, Alzheimer’s, ALS, Schizophrenia,
Parkinson’s, ADHD, concussions…

59. Heat Map “Look At"
1 person everyone

60. Saccadic Blindness

61. Redirected Walking

62. Change Blindness
BBC
Royal Institution

63. Human Perception Takeaways
Our brains continually
reconstruct our perception
of the world in small
sections (fovea, saccades)
Our eyes can be easily
distracted, tricked, and lead
We are frequently blind,
allowing the world to be
changed without us noticing
Our eye-gaze interaction
can reveal how we feel
about anything we see
If we allow this biometric
information to leave our
devices, even anonymously,
we can later be re-identi
fi
ed
This forms a robust model of
our individual personality and
innate reactions that can be
used to manipulate us

64. The Darker Side
• Security of Biometric Signatures
• Emotion-Mapping Personality Pro
fi
les
• Medical-Grade Invasions of Privacy
• Non-consensual Experimentation
• Real-Time Emotional Manipulation

65. The Guardian

66. Identity
You can change your password…
Unique to You:
Sclera (blood vessels)
Retina (vessels, topology)
Lens (distortion)
Iris Pattern (color+IR)
Eye Motion, Shape, Position
The high-
fi
delity info should never
leave the device, even anonymized
(because you can easily be re-identi
fi
ed).
infrared

67. Raw and Derived Eye Tracking Data
Multiple Streams of Concern
Raw 3D eye rotations plus 6D head position -> what did you look at?
Time/Motion analysis -> did you ignore, notice or focus on it?
Pupil analysis -> how excited or motivated were you? (involuntary reflex)
ML classification -> how did you feel? (confused, upset; involuntary)
Eyebrows / eyelids -> how did you respond? (happy, angry; voluntary)
This data also supports medical diagnosis of Autism, Multiple Sclerosis,
Alzheimer’s, ALS, Schizophrenia, Parkinson’s, ADHD, concussions…

68. “Half the money I spend on advertising is wasted
The trouble is, I don’t know which half.”
– John Wanamaker

69. Feedback Loops
The subtle cycle of behavioral manipulation
Improve Model Predicting
Behavior (given stimuli)
Change Environment to Trigger
or Test New Stimuli
Collect Cognitive & Emotional
Responses to Stimuli
In the Lab
In the Lab
In the Lab

71. Does “Opt-In” Cover Privacy Concerns?
One step beyond contracts of adhesion
Consumers will almost certainly opt-into eye
tracking for the obvious benefits without
understanding the risks. Just like “click-wrap!”
First and third party apps could theoretically
off-load this data for almost any other use.
Permission per app helps a bit. But how do
we know which apps to trust or not trust?

72. Does “Opt-In” Cover Privacy Concerns?
It’s several steps too short
1. Ask consumers for permission based on
clearly explained use cases, not just by app
name. Apps must declare ‘intents’ to use
sensitive data for specific use cases too.
2. Require AAA security and E2E encryption
for all data sharing, even first-party.
3. The personal device holds the keys that can
audit or invalidate remote copies of this data.
Permission to use my Eye Tracking for:
Animating my avatar’s eyes
Improving 3D rendering performance
Marketing Studies (anon, off-device)
Diagnostic Health Studies (for ALS)
VR-chat (avatar eyes, E2E encrypted)
AAA = Authentication, Authorization and Accounting
E2E = End to End Encryption

73. "The Game Changers” (Net
fl
ix)
Duracell Battery Commercial
“How Juul made nicotine go viral” (Vox, Youtube)
Have you ever been Manipulated?