This document provides an introduction to extended reality technologies from Mark Billinghurst, the director of the Empathic Computing Lab at the University of South Australia. It outlines Billinghurst's background and research interests. It then provides an overview of the class, including assignments, equipment available, and the lecture schedule. The lecture schedule covers topics such as augmented reality, virtual reality, the metaverse, and the history of AR/VR.
1. INTRODUCTION TO XR
COMP 4010 Lecture One
Mark Billinghurst
July 28th 2022
mark.billinghurst@unisa.edu.au
2. Who am I ..
• Mark Billinghurst
• Director of the Empathic Computing Lab
• Univ. South Australia, Univ. of Auckland
• Conducting research in:
• Collaborative AR, AR/VR Interface Design, HCI
• Empathic Computing
• Previous worked at:
• Google, Amazon, Nokia, British Telecom
• MIT, Univ. of Washington, NAIST, Univ. of Canterbury
• PhD from University of Washington (2002)
9. Assessment
• Assignment 1: Mobile AR (20%)
• Develop a Snap Lens
• AR Tracking and Interaction
• Assignment 2: Interactive VR (30%)
• Create an interactive VR scene
• Oculus Quest, HTC Vive, Rift
• Assignment 3: HMD Experience (40%)
• Develop either VR HMD experience
• Oculus Quest
• Class Participation (10%)
• Watch at least one short YouTube AR/VR video each week
• Be prepared to present video each week
13. The Incredible Disappearing Computer
1960-70’s
Room
1970-80’s
Desk
1980-90’s
Lap
1990-2000’s
Hand
2010 -
Head
14. Graphical User Interfaces
• Separation between real and digital worlds
• WIMP (Windows, Icons, Menus, Pointer) metaphor
15. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments.
Making Interfaces Invisible
(c) Internet of Things
16. Internet of Things (IoT)..
• Embed computing and sensing in real world
• Smart objects, sensors, etc..
(c) Internet of Things
17. Virtual Reality (VR)
• Users immersed in Computer Generated environment
• HMD, gloves, 3D graphics, body tracking
18. The First VR Experience …
https://www.youtube.com/watch?v=pAC5SeNH8jw
19. Virtual Reality Definition
•Defining Characteristics
• Sense of Immersion
• User feels immersed in computer generated space
• Interactive in real-time
• The virtual content can be interacted with
• Independence
• User can have independent view and reaction to environment
20. David Zeltzer’s AIP Cube
Autonomy – User can to react to events
and stimuli.
Interaction – User can interact with
objects and environment.
Presence – User feels immersed through
sensory input and output channels
Interaction
Autonomy
Presence
VR
Zeltzer, D. (1992). Autonomy, interaction, and presence. Presence: Teleoperators & Virtual Environments, 1(1),127-132.
23. Augmented Reality (AR)
• Virtual Images blended with the real world
• See-through HMD, handheld display, viewpoint tracking, etc..
24. Augmented Reality Definition
•Defining Characteristics [Azuma 97]
• Combines Real and Virtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
29. • Weak AR
• Imprecise tracking
• No knowledge of environment
• Limited interactivity
• Handheld AR
• Strong AR
• Very accurate tracking
• Seamless integration into real world
• Natural interaction
• Head mounted AR
Strong vs. Weak AR
32. From Reality to Virtual Reality
Internet of Things Augmented Reality Virtual Reality
Real World Virtual World
33. Milgram’s Mixed Reality (MR) Continuum
Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays
Internet of Things
34. Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
39. Mixed Reality Revisited (2019)
• Interviewed experts + read papers
• ”.. no universally agreed on, one-size-fits-
all definition of MR”
• Six notions of MR
• Continuum, Synonym, Collaboration
• Combination, Alignment, Strong AR
Speicher, M., Hall, B. D., & Nebeling, M. (2019,
May). What is mixed reality?. In Proceedings of the
2019 CHI conference on human factors in
computing systems (pp. 1-15).
41. The Metaverse
• Neal Stephenson’s “SnowCrash” (1992)
”.. When you live in a shithole, there's always the
Metaverse, and in the Metaverse, Hiro
Protagonist is a warrior prince.”
42. Defining the Metaverse
• Real-time, 3D, Interactive, Social, Persistent
AWE 2022 John Riccitiello
CEO, Unity Technologies
43. Metaverse Definition (2007)
• Metaverse Roadmap
• http://metaverseroadmap.org/
• The Metaverse is the convergence of:
• 1) virtually enhanced physical reality
• 2) physically persistent virtual space
44. Metaverse Dimensions
• Augmentation technologies that layer information onto our
perception of the physical environment.
• Simulation refers to technologies that model reality
• Intimate technologies are focused inwardly, on the identity
and actions of the individual or object;
• External technologies are focused outwardly, towards the
world at large;
46. Mirror Worlds
• Simulations of external space/content
• Capturing and sharing surroundings
• Photorealistic content
• Digital twins
Matterport Deep Mirror Google Street View
Soul Machines
47. Lifelogging
• Measuring user’s internal state
• Capturing physiological cues
• Recording everyday life
• Augmenting humans
Apple Fitbit Shimmer
OpenBCI
51. When anything new comes along, everyone, like a child
discovering the world thinks that they’ve invented it, but you
scratch a little and you find a caveman scratching on a wall
is creating virtual reality in a sense.
Morton Helig (Hammit 1993)
52. Early History (30,000 BC - )
The history of VR is rooted in human’s first
attempts to reproduce the world around them
56. 3D Cinema Golden Era (1950-60s)
• Polarized 3D projection or anaglyph (red/blue)
57. Pepper’s Ghost (1862)
• Projecting onto glass to make ghost image appear on stage
• Dates back to Giambattista della Porta (1584)
58. The Master Key (1901) – AR Glass
"It consists of this pair of spectacles. While you
wear them every one you meet will be marked
upon the forehead with a letter indicating his or
her character. The good will bear the letter 'G,'
the evil the letter 'E.' … Thus you may
determine by a single look the true natures of
all those you encounter.”
L. Frank Baum
AR display showing if people are good or evil
59. 1900s – Interactive Experiences
• Early Simulators (<1960s)
• Flight simulation
• Sensorama (1955)
• Early HMDs (1960s)
• Philco, Ivan Sutherland
• Military + University Research (1970-80s)
• US Airforce, NASA, MIT, UNC
• First Commercial Wave (1980-90s)
• VPL, Virtual i-O, Division, Virtuality
• VR Arcades, Virtual Boy
60. Link Trainer (1929 – 1950s)
• Flight Simulator Training
• Full six degree of freedom rotation
• Force feedback and motion control
• Simulated instruments
• Modeling common flight conditions
• Over 500,000 pilots trained
68. Super Cockpit (1965-80’s)
• US Airforce Research Program
• Wright Patterson Air Force Base
• Tom Furness III
• Multisensory
• Visual, auditory, tactile
• Head, eye, speech, and hand input
• Addressing pilot information overload
• Flight controls and tasks too complicated
• Research only
• big system, not safe for ejecting
69.
70. LEEP Optics (1979)
• Large Expanse, Extra Perspective optics
• Developed by Eric Howlett
• Lens design for extremely wide field of view
• High resolution in centre, lower resolution in periphery
• 90o direct FOV, 140o corneal FOV
• Used as basis for most VR HMDs
75. VPL Research (1985 – 1999)
• First Commercial VR Company
• Jaron Lanier, Jean-Jacques Grimaud
• Provided complete systems
• Displays, software, gloves, etc
• DataGlove, EyePhone, AudioSphere
76. The University of North Carolina
at Chapel Hill (1980s-1990s)
7
6
Head-Mounted Displays
Tracking, Haptics, Applications
77. University of Washington (1989 - )
• Human Interface Technology Laboratory (HIT Lab)
• Founded by Tom Furness III
• Many AR/VR Innovations
• Virtual Retinal Display
• ARToolKit AR Tracking library
• GreenSpace shared VR experience
• VR and pain care
• VR and Education
78. First Industrial Use of AR (1990’s)
• 1992: Tom Caudell at Boeing coined the term “AR.”
• Wire harness assembly application begun
• Lead by Tom Caudell, and David Mizell
79. CAVE (1992)
• Projection VR system
• 3-6 wall stereo projection, viewpoint tracking
• Developed at EVL, University of Illinois Chicago
• Commercialized by Mechdyne Corporation(1996)
C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V. Kenyon and J. C. Hart. "The CAVE: Audio Visual
Experience Automatic Virtual Environment", Communications of the ACM, vol. 35(6), 1992, pp. 64–72.
87. Development of AR Tools
• 1996 CyberCode (Rekimoto)
• First matrix code tracking
• 1999 ARToolKit (Kato & Billinghurst)
• Open source tracking library
90. • April 2007 Computer World
• VRVoted 7th on list of 21 biggest technology flops
• MS Bob #1
91. Lessons Learned
• Don’t believe the hype
• Not everything is better inVR
• Many factors determine technology acceptance
• Human Centered Design/Design for users
• Need to move from Demo to Production
• Profitable niche markets first
• Follow the money
92. Mobile Phone AR (2005)
• Mobile Phones
• camera
• processor
• display
• AR on Mobile Phones
• Simple graphics
• Optimized computer vision
• Collaborative Interaction
93. AR Advertising (HIT Lab NZ 2007)
• Txt message to download AR application (200K)
• See virtual content popping out of real paper advert
• Tested May 2007 by Saatchi and Saatchi
95. Eye of judgement (2007)
• Sony Playstation3 game
• First AR console game
• Over 300,000 copies sold
• Used Eye camera + tracking cards
96. 2007 - AR Reaches Mainstream
• MIT Technology Review
• March 2007
• One of the 10 most exciting
technologies
• Economist
• Dec 6th 2007
• Reality, only better
97. Google Searches for AR
• Cross over in 2009, with more interest in AR than VR
98. 2008 - Browser BasedAR
• Flash + camera + 3D graphics
• ARToolKit ported to Flash
• High impact
• High marketing value
• Large potential install base
• 1.6 Billion web users
• Ease of development
• Lots of developers, mature tools
• Low cost of entry
• Browser, web camera
99. Demo: GE Smart Grid
• https://www.youtube.com/watch?v=vJO_AZkCL9U
101. Outdoor Information Overlay
• Mobile phone based
• Tag real world locations
• GPS + Compass input
• Overlay graphics on live video
• Applications
• Travel guide, Advertising, etc.
• Wikitude, Metaio, Layar, etc..
• iOS/Android, Public API released
118. Social Mobile Camera AR Apps (2015 - )
• SnapChat - Lenses, World Lenses
• Cinco de Mayo lens > 225 million views
• Facebook - Camera Effects
• Google – Word Lens/Translate
119. Hololens (2016)
• Integrated system – Windows
• Stereo see-through display
• Depth sensing tracking
• Voice and gesture interaction
• Note: Hololens2 coming September 2019
120. ARKit/ARcore (2017)
• Visual Inertial Odometry (VIO) systems
• Mobile phone pose tracked by
• Camera (Visual), Accelerometer & Gyroscope (Intertial)
• Features
• Plane detection, lighting detection, hardware optimisation
• Links
• https://developer.apple.com/arkit/
• https://developers.google.com/ar/
125. Resources
• Azuma, R. T. (1997). A survey of augmented reality. Presence: teleoperators & virtual
environments, 6(4), 355-385.
• Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent
advances in augmented reality. IEEE computer graphics and applications, 21(6), 34-47.
• Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE
TRANSACTIONS on Information and Systems, 77(12), 1321-1329.
• Billinghurst, M., Clark, A., & Lee, G. (2015). A Survey of Augmented Reality. Foundations and
Trends in Human–Computer Interaction, 8(2-3), 73-272.
• Speicher, M., Hall, B. D., & Nebeling, M. (2019, May). What is mixed reality?. In Proceedings
of the 2019 CHI conference on human factors in computing systems (pp. 1-15).