The second lecture from the Augmented Reality Summer School talk by Mark Billinghurst at the University of South Australia, February 15th - 19th, 2016. This provides an overview of AR Technology.
1. LECTURE 2: AUGMENTED
REALITY TECHNOLOGY
Mark Billinghurst
AR Summer School
February 15th – 19th 2016
University of South Australia
2. Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
3. Augmented RealityTechnology
! Combining Real and Virtual Images
• Display technologies
! Interactive in Real-Time
• Input and interactive technologies
! Registered in 3D
• Viewpoint tracking technologies
Display
Processing
Input Tracking
5. AR Displays
e.g. window
reflections
Virtual Images
seen off windows
e.g. Reach-In
Projection CRT Display
using beamsplitter
Not Head-Mounted
e.g. Shared Space
Magic Book
Liquid Crystal
Displays LCDs
Head-Mounted
Display (HMD)
Primarily Indoor
Environments
e.g. WLVA
and IVRD
Cathode Ray Tube (CRT)
or Virtual Retinal Display (VRD)
Many Military Applications
& Assistive Technologies
Head-Mounted
Display (HMD)
e.g. Head-Up
Display (HUD)
Projection Display
Navigational Aids in Cars
Military Airborne Applications
Not Head Mounted
(e.g. vehicle mounted)
Primarily Outdoor
(Daylight) Environments
AR
Visual Displays
6. Display Technologies
! Types (Bimber/Raskar 2003)
! Head attached
• Head mounted display/projector
! Body attached
• Handheld display/projector
! Spatial
• Spatially aligned projector/monitor
9. Head Mounted Displays (HMD)
• Display and Optics mounted on Head
• May or may not fully occlude real world
• Provide full-color images
• Considerations
• Cumbersome to wear
• Brightness
• Low power consumption
• Resolution limited
• Cost is high?
10. Key Properties of HMD
• Field ofView
• Human eye 95 degrees horizontal, 60/70 degrees vertical
• Resolution
• > 320x240 pixel
• Refresh Rate
• Focus
• Fixed/manual
• Power
• Size
11. Types of Head Mounted Displays
Occluded
See-thru
Multiplexed
18. TheVirtual Retinal Display
• Image scanned onto retina
• Commercialized through Microvision
• Nomad System - www.mvis.com
19. Strengths of optical see-throughAR
• Simpler (cheaper)
• Direct view of real world
• Full resolution, no time delay (for real world)
• Safety
• Lower distortion
• No eye displacement
• but COASTAR video see-through avoids this
25. Strengths ofVideo See-ThroughAR
• True occlusion
• Kiyokawa optical display that supports occlusion
• Digitized image of real world
• Flexibility in composition
• Matchable time delays
• More registration, calibration strategies
• Wide FOV is easier to support
26. Optical vs.VideoAR Summary
• Both have proponents
• Video is more popular today?
• Likely because lack of available optical products
• Depends on application?
• Manufacturing: optical is cheaper
• Medical: video for calibration strategies
33. DisplayTechnology
• Curved Mirror
• off-axis projection
• curved mirrors in front of eye
• high distortion, small eye-box
• Waveguide
• use internal reflection
• unobstructed view of world
• large eye-box
34. See-through thin displays
• Waveguide techniques for thin see-through displays
• Wider FOV, enable AR applications
• Social acceptability
Opinvent Ora
Lumus DK40
49. Virtual Showcase
• Mirrors on a projection table
• Head tracked stereo
• Up to 4 users
• Merges graphic and real objects
• Exhibit/museum applications
• Fraunhofer Institute (2001)
• Bimber, Frohlich
58. Objects Registered in 3D
• Registration
• Positioning virtual object wrt real world
• Tracking
• Continually locating the users viewpoint
• Position (x,y,z), Orientation (r,p,y)
59. Tracking Requirements
• Augmented Reality Information Display
• World Stabilized
• Body Stabilized
• Head Stabilized
Increasing Tracking
Requirements
Head Stabilized Body Stabilized World Stabilized
63. MagneticTracker
• Idea: difference between a magnetic
transmitter and a receiver
• ++: 6DOF, robust
• -- : wired, sensible to metal, noisy, expensive
Flock of Birds (Ascension)
65. Global Positioning System (GPS)
• Created by US in 1978
• Currently 29 satellites
• Satellites send position + time
• GPS Receiver positioning
• 4 satellites need to be visible
• Differential time of arrival
• Triangulation
• Accuracy
• 5-30m+, blocked by weather, buildings etc
66. InertialTracker
• Idea: measuring linear and angular orientation rates
(accelerometer/gyroscope)
• ++: no transmitter, cheap, small, high frequency, wireless
• -- : drift, hysteris only 3DOF
IS300 (Intersense)
Wii Remote
67. Mobile Sensors
• Inertial compass
• Earth’s magnetic field
• Measures absolute orientation
• Accelerometers
• Measures acceleration about axis
• Used for tilt, relative rotation
• Can drift over time
71. HiBallTracking System (3rd Tech)
• Inside-Out Tracker
• $50K USD
• Scalable over large area
• Fast update (2000Hz)
• Latency Less than 1 ms.
• Accurate
• Position 0.4mm RMS
• Orientation 0.02° RMS
72.
73. Marker tracking
• Available for more than 10 years
• Several open source solutions exist
• ARToolKit,ARTag,ATK+, etc
• Fairly simple to implement
• Standard computer vision methods
• A rectangle provides 4 corner points
• Enough for pose estimation!
76. Tracking challenges inARToolKit
False positives and inter-marker confusion
(image by M. Fiala)
Image noise
(e.g. poor lens, block
coding /
compression, neon tube)
Unfocused camera,
motion blur
Dark/unevenly lit
scene, vignetting
Jittering
(Photoshop illustration)
Occlusion
(image by M. Fiala)
77.
78. Markerless Tracking
Magnetic Tracker Inertial
Tracker
Ultrasonic
Tracker
Optical
Tracker
Marker-Based
Tracking
Markerless
Tracking
Specialized
Tracking
Edge-Based
Tracking
Template-Based
Tracking
Interest Point
Tracking
• No more Markers! #Markerless Tracking
Mechanical
Tracker
79. Natural Feature Tracking
• Use Natural Cues of Real Elements
• Edges
• Surface Texture
• Interest Points
• Model or Model-Free
• No visual pollution
Contours
Features Points
Surfaces
84. Marker vs.natural feature tracking
• Marker tracking
• ++ Markers can be an eye-catcher
• ++ Tracking is less demanding
• -- The environment must be instrumented with markers
• -- Markers usually work only when fully in view
• Natural feature tracking
• -- A database of keypoints must be stored/downloaded
• ++ Natural feature targets might catch the attention less
• ++ Natural feature targets are potentially everywhere
• ++ Natural feature targets work also if partially in view
85. Example: Outdoor Hybrid Tracking
• Combines
• computer vision
• natural feature tracking
• inertial gyroscope sensors
• Both correct for each other
• Inertial gyro - provides frame to frame
prediction of camera orientation
• Computer vision - correct for gyro drift
91. The Registration Problem
• Virtual and Real must stay properly aligned
• If not:
• Breaks the illusion that the two coexist
• Prevents acceptance of many serious applications
92. Sources of registration errors
• Static errors
• Optical distortions
• Mechanical misalignments
• Tracker errors
• Incorrect viewing parameters
• Dynamic errors
• System delays (largest source of error)
• 1 ms delay = 1/3 mm registration error
95. Dynamic errors
• Total Delay = 50 + 2 + 33 + 17 = 102 ms
• 1 ms delay = 1/3 mm = 33mm error
Tracking Calculate
Viewpoint
Simulation
Render
Scene
Draw to
Display
x,y,z
r,p,y
Application Loop
20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
96. Reducing dynamic errors (1)
• Reduce system lag
• Faster components/system modules
• Reduce apparent lag
• Image deflection
• Image warping
97. Reducing System Lag
Tracking Calculate
Viewpoint
Simulation
Render
Scene
Draw to
Display
x,y,z
r,p,y
Application Loop
Faster Tracker Faster CPU Faster GPU Faster Display
99. Reducing dynamic errors (2)
• Match input streams (video)
• Delay video of real world to match system lag
• Predictive Tracking
• Inertial sensors helpful
Azuma / Bishop 1994