Mark Billinghurst, profile picture
Uploaded byMark Billinghurst
PDF, PPTX4,474 views

Lecture 9 AR Technology

The document presents a lecture on augmented reality (AR) technology, discussing its history, developments, and applications from the 1960s to today. Key components of AR, such as tracking technologies, display types, and interaction methods are examined, along with various use cases in gaming, medicine, and marketing. The lecture also contrasts strong and weak AR experiences, highlighting current advancements and future growth areas in the AR commercial landscape.

Embed presentation

Download as PDF, PPTX
1 / 141
2 / 141
3 / 141
4 / 141
5 / 141
6 / 141
7 / 141
8 / 141
9 / 141
10 / 141
11 / 141
12 / 141
13 / 141
14 / 141
15 / 141
16 / 141
17 / 141
18 / 141
19 / 141
20 / 141
21 / 141
22 / 141
23 / 141
24 / 141
25 / 141
26 / 141
27 / 141
28 / 141
29 / 141
30 / 141
31 / 141
32 / 141
33 / 141
34 / 141
35 / 141
36 / 141
37 / 141
38 / 141
39 / 141
40 / 141
41 / 141
42 / 141
43 / 141
44 / 141
45 / 141
46 / 141
47 / 141
48 / 141
49 / 141
50 / 141
51 / 141
52 / 141
53 / 141
54 / 141
55 / 141
56 / 141
57 / 141
58 / 141
59 / 141
60 / 141
61 / 141
62 / 141
63 / 141
64 / 141
65 / 141
66 / 141
67 / 141
68 / 141
69 / 141
70 / 141
71 / 141
72 / 141
73 / 141
74 / 141
75 / 141
76 / 141
77 / 141
78 / 141
79 / 141
80 / 141
81 / 141
82 / 141
83 / 141
84 / 141
85 / 141
86 / 141
87 / 141
88 / 141
89 / 141
90 / 141
91 / 141
92 / 141
93 / 141
94 / 141
95 / 141
96 / 141
97 / 141
98 / 141
99 / 141
100 / 141
101 / 141
102 / 141
103 / 141
104 / 141
105 / 141
106 / 141
107 / 141
108 / 141
109 / 141
110 / 141
111 / 141
112 / 141
113 / 141
114 / 141
115 / 141
116 / 141
117 / 141
118 / 141
119 / 141
120 / 141
121 / 141
122 / 141
123 / 141
124 / 141
125 / 141
126 / 141
127 / 141
128 / 141
129 / 141
130 / 141
131 / 141
132 / 141
133 / 141
134 / 141
135 / 141
136 / 141
137 / 141
138 / 141
139 / 141
140 / 141
141 / 141
LECTURE 9: AR TECHNOLOGY COMP 4010 – Virtual Reality Semester 5 – 2019 Bruce Thomas, Mark Billinghurst, Gun Lee University of South Australia Oct 8th 2019
Lecture 8: Recap • Augmented Reality • Defining Characteristics [Azuma 97]: •Combines Real and Virtual Images • Both can be seen at the same time •Interactive in real-time • The virtual content can be interacted with •Registered in 3D • Virtual objects appear fixed in space
Augmented Reality 1977 – Star Wars
History Summary •1960’s – 80’s: Early Experimentation •1980’s – 90’s: Basic Research • Tracking, displays •1995 – 2005: Tools/Applications • Interaction, usability, theory •2005 - : Commercial Applications • Games, Medical, Industry
2005 - Mobile Phone AR •Mobile Phones • camera • processor • display •AR on Mobile Phones • Simple graphics • Optimized computer vision • Collaborative Interaction
2008 - Browser Based AR • Flash + camera + 3D graphics • ARToolKit ported to Flash • High impact • High marketing value • Large potential install base • 1.6 Billion web users • Ease of development • Lots of developers, mature tools • Low cost of entry • Browser, web camera
• Weak AR • Imprecise tracking • No knowledge of environment • Limited interactivity • Handheld AR • Strong AR • Very accurate tracking • Seamless integration into real world • Natural interaction • Head mounted AR Strong vs. Weak AR
Augmented Reality Today •Key Technologies Available • Robust tracking (Computer Vision, GPS/sensors) • Display (Handheld, HMDs) • Input Devices (Kinect, etc) • Developer tools (Vuforia, ARToolKit) •Commercial Business Growing • Gaming, GPS/Mobile, Online Advertisement • >$3Billion USD in 2017
AR Business Today • Marketing • Web-based, mobile • Mobile AR • Geo-located information and service • Driving demand for high end phones • Gaming • Mobile, Physical input (Kinect, PS Move) • Upcoming areas • Manufacturing, Medical, Military
AR Commercial Landscape
• Web based AR • Flash, HTML 5 based AR • Marketing, education • Outdoor Mobile AR • GPS, compass tracking • Viewing Points of Interest in real world • Eg: Junaio, Layar, Wikitude • Handheld AR • Vision based tracking • Marketing, gaming • Location Based Experiences • HMD, fixed screens • Museums, point of sale, advertising Typical AR Experiences
AR TECHNOLOGY
Augmented Reality 1977 – Star Wars
Technology Requirements • Combining Real and Virtual Images • Needs - Display technologies • Interactive in Real-Time • Needs - Input and interactive technologies • Registered in 3D • Needs - Viewpoint tracking technologies Display Processing Input Tracking
AR DISPLAYS
Display Technologies • Types (Bimber/Raskar 2005) • Head attached • Head mounted display/projector • Body attached • Handheld display/projector • Spatial • Spatially aligned projector/monitor
Bimber, O., & Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press. DisplayTaxonomy
Great Books for Background • Spatial Augmented Reality, Displays; Fundamentals & Applications • Both by Oliver Bimber
HEAD MOUNTED DISPLAYS
Head Mounted Displays (HMD) • Display and Optics mounted on Head • May or may not fully occlude real world • Provide full-color images • Things to considerations • Wearability • Brightness • Power consumption • Resolution, Field of View • Cost
Types of Head Mounted Displays Occluded See-thru Multiplexed
Optical See-Through HMD (OSTHMD) Virtual images from monitors Real World Optical Combiners
Optical See-Through HMD
Epson Moverio BT-35E ▪ Stereo see-through display ($1,200 AUD) ▪ 1280 RGB x 720 pixels, 23 degree FOV, 30Hz, ▪ USB-C, HDMI connectors ▪ 5mp camera, GPS, gyro, accelerometer
View Through Optical See-Through HMD Big data visualization on Microsoft HoloLens
Other Optical See-Through Displays • Microsoft HoloLens - $3,000 USD • Wearable computer • Waveguide displays • https://www.microsoft.com/hololens • Meta Meta2 - $1,500 USD • Wide field of view (90 degrees) • Tethered display • https://www.metavision.com/ • Mira Prism - $100 USD • Smart phone based • Wide field of view • https://www.mirareality.com/
Project North Star (Leap Motion) • http://blog.leapmotion.com/northstar/ • Open source OSTHMD • 100 degree FOV, 120 Hz, 2 x 1600x1440 pixel displays • Hand tracking using Leap Motion hardware
Demo: Project North Star • https://www.youtube.com/watch?v=yN0EGBo74j8
Strengths and Weaknesses • Pros • Simpler design (cheaper) • Direct view of real world • Full resolution, no time delay (for real world) • Safety, Lower distortion • No eye displacement • Cons • No occlusion of real world (image see-through) • Wide field of view can be difficult
Optics Design for OSTHMD
DisplayTechnology • Curved Mirror • off-axis projection • curved mirrors in front of eye • high distortion, small eye-box • Waveguide • use internal reflection • unobstructed view of world • large eye-box
Example: Sony Waveguide Display • https://www.youtube.com/watch?v=G2MtI7asLcA
See-throughThin Displays (Waveguide) • Waveguide techniques for thin see-through displays • Wider FOV, enable AR applications • Social acceptability Opinvent Ora Lumus DK40
Example: Sony Smart EyeGlasses • https://www.youtube.com/watch?v=kYPWaMsarss
Video See-Through HMD (VSTHMD) Video cameras Monitors Graphics Combiner Video
Video See-Through HMD
Vuzix Wrap 1200DXAR (Discontinued) ▪ Stereo video see-through display ($1500) ■Twin 852 x 480 LCD displays, 35 deg. FOV ■Stereo VGA cameras ■3 DOF head tracking ■https://www.vuzix.com/Products/LegacyProduct/4
View Through a Video See-Through HMD
Varjo XR-1 • https://varjo.com/products/xr-1/ • Video pass through • 2 x 12 megapixel cameras at 90 Hz, < 15 ms latency • 80 degree FOV, 1440 x 1600 resolution per eye
• https://www.youtube.com/watch?v=vaLMp_mqetI
• Pros • True occlusion • AR pixels block video pixels • Digitized image of real world • Flexibility in composition, match time delays • Wide FOV is easier to support • Wide FOV camera • Cons • No direct view, latency Strengths and Weaknesses
VST vs. OST HMD VSTHMD OSTHMD
Eye Multiplexed Display • See content in periphery • Virtual image “inset” into the real world
Example:Google Glass
ViewThrough Google Glass https://www.youtube.com/watch?v=fNATuCkRWFE
Vuzix M-100 ▪ Monocular multiplexed display ($1000) ■852 x 480 LCD display, 15 deg. FOV ■5 MP camera, HD video ■GPS, gyro, accelerometer
SPATIAL AUGMENTED REALITY
Spatial Augmented Reality • Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber, Rasker “Spatial Augmented Reality”
Projector-based AR Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo Projector Real objects with retroreflective covering User (possibly head-tracked)
Example of Projector-BasedAR Ramesh Raskar, MIT
Example of Projector-BasedAR Ramesh Raskar, MIT
Head Mounted Projector • NVIS P-50 HMPD • 1280x1024/eye • Stereoscopic • 50 degree FOV • www.nvis.com
HMD vs.HMPD Head Mounted Display Head Mounted Projected Display
Tilt5 - https://www.tiltfive.com/ • Stereo head worn projectors • Interactive wand • Rollible retro-reflective sheet • Designed for shared interaction
Demo: Tilt5 https://www.youtube.com/watch?v=XPOgLt_73cY
OTHER AR DISPLAYS
Video MonitorAR Video cameras Monitor Graphics Combiner Video Stereo glasses
Examples
Alternate Displays LCD Panel Laptop PDA
Handheld Displays •Mobile Phones • Camera • Display • Input
OtherTypes ofAR Display • Audio • spatial sound • ambient audio • Tactile • physical sensation • Haptic • virtual touch
AR INTERACTION
Interacting with AR Content • You can see AR images .. how can you interact with it?
Typical Interaction Methods • Handheld Display (phone, tablet) • Touch screen, device motion • Head mounted display • Handheld controller • Speech, gesture, touch • Physical manipulation • Spatial AR • Body motion • Physical manipulation
Gestural interfaces • 1. Micro-gestures • (unistroke, smartPad) • 2. Device-based gestures • (tilt based examples) • 3. Embodied interaction • (eye toy)
Mobile Device Motion • Interact by moving handheld display • Use IMU to detect shaking, rotation, etc • Treat device as controller
Example: AR Paparazzi Game • https://www.youtube.com/watch?v=MIGH5WGMnbs
Tangible Interaction • Use objects in the real world to interact with AR content • Attach AR content to real objects
Example: Stic Stac https://vimeo.com/295222550
HMD Gesture Interaction • Simple click gestures • Head point and air tap • Natural two handed gestures • Full hand tracking, direction manipulation
Hololens Gestures • Set of basic gestures • Air Tap = select, Bloom = open • Easy to remember and execute BloomAir Tap
https://www.youtube.com/watch?v=wogJv5v9x-s
Embodied Interaction • Use whole body to interact with AR • Popular for “Magic Mirror” Interface • Natural Interaction
• https://www.youtube.com/watch?v=UXIP2AoQPlU
AR TRACKING AND REGISTRATION
AR RequiresTracking and Registration • Registration • Positioning virtual object wrt real world • Fixing virtual object on real object when view is fixed • Tracking • Continually locating the users viewpoint when view moving • Position (x,y,z), Orientation (r,p,y)
AR TRACKING
Tracking Requirements • Augmented Reality Information Display • World Stabilized • Body Stabilized • Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized
Tracking Technologies § Active • Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location § Passive • Inertial sensors (compass, accelerometer, gyro) • Computer Vision • Marker based, Natural feature tracking § Hybrid Tracking • Combined sensors (eg Vision + Inertial)
Tracking Types Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking Mechanical Tracker
MechanicalTracker •Idea: mechanical arms with joint sensors •++: high accuracy, haptic feedback •-- : cumbersome, expensive Microscribe
MagneticTracker • Idea: coil generates current when moved in magnetic field. Measuring current gives position and orientation relative to magnetic source. • ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive Flock of Birds (Ascension)
InertialTracker • Idea: measuring linear and angular orientation rates (accelerometer/gyroscope) • ++: no transmitter, cheap, small, high frequency, wireless • -- : drifts over time, hysteresis effect, only 3DOF IS300 (Intersense) Wii Remote
UltrasonicTracker • Idea: time of Flight or phase-Coherence Sound Waves • ++: Small, Cheap • -- : 3DOF, Line of Sight, Low resolution, Affected by environmental conditons (pressure, temperature) Ultrasonic Logitech IS600
Global Positioning System (GPS) • Created by US in 1978 • Currently 29 satellites • Satellites send position + time • GPS Receiver positioning • 4 satellites need to be visible • Differential time of arrival • Triangulation • Accuracy • 5-30m+, blocked by weather, buildings etc.
Mobile Sensors • Inertial compass • Earth’s magnetic field • Measures absolute orientation • Accelerometers • Measures acceleration about axis • Used for tilt, relative rotation • Can drift over time
OPTICAL TRACKING
Why Optical Tracking for AR? • Many AR devices have cameras • Mobile phone/tablet, Video see-through display • Provides precise alignment between video and AR overlay • Using features in video to generate pixel perfect alignment • Real world has many visual features that can be tracked from • Computer Vision well established discipline • Over 40 years of research to draw on • Old non real time algorithms can be run in real time on todays devices
Common AR Optical Tracking Types • Marker Tracking • Tracking known artificial markers/images • e.g. ARToolKit square markers • Markerless Tracking • Tracking from known features in real world • e.g. Vuforia image tracking • Unprepared Tracking • Tracking in unknown environment • e.g. SLAM tracking
Marker Tracking • Available for more than 10 years • Several open source solutions • ARToolKit, ARTag, ATK+, etc • Fairly simple to implement • Standard computer vision methods • A rectangle provides 4 corner points • Enough for pose estimation!
Demo: ARToolKit https://www.youtube.com/watch?v=TqGAqAFlGg0
Marker Based Tracking: ARToolKit http://www.artoolkit.org
Tracking challenges inARToolKit False positives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)
Other MarkerTracking Libraries
But - You can’t cover world with ARToolKit Markers!
Markerless Tracking Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking • No more Markers! èMarkerless Tracking Mechanical Tracker
Natural Feature Tracking • Use Natural Cues of Real Elements • Edges • Surface Texture • Interest Points • Model or Model-Free • No visual pollution Contours Features Points Surfaces
TextureTracking
Tracking by Keypoint Detection • This is what most trackers do… • Targets are detected every frame • Popular because tracking and detection are solved simultaneously Keypoint detection Descriptor creation and matching Outlier Removal Pose estimation and refinement Camera Image Pose Recognition
What is a Keypoint? • It depends on the detector you use! • For high performance use the FAST corner detector • Apply FAST to all pixels of your image • Obtain a set of keypoints for your image • Describe the keypoints Rosten, E., & Drummond, T. (2006, May). Machine learning for high-speed corner detection. In European conference on computer vision (pp. 430-443). Springer Berlin Heidelberg.
FAST Corner Keypoint Detection
Example:FAST Corner Detection https://www.youtube.com/watch?v=pJ2xSrIXy_s
Descriptors • Describe the Keypoint features • Can use SIFT • Estimate the dominant keypoint orientation using gradients • Compensate for detected orientation • Describe the keypoints in terms of the gradients surrounding it Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D., Real-Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010
Demo: Vuforia Texture Tracking https://www.youtube.com/watch?v=1Qf5Qew5zSU
Edge Based Tracking • Example: RAPiD [Drummond et al. 02] • Initialization, Control Points, Pose Prediction (Global Method)
Demo: Edge Based Tracking
Line Based Tracking • Visual Servoing [Comport et al. 2004]
Model BasedTracking • Tracking from 3D object shape • Example: OpenTL - www.opentl.org • General purpose library for model based visual tracking
Demo: OpenTL Model Tracking
Demo: OpenTL Face Tracking https://www.youtube.com/watch?v=WIoGdhkfNVE
Marker vs.Natural FeatureTracking • Marker tracking • Usually requires no database to be stored • Markers can be an eye-catcher • Tracking is less demanding • The environment must be instrumented • Markers usually work only when fully in view • Natural feature tracking • A database of keypoints must be stored/downloaded • Natural feature targets might catch the attention less • Natural feature targets are potentially everywhere • Natural feature targets work also if partially in view
Tracking from an Unknown Environment • What to do when you don’t know any features? • Very important problem in mobile robotics - Where am I? • SLAM • Simultaneously Localize And Map the environment • Goal: to recover both camera pose and map structure while initially knowing neither. • Mapping: • Building a map of the environment which the robot is in • Localisation: • Navigating this environment using the map while keeping track of the robot’s relative position and orientation
Visual SLAM • Early SLAM systems (1986 - ) • Computer visions and sensors (e.g. IMU, laser, etc.) • One of the most important algorithms in Robotics • Visual SLAM • Using cameras only, such as stereo view • MonoSLAM (single camera) developed in 2007 (Davidson)
Example:Kudan MonoSLAM https://www.youtube.com/watch?v=HdNtiabm3k0
How SLAMWorks • Three main steps 1. Tracking a set of points through successive camera frames 2. Using these tracks to triangulate their 3D position 3. Simultaneously use the estimated point locations to calculate the camera pose which could have observed them • By observing a sufficient number of points can solve for for both structure and motion (camera path and scene structure).
Evolution of SLAM Systems • MonoSLAM (Davidson, 2007) • Real time SLAM from single camera • PTAM (Klein, 2009) • First SLAM implementation on mobile phone • FAB-MAP (Cummins, 2008) • Probabilistic Localization and Mapping • DTAM (Newcombe, 2011) • 3D surface reconstruction from every pixel in image • KinectFusion (Izadi, 2011) • Realtime dense surface mapping and tracking using RGB-D
Demo:MonoSLAM https://www.youtube.com/watch?v=saUE7JHU3P0
LSD-SLAM (Engel 2014) • A novel, direct monocular SLAM technique • Uses image intensities both for tracking and mapping. • The camera is tracked using direct image alignment, while • Geometry is estimated as semi-dense depth maps • Supports very large scale tracking • Runs in real time on CPU and smartphone
Demo:LSD-SLAM https://www.youtube.com/watch?v=GnuQzP3gty4
Direct Method vs.Feature Based • Direct uses all information in image, cf feature based approach that only use small patches around corners and edges
Applications of SLAM Systems • Many possible applications • Augmented Reality camera tracking • Mobile robot localisation • Real world navigation aid • 3D scene reconstruction • 3D Object reconstruction • Etc.. • Assumptions • Camera moves through an unchanging scene • So not suitable for person tracking, gesture recognition • Both involve non-rigidly deforming objects and a non-static map
ARKit – Visual Inertial Odometry • Uses both computer vision + inertial sensing • Tracking position twice • Computer Vision – feature tracking, 2D plane tracking • Inertial sensing – using the phone IMU • Output combined via Kalman filter • Determine which output is most accurate • Pass pose to ARKit SDK • Each system compliments the other • Computer vision – needs visual features • IMU - drifts over time, doesn’t need features
ARKit –Visual Inertial Odometry • Slow camera • Fast IMU • If camera drops out IMU takes over • Camera corrects IMU errors
ARKit Demo • https://www.youtube.com/watch?v=dMEWp45WAUg
REGISTRATION
Spatial Registration
The Registration Problem • Virtual and Real content must stay properly aligned • If not: • Breaks the illusion that the two coexist • Prevents acceptance of many serious applications t = 0 seconds t = 1 second
Sources of Registration Errors •Static errors • Optical distortions (in HMD) • Mechanical misalignments • Tracker errors • Incorrect viewing parameters •Dynamic errors • System delays (largest source of error) • 1 ms delay = 1/3 mm registration error
Reducing Static Errors •Distortion compensation • For lens or display distortions •Manual adjustments • Have user manually alighn AR andVR content •View-based or direct measurements • Have user measure eye position •Camera calibration (video AR) • Measuring camera properties
View Based Calibration (Azuma 94)
Dynamic errors • Total Delay = 50 + 2 + 33 + 17 = 102 ms • 1 ms delay = 1/3 mm = 33mm error Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
Reducing dynamic errors (1) •Reduce system lag •Faster components/system modules •Reduce apparent lag •Image deflection •Image warping
Reducing System Lag Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display
ReducingApparent Lag Tracking Update x,y,z r,p,y Virtual Display Physical Display (640x480) 1280 x 960 Last known position Virtual Display Physical Display (640x480) 1280 x 960 Latest position Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop
Reducing dynamic errors (2) • Match video + graphics input streams (video AR) • Delay video of real world to match system lag • User doesn’t notice • Predictive Tracking • Inertial sensors helpful Azuma / Bishop 1994
PredictiveTracking Time Position Past Future Can predict up to 80 ms in future (Holloway) Now
PredictiveTracking (Azuma 94)
Wrap-up • Tracking and Registration are key problems • Registration error • Measures against static error • Measures against dynamic error • AR typically requires multiple tracking technologies • Computer vision most popular • Research Areas: • SLAM systems, Deformable models, Mobile outdoor tracking
www.empathiccomputing.org @marknb00 mark.billinghurst@unisa.edu.au

More Related Content

PDF
COMP 4010 - Lecture 7: Introduction to Augmented Reality
byMark Billinghurst
 
PDF
Grand Challenges for Mixed Reality
byMark Billinghurst
 
PDF
Lecture 8 Introduction to Augmented Reality
byMark Billinghurst
 
PDF
COMP 4010 - Lecture 7: Introduction to Augmented Reality
byMark Billinghurst
 
PDF
Comp4010 lecture3-AR Technology
byMark Billinghurst
 
PDF
2022 COMP4010 Lecture4: AR Interaction
byMark Billinghurst
 
PDF
Comp4010 Lecture4 AR Tracking and Interaction
byMark Billinghurst
 
PDF
COMP 4010 Lecture10: AR Tracking
byMark Billinghurst
 
COMP 4010 - Lecture 7: Introduction to Augmented Reality
byMark Billinghurst
 
Grand Challenges for Mixed Reality
byMark Billinghurst
 
Lecture 8 Introduction to Augmented Reality
byMark Billinghurst
 
COMP 4010 - Lecture 7: Introduction to Augmented Reality
byMark Billinghurst
 
Comp4010 lecture3-AR Technology
byMark Billinghurst
 
2022 COMP4010 Lecture4: AR Interaction
byMark Billinghurst
 
Comp4010 Lecture4 AR Tracking and Interaction
byMark Billinghurst
 
COMP 4010 Lecture10: AR Tracking
byMark Billinghurst
 

What's hot

PDF
2022 COMP4010 Lecture2: Perception
byMark Billinghurst
 
PDF
Comp4010 2021 Lecture2-Perception
byMark Billinghurst
 
PDF
COMP 4010 - Lecture 8 AR Technology
byMark Billinghurst
 
PDF
Comp 4010 2021 Lecture1-Introduction to XR
byMark Billinghurst
 
PDF
Application in Augmented and Virtual Reality
byMark Billinghurst
 
PDF
COMP 4010: Lecture8 - AR Technology
byMark Billinghurst
 
PDF
Building AR and VR Experiences
byMark Billinghurst
 
PDF
Comp4010 Lecture5 Interaction and Prototyping
byMark Billinghurst
 
PDF
Comp4010 Lecture10 VR Interface Design
byMark Billinghurst
 
PDF
Developing AR and VR Experiences with Unity
byMark Billinghurst
 
PDF
Comp4010 Lecture8 Introduction to VR
byMark Billinghurst
 
PDF
2022 COMP4010 Lecture3: AR Technology
byMark Billinghurst
 
PDF
Designing Usable Interface
byMark Billinghurst
 
PDF
2022 COMP4010 Lecture 6: Designing AR Systems
byMark Billinghurst
 
PDF
2022 COMP 4010 Lecture 7: Introduction to VR
byMark Billinghurst
 
PPTX
Augment your reality with ARCore !
byWajdi Ben Rabah
 
PDF
COMP 4010 - Lecture4 VR Technology - Visual and Haptic Displays
byMark Billinghurst
 
PDF
Comp4010 lecture11 VR Applications
byMark Billinghurst
 
PDF
Developing VR Experiences with Unity
byMark Billinghurst
 
PDF
2013 Lecture3: AR Tracking
byMark Billinghurst
 
2022 COMP4010 Lecture2: Perception
byMark Billinghurst
 
Comp4010 2021 Lecture2-Perception
byMark Billinghurst
 
COMP 4010 - Lecture 8 AR Technology
byMark Billinghurst
 
Comp 4010 2021 Lecture1-Introduction to XR
byMark Billinghurst
 
Application in Augmented and Virtual Reality
byMark Billinghurst
 
COMP 4010: Lecture8 - AR Technology
byMark Billinghurst
 
Building AR and VR Experiences
byMark Billinghurst
 
Comp4010 Lecture5 Interaction and Prototyping
byMark Billinghurst
 
Comp4010 Lecture10 VR Interface Design
byMark Billinghurst
 
Developing AR and VR Experiences with Unity
byMark Billinghurst
 
Comp4010 Lecture8 Introduction to VR
byMark Billinghurst
 
2022 COMP4010 Lecture3: AR Technology
byMark Billinghurst
 
Designing Usable Interface
byMark Billinghurst
 
2022 COMP4010 Lecture 6: Designing AR Systems
byMark Billinghurst
 
2022 COMP 4010 Lecture 7: Introduction to VR
byMark Billinghurst
 
Augment your reality with ARCore !
byWajdi Ben Rabah
 
COMP 4010 - Lecture4 VR Technology - Visual and Haptic Displays
byMark Billinghurst
 
Comp4010 lecture11 VR Applications
byMark Billinghurst
 
Developing VR Experiences with Unity
byMark Billinghurst
 
2013 Lecture3: AR Tracking
byMark Billinghurst
 

Similar to Lecture 9 AR Technology

PDF
A Survey of Augmented Reality
byMark Billinghurst
 
PPTX
Augmented Reality
byAkash J Nair
 
PPTX
Augmented reality
byNarendra kumar Jha
 
PPT
Augmented reality(my ppt)
bySrilakshmi Alla
 
PPTX
Augmented reality(ar) seminar
byHarshith Booragadda
 
PPT
Augmented Reality ppt
byKhyati Ganatra
 
PPTX
Augmented Reality By Jaseem Bhutto
byJaseem Bhutto
 
PDF
Mobile AR Lecture 2 - Technology
byMark Billinghurst
 
PDF
2016 AR Summer School Lecture2
byMark Billinghurst
 
PDF
2014 COSC 426 Lecture 2: Augmented Reality Technology
byMark Billinghurst
 
PDF
COMP 4010 Lecture9 AR Displays
byMark Billinghurst
 
PDF
2016 AR Summer School - Lecture1
byMark Billinghurst
 
PDF
2013 426 Lecture 2: Augmented Reality Technology
byMark Billinghurst
 
PPTX
AR/VR basics
byAbhishek Singhal [L.I.O.N]
 
PPT
Augmented Reality
byMahesh Mohan
 
PPTX
Workshop: AR Glasses and their Peculiarities
byMartin Lechner
 
PDF
2016 AR Summer School - Lecture 5
byMark Billinghurst
 
PDF
426 lecture1: Introduction to AR
byMark Billinghurst
 
PPTX
Augumented reallity
byShashwat Shriparv
 
PDF
3° RALI-Lisboa. SEEing THROUGH: enabling Augmented Reality applications from ...
bySaraMautino
 
A Survey of Augmented Reality
byMark Billinghurst
 
Augmented Reality
byAkash J Nair
 
Augmented reality
byNarendra kumar Jha
 
Augmented reality(my ppt)
bySrilakshmi Alla
 
Augmented reality(ar) seminar
byHarshith Booragadda
 
Augmented Reality ppt
byKhyati Ganatra
 
Augmented Reality By Jaseem Bhutto
byJaseem Bhutto
 
Mobile AR Lecture 2 - Technology
byMark Billinghurst
 
2016 AR Summer School Lecture2
byMark Billinghurst
 
2014 COSC 426 Lecture 2: Augmented Reality Technology
byMark Billinghurst
 
COMP 4010 Lecture9 AR Displays
byMark Billinghurst
 
2016 AR Summer School - Lecture1
byMark Billinghurst
 
2013 426 Lecture 2: Augmented Reality Technology
byMark Billinghurst
 
AR/VR basics
byAbhishek Singhal [L.I.O.N]
 
Augmented Reality
byMahesh Mohan
 
Workshop: AR Glasses and their Peculiarities
byMartin Lechner
 
2016 AR Summer School - Lecture 5
byMark Billinghurst
 
426 lecture1: Introduction to AR
byMark Billinghurst
 
Augumented reallity
byShashwat Shriparv
 
3° RALI-Lisboa. SEEing THROUGH: enabling Augmented Reality applications from ...
bySaraMautino
 

More from Mark Billinghurst

PDF
Rapid Prototyping for XR: Lecture 1 Introduction to Prototyping
byMark Billinghurst
 
PDF
NeuroXR: Current Research and Opportunities
byMark Billinghurst
 
PDF
Research Directions in Heads-Up Computing
byMark Billinghurst
 
PDF
Rapid Prototyping: A lecture on prototyping techniques for interface design
byMark Billinghurst
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
byMark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 6 - AI for Prototyping and Research Directi...
byMark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 4 - High Level Prototyping.
byMark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 5 - Cross Platform Development
byMark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 2 - Low Fidelity Prototyping.
byMark Billinghurst
 
PDF
Rapid Prototyping for XR: Lecture 3 - Video and Paper Prototyping
byMark Billinghurst
 
PDF
Empathic Computing: Creating Shared Understanding
byMark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture 8 - Electroencephalography (EEG) Basics
byMark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture12 - OpenVibe Tutorial
byMark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture14 - Evaluation
byMark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture13 - Neurotechnology for Enhanced Interaction ...
byMark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture16- Cognixion Axon-R
byMark Billinghurst
 
PDF
IVE 2024 Short Course Lecture15 - Measuring Cybersickness
byMark Billinghurst
 
PDF
IVE 2024 Short Course Lecture10 - Multimodal Emotion Recognition in Conversat...
byMark Billinghurst
 
PDF
IVE 2024 Short Course Lecture 9 - Empathic Computing in VR
byMark Billinghurst
 
PDF
IVE 2024 Short Course - Lecture18- Hacking Emotions in VR Collaboration.
byMark Billinghurst
 
Rapid Prototyping for XR: Lecture 1 Introduction to Prototyping
byMark Billinghurst
 
NeuroXR: Current Research and Opportunities
byMark Billinghurst
 
Research Directions in Heads-Up Computing
byMark Billinghurst
 
Rapid Prototyping: A lecture on prototyping techniques for interface design
byMark Billinghurst
 
Reach Out and Touch Someone: Haptics and Empathic Computing
byMark Billinghurst
 
Rapid Prototyping for XR: Lecture 6 - AI for Prototyping and Research Directi...
byMark Billinghurst
 
Rapid Prototyping for XR: Lecture 4 - High Level Prototyping.
byMark Billinghurst
 
Rapid Prototyping for XR: Lecture 5 - Cross Platform Development
byMark Billinghurst
 
Rapid Prototyping for XR: Lecture 2 - Low Fidelity Prototyping.
byMark Billinghurst
 
Rapid Prototyping for XR: Lecture 3 - Video and Paper Prototyping
byMark Billinghurst
 
Empathic Computing: Creating Shared Understanding
byMark Billinghurst
 
IVE 2024 Short Course - Lecture 8 - Electroencephalography (EEG) Basics
byMark Billinghurst
 
IVE 2024 Short Course - Lecture12 - OpenVibe Tutorial
byMark Billinghurst
 
IVE 2024 Short Course - Lecture14 - Evaluation
byMark Billinghurst
 
IVE 2024 Short Course - Lecture13 - Neurotechnology for Enhanced Interaction ...
byMark Billinghurst
 
IVE 2024 Short Course - Lecture16- Cognixion Axon-R
byMark Billinghurst
 
IVE 2024 Short Course Lecture15 - Measuring Cybersickness
byMark Billinghurst
 
IVE 2024 Short Course Lecture10 - Multimodal Emotion Recognition in Conversat...
byMark Billinghurst
 
IVE 2024 Short Course Lecture 9 - Empathic Computing in VR
byMark Billinghurst
 
IVE 2024 Short Course - Lecture18- Hacking Emotions in VR Collaboration.
byMark Billinghurst
 

Recently uploaded

PPTX
UNIT 1- Introduction to Basic of Computer Fundamentals
bypawarbhaktiit
 
PDF
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
PDF
Mercury Computer Systems & The Cell Broadband Engine
bySlide_N
 
PDF
TrustArc Webinar - Unpacking India’s DPDP Act: What You Should Know
byTrustArc
 
PDF
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
PDF
Best 10 Dedicated Servers in Amsterdam, Netherlands (2026 Enterprise Edition)...
byAtal Networks
 
PDF
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
PDF
Profiting from Technological Innovation: Managing Intellectual Property to Bu...
byDavid Teece
 
PDF
January Patch Tuesday
byIvanti
 
PPTX
Webinar: EVerest for Production: Accelerating EV Charger Development w/ Indus...
byDanBrown980551
 
PDF
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
PDF
Pervasive Encryption - Keeping Data Secure.pdf
byPrecisely
 
PDF
Hart, Inc. M&A Data Transition Checklist
byHart, Inc.
 
PDF
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
PDF
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
PDF
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
PPTX
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
PDF
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
PDF
Philippine Agricultural Engineering Standard_Hand Tractor .pdf
by6xfrnmdp4b
 
PPTX
computer science class 11 ( phishing and fraud mails) .pptx
bybalajichess314
 
UNIT 1- Introduction to Basic of Computer Fundamentals
bypawarbhaktiit
 
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
Mercury Computer Systems & The Cell Broadband Engine
bySlide_N
 
TrustArc Webinar - Unpacking India’s DPDP Act: What You Should Know
byTrustArc
 
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
Best 10 Dedicated Servers in Amsterdam, Netherlands (2026 Enterprise Edition)...
byAtal Networks
 
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
Profiting from Technological Innovation: Managing Intellectual Property to Bu...
byDavid Teece
 
January Patch Tuesday
byIvanti
 
Webinar: EVerest for Production: Accelerating EV Charger Development w/ Indus...
byDanBrown980551
 
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
Pervasive Encryption - Keeping Data Secure.pdf
byPrecisely
 
Hart, Inc. M&A Data Transition Checklist
byHart, Inc.
 
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
Just Eat Data Scraping For Restaurant Reviews And Pricing.pdf
byfusiondata2026
 
Apache Kafka 101 for Techies in IT dep.pptx
byamulyareddyk97
 
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
Philippine Agricultural Engineering Standard_Hand Tractor .pdf
by6xfrnmdp4b
 
computer science class 11 ( phishing and fraud mails) .pptx
bybalajichess314
 

Lecture 9 AR Technology

  • 1.
    LECTURE 9: AR TECHNOLOGY COMP4010 – Virtual Reality Semester 5 – 2019 Bruce Thomas, Mark Billinghurst, Gun Lee University of South Australia Oct 8th 2019
  • 2.
    Lecture 8: Recap •Augmented Reality • Defining Characteristics [Azuma 97]: •Combines Real and Virtual Images • Both can be seen at the same time •Interactive in real-time • The virtual content can be interacted with •Registered in 3D • Virtual objects appear fixed in space
  • 3.
  • 4.
    History Summary •1960’s –80’s: Early Experimentation •1980’s – 90’s: Basic Research • Tracking, displays •1995 – 2005: Tools/Applications • Interaction, usability, theory •2005 - : Commercial Applications • Games, Medical, Industry
  • 5.
    2005 - MobilePhone AR •Mobile Phones • camera • processor • display •AR on Mobile Phones • Simple graphics • Optimized computer vision • Collaborative Interaction
  • 6.
    2008 - BrowserBased AR • Flash + camera + 3D graphics • ARToolKit ported to Flash • High impact • High marketing value • Large potential install base • 1.6 Billion web users • Ease of development • Lots of developers, mature tools • Low cost of entry • Browser, web camera
  • 7.
    • Weak AR •Imprecise tracking • No knowledge of environment • Limited interactivity • Handheld AR • Strong AR • Very accurate tracking • Seamless integration into real world • Natural interaction • Head mounted AR Strong vs. Weak AR
  • 8.
    Augmented Reality Today •KeyTechnologies Available • Robust tracking (Computer Vision, GPS/sensors) • Display (Handheld, HMDs) • Input Devices (Kinect, etc) • Developer tools (Vuforia, ARToolKit) •Commercial Business Growing • Gaming, GPS/Mobile, Online Advertisement • >$3Billion USD in 2017
  • 9.
    AR Business Today •Marketing • Web-based, mobile • Mobile AR • Geo-located information and service • Driving demand for high end phones • Gaming • Mobile, Physical input (Kinect, PS Move) • Upcoming areas • Manufacturing, Medical, Military
  • 11.
  • 12.
    • Web basedAR • Flash, HTML 5 based AR • Marketing, education • Outdoor Mobile AR • GPS, compass tracking • Viewing Points of Interest in real world • Eg: Junaio, Layar, Wikitude • Handheld AR • Vision based tracking • Marketing, gaming • Location Based Experiences • HMD, fixed screens • Museums, point of sale, advertising Typical AR Experiences
  • 13.
  • 14.
  • 15.
    Technology Requirements • CombiningReal and Virtual Images • Needs - Display technologies • Interactive in Real-Time • Needs - Input and interactive technologies • Registered in 3D • Needs - Viewpoint tracking technologies Display Processing Input Tracking
  • 16.
  • 17.
    Display Technologies • Types(Bimber/Raskar 2005) • Head attached • Head mounted display/projector • Body attached • Handheld display/projector • Spatial • Spatially aligned projector/monitor
  • 18.
    Bimber, O., &Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press. DisplayTaxonomy
  • 19.
    Great Books forBackground • Spatial Augmented Reality, Displays; Fundamentals & Applications • Both by Oliver Bimber
  • 20.
  • 21.
    Head Mounted Displays(HMD) • Display and Optics mounted on Head • May or may not fully occlude real world • Provide full-color images • Things to considerations • Wearability • Brightness • Power consumption • Resolution, Field of View • Cost
  • 22.
    Types of HeadMounted Displays Occluded See-thru Multiplexed
  • 23.
    Optical See-Through HMD(OSTHMD) Virtual images from monitors Real World Optical Combiners
  • 24.
  • 25.
    Epson Moverio BT-35E ▪Stereo see-through display ($1,200 AUD) ▪ 1280 RGB x 720 pixels, 23 degree FOV, 30Hz, ▪ USB-C, HDMI connectors ▪ 5mp camera, GPS, gyro, accelerometer
  • 26.
    View Through OpticalSee-Through HMD Big data visualization on Microsoft HoloLens
  • 27.
    Other Optical See-ThroughDisplays • Microsoft HoloLens - $3,000 USD • Wearable computer • Waveguide displays • https://www.microsoft.com/hololens • Meta Meta2 - $1,500 USD • Wide field of view (90 degrees) • Tethered display • https://www.metavision.com/ • Mira Prism - $100 USD • Smart phone based • Wide field of view • https://www.mirareality.com/
  • 28.
    Project North Star(Leap Motion) • http://blog.leapmotion.com/northstar/ • Open source OSTHMD • 100 degree FOV, 120 Hz, 2 x 1600x1440 pixel displays • Hand tracking using Leap Motion hardware
  • 29.
    Demo: Project NorthStar • https://www.youtube.com/watch?v=yN0EGBo74j8
  • 30.
    Strengths and Weaknesses •Pros • Simpler design (cheaper) • Direct view of real world • Full resolution, no time delay (for real world) • Safety, Lower distortion • No eye displacement • Cons • No occlusion of real world (image see-through) • Wide field of view can be difficult
  • 31.
  • 32.
    DisplayTechnology • Curved Mirror •off-axis projection • curved mirrors in front of eye • high distortion, small eye-box • Waveguide • use internal reflection • unobstructed view of world • large eye-box
  • 33.
    Example: Sony WaveguideDisplay • https://www.youtube.com/watch?v=G2MtI7asLcA
  • 34.
    See-throughThin Displays (Waveguide) •Waveguide techniques for thin see-through displays • Wider FOV, enable AR applications • Social acceptability Opinvent Ora Lumus DK40
  • 35.
    Example: Sony SmartEyeGlasses • https://www.youtube.com/watch?v=kYPWaMsarss
  • 36.
    Video See-Through HMD(VSTHMD) Video cameras Monitors Graphics Combiner Video
  • 37.
  • 38.
    Vuzix Wrap 1200DXAR(Discontinued) ▪ Stereo video see-through display ($1500) ■Twin 852 x 480 LCD displays, 35 deg. FOV ■Stereo VGA cameras ■3 DOF head tracking ■https://www.vuzix.com/Products/LegacyProduct/4
  • 39.
    View Through aVideo See-Through HMD
  • 40.
    Varjo XR-1 • https://varjo.com/products/xr-1/ •Video pass through • 2 x 12 megapixel cameras at 90 Hz, < 15 ms latency • 80 degree FOV, 1440 x 1600 resolution per eye
  • 41.
  • 42.
    • Pros • Trueocclusion • AR pixels block video pixels • Digitized image of real world • Flexibility in composition, match time delays • Wide FOV is easier to support • Wide FOV camera • Cons • No direct view, latency Strengths and Weaknesses
  • 43.
    VST vs. OSTHMD VSTHMD OSTHMD
  • 44.
    Eye Multiplexed Display •See content in periphery • Virtual image “inset” into the real world
  • 45.
  • 46.
  • 48.
    Vuzix M-100 ▪ Monocularmultiplexed display ($1000) ■852 x 480 LCD display, 15 deg. FOV ■5 MP camera, HD video ■GPS, gyro, accelerometer
  • 49.
  • 50.
    Spatial Augmented Reality •Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber, Rasker “Spatial Augmented Reality”
  • 51.
    Projector-based AR Examples: Raskar, MITMedia Lab Inami, Tachi Lab, U. Tokyo Projector Real objects with retroreflective covering User (possibly head-tracked)
  • 52.
  • 53.
  • 54.
    Head Mounted Projector •NVIS P-50 HMPD • 1280x1024/eye • Stereoscopic • 50 degree FOV • www.nvis.com
  • 55.
    HMD vs.HMPD Head MountedDisplay Head Mounted Projected Display
  • 56.
    Tilt5 - https://www.tiltfive.com/ •Stereo head worn projectors • Interactive wand • Rollible retro-reflective sheet • Designed for shared interaction
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
    Handheld Displays •Mobile Phones •Camera • Display • Input
  • 63.
    OtherTypes ofAR Display •Audio • spatial sound • ambient audio • Tactile • physical sensation • Haptic • virtual touch
  • 64.
  • 65.
    Interacting with ARContent • You can see AR images .. how can you interact with it?
  • 66.
    Typical Interaction Methods •Handheld Display (phone, tablet) • Touch screen, device motion • Head mounted display • Handheld controller • Speech, gesture, touch • Physical manipulation • Spatial AR • Body motion • Physical manipulation
  • 67.
    Gestural interfaces • 1.Micro-gestures • (unistroke, smartPad) • 2. Device-based gestures • (tilt based examples) • 3. Embodied interaction • (eye toy)
  • 68.
    Mobile Device Motion •Interact by moving handheld display • Use IMU to detect shaking, rotation, etc • Treat device as controller
  • 69.
    Example: AR PaparazziGame • https://www.youtube.com/watch?v=MIGH5WGMnbs
  • 70.
    Tangible Interaction • Useobjects in the real world to interact with AR content • Attach AR content to real objects
  • 71.
  • 72.
    HMD Gesture Interaction •Simple click gestures • Head point and air tap • Natural two handed gestures • Full hand tracking, direction manipulation
  • 73.
    Hololens Gestures • Setof basic gestures • Air Tap = select, Bloom = open • Easy to remember and execute BloomAir Tap
  • 74.
  • 75.
    Embodied Interaction • Usewhole body to interact with AR • Popular for “Magic Mirror” Interface • Natural Interaction
  • 76.
  • 77.
  • 78.
    AR RequiresTracking andRegistration • Registration • Positioning virtual object wrt real world • Fixing virtual object on real object when view is fixed • Tracking • Continually locating the users viewpoint when view moving • Position (x,y,z), Orientation (r,p,y)
  • 79.
  • 80.
    Tracking Requirements • AugmentedReality Information Display • World Stabilized • Body Stabilized • Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized
  • 81.
    Tracking Technologies § Active •Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location § Passive • Inertial sensors (compass, accelerometer, gyro) • Computer Vision • Marker based, Natural feature tracking § Hybrid Tracking • Combined sensors (eg Vision + Inertial)
  • 82.
  • 83.
    MechanicalTracker •Idea: mechanical armswith joint sensors •++: high accuracy, haptic feedback •-- : cumbersome, expensive Microscribe
  • 84.
    MagneticTracker • Idea: coilgenerates current when moved in magnetic field. Measuring current gives position and orientation relative to magnetic source. • ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive Flock of Birds (Ascension)
  • 85.
    InertialTracker • Idea: measuringlinear and angular orientation rates (accelerometer/gyroscope) • ++: no transmitter, cheap, small, high frequency, wireless • -- : drifts over time, hysteresis effect, only 3DOF IS300 (Intersense) Wii Remote
  • 86.
    UltrasonicTracker • Idea: timeof Flight or phase-Coherence Sound Waves • ++: Small, Cheap • -- : 3DOF, Line of Sight, Low resolution, Affected by environmental conditons (pressure, temperature) Ultrasonic Logitech IS600
  • 87.
    Global Positioning System(GPS) • Created by US in 1978 • Currently 29 satellites • Satellites send position + time • GPS Receiver positioning • 4 satellites need to be visible • Differential time of arrival • Triangulation • Accuracy • 5-30m+, blocked by weather, buildings etc.
  • 88.
    Mobile Sensors • Inertialcompass • Earth’s magnetic field • Measures absolute orientation • Accelerometers • Measures acceleration about axis • Used for tilt, relative rotation • Can drift over time
  • 89.
  • 90.
    Why Optical Trackingfor AR? • Many AR devices have cameras • Mobile phone/tablet, Video see-through display • Provides precise alignment between video and AR overlay • Using features in video to generate pixel perfect alignment • Real world has many visual features that can be tracked from • Computer Vision well established discipline • Over 40 years of research to draw on • Old non real time algorithms can be run in real time on todays devices
  • 91.
    Common AR OpticalTracking Types • Marker Tracking • Tracking known artificial markers/images • e.g. ARToolKit square markers • Markerless Tracking • Tracking from known features in real world • e.g. Vuforia image tracking • Unprepared Tracking • Tracking in unknown environment • e.g. SLAM tracking
  • 92.
    Marker Tracking • Availablefor more than 10 years • Several open source solutions • ARToolKit, ARTag, ATK+, etc • Fairly simple to implement • Standard computer vision methods • A rectangle provides 4 corner points • Enough for pose estimation!
  • 93.
  • 94.
    Marker Based Tracking:ARToolKit http://www.artoolkit.org
  • 95.
    Tracking challenges inARToolKit Falsepositives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)
  • 96.
  • 97.
    But - Youcan’t cover world with ARToolKit Markers!
  • 98.
    Markerless Tracking Magnetic TrackerInertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking • No more Markers! èMarkerless Tracking Mechanical Tracker
  • 99.
    Natural Feature Tracking •Use Natural Cues of Real Elements • Edges • Surface Texture • Interest Points • Model or Model-Free • No visual pollution Contours Features Points Surfaces
  • 100.
  • 101.
    Tracking by KeypointDetection • This is what most trackers do… • Targets are detected every frame • Popular because tracking and detection are solved simultaneously Keypoint detection Descriptor creation and matching Outlier Removal Pose estimation and refinement Camera Image Pose Recognition
  • 102.
    What is aKeypoint? • It depends on the detector you use! • For high performance use the FAST corner detector • Apply FAST to all pixels of your image • Obtain a set of keypoints for your image • Describe the keypoints Rosten, E., & Drummond, T. (2006, May). Machine learning for high-speed corner detection. In European conference on computer vision (pp. 430-443). Springer Berlin Heidelberg.
  • 103.
  • 104.
  • 105.
    Descriptors • Describe theKeypoint features • Can use SIFT • Estimate the dominant keypoint orientation using gradients • Compensate for detected orientation • Describe the keypoints in terms of the gradients surrounding it Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D., Real-Time Detection and Tracking for Augmented Reality on Mobile Phones. IEEE Transactions on Visualization and Computer Graphics, May/June, 2010
  • 106.
    Demo: Vuforia TextureTracking https://www.youtube.com/watch?v=1Qf5Qew5zSU
  • 107.
    Edge Based Tracking •Example: RAPiD [Drummond et al. 02] • Initialization, Control Points, Pose Prediction (Global Method)
  • 108.
  • 109.
    Line Based Tracking •Visual Servoing [Comport et al. 2004]
  • 110.
    Model BasedTracking • Trackingfrom 3D object shape • Example: OpenTL - www.opentl.org • General purpose library for model based visual tracking
  • 111.
  • 112.
    Demo: OpenTL FaceTracking https://www.youtube.com/watch?v=WIoGdhkfNVE
  • 113.
    Marker vs.Natural FeatureTracking •Marker tracking • Usually requires no database to be stored • Markers can be an eye-catcher • Tracking is less demanding • The environment must be instrumented • Markers usually work only when fully in view • Natural feature tracking • A database of keypoints must be stored/downloaded • Natural feature targets might catch the attention less • Natural feature targets are potentially everywhere • Natural feature targets work also if partially in view
  • 114.
    Tracking from anUnknown Environment • What to do when you don’t know any features? • Very important problem in mobile robotics - Where am I? • SLAM • Simultaneously Localize And Map the environment • Goal: to recover both camera pose and map structure while initially knowing neither. • Mapping: • Building a map of the environment which the robot is in • Localisation: • Navigating this environment using the map while keeping track of the robot’s relative position and orientation
  • 115.
    Visual SLAM • EarlySLAM systems (1986 - ) • Computer visions and sensors (e.g. IMU, laser, etc.) • One of the most important algorithms in Robotics • Visual SLAM • Using cameras only, such as stereo view • MonoSLAM (single camera) developed in 2007 (Davidson)
  • 116.
  • 117.
    How SLAMWorks • Threemain steps 1. Tracking a set of points through successive camera frames 2. Using these tracks to triangulate their 3D position 3. Simultaneously use the estimated point locations to calculate the camera pose which could have observed them • By observing a sufficient number of points can solve for for both structure and motion (camera path and scene structure).
  • 118.
    Evolution of SLAMSystems • MonoSLAM (Davidson, 2007) • Real time SLAM from single camera • PTAM (Klein, 2009) • First SLAM implementation on mobile phone • FAB-MAP (Cummins, 2008) • Probabilistic Localization and Mapping • DTAM (Newcombe, 2011) • 3D surface reconstruction from every pixel in image • KinectFusion (Izadi, 2011) • Realtime dense surface mapping and tracking using RGB-D
  • 119.
  • 120.
    LSD-SLAM (Engel 2014) •A novel, direct monocular SLAM technique • Uses image intensities both for tracking and mapping. • The camera is tracked using direct image alignment, while • Geometry is estimated as semi-dense depth maps • Supports very large scale tracking • Runs in real time on CPU and smartphone
  • 121.
  • 122.
    Direct Method vs.FeatureBased • Direct uses all information in image, cf feature based approach that only use small patches around corners and edges
  • 123.
    Applications of SLAMSystems • Many possible applications • Augmented Reality camera tracking • Mobile robot localisation • Real world navigation aid • 3D scene reconstruction • 3D Object reconstruction • Etc.. • Assumptions • Camera moves through an unchanging scene • So not suitable for person tracking, gesture recognition • Both involve non-rigidly deforming objects and a non-static map
  • 124.
    ARKit – VisualInertial Odometry • Uses both computer vision + inertial sensing • Tracking position twice • Computer Vision – feature tracking, 2D plane tracking • Inertial sensing – using the phone IMU • Output combined via Kalman filter • Determine which output is most accurate • Pass pose to ARKit SDK • Each system compliments the other • Computer vision – needs visual features • IMU - drifts over time, doesn’t need features
  • 125.
    ARKit –Visual InertialOdometry • Slow camera • Fast IMU • If camera drops out IMU takes over • Camera corrects IMU errors
  • 126.
  • 127.
  • 128.
  • 129.
    The Registration Problem •Virtual and Real content must stay properly aligned • If not: • Breaks the illusion that the two coexist • Prevents acceptance of many serious applications t = 0 seconds t = 1 second
  • 130.
    Sources of RegistrationErrors •Static errors • Optical distortions (in HMD) • Mechanical misalignments • Tracker errors • Incorrect viewing parameters •Dynamic errors • System delays (largest source of error) • 1 ms delay = 1/3 mm registration error
  • 131.
    Reducing Static Errors •Distortioncompensation • For lens or display distortions •Manual adjustments • Have user manually alighn AR andVR content •View-based or direct measurements • Have user measure eye position •Camera calibration (video AR) • Measuring camera properties
  • 132.
  • 133.
    Dynamic errors • TotalDelay = 50 + 2 + 33 + 17 = 102 ms • 1 ms delay = 1/3 mm = 33mm error Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
  • 134.
    Reducing dynamic errors(1) •Reduce system lag •Faster components/system modules •Reduce apparent lag •Image deflection •Image warping
  • 135.
    Reducing System Lag TrackingCalculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display
  • 136.
    ReducingApparent Lag Tracking Update x,y,z r,p,y Virtual Display Physical Display (640x480) 1280x 960 Last known position Virtual Display Physical Display (640x480) 1280 x 960 Latest position Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop
  • 137.
    Reducing dynamic errors(2) • Match video + graphics input streams (video AR) • Delay video of real world to match system lag • User doesn’t notice • Predictive Tracking • Inertial sensors helpful Azuma / Bishop 1994
  • 138.
  • 139.
  • 140.
    Wrap-up • Tracking andRegistration are key problems • Registration error • Measures against static error • Measures against dynamic error • AR typically requires multiple tracking technologies • Computer vision most popular • Research Areas: • SLAM systems, Deformable models, Mobile outdoor tracking
  • 141.