Download as PDF, PPTX
![Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.](https://image.slidesharecdn.com/lecture2-160228101635/75/2016-AR-Summer-School-Lecture2-2-2048.jpg)
![Edge Based Tracking
• RAPiD [Drummond et al. 02]
• Initialization, Control Points, Pose Prediction (Global Method)](https://image.slidesharecdn.com/lecture2-160228101635/75/2016-AR-Summer-School-Lecture2-81-2048.jpg)
![Line Based Tracking
• Visual Servoing [Comport et al. 2004]](https://image.slidesharecdn.com/lecture2-160228101635/75/2016-AR-Summer-School-Lecture2-82-2048.jpg)
Uploaded byMark Billinghurst
2016 AR Summer School Lecture2
This document is a lecture on augmented reality (AR) technology presented by Mark Billinghurst at the University of South Australia. It covers definitions, display technologies, tracking methods, and registration challenges essential to the functioning of AR systems. The lecture highlights various types of AR displays, their applications, and the importance of accurate tracking and registration for effective augmented experiences.
More Related Content
Similar to 2016 AR Summer School Lecture2
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
2016 AR Summer School Lecture2
- 1. LECTURE 2: AUGMENTED REALITYTECHNOLOGY Mark Billinghurst AR Summer School February 15th – 19th 2016 University of South Australia
- 2. Augmented Reality Definition • Defining Characteristics [Azuma 97] • Combines Real andVirtual Images • Both can be seen at the same time • Interactive in real-time • The virtual content can be interacted with • Registered in 3D • Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 3. Augmented RealityTechnology ! Combining Realand Virtual Images • Display technologies ! Interactive in Real-Time • Input and interactive technologies ! Registered in 3D • Viewpoint tracking technologies Display Processing Input Tracking
- 4.
- 5. AR Displays e.g. window reflections VirtualImages seen off windows e.g. Reach-In Projection CRT Display using beamsplitter Not Head-Mounted e.g. Shared Space Magic Book Liquid Crystal Displays LCDs Head-Mounted Display (HMD) Primarily Indoor Environments e.g. WLVA and IVRD Cathode Ray Tube (CRT) or Virtual Retinal Display (VRD) Many Military Applications & Assistive Technologies Head-Mounted Display (HMD) e.g. Head-Up Display (HUD) Projection Display Navigational Aids in Cars Military Airborne Applications Not Head Mounted (e.g. vehicle mounted) Primarily Outdoor (Daylight) Environments AR Visual Displays
- 6. Display Technologies ! Types (Bimber/Raskar2003) ! Head attached • Head mounted display/projector ! Body attached • Handheld display/projector ! Spatial • Spatially aligned projector/monitor
- 7.
- 8.
- 9. Head Mounted Displays(HMD) • Display and Optics mounted on Head • May or may not fully occlude real world • Provide full-color images • Considerations • Cumbersome to wear • Brightness • Low power consumption • Resolution limited • Cost is high?
- 10. Key Properties ofHMD • Field ofView • Human eye 95 degrees horizontal, 60/70 degrees vertical • Resolution • > 320x240 pixel • Refresh Rate • Focus • Fixed/manual • Power • Size
- 11. Types of HeadMounted Displays Occluded See-thru Multiplexed
- 12. ImmersiveVRArchitecture Head! Tracker Host! Processor Data Base! Model Rendering! Engine Frame! Buffer head position/orientation tonetwork Display! Driver Non see- thru! Image source & optics virtual object Virtual World
- 13. See-thruARArchitecture Head! Tracker Host! Processor Data Base! Model Rendering! Engine Frame! Buffer head position/orientation tonetwork Display! Driver see-thru! combiner Virtual Image superimpos ed! over real world object real world Image source
- 14. Optical see-through head-mounteddisplay Virtual images from monitors Real World Optical Combiners
- 15.
- 16. Epson Moverio BT-200 ▪ Stereosee-through display ($700) ▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g ▪ Android Powered, separate controller ▪ VGA camera, GPS, gyro, accelerometer
- 17.
- 18. TheVirtual Retinal Display • Image scanned onto retina • Commercialized through Microvision • Nomad System - www.mvis.com
- 19. Strengths of opticalsee-throughAR • Simpler (cheaper) • Direct view of real world • Full resolution, no time delay (for real world) • Safety • Lower distortion • No eye displacement • but COASTAR video see-through avoids this
- 20. VideoARArchitecture Head! Tracker Host! Processor Graphics! renderer Digital! Mixer Frame! Buffer head position/orientation to network Display! Driver Nonsee- thru! Image source & optics Head- mounted camera aligned to display optics Video! Processor Video image of real world Virtual image inset into video of real world
- 21.
- 22.
- 23. Vuzix Wrap 1200DXAR ▪ Stereovideo see-through display ($1500) ■ Twin 852 x 480 LCD displays, 35 deg. FOV ■ StereoVGA cameras ■ 3 DOF head tracking
- 24.
- 25. Strengths ofVideo See-ThroughAR • Trueocclusion • Kiyokawa optical display that supports occlusion • Digitized image of real world • Flexibility in composition • Matchable time delays • More registration, calibration strategies • Wide FOV is easier to support
- 26. Optical vs.VideoAR Summary • Bothhave proponents • Video is more popular today? • Likely because lack of available optical products • Depends on application? • Manufacturing: optical is cheaper • Medical: video for calibration strategies
- 27. Eye multiplexedARArchitecture Head! Tracker Host! Processor Data Base! Model Rendering! Engine Frame! Buffer headposition/orientation to network Display! Driver Virtual Image inset into! real world scene real world Opaque! Image source
- 28.
- 29.
- 30.
- 32. Vuzix M-100 ▪ Monocular multiplexeddisplay ($1000) ■ 852 x 480 LCD display, 15 deg. FOV ■ 5 MP camera, HD video ■ GPS, gyro, accelerometer
- 33. DisplayTechnology • Curved Mirror • off-axis projection • curved mirrors in front of eye • high distortion, small eye-box • Waveguide • use internal reflection • unobstructed view of world • large eye-box
- 34. See-through thin displays • Waveguide techniques for thin see-through displays • Wider FOV, enable AR applications • Social acceptability Opinvent Ora Lumus DK40
- 35.
- 36. SpatialAugmented Reality • Project ontoirregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber, Rasker “Spatial Augmented Reality”
- 37. Projector-basedAR Examples: Raskar, MIT MediaLab Inami, Tachi Lab, U. Tokyo Projector Real objects with retroreflective covering User (possibly head-tracked)
- 38. Example of projector-basedAR RameshRaskar, UNC, MERL
- 39. Example of projector-basedAR RameshRaskar, UNC Chapel Hill
- 40. Head Mounted Projector • NVIS P-50 HMPD • 1280x1024/eye • Stereoscopic • 50 degree FOV • www.nvis.com
- 41. HMD vs.HMPD Head MountedDisplay Head Mounted Projected Display
- 42. CastAR - http://technicalillusions.com/ • Stereohead worn projectors • Interactive wand • Rollable retro-reflective sheet
- 43. • Designed for sharedinteraction
- 44. Pico Projectors • Microvision -www.mvis.com • 3M, Samsung, Philips, etc
- 45. MIT Sixth Sense • Bodyworn camera and projector • http://www.pranavmistry.com/projects/sixthsense/
- 46.
- 47. Video MonitorAR Video cameras Monitor GraphicsCombiner Video Stereo glasses
- 48.
- 49. Virtual Showcase • Mirrors ona projection table • Head tracked stereo • Up to 4 users • Merges graphic and real objects • Exhibit/museum applications • Fraunhofer Institute (2001) • Bimber, Frohlich
- 50. Augmented Paleontology Bimber et.al. IEEE Computer Sept. 2002
- 51. Alternate Displays LCD PanelLaptop PDA
- 52.
- 53. OtherTypes ofAR Display • Audio • spatial sound • ambient audio • Tactile • physical sensation • Haptic • virtual touch
- 54. Haptic Input • ARHaptic Workbench • CSIRO 2003 – Adcock et. al.
- 55. Phantom • Sensable Technologies (www.sensable.com) • 6DOF Force Feedback Device
- 56. AR Haptic Interface • Phantom,ARToolKit, Magellan
- 57.
- 58. Objects Registered in3D • Registration • Positioning virtual object wrt real world • Tracking • Continually locating the users viewpoint • Position (x,y,z), Orientation (r,p,y)
- 59. Tracking Requirements • Augmented RealityInformation Display • World Stabilized • Body Stabilized • Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized
- 60. Tracking Technologies " Active • Mechanical, Magnetic, Ultrasonic • GPS, Wifi, cell location " Passive • Inertial sensors (compass, accelerometer, gyro) • Computer Vision • Marker based, Natural feature tracking " Hybrid Tracking • Combined sensors (eg Vision + Inertial)
- 61.
- 62. MechanicalTracker • Idea: mechanical armswith joint sensors • ++: high accuracy, haptic feedback • -- : cumbersome, expensive Microscribe
- 63. MagneticTracker • Idea: difference betweena magnetic transmitter and a receiver • ++: 6DOF, robust • -- : wired, sensible to metal, noisy, expensive Flock of Birds (Ascension)
- 64.
- 65. Global Positioning System(GPS) • Created by US in 1978 • Currently 29 satellites • Satellites send position + time • GPS Receiver positioning • 4 satellites need to be visible • Differential time of arrival • Triangulation • Accuracy • 5-30m+, blocked by weather, buildings etc
- 66. InertialTracker • Idea: measuring linearand angular orientation rates (accelerometer/gyroscope) • ++: no transmitter, cheap, small, high frequency, wireless • -- : drift, hysteris only 3DOF IS300 (Intersense) Wii Remote
- 67. Mobile Sensors • Inertialcompass • Earth’s magnetic field • Measures absolute orientation • Accelerometers • Measures acceleration about axis • Used for tilt, relative rotation • Can drift over time
- 68.
- 69. OpticalTrackingTechnologies • Scalable active trackers • InterSenseIS-900, 3rd Tech HiBall • Passive optical computer vision • Line of sight, may require landmarks • Can be brittle. • Computer vision is computationally-intensive 3rd Tech, Inc.
- 70.
- 71. HiBallTracking System (3rdTech) • Inside-Out Tracker • $50K USD • Scalable over large area • Fast update (2000Hz) • Latency Less than 1 ms. • Accurate • Position 0.4mm RMS • Orientation 0.02° RMS
- 73. Marker tracking • Availablefor more than 10 years • Several open source solutions exist • ARToolKit,ARTag,ATK+, etc • Fairly simple to implement • Standard computer vision methods • A rectangle provides 4 corner points • Enough for pose estimation!
- 74. Marker Based Tracking:ARToolKit http://artoolkit.sourceforge.net/
- 75.
- 76. Tracking challenges inARToolKit Falsepositives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)
- 78. Markerless Tracking Magnetic TrackerInertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking • No more Markers! #Markerless Tracking Mechanical Tracker
- 79. Natural Feature Tracking • UseNatural Cues of Real Elements • Edges • Surface Texture • Interest Points • Model or Model-Free • No visual pollution Contours Features Points Surfaces
- 80.
- 81. Edge Based Tracking • RAPiD [Drummond et al. 02] • Initialization, Control Points, Pose Prediction (Global Method)
- 82. Line Based Tracking • Visual Servoing [Comport et al. 2004]
- 83. Model BasedTracking • OpenTL -www.opentl.org • General purpose library for model based visual tracking
- 84. Marker vs.natural featuretracking • Marker tracking • ++ Markers can be an eye-catcher • ++ Tracking is less demanding • -- The environment must be instrumented with markers • -- Markers usually work only when fully in view • Natural feature tracking • -- A database of keypoints must be stored/downloaded • ++ Natural feature targets might catch the attention less • ++ Natural feature targets are potentially everywhere • ++ Natural feature targets work also if partially in view
- 85. Example: Outdoor HybridTracking • Combines • computer vision • natural feature tracking • inertial gyroscope sensors • Both correct for each other • Inertial gyro - provides frame to frame prediction of camera orientation • Computer vision - correct for gyro drift
- 86. OutdoorARTracking System You, Neumann,Azumaoutdoor AR system (1999)
- 87. Robust OutdoorTracking • Hybrid Tracking • ComputerVision, GPS, inertial • Going Out • Reitmayr & Drummond (Univ. Cambridge) Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reaity. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.
- 88.
- 89.
- 90.
- 91. The Registration Problem • Virtual and Real must stay properly aligned • If not: • Breaks the illusion that the two coexist • Prevents acceptance of many serious applications
- 92. Sources of registrationerrors • Static errors • Optical distortions • Mechanical misalignments • Tracker errors • Incorrect viewing parameters • Dynamic errors • System delays (largest source of error) • 1 ms delay = 1/3 mm registration error
- 93. Reducing static errors • Distortioncompensation • Manual adjustments • View-based or direct measurements • Camera calibration (video)
- 94. View Based Calibration(Azuma 94)
- 95. Dynamic errors • TotalDelay = 50 + 2 + 33 + 17 = 102 ms • 1 ms delay = 1/3 mm = 33mm error Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop 20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
- 96. Reducing dynamic errors(1) • Reduce system lag • Faster components/system modules • Reduce apparent lag • Image deflection • Image warping
- 97. Reducing System Lag TrackingCalculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display
- 98. ReducingApparent Lag Tracking Update x,y,z r,p,y Virtual Display Physical Display (640x480) 1280x 960 Last known position Virtual Display Physical Display (640x480) 1280 x 960 Latest position Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop
- 99. Reducing dynamic errors(2) • Match input streams (video) • Delay video of real world to match system lag • Predictive Tracking • Inertial sensors helpful Azuma / Bishop 1994
- 100. PredictiveTracking Time Position Past Future Can predictup to 80 ms in future (Holloway) Now
- 101.
- 102. Wrap-up • Tracking and Registrationare key problems • Registration error • Measures against static error • Measures against dynamic error • AR typically requires multiple tracking technologies • Research Areas: • Hybrid Markerless Techniques, Deformable Surface, Mobile, Outdoors
- 103.