Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
LECTURE 2:
AR TECHNOLOGY
Mark Billinghurst
mark.billinghurst@unisa.edu.au
Zi Siang See
zisiang@reina.com.my
November 29th-...
TECHNOLOGY
Augmented Reality Definition
• Defining Characteristics
• Combines Real andVirtual Images
• Display Technology
• Interacti...
DISPLAY
Display Technologies
! Types (Bimber/Raskar 2003)
! Head attached
•  Head mounted display/projector
! Body attached
•  Han...
DisplayTaxonomy
Types of Head Mounted Displays
Occluded
See-thru
Multiplexed
Optical see-through HMD
Virtual images
from monitors
Real
World
Optical
Combiners
Epson Moverio BT-200
▪  Stereo see-through display ($700)
▪  960 x 540 pixels, 23 degree FOV, 60Hz, 88g
▪  Android Powered...
Strengths of optical see-through
•  Simpler (cheaper)
•  Direct view of real world
•  Full resolution, no time delay (for ...
Video see-through HMD
Video
cameras
Monitors
Graphics
Combiner
Video
Vuzix Wrap 1200DXAR
▪  Stereo video see-through display ($1500)
■  Twin 852 x 480 LCD displays, 35 deg. FOV
■  StereoVGA c...
Strengths of Video See-Through
•  True occlusion
•  Block image of real world
•  Digitized image of real world
•  Flexibil...
Multiplexed Displays
•  Above or below line of sight
•  Strengths
•  User has unobstructed view of real world
•  Simple op...
Google Glass
▪  Monocular see-through multiplexed display
▪  640 x 360 microprojector, 15 degree FOV
▪  5 MP camera, gyro,...
DisplayTechnology
•  Curved Mirror
•  off-axis projection
•  curved mirrors in front of eye
•  high distortion, small eye-...
See-through thin displays
•  Waveguide techniques for thin see-through displays
•  Wider FOV, enable AR applications
•  So...
SpatialAugmented Reality
• Project onto irregular surfaces
• Geometric Registration
• Projector blending, High dynamic ran...
Projector-basedAR
Examples:
Raskar, MIT Media Lab
Inami, Tachi Lab, U. Tokyo
Projector
Real objects
with retroreflective
c...
HMD vs.HMPD
Head Mounted Display Head Mounted Projected Display
CastAR - http://technicalillusions.com/
• Stereo head worn projectors
• Interactive wand
• Rollable retro-reflective sheet
• Designed for shared interaction
Video MonitorAR
Video
cameras Monitor
Graphics Combiner
Video
Stereo
glasses
Examples
Handheld Displays
• Mobile Phones
• Camera
• Display
• Input
TRACKING
Objects Registered in 3D
• Registration
• Positioning virtual object wrt real world
• Tracking
• Continually locating the ...
Tracking Technologies
"  Active
•  Mechanical, Magnetic, Ultrasonic
•  GPS, Wifi, cell location
"  Passive
•  Inertial sen...
Tracking Types
Magnetic
Tracker
Inertial
Tracker
Ultrasonic
Tracker
Optical
Tracker
Marker-Based
Tracking
Markerless
Track...
Example:Marker tracking
•  Available for more than 10 years
•  Several open source solutions exist
•  ARToolKit, ARTag, AT...
Marker Based Tracking: ARToolKit
http://artoolkit.sourceforge.net/
Coordinate Systems
Tracking challenges inARToolKit
False positives and inter-marker confusion
(image by M. Fiala)
Image noise
(e.g. poor lens...
Markerless Tracking
Magnetic Tracker Inertial
Tracker
Ultrasonic
Tracker
Optical
Tracker
Marker-Based
Tracking
Markerless
...
Natural Feature Tracking
• Use Natural Cues of Real Elements
•  Edges
•  Surface Texture
•  Interest Points
• Model or Mod...
TextureTracking
Edge Based Tracking
•  RAPiD [Drummond et al. 02]
•  Initialization, Control Points, Pose Prediction (Global Method)
Line Based Tracking
• Visual Servoing [Comport et al. 2004]
Model BasedTracking
• OpenTL - www.opentl.org
•  General purpose library for model based visual tracking
Marker vs.natural feature tracking
•  Marker tracking
•  ++ Markers can be an eye-catcher
•  ++ Tracking is less demanding...
Example: Outdoor Hybrid Tracking
• Combines
• computer vision
•  natural feature tracking
• inertial gyroscope sensors
• B...
Robust OutdoorTracking
• Hybrid Tracking
•  ComputerVision, GPS, inertial
• Going Out
•  Reitmayr & Drummond (Univ. Cambri...
Handheld Display
INTERACTION
• Interface Components
• Physical components
• Display elements
• Visual/audio
• Interaction metaphors
Physical
Elements
D...
AR Design Space
Reality Virtual Reality
Augmented Reality
Physical Design Virtual Design
Interface Design Path
1/ Prototype Demonstration
2/ Adoption of Interaction Techniques from other
interface metaphors
3/ D...
Interaction Development
• Information Browsing
• Camera movement
• Limited interaction
• 3D AR Interaction
• HMD, hand tra...
Tangible User Interfaces (Ishii 97)
• Augmented Surfaces
• Rekimoto 1998
• Multiple projection surfaces
• Tangible prop in...
Other Examples
• Triangles (Gorbert 1998)
• Triangular based story telling
• ActiveCube (Kitamura 2000-)
• Cubes with sens...
Lessons fromTangible Interfaces
• Benefits
• Physical objects make us smart (affordances)
• Objects aid collaboration
• Ob...
Orthogonal Nature ofAR Interfaces
TangibleAR Interaction
• AR overcomes limitation of TUIs
• enhance display possibilities
• merge task/display space
• prov...
Tangible AR Design Principles
• Tangible AR Interfaces use TUI principles
• Physical controllers for moving virtual conten...
VOMAR -TangibleAR Interface
• Use of natural physical object
to control virtual objects
• Physical objects
• Catalog book:...
Interaction Evolution
• Browsing Interfaces
• simple (conceptually!), unobtrusive
• 3D AR Interfaces
• expressive, creativ...
AR APPLICATIONS
•  Web based AR
•  Flash, HTML 5 based AR
•  Marketing, education
•  Outdoor Mobile AR
•  GPS, compass tracking
•  Viewing...
CityViewARApplication
•  Visualize Christchurch before the earthquakes
User Experience
• MultipleViews
• MapView,ARView, ListView
• Multiple Data Types
• 2D images, 3D content, text, panoramas
Warp Runner
• Puzzle solving game
• Deform real world terrain
Demo:colAR
• Turn colouring books pages into AR scenes
• Markerless tracking, use your own colours..
• Try it yourself: ht...
What Makes a GoodAR Experience?
• Compelling
• Engaging,‘Magic’ moment
• Intuitive, ease of use
• Uses existing skills
• A...
Conclusion
• AR seamlessly blends real and virtual imagery
•  Interactive in real time, fixed in space
• AR has developed ...
Upcoming SlideShare
Loading in …5
×

Mobile AR Lecture 2 - Technology

1,421 views

Published on

Lecture 2 from a course on Mobile Based Augmented Reality Development taught by Mark Billinghurst and Zi Siang See on November 29th and 30th 2015 at Johor Bahru in Malaysia. This lecture provides an introduction to Mobile AR Technology. Look for the other 9 lectures in the course.

Published in: Technology

Mobile AR Lecture 2 - Technology

  1. 1. LECTURE 2: AR TECHNOLOGY Mark Billinghurst mark.billinghurst@unisa.edu.au Zi Siang See zisiang@reina.com.my November 29th-30th 2015 Mobile-Based Augmented Reality Development
  2. 2. TECHNOLOGY
  3. 3. Augmented Reality Definition • Defining Characteristics • Combines Real andVirtual Images • Display Technology • Interactive in real-time • Interaction Technology • Registered in 3D • Tracking Technology
  4. 4. DISPLAY
  5. 5. Display Technologies ! Types (Bimber/Raskar 2003) ! Head attached •  Head mounted display/projector ! Body attached •  Handheld display/projector ! Spatial •  Spatially aligned projector/monitor
  6. 6. DisplayTaxonomy
  7. 7. Types of Head Mounted Displays Occluded See-thru Multiplexed
  8. 8. Optical see-through HMD Virtual images from monitors Real World Optical Combiners
  9. 9. Epson Moverio BT-200 ▪  Stereo see-through display ($700) ▪  960 x 540 pixels, 23 degree FOV, 60Hz, 88g ▪  Android Powered, separate controller ▪  VGA camera, GPS, gyro, accelerometer
  10. 10. Strengths of optical see-through •  Simpler (cheaper) •  Direct view of real world •  Full resolution, no time delay (for real world) •  Safety •  Lower distortion •  No eye displacement •  see directly through display
  11. 11. Video see-through HMD Video cameras Monitors Graphics Combiner Video
  12. 12. Vuzix Wrap 1200DXAR ▪  Stereo video see-through display ($1500) ■  Twin 852 x 480 LCD displays, 35 deg. FOV ■  StereoVGA cameras ■  3 DOF head tracking
  13. 13. Strengths of Video See-Through •  True occlusion •  Block image of real world •  Digitized image of real world •  Flexibility in composition •  Matchable time delays •  More registration, calibration strategies •  Wide FOV is easier to support •  wide FOV camera
  14. 14. Multiplexed Displays •  Above or below line of sight •  Strengths •  User has unobstructed view of real world •  Simple optics/cheap •  Weaknesses •  Direct information overlay difficult •  Display/camera offset from eyeline •  Wide FOV difficult
  15. 15. Google Glass ▪  Monocular see-through multiplexed display ▪  640 x 360 microprojector, 15 degree FOV ▪  5 MP camera, gyro, accelerometer
  16. 16. DisplayTechnology •  Curved Mirror •  off-axis projection •  curved mirrors in front of eye •  high distortion, small eye-box •  Waveguide •  use internal reflection •  unobstructed view of world •  large eye-box
  17. 17. See-through thin displays •  Waveguide techniques for thin see-through displays •  Wider FOV, enable AR applications •  Social acceptability Opinvent Ora Lumus DK40
  18. 18. SpatialAugmented Reality • Project onto irregular surfaces • Geometric Registration • Projector blending, High dynamic range • Book: Bimber “Spatial Augmented Reality”
  19. 19. Projector-basedAR Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo Projector Real objects with retroreflective covering User (possibly head-tracked)
  20. 20. HMD vs.HMPD Head Mounted Display Head Mounted Projected Display
  21. 21. CastAR - http://technicalillusions.com/ • Stereo head worn projectors • Interactive wand • Rollable retro-reflective sheet
  22. 22. • Designed for shared interaction
  23. 23. Video MonitorAR Video cameras Monitor Graphics Combiner Video Stereo glasses
  24. 24. Examples
  25. 25. Handheld Displays • Mobile Phones • Camera • Display • Input
  26. 26. TRACKING
  27. 27. Objects Registered in 3D • Registration • Positioning virtual object wrt real world • Tracking • Continually locating the users viewpoint •  Position (x,y,z), Orientation (r,p,y)
  28. 28. Tracking Technologies "  Active •  Mechanical, Magnetic, Ultrasonic •  GPS, Wifi, cell location "  Passive •  Inertial sensors (compass, accelerometer, gyro) •  Computer Vision •  Marker based, Natural feature tracking "  Hybrid Tracking •  Combined sensors (eg Vision + Inertial)
  29. 29. Tracking Types Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking Mechanical Tracker
  30. 30. Example:Marker tracking •  Available for more than 10 years •  Several open source solutions exist •  ARToolKit, ARTag, ATK+, etc •  Fairly simple to implement •  Standard computer vision methods •  A rectangle provides 4 corner points •  Enough for pose estimation!
  31. 31. Marker Based Tracking: ARToolKit http://artoolkit.sourceforge.net/
  32. 32. Coordinate Systems
  33. 33. Tracking challenges inARToolKit False positives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)
  34. 34. Markerless Tracking Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking •  No more Markers! #Markerless Tracking Mechanical Tracker
  35. 35. Natural Feature Tracking • Use Natural Cues of Real Elements •  Edges •  Surface Texture •  Interest Points • Model or Model-Free • No visual pollution Contours Features Points Surfaces
  36. 36. TextureTracking
  37. 37. Edge Based Tracking •  RAPiD [Drummond et al. 02] •  Initialization, Control Points, Pose Prediction (Global Method)
  38. 38. Line Based Tracking • Visual Servoing [Comport et al. 2004]
  39. 39. Model BasedTracking • OpenTL - www.opentl.org •  General purpose library for model based visual tracking
  40. 40. Marker vs.natural feature tracking •  Marker tracking •  ++ Markers can be an eye-catcher •  ++ Tracking is less demanding •  -- The environment must be instrumented with markers •  -- Markers usually work only when fully in view •  Natural feature tracking •  -- A database of keypoints must be stored/downloaded •  ++ Natural feature targets might catch the attention less •  ++ Natural feature targets are potentially everywhere •  ++ Natural feature targets work also if partially in view
  41. 41. Example: Outdoor Hybrid Tracking • Combines • computer vision •  natural feature tracking • inertial gyroscope sensors • Both correct for each other • Inertial gyro - provides frame to frame prediction of camera orientation • Computer vision - correct for gyro drift
  42. 42. Robust OutdoorTracking • Hybrid Tracking •  ComputerVision, GPS, inertial • Going Out •  Reitmayr & Drummond (Univ. Cambridge) Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reality. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.
  43. 43. Handheld Display
  44. 44. INTERACTION
  45. 45. • Interface Components • Physical components • Display elements • Visual/audio • Interaction metaphors Physical Elements Display ElementsInteraction MetaphorInput Output AR Interface Elements
  46. 46. AR Design Space Reality Virtual Reality Augmented Reality Physical Design Virtual Design
  47. 47. Interface Design Path 1/ Prototype Demonstration 2/ Adoption of Interaction Techniques from other interface metaphors 3/ Development of new interface metaphors appropriate to the medium 4/ Development of formal theoretical models for predicting and modeling user actions Desktop WIMP Virtual Reality Augmented Reality
  48. 48. Interaction Development • Information Browsing • Camera movement • Limited interaction • 3D AR Interaction • HMD, hand tracking • 3D UI/VR techniques • Specialized input devices
  49. 49. Tangible User Interfaces (Ishii 97) • Augmented Surfaces • Rekimoto 1998 • Multiple projection surfaces • Tangible prop interaction • i/O Brush (2004) • Ryokai, Marti, Ishii • Sensor enhanced real brush
  50. 50. Other Examples • Triangles (Gorbert 1998) • Triangular based story telling • ActiveCube (Kitamura 2000-) • Cubes with sensors
  51. 51. Lessons fromTangible Interfaces • Benefits • Physical objects make us smart (affordances) • Objects aid collaboration • Objects increase understanding (cognitive artifacts) • Limitations • Difficult to change object properties • Limited display capabilities (2D surface) • Separation between object and display
  52. 52. Orthogonal Nature ofAR Interfaces
  53. 53. TangibleAR Interaction • AR overcomes limitation of TUIs • enhance display possibilities • merge task/display space • provide public and private views • TUI + AR = Tangible AR • Apply TUI methods to AR interface design
  54. 54. Tangible AR Design Principles • Tangible AR Interfaces use TUI principles • Physical controllers for moving virtual content • Support for spatial 3D interaction techniques • Support for multi-handed interaction • Match object affordances to task requirements • Support parallel activity with multiple objects • Allow collaboration between multiple users
  55. 55. VOMAR -TangibleAR Interface • Use of natural physical object to control virtual objects • Physical objects • Catalog book: •  Turn over the page • Paddle operation: •  Push, shake, incline, hit, scoop Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000). Virtual object manipulation on a table-top AR environment. In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium on (pp. 111-119). Ieee.
  56. 56. Interaction Evolution • Browsing Interfaces • simple (conceptually!), unobtrusive • 3D AR Interfaces • expressive, creative, require attention • Tangible Interfaces • Embedded into conventional environments • Tangible AR • Combines TUI input + AR display
  57. 57. AR APPLICATIONS
  58. 58. •  Web based AR •  Flash, HTML 5 based AR •  Marketing, education •  Outdoor Mobile AR •  GPS, compass tracking •  Viewing Points of Interest in real world •  Eg: Junaio, Layar, Wikitude •  Handheld AR •  Vision based tracking •  Marketing, gaming •  Location Based Experiences •  HMD, fixed screens •  Museums, point of sale, advertising Typical AR Experiences
  59. 59. CityViewARApplication •  Visualize Christchurch before the earthquakes
  60. 60. User Experience • MultipleViews • MapView,ARView, ListView • Multiple Data Types • 2D images, 3D content, text, panoramas
  61. 61. Warp Runner • Puzzle solving game • Deform real world terrain
  62. 62. Demo:colAR • Turn colouring books pages into AR scenes • Markerless tracking, use your own colours.. • Try it yourself: http://www.colARapp.com/
  63. 63. What Makes a GoodAR Experience? • Compelling • Engaging,‘Magic’ moment • Intuitive, ease of use • Uses existing skills • Anchored in physical world • Seamless combination of real and digital
  64. 64. Conclusion • AR seamlessly blends real and virtual imagery •  Interactive in real time, fixed in space • AR has developed into a mass market technology •  Education, engineering, entertainment • The technologies to create AR are available •  Display, tracking, interaction

×