Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Survey of Augmented Reality

11,555 views

Published on

A presentation given by Mark Billinghurst on April 21st 2015 at the CHI 2015 conference. This talk presents highlights from the journal paper:

M. Billinghurst, A. Clark, and G. Lee. A Survey
of Augmented Reality, Foundations and
Trends in Human-Computer Interaction.
Vol. 8, No. 1 (2015) 1–202, 2015

Available at :http://www.nowpublishers.com/article/Details/HCI-049

Published in: Technology

A Survey of Augmented Reality

  1. 1. A Survey of Augmented Reality Mark Billinghurst mark.billinghurst@hitlabnz.org April 21st 2014 mark.billinghurst@hitlabnz.org
  2. 2. Highlights from: M. Billinghurst, A. Clark, and G. Lee. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction. Vol. 8, No. 1 (2015) 1–202, 2015 http://www.nowpublishers.com/article/Details/HCI-049
  3. 3. Introduction
  4. 4. 1977 – Star Wars
  5. 5. Augmented Reality Definition !  Defining Characteristics [Azuma 97] ! Combines Real and Virtual Images -  Both can be seen at the same time ! Interactive in real-time -  The virtual content can be interacted with ! Registered in 3D -  Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
  6. 6. 2008 - CNN
  7. 7. !  Put AR pictures here Augmented Reality Examples
  8. 8. Virtual Reality !  Immersive VR !  Head mounted display, gloves !  Separation from the real world
  9. 9. AR vs VR
  10. 10. Making Interfaces Invisible Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.
  11. 11. From Reality to Virtual Reality Ubiquitous Computing Augmented Reality Virtual Reality
  12. 12. Milgram’s Reality-Virtuality continuum Mixed Reality Reality - Virtuality (RV) Continuum Real Environment Augmented Reality (AR) Augmented Virtuality (AV) Virtual Environment "...anywhere between the extrema of the virtuality continuum." P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
  13. 13. Metaverse !  Neal Stephenson’s “SnowCrash” !  The Metaverse is the convergence of: !  1) virtually enhanced physical reality !  2) physically persistent virtual space !  Metaverse Roadmap !  http://metaverseroadmap.org/
  14. 14. Summary !  Augmented Reality has three key features !  Combines Real and Virtual Images !  Interactive in real-time !  Registered in 3D !  AR can be classified alongside other technologies !  Invisible Interfaces !  Milgram’s Mixed Reality continuum !  Stephenson’s MetaVerse
  15. 15. History
  16. 16. Pepper’s Ghost (1862) !  Dates back to Giambattista della Porta (1584)
  17. 17. Early HMDs and HUDs (1960’s) Sutherland / Sproull’s see- through HMD (1965) Bucaneer HUD (1958)
  18. 18. AR History !  1960’s – 80’s: Early Experimentation •  Military, Academic labs !  1980’s – 90’s: Basic Research •  Tracking, Displays !  1995 – 2005: Tools/Applications •  Interaction, Usability, Theory !  2005 - : Commercial Applications •  Games, Medical, Industry, Mobile
  19. 19. Important Milestones !  1992: T. Caudell coined term “AR” (Boeing) !  1996: First collaborative AR systems !  1999: ARToolKit released – tracking library !  1997: Feiner First outdoor AR system !  1997 – 2001: Mixed Reality Systems Lab (Japan) !  1999 – 2003: ARVIKA project (Germany) !  2002: ISMAR conference started
  20. 20. AR Enters Mainstream (2007 -) !  Magazines !  MIT Tech. Review (Mar 2007) -  10 most exciting technologies !  Economist (Dec. 2007) -  Reality, only better !  Games !  Sony “Eye of Judgement” -  300,000+ units shipped !  Broadcast TV !  Sports broadcasting
  21. 21. Google Searches for AR
  22. 22. Browser Based AR (2008 - ) !  Flash + Camera + 3D graphics !  High impact !  High marketing value !  Large potential install base !  1.6 Billion web users !  Ease of development !  Lots of developers, mature tools !  Low cost of entry !  Browser, web camera
  23. 23. Mobile AR (2005 - ) !  Mobile Phones !  Camera, processor, display !  Computer vision based AR !  Advertising !  HIT Lab NZ (2007) !  AR print advertisement !  Txt to download app
  24. 24. Mobile Outdoor AR (2009 - ) !  Mobile phones with GPS !  Tag real world locations !  GPS + Compass input !  Overlay graphics data on live video !  Applications !  Travel guide, Advertising, etc !  Wikitude, Layar, Junaio, etc.. !  Android/iOS based, Public API released Motorola Droid
  25. 25. AR Today ! Key Technologies Available -  Robust tracking (Computer Vision, GPS/sensors) -  Display (Handheld, HMDs) -  Input Devices (Kinect, etc) -  Developer tools (Qualcomm, Metaio, Wikitude) ! Commercial Business Growing -  Gaming, GPS/Mobile, Online Advertisement •  >$5 Billion USD by 2016 (Markets andMarkets) •  >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)
  26. 26. AR Business Today !  Marketing !  Web-based, mobile !  Mobile AR !  Geo-located information and service !  Gaming !  Mobile, Physical input (Kinect, PS Move) !  Upcoming areas !  Manufacturing, Medical, Military
  27. 27. Summary !  Augmented Reality has a long history going back to the 1960’s !  Interest in AR has exploded over the last few years and is being commercialized quickly !  AR is growing in a number of areas !  Mobile AR !  Web based AR !  Marketing experiences
  28. 28. Technology
  29. 29. Augmented Reality Definition !  Defining Characteristics ! Combines Real and Virtual Images -  Display Technology ! Interactive in real-time -  Interaction Technology ! Registered in 3D -  Tracking Technology
  30. 30. Display
  31. 31. Display Technologies " Types (Bimber/Raskar 2003) " Head attached •  Head mounted display/projector " Body attached •  Handheld display/projector " Spatial •  Spatially aligned projector/monitor
  32. 32. Display Taxonomy
  33. 33. Types of Head Mounted Displays Occluded See-thru Multiplexed
  34. 34. Optical see-through HMD Virtual images from monitors Real World Optical Combiners
  35. 35. Epson Moverio BT-200 ▪  Stereo see-through display ($700) ▪  960 x 540 pixels, 23 degree FOV, 60Hz, 88g ▪  Android Powered, separate controller ▪  VGA camera, GPS, gyro, accelerometer
  36. 36. Strengths of optical see-through !  Simpler (cheaper) !  Direct view of real world !  Full resolution, no time delay (for real world) !  Safety !  Lower distortion !  No eye displacement !  see directly through display
  37. 37. Video see-through HMD Video cameras Monitors Graphics Combiner Video
  38. 38. Vuzix Wrap 1200DXAR ▪  Stereo video see-through display ($1500) ■ Twin 852 x 480 LCD displays, 35 deg. FOV ■ Stereo VGA cameras ■ 3 DOF head tracking
  39. 39. Strengths of Video See-Through !  True occlusion !  Block image of real world !  Digitized image of real world !  Flexibility in composition !  Matchable time delays !  More registration, calibration strategies !  Wide FOV is easier to support !  wide FOV camera
  40. 40. Multiplexed Displays !  Above or below line of sight !  Strengths !  User has unobstructed view of real world !  Simple optics/cheap !  Weaknesses !  Direct information overlay difficult •  Display/camera offset from eyeline !  Wide FOV difficult
  41. 41. Google Glass ▪  Monocular see-through multiplexed display ▪  640 x 360 microprojector, 15 degree FOV ▪  5 MP camera, gyro, accelerometer
  42. 42. Display Technology !  Curved Mirror !  off-axis projection !  curved mirrors in front of eye !  high distortion, small eye-box !  Waveguide !  use internal reflection !  unobstructed view of world !  large eye-box
  43. 43. See-through thin displays !  Waveguide techniques for thin see-through displays !  Wider FOV, enable AR applications !  Social acceptability Opinvent Ora Lumus DK40
  44. 44. Spatial Augmented Reality !  Project onto irregular surfaces !  Geometric Registration !  Projector blending, High dynamic range !  Book: Bimber, Rasker “Spatial Augmented Reality”
  45. 45. Projector-based AR Examples: Raskar, MIT Media Lab Inami, Tachi Lab, U. Tokyo Projector Real objects with retroreflective covering User (possibly head-tracked)
  46. 46. Example of projector-based AR Ramesh Raskar (MIT Media Lab)
  47. 47. Head Mounted Projector !  NVIS P-50 HMPD !  1280x1024/eye !  Stereoscopic !  50 degree FOV !  www.nvis.com
  48. 48. HMD vs. HMPD Head Mounted Display Head Mounted Projected Display
  49. 49. CastAR - http://technicalillusions.com/ !  Stereo head worn projectors !  Interactive wand !  Rollable retro-reflective sheet
  50. 50. !  Designed for shared interaction
  51. 51. Video Monitor AR Video cameras Monitor Graphics Combiner Video Stereo glasses
  52. 52. Examples
  53. 53. Handheld Displays !  Mobile Phones !  Camera !  Display !  Input
  54. 54. Tracking
  55. 55. Objects Registered in 3D !  Registration !  Positioning virtual object wrt real world !  Tracking !  Continually locating the users viewpoint -  Position (x,y,z), Orientation (r,p,y)
  56. 56. Tracking Technologies !  Active •  Mechanical, Magnetic, Ultrasonic •  GPS, Wifi, cell location !  Passive •  Inertial sensors (compass, accelerometer, gyro) •  Computer Vision •  Marker based, Natural feature tracking !  Hybrid Tracking •  Combined sensors (eg Vision + Inertial)
  57. 57. Tracking Types Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking Mechanical Tracker
  58. 58. Example: Marker tracking !  Available for more than 10 years !  Several open source solutions exist !  ARToolKit, ARTag, ATK+, etc !  Fairly simple to implement !  Standard computer vision methods !  A rectangle provides 4 corner points !  Enough for pose estimation!
  59. 59. Marker Based Tracking: ARToolKit http://artoolkit.sourceforge.net/
  60. 60. Coordinate Systems
  61. 61. Tracking challenges in ARToolKit False positives and inter-marker confusion (image by M. Fiala) Image noise (e.g. poor lens, block coding / compression, neon tube) Unfocused camera, motion blur Dark/unevenly lit scene, vignetting Jittering (Photoshop illustration) Occlusion (image by M. Fiala)
  62. 62. Markerless Tracking Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking !  No more Markers! #Markerless Tracking Mechanical Tracker
  63. 63. Natural Feature Tracking !  Use Natural Cues of Real Elements !  Edges !  Surface Texture !  Interest Points !  Model or Model-Free !  No visual pollution Contours Features Points Surfaces
  64. 64. Texture Tracking
  65. 65. Edge Based Tracking !  RAPiD [Drummond et al. 02] !  Initialization, Control Points, Pose Prediction (Global Method)
  66. 66. Line Based Tracking !  Visual Servoing [Comport et al. 2004]
  67. 67. Model Based Tracking !  OpenTL - www.opentl.org !  General purpose library for model based visual tracking
  68. 68. Marker vs. natural feature tracking !  Marker tracking !  ++ Markers can be an eye-catcher !  ++ Tracking is less demanding !  -- The environment must be instrumented with markers !  -- Markers usually work only when fully in view !  Natural feature tracking !  -- A database of keypoints must be stored/downloaded !  ++ Natural feature targets might catch the attention less !  ++ Natural feature targets are potentially everywhere !  ++ Natural feature targets work also if partially in view
  69. 69. Example: Outdoor Hybrid Tracking !  Combines !  computer vision -  natural feature tracking !  inertial gyroscope sensors !  Both correct for each other !  Inertial gyro - provides frame to frame prediction of camera orientation !  Computer vision - correct for gyro drift
  70. 70. Robust Outdoor Tracking !  Hybrid Tracking !  Computer Vision, GPS, inertial !  Going Out !  Reitmayr & Drummond (Univ. Cambridge) Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reality. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.
  71. 71. Handheld Display
  72. 72. Interaction
  73. 73. !  Interface Components ! Physical components ! Display elements -  Visual/audio ! Interaction metaphors Physical Elements Display ElementsInteraction MetaphorInput Output AR Interface Elements
  74. 74. AR Design Space Reality Virtual Reality Augmented Reality Physical Design Virtual Design
  75. 75. Interface Design Path 1/ Prototype Demonstration 2/ Adoption of Interaction Techniques from other interface metaphors 3/ Development of new interface metaphors appropriate to the medium 4/ Development of formal theoretical models for predicting and modeling user actions Desktop WIMP Virtual Reality Augmented Reality
  76. 76. Interaction Development !  Information Browsing !  Camera movement !  Limited interaction !  3D AR Interaction !  HMD, hand tracking !  3D UI/VR techniques !  Specialized input devices
  77. 77. Tangible User Interfaces (Ishii 97) !  Augmented Surfaces !  Rekimoto 1998 !  Multiple projection surfaces !  Tangible prop interaction !  i/O Brush (2004) !  Ryokai, Marti, Ishii !  Sensor enhanced real brush
  78. 78. Other Examples !  Triangles (Gorbert 1998) !  Triangular based story telling !  ActiveCube (Kitamura 2000-) !  Cubes with sensors
  79. 79. Lessons from Tangible Interfaces !  Benefits !  Physical objects make us smart (affordances) !  Objects aid collaboration !  Objects increase understanding (cognitive artifacts) !  Limitations !  Difficult to change object properties !  Limited display capabilities (2D surface) !  Separation between object and display
  80. 80. Orthogonal Nature of AR Interfaces
  81. 81. Tangible AR Interaction !  AR overcomes limitation of TUIs !  enhance display possibilities !  merge task/display space !  provide public and private views !  TUI + AR = Tangible AR !  Apply TUI methods to AR interface design
  82. 82. Tangible AR Design Principles !  Tangible AR Interfaces use TUI principles !  Physical controllers for moving virtual content !  Support for spatial 3D interaction techniques !  Support for multi-handed interaction !  Match object affordances to task requirements !  Support parallel activity with multiple objects !  Allow collaboration between multiple users
  83. 83. VOMAR - Tangible AR Interface !  Use of natural physical object to control virtual objects !  Physical objects !  Catalog book: -  Turn over the page !  Paddle operation: -  Push, shake, incline, hit, scoop Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000). Virtual object manipulation on a table-top AR environment. In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium on (pp. 111-119). Ieee.
  84. 84. Wrap-up !  Browsing Interfaces !  simple (conceptually!), unobtrusive !  3D AR Interfaces !  expressive, creative, require attention !  Tangible Interfaces !  Embedded into conventional environments !  Tangible AR !  Combines TUI input + AR display
  85. 85. Typical Applications
  86. 86. !  Web based AR !  Flash, HTML 5 based AR !  Marketing, education !  Outdoor Mobile AR !  GPS, compass tracking !  Viewing Points of Interest in real world !  Eg: Junaio, Layar, Wikitude !  Handheld AR !  Vision based tracking !  Marketing, gaming !  Location Based Experiences !  HMD, fixed screens !  Museums, point of sale, advertising Typical AR Experiences
  87. 87. CityViewAR Application !  Visualize Christchurch before the earthquakes
  88. 88. User Experience !  Multiple Views !  Map View, AR View, List View !  Multiple Data Types !  2D images, 3D content, text, panoramas
  89. 89. Demo: Warp Runner !  Puzzle solving game !  Deform real world terrain
  90. 90. Demo: colAR !  Turn colouring books pages into AR scenes !  Markerless tracking, use your own colours.. !  Try it yourself: http://www.colARapp.com/
  91. 91. What Makes a Good AR Experience? !  Compelling !  Engaging, ‘Magic’ moment !  Intuitive, ease of use !  Uses existing skills !  Anchored in physical world !  Seamless combination of real and digital
  92. 92. Research Opportunities
  93. 93. Previous Research Themes !  Zhou et. al. (2008) analyzed all ISMAR papers Zhou, F., Duh, H. B. L., & Billinghurst, M. (2008). Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings ISMAR 2008, (pp. 193-202). IEEE Computer Society.
  94. 94. Possible Research Directions !  Tracking !  Wide area, Reliable indoor, Ubiquitous tracking !  Interaction !  Intelligent systems, Gesture, Collaborative systems !  Displays !  Wide FOV, Retinal Scanning, Contact Lens !  Social Acceptance !  Wearable AR, Handheld AR, Social Interactions
  95. 95. Wide Area Tracking !  Process !  Combine panorama’s into point cloud model (offline) !  Initialize camera tracking from point cloud !  Update pose by aligning camera image to point cloud !  Accurate to 25 cm, 0.5 degree over wide area Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking.In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.
  96. 96. Gesture Based Interaction !  Use free hand gestures to interact !  Depth camera, scene capture !  Multimodal input !  Combining speech and gesture HIT Lab NZ Microsoft Hololens Meta SpaceGlasses
  97. 97. Wide FOV Displays !  Wide FOV see-through display for AR !  LCD panel + edge light point light sources !  110 degree FOV Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014 Emerging Technologies (p. 20). ACM.
  98. 98. Social Acceptance !  People don’t want to look silly !  Only 12% of 4,600 adults would be willing to wear AR glasses !  20% of mobile AR browser users experience social issues !  Acceptance more due to Social than Technical issues !  Needs further study (ethnographic, field tests, longitudinal)
  99. 99. Conclusions
  100. 100. Conclusion !  AR seamlessly blends real and virtual imagery !  Interactive in real time, fixed in space !  AR has developed into a mass market technology !  Education, engineering, entertainment !  The technologies to create AR are available !  Display, tracking, interaction !  There are opportunities for significant research
  101. 101. Gartner Hype Cycle (2014)
  102. 102. Market Forecast Up to $120B by 2020 – 5 x VR market
  103. 103. User Forecast
  104. 104. More Information •  Mark Billinghurst –  Email: mark.billinghurst@hitlabnz.org –  Twitter: @marknb00 •  Website –  http://www.hitlabnz.org/

×