SVR2011 Keynote


Published on

Mark Billinghurst's keynote talk given at the SVR 2011 conference on Augmented and Virtual Reality in Brazil - May 24th 2011

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SVR2011 Keynote

  1. 1. Research Directionsin Augmented Reality Mark Billinghurst The HIT Lab NZUniversity of Canterbury
  2. 2. Augmented Reality DefinitionDefining Characteristics [Azuma 97] Combines Real and Virtual Images - Both can be seen at the same time Interactive in real-time - The virtual content can be interacted with Registered in 3D - Virtual objects appear fixed in space
  3. 3. Augmented Reality Examples Put AR pictures here
  4. 4. Virtual Reality1989…
  5. 5. Virtual RealityImmersive VR Head mounted display, gloves Separation from the real world
  6. 6. AR vs VRVirtual Reality: Replaces Reality Scene Generation: requires realistic images Display Device: fully immersive, wide FOV Tracking and Sensing: low accuracy is okayAugmented Reality: Enhances Reality Scene Generation: minimal rendering okay Display Device: non-immersive, small FOV Tracking and Sensing: high accuracy needed
  7. 7. Milgram’s Reality-Virtuality continuum Mixed Reality Real Augmented Augmented VirtualEnvironment Reality (AR) Virtuality (AV) Environment Reality - Virtuality (RV) Continuum
  8. 8. AR History
  9. 9. AR Beginnings1960’s: Sutherland / Sproull’sfirst HMD system was see-through
  10. 10. 1960 - 80’s: US Air Force SuperCockpit (T. Furness)
  11. 11. Early 1990’s: Boeing coined theterm “AR.” Wire harnessassembly application begun (T.Caudell, D. Mizell).Early to mid 1990’s: UNCultrasound visualization projectEarly 1990’s: Boeing coined the term “AR.” Wire harnessassembly application begun (T. Caudell, D. Mizell).1994 - : UNC Research Motion stabilized display, Hybrid tracking, Ultrasound visualization
  12. 12. A Brief History of AR1996: MIT Wearable Computing efforts1998: Dedicated conferences beginLate 90’s: Collaboration, outdoor, interactionLate 90’s: Augmented sports broadcasts1998 - 2001: Mixed Reality Systems Lab
  13. 13. History Summary1960’s – 80’s: Early Experimentation1980’s – 90’s: Basic Research Tracking, displays1995 – 2005: Tools/Applications Interaction, usability, theory2005 - : Commercial Applications Games, Medical, Industry
  14. 14. Medical AR Trials Sauer et al. 2000 at Siemens Corporate Research, NJ Stereo video see throughF. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with aSingle-Camera Tracker: System Design and Needle Biopsy Phantom Trial, MICCAI 2002
  15. 15. AR Reaches Mainstream MIT Technology Review March 2007 list of the 10 most exciting technologies Economist Dec 6th 2007 Reality, only better
  16. 16. Virtual Reality, Augmented Reality
  17. 17. Esquire MagazineDec 2009 issue12 pages AR content
  18. 18. Trend One: Browser Based ARAdobe Flash + camera + 3D graphicsHigh impact High marketing valueLarge potential install base 1.6 Billion web usersEase of development Lots of developers, mature toolsLow cost of entry Browser, web camera
  19. 19. 1983 – Star Wars
  20. 20. 1999: AR Face to Face Collaboration
  21. 21. 1998: SGI O2 2008: Nokia N95CPU: 300 Mhz CPU: 332 MhzHDD; 9GB HDD; 8GBRAM: 512 mb RAM: 128 mbCamera: VGA 30fps Camera: VGA 30 fpsGraphics: 500K poly/sec Graphics: 2m poly/sec
  22. 22. Trend Two: Mobile Phone ARMobile Phones camera, sensors processor displayAR on Mobile Phones Simple graphics Optimized computer vision Collaborative Interaction
  23. 23. Collaborative ARAR Tennis Shared AR content Two user game Audio + haptic feedback Bluetooth networking
  24. 24. Location Aware PhonesMotorola Droid Nokia Navigator
  25. 25. 2009 - Outdoor Information OverlayMobile phone basedTag real world locations GPS + Compass input Overlay graphics data on live videoApplications Travel guide, Advertising, etcWikitude, Layar, Junaio, etc.. Android based, Public API released
  26. 26. Layar ( based data GPS + compass location Map + camera viewAR Layers on real world Customized data Audio, 3D, 2D contentEasy authoringAndroid, iPhone
  27. 27. Android AR PlatformArchitectural ApplicationLoads 3D models a OBJ/MTL formatPositions content in space GPS, compassIntuitive user interface toolkit to modify the modelConnects to back end model database
  28. 28. Mobile Outdoor AR
  29. 29. Client/Server Web Interface Add models Web applicationjava and php server Android applicationDatabase server Postgres
  30. 30. $784 million USD in 2014
  31. 31. SummaryAugmented Reality has a long history goingback to the 1960’sInterest in AR has exploded over the last twoyears and is being commercialized quicklyAR is growing in a number of areas Mobile AR Web based AR Advertising experiences
  32. 32. Looking to the Future
  33. 33. What’s Next? Sony CSL © 2004
  34. 34. “The product is no longer the basis of value. The experience is.” Venkat Ramaswamy The Future of Competition.
  35. 35. PS3 - Eye of JudgementComputer Vision TrackingCard based battle gameCollaborative AROctober 24th 2007
  36. 36. Building Compelling AR Experiences experiences Usability applications Interaction tools Authoring components Tracking, Display Sony CSL © 2004
  37. 37. AR Components
  38. 38. Building Compelling AR Experiences experiences applications tools components Tracking, Display Sony CSL © 2004
  39. 39. Low Level AR LibrariesARToolKit Enhancements Occlusion HandlingSSTT Simple Spatial Template TrackingOpira Robust Natural Feature Tracking
  40. 40. Markerless Tracking
  41. 41. AR Tools
  42. 42. Building Compelling AR Experiences experiences applications tools Authoring components Tracking, Display Sony CSL © 2004
  43. 43. AR AuthoringSoftware Libraries OSGART, Studierstube, MXRToolKitPlugin to existing software DART (Macromedia Director)Stand Alone AMIRE, etcNext Generation iaTAR (Tangible AR)
  44. 44. mARx Plug-in3D Studio Max Plug-inCan model and view AR content at the same time
  45. 45. BuildAR alone applicationVisual interface for AR model viewing applicationEnables non-programmers to build AR scenes
  46. 46. AR Applications
  47. 47. Building Compelling AR Experiences experiences applications Interaction tools Authoring components Tracking, Display Sony CSL © 2004
  48. 48. AR Design PrinciplesInterface Components Physical components Display elements - Visual/audio Interaction metaphors Physical Display Elements Interaction Elements Metaphor Input Output
  49. 49. Tangible User Interfaces (Ishii 97)Create digital shadows forphysical objectsForeground graspable UIBackground ambient interfaces
  50. 50. Tangible AR MetaphorAR overcomes limitation of TUIs enhance display possibilities merge task/display space provide public and private viewsTUI + AR = Tangible AR Apply TUI methods to AR interface design
  51. 51. Tangible AR Design PrinciplesTangible AR Interfaces use TUI principles Physical controllers for moving virtual content Support for spatial 3D interaction techniques Support for multi-handed interaction Match object affordances to task requirements Support parallel activity with multiple objects Allow collaboration between multiple users
  52. 52. Case Study: 3D AR LensGoal: Develop a lens based AR interface MagicLenses Developed at Xerox PARC in 1993 View a region of the workspace differently to the rest Overlap MagicLenses to create composite effects
  53. 53. 3D MagicLensesMagicLenses extended to 3D (Veiga et. al. 96) Volumetric and flat lenses
  54. 54. AR Lens Design PrinciplesPhysical Components Lens handle - Virtual lens attached to real objectDisplay Elements Lens view - Reveal layers in datasetInteraction Metaphor Physically holding lens
  55. 55. 3D AR Lenses: Model ViewerDisplays models made up of multiple partsEach part can be shown or hidden through the lensAllows the user to peer inside the modelMaintains focus + context
  56. 56. AR Lens Demo
  57. 57. HMD vs Handheld AR Interface Wearable AR HandHeld AR Output: Display Input & Output Input
  58. 58. Handheld Interface MetaphorsTangible AR Lens Viewing Look through screen into AR scene Interact with screen to interact with AR content - Eg Invisible TrainTangible AR Lens Manipulation Select AR object and attach to device Use the motion of the device as input - Eg AR Lego
  59. 59. Next Interaction TechniquesNatural Gestures Depth sensing Natural body inputMultimodal Speech + gesture
  60. 60. AR Experiences
  61. 61. Building Compelling AR Experiences experiences Usability applications Interaction tools Authoring components Tracking, Display Sony CSL © 2004
  62. 62. Survey of AR PapersEdward Swan (2005)Surveyed major conference/journals (1992-2004) - Presence, ISMAR, ISWC, IEEE VRSummary 1104 total papers 266 AR papers 38 AR HCI papers (Interaction) 21 AR user studiesOnly 21 from 266 AR papers have formal user study(<8% of all AR papers)
  63. 63. Types of ExperimentsPerception How is virtual content perceived ? What perceptual cues are most important ?Interaction How can users interact with virtual content ? Which interaction techniques are most efficient ?Collaboration How is collaboration in AR interface different ? Which collaborative cues can be conveyed best ?
  64. 64. AR Browser InterfaceLayar ( show POI on real worldTypical Interface Elements Live camera view Radar view Virtual graphics of POI 2D map view Information area
  65. 65. NavigationHow useful is AR viewfor navigation ego- vs. exo-centricExperiment AR only Map only AR + map
  66. 66. Experiment DesignConditions AR: Using only an AR view 2D-map: Using only a top down 2D map view AR+2D-map: Using both an AR and 2D map viewMeasures Time to complete, Distance travelled User preference, subjective measures
  67. 67. Paths Walkedthree different paths walked around campusbetween buildings/under trees
  68. 68. Performance MeasuresAR+Map AR+Map Map Map AR AR 0 200 400 600 800 1000 0 200 400 600 800 1000 1200 1400 Average Time Taken (sec) Average Distance Travelled (m) No difference between conditions
  69. 69. Path Trails
  70. 70. User FeedbackAR + Map easy to identifypoints of interestAR only hard to knowwhere things wereLiked being able toswitch between modesAR+Map preferred best
  71. 71. Typical User Comments“With the AR mode, I didn’t know where any of thebuildings were, a couple of times I went round in acircle because I didn’t know where things were.”“I found the map interface the best one to usebecause you are actually able to see the physicalobjects around you"“I used the map at the beginning to understand wherethe buildings were and the AR between each point”
  72. 72. Navigation ConclusionAR alone provides no improvement Lack of depth cues Difficult to create spatial awarenessAR + Map preferred interface Map for creating mental mode AR for near navigation
  73. 73. Conclusions
  74. 74. “We’re living in the experience economy and the customer is the star of the show. If I’m going to spend thousands of dollars on something. I want the whole experience to be a fairy-tale” Milton Pedraza, The Luxury Institute Illustrative
  75. 75. Building Compelling AR Experiences experiences Usability applications Interaction tools Authoring components Tracking, Display Sony CSL © 2004
  76. 76. ConclusionsAR is on the verge of commercializationThere are interesting research opportunities in Developing AR Component Technology Build Easy to Use Tools Identify Application Domains Develop Compelling AR Experiences
  77. 77. More Information• Mark Billinghurst –• Websites – – – –