Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DESIGNING OUTSTANDING
AUGMENTED REALITY
EXPERIENCES
Mark Billinghurst
mark.billinghurst@unisa.edu.au
May 17th 2016
Shenzen...
•  adsf
•  jg
The Digital Divide
•  Screens are windows into digital spaces
•  Separation between digital and physical world
Augmented Reality
Vuforia Smart Terrain
Augmented Reality
Technology that:
1.  Combines Real andVirtual Images
• Both can be seen at the same time
2.  Interactive...
•  Web based AR
•  Flash, HTML 5
•  Marketing, education
•  Outdoor Mobile AR
•  GPS, compass tracking
•  Viewing Points o...
AR Business Today
• Around $600 Million USD in 2014 (>$2B 2016)
• 70-80+% Games and Marketing
Market Projections
cf. 2014 computer game market = $84 Billion USD
DESIGNING AR
EXPERIENCES
What Makes a Good AR Experience?
• Compelling
• Engaging,‘Magic’ moment
• Intuitive, ease of use
• Uses existing skills
• ...
Example: colAR (Quiver)
• Turn colouring book pages into AR scenes
• Markerless tracking, use your own colours..
• Try it ...
• Interface Components
• Physical components
• Display elements
• Visual/audio
• Interaction metaphors
Physical
Elements
D...
AR Interface Design
• Combines physical + virtual object design
• Physical
• Tangible controllers and objects
• Virtual
• ...
Physical Design
• Industrial Design
• Type of Objects
•  Purposely built – affordances
•  “Found” – repurposed
•  Existing...
How are these used?
”… the perceived and actual properties of the
thing, primarily those fundamental properties
that determine just how the th...
Physical vs. Virtual Affordances
• Physical Affordances
•  Look and feel of real objects
•  Shape, texture, colour, weight...
Design of AR Enhanced Objects
• Make affordances obvious
• Object affordances visible
• Give feedback
• Provide constraint...
Example: Haunted Book/AR Book
•  Camera hidden in lamp object
•  AR content seamlessly integrated into real book
•  Natura...
Interface Design Path
1/ Demonstration:Working Prototype
2/ Copying:Adoption of Interaction Techniques
from other interfac...
Example: VR Interfaces
•  Copying: Virtual Windows/keyboards
•  Creation: World in Miniature
AR Interaction Metaphors
• AR Lens/Window
• simple (conceptually!), unobtrusive
• 3D User Interfaces (VR)
• expressive, cr...
AR Lens
• Information is registered to
real-world context
•  Hand held AR displays
• Interaction
•  2D/3D virtual viewpoin...
3D AR Interfaces
•  Virtual objects displayed in 3D
physical space and manipulated
•  HMDs and 6DOF head-tracking
•  6DOF ...
Tangible User Interfaces (Ishii 97)
• Create digital shadows
for physical objects
• Foreground
• graspable UI
• Background...
Tangible Interface: Augmented Groove
• Collaborative Instrument
• Physically Based Interaction
• Map actions to Midi outpu...
Lessons from Tangible Interfaces
• Advantages
•  Physical objects make us smart
•  Objects aid collaboration
•  Objects in...
Back to the Real World
• AR overcomes limitation of TUIs
•  enhance display possibilities
•  merge task/display space
•  p...
Tangible AR Design Principles
• Tangible AR Interfaces use TUI principles
• Physical controllers for moving virtual conten...
Tangible AR: Tiles
• Tiles semantics
• data tiles
• operation tiles
• Operation on tiles
• proximity
• spatial arrangement...
Example:LevelHead
Case Study:LevelHead
• Physical Components
•  Real blocks
• Display Elements
•  Virtual person and rooms
• Interaction Met...
Design for Technology Limitations
• Understand the platforms used and design for limitations
•  Hardware, software platfor...
Seamless Design
• Design to reduce seams in the user experience
•  Eg: AR tracking failure, change in interaction mode
• P...
Consider the Whole User
Consider Your User
• Mobile Phone AR User
• Probably Mobile
• One hand interaction
• Short application use
• Need to be ab...
Social Acceptance
• People don’t want to look silly
•  Only 12% of 4,600 adults would be willing to wear AR glasses
•  20%...
TAT Augmented ID
Google Glass Anyone?
Designing AR Experiences
• Create a compelling experience
• Intuitive and ease of use
• Anchor in the real world
• Design ...
RESEARCH DIRECTIONS
Looking to the Future
What’s Next?
Research Enables New Experiences
• Gesture/multimodal Input
• Natural user interaction
• Collaborative experiences
• Captu...
Gesture Based Interaction
• Use free hand gestures to interact
•  Depth camera, scene capture
• Multimodal input
•  Combin...
Social Panoramas
• Google Glass
• Capture live image panorama (compass + camera)
• Remote device (tablet)
• Immersive view...
Empathy Glasses
•  Combine together eye-tracking, display, face expression
•  Implicit cues – eye gaze, face expression
++...
Remote Collboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaborat...
Holoportation
•  Augmented Reality + 3D capture + high bandwidth
•  http://research.microsoft.com/en-us/projects/holoporta...
CONCLUSION
Conclusions
• AR enables connection back to real world
• Great AR experiences need good design
• Physical + virtual compon...
www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au
Designing Outstanding AR Experiences
Designing Outstanding AR Experiences
Designing Outstanding AR Experiences
Upcoming SlideShare
Loading in …5
×

Designing Outstanding AR Experiences

1,983 views

Published on

Presentation on how to design great AR experiences. Given by Mark Billinghurst at the Huawei STW 2016 conference on May 17th, 2016 in Shenzen, China.

Published in: Technology
  • You have to choose carefully. ⇒ www.HelpWriting.net ⇐ offers a professional writing service. I highly recommend them. The papers are delivered on time and customers are their first priority. This is their website: ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! I have searched hard to find a reliable and best research paper writing service and finally i got a good option for my needs as ⇒ www.WritePaper.info ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I can advise you this service - HelpWriting.net Bought essay here. No problem.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Designing Outstanding AR Experiences

  1. 1. DESIGNING OUTSTANDING AUGMENTED REALITY EXPERIENCES Mark Billinghurst mark.billinghurst@unisa.edu.au May 17th 2016 Shenzen, China
  2. 2. •  adsf
  3. 3. •  jg
  4. 4. The Digital Divide •  Screens are windows into digital spaces •  Separation between digital and physical world
  5. 5. Augmented Reality Vuforia Smart Terrain
  6. 6. Augmented Reality Technology that: 1.  Combines Real andVirtual Images • Both can be seen at the same time 2.  Interactive in real-time • The virtual content can be interacted with 3.  Registered in 3D • Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
  7. 7. •  Web based AR •  Flash, HTML 5 •  Marketing, education •  Outdoor Mobile AR •  GPS, compass tracking •  Viewing Points of Interest in real world •  Handheld AR •  Vision based tracking •  Marketing, gaming •  Location Based Experiences •  HMD, fixed screens •  Museums, point of sale, advertising Typical AR Experiences
  8. 8. AR Business Today • Around $600 Million USD in 2014 (>$2B 2016) • 70-80+% Games and Marketing
  9. 9. Market Projections cf. 2014 computer game market = $84 Billion USD
  10. 10. DESIGNING AR EXPERIENCES
  11. 11. What Makes a Good AR Experience? • Compelling • Engaging,‘Magic’ moment • Intuitive, ease of use • Uses existing skills • Anchored in physical world • Seamless combination of real and digital
  12. 12. Example: colAR (Quiver) • Turn colouring book pages into AR scenes • Markerless tracking, use your own colours.. • Try it yourself: http://www.colARapp.com/
  13. 13. • Interface Components • Physical components • Display elements • Visual/audio • Interaction metaphors Physical Elements Display ElementsInteraction MetaphorInput Output AR Interface Elements
  14. 14. AR Interface Design • Combines physical + virtual object design • Physical • Tangible controllers and objects • Virtual • Virtual graphics and audio
  15. 15. Physical Design • Industrial Design • Type of Objects •  Purposely built – affordances •  “Found” – repurposed •  Existing – already at use in marketplace
  16. 16. How are these used?
  17. 17. ”… the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. [...] Affordances provide strong clues to the operations of things.” (Norman, The Psychology of EverydayThings 1988, p.9) Affordance
  18. 18. Physical vs. Virtual Affordances • Physical Affordances •  Look and feel of real objects •  Shape, texture, colour, weight, etc •  Industrial Design • Virtual Affordance •  Look of virtual objects •  Copy real objects •  Interface Design
  19. 19. Design of AR Enhanced Objects • Make affordances obvious • Object affordances visible • Give feedback • Provide constraints • Use natural mapping • Use good cognitive model
  20. 20. Example: Haunted Book/AR Book •  Camera hidden in lamp object •  AR content seamlessly integrated into real book •  Natural page turning/manipulation interaction
  21. 21. Interface Design Path 1/ Demonstration:Working Prototype 2/ Copying:Adoption of Interaction Techniques from other interface metaphors 3/ Creation: Development of new interface metaphors appropriate to the medium 4/ Theory: Development of formal theoretical models for predicting and modeling user actions Desktop WIMP Virtual Reality Augmented Reality
  22. 22. Example: VR Interfaces •  Copying: Virtual Windows/keyboards •  Creation: World in Miniature
  23. 23. AR Interaction Metaphors • AR Lens/Window • simple (conceptually!), unobtrusive • 3D User Interfaces (VR) • expressive, creative, require attention • Tangible Interfaces (TUI) • Embedded into conventional environments • Tangible AR • Combines TUI input + AR display
  24. 24. AR Lens • Information is registered to real-world context •  Hand held AR displays • Interaction •  2D/3D virtual viewpoint control •  Limited input/interactivity • Applications •  Context-aware information Tourism, gaming
  25. 25. 3D AR Interfaces •  Virtual objects displayed in 3D physical space and manipulated •  HMDs and 6DOF head-tracking •  6DOF hand trackers for input •  Interaction •  Viewpoint control •  Traditional 3D interaction: •  manipulation, selection, etc. •  VR techniques
  26. 26. Tangible User Interfaces (Ishii 97) • Create digital shadows for physical objects • Foreground • graspable UI • Background • ambient interfaces
  27. 27. Tangible Interface: Augmented Groove • Collaborative Instrument • Physically Based Interaction • Map actions to Midi output •  Translation, rotation •  Tilt, shake
  28. 28. Lessons from Tangible Interfaces • Advantages •  Physical objects make us smart •  Objects aid collaboration •  Objects increase understanding • Disadvantages •  Difficult to change object properties •  Limited display capabilities •  Separation between object and display
  29. 29. Back to the Real World • AR overcomes limitation of TUIs •  enhance display possibilities •  merge task/display space •  provide public and private views • TUI + AR = Tangible AR •  Apply TUI methods to AR interface design •  TUI for input, AR for output
  30. 30. Tangible AR Design Principles • Tangible AR Interfaces use TUI principles • Physical controllers for moving virtual content • Support for spatial 3D interaction techniques • Time and space multiplexed interaction • Support for multi-handed interaction • Match object affordances to task requirements • Support parallel activity with multiple objects • Allow collaboration between multiple users
  31. 31. Tangible AR: Tiles • Tiles semantics • data tiles • operation tiles • Operation on tiles • proximity • spatial arrangements • space-multiplexed Poupyrev, I., Tan, D., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001). Tiles: A mixed reality authoring interface. In INTERACT 2001 Conference on Human Computer Interaction (pp. 334-341).
  32. 32. Example:LevelHead
  33. 33. Case Study:LevelHead • Physical Components •  Real blocks • Display Elements •  Virtual person and rooms • Interaction Metaphor •  Blocks are rooms
  34. 34. Design for Technology Limitations • Understand the platforms used and design for limitations •  Hardware, software platforms • Eg Handheld AR game with visual tracking •  Use large screen icons •  Consider screen reflectivity •  Support one-hand interaction •  Consider the natural viewing angle •  Do not tire users out physically •  Do not encourage fast actions •  Keep at least one tracking surface in view Art of Defense Game
  35. 35. Seamless Design • Design to reduce seams in the user experience •  Eg: AR tracking failure, change in interaction mode • Paparazzi Game •  Change between AR tracking to accelerometer input Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games, Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality-- Arts, Media, and Humanities, p.19-28, October 26-29, 2011
  36. 36. Consider the Whole User
  37. 37. Consider Your User • Mobile Phone AR User • Probably Mobile • One hand interaction • Short application use • Need to be able to multitask • Use in outdoor or indoor environment • Want to enhance interaction with real world
  38. 38. Social Acceptance • People don’t want to look silly •  Only 12% of 4,600 adults would be willing to wear AR glasses •  20% of mobile AR browser users experience social issues • Acceptance more due to Social than Technical issues •  Needs further study (ethnographic, field tests, longitudinal)
  39. 39. TAT Augmented ID
  40. 40. Google Glass Anyone?
  41. 41. Designing AR Experiences • Create a compelling experience • Intuitive and ease of use • Anchor in the real world • Design affordances for physical + virtual elements • Create the appropriate Interaction Metaphor • Design for technology limitations • Consider the whole user (Social, cultural, ..)
  42. 42. RESEARCH DIRECTIONS
  43. 43. Looking to the Future What’s Next?
  44. 44. Research Enables New Experiences • Gesture/multimodal Input • Natural user interaction • Collaborative experiences • Capturing rich communication cues • Experience capture • Sharing surroundings • Empathic Computing • Creating understanding • Etc..
  45. 45. Gesture Based Interaction • Use free hand gestures to interact •  Depth camera, scene capture • Multimodal input •  Combining speech and gesture HIT Lab NZ Microsoft Hololens Meta SpaceGlasses
  46. 46. Social Panoramas • Google Glass • Capture live image panorama (compass + camera) • Remote device (tablet) • Immersive viewing, live annotation Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on (pp. 303-304). IEEE.
  47. 47. Empathy Glasses •  Combine together eye-tracking, display, face expression •  Implicit cues – eye gaze, face expression ++ Pupil Labs Epson BT-200 AffectiveWear
  48. 48. Remote Collboration • Eye gaze pointer and remote pointing • Face expression display • Implicit cues for remote collaboration
  49. 49. Holoportation •  Augmented Reality + 3D capture + high bandwidth •  http://research.microsoft.com/en-us/projects/holoportation/
  50. 50. CONCLUSION
  51. 51. Conclusions • AR enables connection back to real world • Great AR experiences need good design • Physical + virtual components, interaction metaphor • Design for technology limitations • Need to consider whole user • Many directions for future research • Natural interaction, collaboration, experience capture
  52. 52. www.empathiccomputing.org @marknb00 mark.billinghurst@unisa.edu.au

×