2013 Lecture 6: AR User Interface Design Guidelines


Published on

COSC 426 Lecture 6: on AR User Interface Design Guidelines. Lecture taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury on August 16th 2013

Published in: Technology, News & Politics
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2013 Lecture 6: AR User Interface Design Guidelines

  1. 1. COSC 426: Augmented Reality Mark Billinghurst mark.billinghurst@hitlabnz.org August 16th 2013 Lecture 6: AR User Interface Guidelines
  2. 2. AR Authoring Tools   Low Level Software Libraries   osgART, Studierstube, MXRToolKit   Plug-ins to existing software   DART (Macromedia Director), mARx, Unity,   Stand Alone   AMIRE, BuildAR, Metaio Creator etc   Rapid Prototyping Tools   Flash, OpenFrameworks, Processing, Arduino, etc   Next Generation   iaTAR (Tangible AR)
  3. 3. BuildAR   http://www.buildar.co.nz/   Stand alone application   Visual interface for AR model viewing application   Enables non-programmers to build AR scenes
  4. 4. Scene Graph Example
  5. 5. osgART Approach: AR Scene Graph Video Geode Root Transform 3D Object Virtual Camera Projection matrix from tracker calibration Transformation matrix updated from marker tracking in realtimeVideo Layer Full-screen quad with live texture updated from Video source Orthographic projection
  6. 6. AR Interaction   Designing AR System = Interface Design   Using different input and output technologies   Objective is a high quality of user experience   Ease of use and learning   Performance and satisfaction
  7. 7. AR Information Browsers   Information is registered to real-world context   Hand held AR displays   Interaction   Manipulation of a window into information space   Applications   Context-aware information displays Rekimoto, et al. 1997
  8. 8. 3D AR Interfaces   Virtual objects displayed in 3D physical space and manipulated   HMDs and 6DOF head-tracking   6DOF hand trackers for input   Interaction   Viewpoint control   Traditional 3D user interface interaction: manipulation, selection, etc. Kiyokawa, et al. 2000
  9. 9. Augmented Surfaces   Rekimoto, et al. 1998   Front projection   Marker-based tracking   Multiple projection surfaces
  10. 10. Tangible AR: Time-multiplexed Interaction   Use of natural physical object manipulations to control virtual objects   VOMAR Demo   Catalog book: -  Turn over the page   Paddle operation: -  Push, shake, incline, hit, scoop
  11. 11.   Space-multiplexed   Many devices each with one function -  Quicker to use, more intuitive, clutter -  Real Toolbox   Time-multiplexed   One device with many functions -  Space efficient -  mouse
  12. 12. Object Based Interaction: MagicCup   Intuitive Virtual Object Manipulation on a Table-Top Workspace   Time multiplexed   Multiple Markers -  Robust Tracking   Tangible User Interface -  Intuitive Manipulation   Stereo Display -  Good Presence
  13. 13. Our system   Main table, Menu table, Cup interface
  14. 14. Wrap-up   Browsing Interfaces   simple (conceptually!), unobtrusive   3D AR Interfaces   expressive, creative, require attention   Tangible Interfaces   Embedded into conventional environments   Tangible AR   Combines TUI input + AR display
  15. 15. Designing AR Interfaces
  16. 16. The Interaction Design Process
  17. 17. How Would You Design This?   Put nice AR Picture here – and video
  18. 18. Or This?
  19. 19. experiences applications tools components Sony CSL © 2004 Building Compelling AR Experiences Tracking, Display Authoring Interaction
  20. 20. AR Interaction Design   Designing AR System = Interface Design   Using different input and output technologies   Objective is a high quality of user experience   Ease of use and learning   Performance and satisfaction
  21. 21. AR UI Design   Consider your user   Follow good HCI principles   Adapt HCI guidelines for AR   Design to device constraints   Using Design Patterns to Inform Design   Design for you interface metaphor   Design for evaluation
  22. 22. Consider Your User   Consider context of user   Physical, social, emotional, cognitive, etc   Mobile Phone AR User   Probably Mobile   One hand interaction   Short application use   Need to be able to multitask   Use in outdoor or indoor environment   Want to enhance interaction with real world
  23. 23. Good HCI Principles  Affordance  Reducing cognitive overload  Low physical effort  Learnability  User satisfaction  Flexibility in use  Responsiveness and feedback  Error tolerance
  24. 24. Norman’s Principles of Good Practice   Ensure a high degree of visibility -  allow the user to work out the current state of the system and the range of actions possible.   Provide feedback -  continuous, clear information about the results of actions.   Present a good conceptual model -  allow the user to build up a picture of the way the system holds together, the relationships between its different parts and how to move from one state to the next.   Offer good mappings -  aim for clear, natural relationships between actions the user performs and the results they achieve.
  25. 25. Adapting Existing Guidelines   Mobile Phone AR   Phone HCI Guidelines   Mobile HCI Guidelines   HMD Based AR   3D User Interface Guidelines   VR Interface Guidelines   Desktop AR   Desktop UI Guidelines
  26. 26. iPhone Guidelines   Make it obvious how to use your content.   Avoid clutter, unused blank space, and busy backgrounds.   Minimize required user input.   Express essential information succinctly.   Provide a fingertip-sized target area for all links and controls.   Avoid unnecessary interactivity.   Provide feedback when necessary
  27. 27. Applying Principles to Mobile AR   Clean   Large Video View   Large Icons   Text Overlay   Feedback
  28. 28. AR vs. Non AR Design   Design Guidelines   Design for 3D graphics + Interaction   Consider elements of physical world   Support implicit interaction Characteristics Non-AR Interfaces AR Interfaces Object Graphics Mainly 2D Mainly 3D Object Types Mainly virtual objects Both virtual and physical objects Object behaviors Mainly passive objects Both passive and active objects Communication Mainly simple Mainly complex HCI methods Mainly explicit Both explicit and implicit
  29. 29. Maps vs. Junaio   Google Maps   2D, mouse driven, text/image heavy, exocentric   Junaio   3D, location driven, simple graphics, egocentric
  30. 30. Design to Device Constraints   Understand the platforms and design for limitations   Hardware, software platforms   Eg Handheld AR game with visual tracking   Use large screen icons   Consider screen reflectivity   Support one-hand interaction   Consider the natural viewing angle   Do not tire users out physically   Do not encourage fast actions   Keep at least one tracking surface in view 31 Art of Defense Game
  31. 31. Design Patterns “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice.” – Christopher Alexander et al. Use Design Patterns to Address Reoccurring Problems C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.
  32. 32. Handheld AR Design Patterns Title Meaning Embodied Skills Device Metaphors Using metaphor to suggest available player actions Body A&S Naïve physics Control Mapping Intuitive mapping between physical and digital objects Body A&S Naïve physics Seamful Design Making sense of and integrating the technological seams through game design Body A&S World Consistency Whether the laws and rules in physical world hold in digital world Naïve physics Environmental A&S Landmarks Reinforcing the connection between digital-physical space through landmarks Environmental A&S Personal Presence The way that a player is represented in the game decides how much they feel like living in the digital game world Environmental A&S Naïve physics Living Creatures Game characters that are responsive to physical, social events that mimic behaviours of living beings Social A&S Body A&S Body constraints Movement of one’s body position constrains another player’s action Body A&S Social A&S
  33. 33. Example: Seamless Design   Design to reduce seams in the user experience   Eg: AR tracking failure, change in interaction mode   Paparazzi Game   Change between AR tracking to accelerometer input
  34. 34. Example: Living Creatures   Virtual creatures should respond to real world events   eg. Player motion, wind, light, etc   Creates illusion creatures are alive in the real world   Sony EyePet   Responds to player blowing on creature 35
  35. 35. Designing for Children  Development Psychology Factors •  Motor Abilities •  Spatial Abilities •  Logic Abilities •  Attention Abilities Radu, Iulian, and Blair MacIntyre. "Using children's developmental psychology to guide augmented-reality design and usability." Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on. IEEE, 2012.
  36. 36. Motor Abilities Skill Type Challenging AR Interaction Multiple hand coordination Holding phone in one hand and using another hand to move marker Hand-eye coordination Using a marker to intercept a moving object Fine motor skills Moving a marker on a specified path Gross motor skills and endurance Turning body around to look at a panorama
  37. 37. Spatial Abilities Skill Type Challenging AR Interaction Spatial memory Remembering the configuration of a large virtual space while handheld screen shows a limited view Spatial Perception Understanding when a virtual item is on top of a physical item Spatial Visualization Predict when virtual objects are visible by other people or virtual characters
  38. 38. Attention and Logic Skill Type Challenging AR Interaction Divided attention Playing an AR game, and making sure to keep marker in view so tracking is not lost Selective and executive attention Playing an AR game while moving outdoors Skill Type Challenging AR Interaction Remembering and reversing Remembering how to recover from tracking loss Abstract over concrete thinking Understanding that virtual objects are computer generated, and they do not need to obey physical laws Attention Abilities Logic and Memory
  39. 39. Design Considerations  Combining Real and Virtual Images •  Perceptual issues  Interactive in Real-Time •  Interaction issues  Registered in 3D •  Technology issues
  40. 40. AR Perceptual Issues
  41. 41. AR and Perception  Creating the illusion that virtual images are seamlessly part of the real world •  Must match real and virtual cues •  Depth, occlusion, lighting, shadows..
  42. 42. AR as Perception Problem   Goal of AR to fool human senses – create illusion that real and virtual are merged   Depth   Size   Occlusion   Shadows   Relative motion   Etc..
  43. 43. Perceptual Issues  Combining multiple display modes •  Direct View, Stereo Video View, Graphics View  Conflict between display modes •  Mismatch between depth cues
  44. 44. Perceptual Issues  Static and Dynamic registration mismatch  Restricted Field of View  Mismatch of Resolution and Image clarity  Luminance mismatch  Contrast mismatch  Size and distance mismatch  Limited depth resolution  Vertical alignment mismatches  Viewpoint dependency mismatch
  45. 45. Depth Cues  Pictorial: visual cues •  Occlusion, texture, relative brightness  Kinetic: motion cues •  Relative motion parallax, motion perspective  Physiological: motion cues •  Convergence, accommodation  Binocular disparity •  Two different eye images
  46. 46. Use the Following Depth Cues   Movement parallax.   Icon/Object size (for close objects)   Linear perspective   To add side perspective bar.   Overlapping   Works if the objects are big enough   Shades and shadows.   Depends on the available computation
  47. 47. Provide Perspective Cue   Eg ground plane grid
  48. 48. Depth Perception
  49. 49. Information Presentation •  Amount of information •  Clutter, complexity •  Representation of information •  Navigation cues, POI representation •  Placement of information •  Head, body, world stabilized •  View combination •  Multiple views
  50. 50. Twitter 360   www.twitter-360.com   iPhone application   See geo-located tweets in real world   Twitter.com supports geo tagging
  51. 51. Wikitude – www.mobilizy.com Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah
  52. 52. Information Filtering
  53. 53. Information Filtering
  54. 54. Outdoor AR: Limited FOV
  55. 55. Possible solutions   Overview + Detail   spatial separation; two views   Focus + Context   merges both views into one view   Zooming   temporal separation
  56. 56.   TU Graz – HIT Lab NZ - collaboration   Zooming panorama   Zooming Map Zooming Views
  57. 57. AR Interaction Issues
  58. 58.   Interface Components  Physical components  Display elements -  Visual/audio  Interaction metaphors Physical Elements Display ElementsInteraction MetaphorInput Output AR Design Elements
  59. 59. Physical Elements
  60. 60. AR Design Space Reality Virtual Reality Augmented Reality Physical Design Virtual Design
  61. 61. Design of Objects   Objects   Purposely built – affordances   “Found” – repurposed   Existing – already at use in marketplace   Affordance   The quality of an object allowing an action- relationship with an actor   An attribute of an object that allows people to know how to use it -  e.g. a door handle affords pulling
  62. 62. Norman on Affordances "...the term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. [...] Affordances provide strong clues to the operations of things. Plates are for pushing. Knobs are for turning. Slots are for inserting things into. Balls are for throwing .. " (Norman, The Psychology of Everyday Things 1988, p.9)
  63. 63. Physical vs. Virtual Affordances   Physical affordances -  Physical and material aspects of real object  Virtual affordance -  Visual and perceived aspects of digital objects  AR is mixture of physical and virtual affordances   Physical -  Tangible controllers and objects   Virtual -  Virtual graphics and audio - 
  64. 64. Affordance Framework William W. Gaver. 1991. Technology affordances. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '91), Scott P. Robertson, Gary M. Olson, and Judith S. Olson (Eds.). ACM, New York, NY, USA, 79-84.
  65. 65. Affordance Led Design   Make affordances perceivable   Provide visual, haptic, tactile, auditory cues   Affordance Led Usability   Give feedback   Provide constraints   Use natural mapping   Use good cognitive model
  66. 66. Example: AR Chemistry   Tangible AR chemistry education (Fjeld) Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible interaction approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich, Switzerland.
  67. 67. Input Devices   Form informs function and use
  68. 68. Picking up an Atom
  69. 69. AR Interaction Metaphors
  70. 70.   Interface Components  Physical components  Display elements -  Visual/audio  Interaction metaphors Physical Elements Display ElementsInteraction MetaphorInput Output AR Design Elements
  71. 71. Interface Design Path 1/ Prototype Demonstration 2/ Adoption of Interaction Techniques from other interface metaphors 3/ Development of new interface metaphors appropriate to the medium 4/ Development of formal theoretical models for predicting and modeling user actions Desktop WIMP Virtual Reality Augmented Reality
  72. 72. Interface metaphors   Designed to be similar to a physical entity but also has own properties   e.g. desktop metaphor, search engine   Exploit user’s familiar knowledge, helping them to understand ‘the unfamiliar’   Conjures up the essence of the unfamiliar activity, enabling users to leverage of this to understand more aspects of the unfamiliar functionality   People find it easier to learn and talk about what they are doing at the computer interface in terms familiar to them
  73. 73. Example: The spreadsheet   Analogous to ledger sheet   Interactive and computational   Easy to understand   Greatly extending what accountants and others could do www.bricklin.com/history/refcards.htm
  74. 74. Why was it so good?   It was simple, clear, and obvious to the users how to use the application and what it could do   “it is just a tool to allow others to work out their ideas and reduce the tedium of repeating the same calculations.”   capitalized on user’s familiarity with ledger sheets   Got the computer to perform a range of different calculations in response to user input
  75. 75. The Star interface
  76. 76. Benefits of interface metaphors   Makes learning new systems easier   Helps users understand the underlying conceptual model   Can be innovative and enable the realm of computers and their applications to be made more accessible to a greater diversity of users
  77. 77. Problems with interface metaphors (Nielson, 1990)   Break conventional and cultural rules   e.g., recycle bin placed on desktop   Can constrain designers in the way they conceptualize a problem   Conflict with design principles   Forces users to only understand the system in terms of the metaphor   Designers can inadvertently use bad existing designs and transfer the bad parts over   Limits designers’ imagination with new conceptual models
  78. 78. Microsoft Bob
  79. 79.   PSDoom – killing processes
  80. 80. Back to the Real World   AR overcomes limitation of TUIs   enhance display possibilities   merge task/display space   provide public and private views   TUI + AR = Tangible AR   Apply TUI methods to AR interface design
  81. 81. Tangible AR Design Principles   Tangible AR Interfaces use TUI principles   Physical controllers for moving virtual content   Support for spatial 3D interaction techniques   Time and space multiplexed interaction   Support for multi-handed interaction   Match object affordances to task requirements   Support parallel activity with multiple objects   Allow collaboration between multiple users