2013 Lecture 6: AR User Interface Design Guidelines

1. COSC 426: Augmented Reality Mark Billinghurst mark.billinghurst@hitlabnz.org August 16th 2013 Lecture 6: AR User Interface Guidelines

2. AR Authoring Tools   Low Level Software Libraries   osgART, Studierstube, MXRToolKit   Plug-ins to existing software   DART (Macromedia Director), mARx, Unity,   Stand Alone   AMIRE, BuildAR, Metaio Creator etc   Rapid Prototyping Tools   Flash, OpenFrameworks, Processing, Arduino, etc   Next Generation   iaTAR (Tangible AR)

3. BuildAR   http://www.buildar.co.nz/   Stand alone application   Visual interface for AR model viewing application   Enables non-programmers to build AR scenes

4. Scene Graph Example

5. osgART Approach: AR Scene Graph Video Geode Root Transform 3D Object Virtual Camera Projection matrix from tracker calibration Transformation matrix updated from marker tracking in realtimeVideo Layer Full-screen quad with live texture updated from Video source Orthographic projection

6. AR Interaction   Designing AR System = Interface Design   Using different input and output technologies   Objective is a high quality of user experience   Ease of use and learning   Performance and satisfaction

7. AR Information Browsers   Information is registered to real-world context   Hand held AR displays   Interaction   Manipulation of a window into information space   Applications   Context-aware information displays Rekimoto, et al. 1997

8. 3D AR Interfaces   Virtual objects displayed in 3D physical space and manipulated   HMDs and 6DOF head-tracking   6DOF hand trackers for input   Interaction   Viewpoint control   Traditional 3D user interface interaction: manipulation, selection, etc. Kiyokawa, et al. 2000

9. Augmented Surfaces   Rekimoto, et al. 1998   Front projection   Marker-based tracking   Multiple projection surfaces

10. Tangible AR: Time-multiplexed Interaction   Use of natural physical object manipulations to control virtual objects   VOMAR Demo   Catalog book: -  Turn over the page   Paddle operation: -  Push, shake, incline, hit, scoop

11.   Space-multiplexed   Many devices each with one function -  Quicker to use, more intuitive, clutter -  Real Toolbox   Time-multiplexed   One device with many functions -  Space efficient -  mouse

12. Object Based Interaction: MagicCup   Intuitive Virtual Object Manipulation on a Table-Top Workspace   Time multiplexed   Multiple Markers -  Robust Tracking   Tangible User Interface -  Intuitive Manipulation   Stereo Display -  Good Presence

13. Our system   Main table, Menu table, Cup interface

14. Wrap-up   Browsing Interfaces   simple (conceptually!), unobtrusive   3D AR Interfaces   expressive, creative, require attention   Tangible Interfaces   Embedded into conventional environments   Tangible AR   Combines TUI input + AR display

15. Designing AR Interfaces

16. The Interaction Design Process

17. How Would You Design This?   Put nice AR Picture here – and video

18. Or This?

20. AR Interaction Design   Designing AR System = Interface Design   Using different input and output technologies   Objective is a high quality of user experience   Ease of use and learning   Performance and satisfaction

21. AR UI Design   Consider your user   Follow good HCI principles   Adapt HCI guidelines for AR   Design to device constraints   Using Design Patterns to Inform Design   Design for you interface metaphor   Design for evaluation

22. Consider Your User   Consider context of user   Physical, social, emotional, cognitive, etc   Mobile Phone AR User   Probably Mobile   One hand interaction   Short application use   Need to be able to multitask   Use in outdoor or indoor environment   Want to enhance interaction with real world

23. Good HCI Principles  Affordance  Reducing cognitive overload  Low physical effort  Learnability  User satisfaction  Flexibility in use  Responsiveness and feedback  Error tolerance

24. Norman’s Principles of Good Practice   Ensure a high degree of visibility -  allow the user to work out the current state of the system and the range of actions possible.   Provide feedback -  continuous, clear information about the results of actions.   Present a good conceptual model -  allow the user to build up a picture of the way the system holds together, the relationships between its different parts and how to move from one state to the next.   Offer good mappings -  aim for clear, natural relationships between actions the user performs and the results they achieve.

25. Adapting Existing Guidelines   Mobile Phone AR   Phone HCI Guidelines   Mobile HCI Guidelines   HMD Based AR   3D User Interface Guidelines   VR Interface Guidelines   Desktop AR   Desktop UI Guidelines

26. iPhone Guidelines   Make it obvious how to use your content.   Avoid clutter, unused blank space, and busy backgrounds.   Minimize required user input.   Express essential information succinctly.   Provide a fingertip-sized target area for all links and controls.   Avoid unnecessary interactivity.   Provide feedback when necessary

27. Applying Principles to Mobile AR   Clean   Large Video View   Large Icons   Text Overlay   Feedback

28. AR vs. Non AR Design   Design Guidelines   Design for 3D graphics + Interaction   Consider elements of physical world   Support implicit interaction Characteristics Non-AR Interfaces AR Interfaces Object Graphics Mainly 2D Mainly 3D Object Types Mainly virtual objects Both virtual and physical objects Object behaviors Mainly passive objects Both passive and active objects Communication Mainly simple Mainly complex HCI methods Mainly explicit Both explicit and implicit

29. Maps vs. Junaio   Google Maps   2D, mouse driven, text/image heavy, exocentric   Junaio   3D, location driven, simple graphics, egocentric

30. Design to Device Constraints   Understand the platforms and design for limitations   Hardware, software platforms   Eg Handheld AR game with visual tracking   Use large screen icons   Consider screen reflectivity   Support one-hand interaction   Consider the natural viewing angle   Do not tire users out physically   Do not encourage fast actions   Keep at least one tracking surface in view 31 Art of Defense Game

31. Design Patterns “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice.” – Christopher Alexander et al. Use Design Patterns to Address Reoccurring Problems C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.

32. Handheld AR Design Patterns Title Meaning Embodied Skills Device Metaphors Using metaphor to suggest available player actions Body A&S Naïve physics Control Mapping Intuitive mapping between physical and digital objects Body A&S Naïve physics Seamful Design Making sense of and integrating the technological seams through game design Body A&S World Consistency Whether the laws and rules in physical world hold in digital world Naïve physics Environmental A&S Landmarks Reinforcing the connection between digital-physical space through landmarks Environmental A&S Personal Presence The way that a player is represented in the game decides how much they feel like living in the digital game world Environmental A&S Naïve physics Living Creatures Game characters that are responsive to physical, social events that mimic behaviours of living beings Social A&S Body A&S Body constraints Movement of one’s body position constrains another player’s action Body A&S Social A&S

33. Example: Seamless Design   Design to reduce seams in the user experience   Eg: AR tracking failure, change in interaction mode   Paparazzi Game   Change between AR tracking to accelerometer input

34. Example: Living Creatures   Virtual creatures should respond to real world events   eg. Player motion, wind, light, etc   Creates illusion creatures are alive in the real world   Sony EyePet   Responds to player blowing on creature 35

35. Designing for Children  Development Psychology Factors •  Motor Abilities •  Spatial Abilities •  Logic Abilities •  Attention Abilities Radu, Iulian, and Blair MacIntyre. "Using children's developmental psychology to guide augmented-reality design and usability." Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on. IEEE, 2012.

36. Motor Abilities Skill Type Challenging AR Interaction Multiple hand coordination Holding phone in one hand and using another hand to move marker Hand-eye coordination Using a marker to intercept a moving object Fine motor skills Moving a marker on a specified path Gross motor skills and endurance Turning body around to look at a panorama

37. Spatial Abilities Skill Type Challenging AR Interaction Spatial memory Remembering the configuration of a large virtual space while handheld screen shows a limited view Spatial Perception Understanding when a virtual item is on top of a physical item Spatial Visualization Predict when virtual objects are visible by other people or virtual characters

38. Attention and Logic Skill Type Challenging AR Interaction Divided attention Playing an AR game, and making sure to keep marker in view so tracking is not lost Selective and executive attention Playing an AR game while moving outdoors Skill Type Challenging AR Interaction Remembering and reversing Remembering how to recover from tracking loss Abstract over concrete thinking Understanding that virtual objects are computer generated, and they do not need to obey physical laws Attention Abilities Logic and Memory

39. Design Considerations  Combining Real and Virtual Images •  Perceptual issues  Interactive in Real-Time •  Interaction issues  Registered in 3D •  Technology issues

40. AR Perceptual Issues

41. AR and Perception  Creating the illusion that virtual images are seamlessly part of the real world •  Must match real and virtual cues •  Depth, occlusion, lighting, shadows..

42. AR as Perception Problem   Goal of AR to fool human senses – create illusion that real and virtual are merged   Depth   Size   Occlusion   Shadows   Relative motion   Etc..

43. Perceptual Issues  Combining multiple display modes •  Direct View, Stereo Video View, Graphics View  Conflict between display modes •  Mismatch between depth cues

44. Perceptual Issues  Static and Dynamic registration mismatch  Restricted Field of View  Mismatch of Resolution and Image clarity  Luminance mismatch  Contrast mismatch  Size and distance mismatch  Limited depth resolution  Vertical alignment mismatches  Viewpoint dependency mismatch

45. Depth Cues  Pictorial: visual cues •  Occlusion, texture, relative brightness  Kinetic: motion cues •  Relative motion parallax, motion perspective  Physiological: motion cues •  Convergence, accommodation  Binocular disparity •  Two different eye images

46. Use the Following Depth Cues   Movement parallax.   Icon/Object size (for close objects)   Linear perspective   To add side perspective bar.   Overlapping   Works if the objects are big enough   Shades and shadows.   Depends on the available computation

47. Provide Perspective Cue   Eg ground plane grid

48. Depth Perception

49. Information Presentation •  Amount of information •  Clutter, complexity •  Representation of information •  Navigation cues, POI representation •  Placement of information •  Head, body, world stabilized •  View combination •  Multiple views

50. Twitter 360   www.twitter-360.com   iPhone application   See geo-located tweets in real world   Twitter.com supports geo tagging

51. Wikitude – www.mobilizy.com Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah Blah

52. Information Filtering

53. Information Filtering

54. Outdoor AR: Limited FOV

55. Possible solutions   Overview + Detail   spatial separation; two views   Focus + Context   merges both views into one view   Zooming   temporal separation

56.   TU Graz – HIT Lab NZ - collaboration   Zooming panorama   Zooming Map Zooming Views

57. AR Interaction Issues

58.   Interface Components  Physical components  Display elements -  Visual/audio  Interaction metaphors Physical Elements Display ElementsInteraction MetaphorInput Output AR Design Elements

59. Physical Elements

60. AR Design Space Reality Virtual Reality Augmented Reality Physical Design Virtual Design

61. Design of Objects   Objects   Purposely built – affordances   “Found” – repurposed   Existing – already at use in marketplace   Affordance   The quality of an object allowing an action- relationship with an actor   An attribute of an object that allows people to know how to use it -  e.g. a door handle affords pulling

62. Norman on Affordances "...the term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used. [...] Affordances provide strong clues to the operations of things. Plates are for pushing. Knobs are for turning. Slots are for inserting things into. Balls are for throwing .. " (Norman, The Psychology of Everyday Things 1988, p.9)

63. Physical vs. Virtual Affordances   Physical affordances -  Physical and material aspects of real object  Virtual affordance -  Visual and perceived aspects of digital objects  AR is mixture of physical and virtual affordances   Physical -  Tangible controllers and objects   Virtual -  Virtual graphics and audio - 

64. Affordance Framework William W. Gaver. 1991. Technology affordances. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '91), Scott P. Robertson, Gary M. Olson, and Judith S. Olson (Eds.). ACM, New York, NY, USA, 79-84.

65. Affordance Led Design   Make affordances perceivable   Provide visual, haptic, tactile, auditory cues   Affordance Led Usability   Give feedback   Provide constraints   Use natural mapping   Use good cognitive model

66. Example: AR Chemistry   Tangible AR chemistry education (Fjeld) Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible interaction approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich, Switzerland.

67. Input Devices   Form informs function and use

68. Picking up an Atom

69. AR Interaction Metaphors

70.   Interface Components  Physical components  Display elements -  Visual/audio  Interaction metaphors Physical Elements Display ElementsInteraction MetaphorInput Output AR Design Elements

71. Interface Design Path 1/ Prototype Demonstration 2/ Adoption of Interaction Techniques from other interface metaphors 3/ Development of new interface metaphors appropriate to the medium 4/ Development of formal theoretical models for predicting and modeling user actions Desktop WIMP Virtual Reality Augmented Reality

72. Interface metaphors   Designed to be similar to a physical entity but also has own properties   e.g. desktop metaphor, search engine   Exploit user’s familiar knowledge, helping them to understand ‘the unfamiliar’   Conjures up the essence of the unfamiliar activity, enabling users to leverage of this to understand more aspects of the unfamiliar functionality   People find it easier to learn and talk about what they are doing at the computer interface in terms familiar to them

73. Example: The spreadsheet   Analogous to ledger sheet   Interactive and computational   Easy to understand   Greatly extending what accountants and others could do www.bricklin.com/history/refcards.htm

74. Why was it so good?   It was simple, clear, and obvious to the users how to use the application and what it could do   “it is just a tool to allow others to work out their ideas and reduce the tedium of repeating the same calculations.”   capitalized on user’s familiarity with ledger sheets   Got the computer to perform a range of different calculations in response to user input

75. The Star interface

76. Benefits of interface metaphors   Makes learning new systems easier   Helps users understand the underlying conceptual model   Can be innovative and enable the realm of computers and their applications to be made more accessible to a greater diversity of users

77. Problems with interface metaphors (Nielson, 1990)   Break conventional and cultural rules   e.g., recycle bin placed on desktop   Can constrain designers in the way they conceptualize a problem   Conflict with design principles   Forces users to only understand the system in terms of the metaphor   Designers can inadvertently use bad existing designs and transfer the bad parts over   Limits designers’ imagination with new conceptual models

78. Microsoft Bob

79.   PSDoom – killing processes

80. Back to the Real World   AR overcomes limitation of TUIs   enhance display possibilities   merge task/display space   provide public and private views   TUI + AR = Tangible AR   Apply TUI methods to AR interface design

81. Tangible AR Design Principles   Tangible AR Interfaces use TUI principles   Physical controllers for moving virtual content   Support for spatial 3D interaction techniques   Time and space multiplexed interaction   Support for multi-handed interaction   Match object affordances to task requirements   Support parallel activity with multiple objects   Allow collaboration between multiple users