COSC 426 Lecture 6: on AR User Interface Design Guidelines. Lecture taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury on August 16th 2013

1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
August 16th 2013
Lecture 6: AR User Interface Guidelines

2. AR Authoring Tools
 Low Level Software Libraries
 osgART, Studierstube, MXRToolKit
 Plug-ins to existing software
 DART (Macromedia Director), mARx, Unity,
 Stand Alone
 AMIRE, BuildAR, Metaio Creator etc
 Rapid Prototyping Tools
 Flash, OpenFrameworks, Processing, Arduino, etc
 Next Generation
 iaTAR (Tangible AR)

3. BuildAR
 http://www.buildar.co.nz/
 Stand alone application
 Visual interface for AR model viewing application
 Enables non-programmers to build AR scenes

4. Scene Graph Example

5. osgART Approach: AR Scene Graph
Video
Geode
Root
Transform
3D Object
Virtual
Camera
Projection matrix from
tracker calibration
Transformation matrix
updated from marker
tracking in realtimeVideo
Layer
Full-screen quad
with live texture
updated from
Video source
Orthographic
projection

6. AR Interaction
 Designing AR System = Interface Design
 Using different input and output technologies
 Objective is a high quality of user experience
 Ease of use and learning
 Performance and satisfaction

7. AR Information Browsers
 Information is registered to
real-world context
 Hand held AR displays
 Interaction
 Manipulation of a window
into information space
 Applications
 Context-aware information displays
Rekimoto, et al. 1997

8. 3D AR Interfaces
 Virtual objects displayed in 3D
physical space and manipulated
 HMDs and 6DOF head-tracking
 6DOF hand trackers for input
 Interaction
 Viewpoint control
 Traditional 3D user interface
interaction: manipulation, selection,
etc.
Kiyokawa, et al. 2000

9. Augmented Surfaces
 Rekimoto, et al. 1998
 Front projection
 Marker-based tracking
 Multiple projection surfaces

10. Tangible AR: Time-multiplexed Interaction
 Use of natural physical object manipulations to
control virtual objects
 VOMAR Demo
 Catalog book:
- Turn over the page
 Paddle operation:
- Push, shake, incline, hit, scoop

11.  Space-multiplexed
 Many devices each with one function
- Quicker to use, more intuitive, clutter
- Real Toolbox
 Time-multiplexed
 One device with many functions
- Space efficient
- mouse

12. Object Based Interaction: MagicCup
 Intuitive Virtual Object Manipulation
on a Table-Top Workspace
 Time multiplexed
 Multiple Markers
- Robust Tracking
 Tangible User Interface
- Intuitive Manipulation
 Stereo Display
- Good Presence

13. Our system
 Main table, Menu table, Cup interface

15. Wrap-up
 Browsing Interfaces
 simple (conceptually!), unobtrusive
 3D AR Interfaces
 expressive, creative, require attention
 Tangible Interfaces
 Embedded into conventional environments
 Tangible AR
 Combines TUI input + AR display

16. Designing AR Interfaces

17. The Interaction Design Process

18. How Would You Design This?
 Put nice AR Picture here – and video

19. Or This?

20. experiences
applications
tools
components
Sony CSL © 2004
Building Compelling AR Experiences
Tracking, Display
Authoring
Interaction

21. AR Interaction Design
 Designing AR System = Interface Design
 Using different input and output technologies
 Objective is a high quality of user experience
 Ease of use and learning
 Performance and satisfaction

22. AR UI Design
 Consider your user
 Follow good HCI principles
 Adapt HCI guidelines for AR
 Design to device constraints
 Using Design Patterns to Inform Design
 Design for you interface metaphor
 Design for evaluation

23. Consider Your User
 Consider context of user
 Physical, social, emotional, cognitive, etc
 Mobile Phone AR User
 Probably Mobile
 One hand interaction
 Short application use
 Need to be able to multitask
 Use in outdoor or indoor environment
 Want to enhance interaction with real world

24. Good HCI Principles
 Affordance
 Reducing cognitive overload
 Low physical effort
 Learnability
 User satisfaction
 Flexibility in use
 Responsiveness and feedback
 Error tolerance

25. Norman’s Principles of Good Practice
 Ensure a high degree of visibility
- allow the user to work out the current state of the system and
the range of actions possible.
 Provide feedback
- continuous, clear information about the results of actions.
 Present a good conceptual model
- allow the user to build up a picture of the way the system holds
together, the relationships between its different parts and how
to move from one state to the next.
 Offer good mappings
- aim for clear, natural relationships between actions the user
performs and the results they achieve.

26. Adapting Existing Guidelines
 Mobile Phone AR
 Phone HCI Guidelines
 Mobile HCI Guidelines
 HMD Based AR
 3D User Interface Guidelines
 VR Interface Guidelines
 Desktop AR
 Desktop UI Guidelines

27. iPhone Guidelines
 Make it obvious how to use your content.
 Avoid clutter, unused blank space, and busy
backgrounds.
 Minimize required user input.
 Express essential information succinctly.
 Provide a fingertip-sized target area for all links and
controls.
 Avoid unnecessary interactivity.
 Provide feedback when necessary

28. Applying Principles to Mobile AR
 Clean
 Large Video View
 Large Icons
 Text Overlay
 Feedback

29. AR vs. Non AR Design
 Design Guidelines
 Design for 3D graphics + Interaction
 Consider elements of physical world
 Support implicit interaction
Characteristics Non-AR Interfaces AR Interfaces
Object Graphics Mainly 2D Mainly 3D
Object Types Mainly virtual objects Both virtual and physical objects
Object behaviors Mainly passive objects Both passive and active objects
Communication Mainly simple Mainly complex
HCI methods Mainly explicit Both explicit and implicit

30. Maps vs. Junaio
 Google Maps
 2D, mouse driven, text/image heavy, exocentric
 Junaio
 3D, location driven, simple graphics, egocentric

31. Design to Device Constraints
 Understand the platforms and design for limitations
 Hardware, software platforms
 Eg Handheld AR game with visual tracking
 Use large screen icons
 Consider screen reflectivity
 Support one-hand interaction
 Consider the natural viewing angle
 Do not tire users out physically
 Do not encourage fast actions
 Keep at least one tracking surface in view
31
Art of Defense Game

32. Design Patterns
“Each pattern describes a problem which occurs
over and over again in our environment, and then
describes the core of the solution to that problem in
such a way that you can use this solution a million
times over, without ever doing it the same way twice.”
– Christopher Alexander et al.
Use Design Patterns to Address Reoccurring Problems
C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.

33. Handheld AR Design Patterns
Title Meaning Embodied Skills
Device Metaphors Using metaphor to suggest available
player actions
Body A&S Naïve physics
Control Mapping Intuitive mapping between physical and
digital objects
Body A&S Naïve physics
Seamful Design Making sense of and integrating the
technological seams through game
design
Body A&S
World
Consistency
Whether the laws and rules in
physical world hold in digital world
Naïve physics
Environmental A&S
Landmarks Reinforcing the connection between
digital-physical space through landmarks
Environmental A&S
Personal Presence The way that a player is represented in
the game decides how much they feel
like living in the digital game world
Environmental A&S
Naïve physics
Living Creatures Game characters that are responsive to
physical, social events that mimic
behaviours of living beings
Social A&S Body A&S
Body constraints Movement of one’s body position
constrains another player’s action
Body A&S Social A&S

34. Example: Seamless Design
 Design to reduce seams in the user experience
 Eg: AR tracking failure, change in interaction mode
 Paparazzi Game
 Change between AR tracking to accelerometer input

35. Example: Living Creatures
 Virtual creatures should respond to real world events
 eg. Player motion, wind, light, etc
 Creates illusion creatures are alive in the real world
 Sony EyePet
 Responds to player blowing on creature
35

36. Designing for Children
 Development Psychology Factors
• Motor Abilities
• Spatial Abilities
• Logic Abilities
• Attention Abilities
Radu, Iulian, and Blair MacIntyre. "Using children's developmental psychology to
guide augmented-reality design and usability." Mixed and Augmented Reality
(ISMAR), 2012 IEEE International Symposium on. IEEE, 2012.

37. Motor Abilities
Skill Type Challenging AR Interaction
Multiple hand coordination Holding phone in one hand and
using another hand to move
marker
Hand-eye coordination Using a marker to intercept a
moving object
Fine motor skills Moving a marker on a specified
path
Gross motor skills and endurance Turning body around to look at a
panorama

38. Spatial Abilities
Skill Type Challenging AR Interaction
Spatial memory Remembering the configuration of a large
virtual space while handheld screen shows
a limited view
Spatial Perception Understanding when a virtual item is on top
of a physical item
Spatial Visualization Predict when virtual objects are visible by
other people or virtual characters

39. Attention and Logic
Skill Type Challenging AR Interaction
Divided attention Playing an AR game, and making sure to
keep marker in view so tracking is not lost
Selective and executive
attention
Playing an AR game while moving outdoors
Skill Type Challenging AR Interaction
Remembering and reversing Remembering how to recover from tracking
loss
Abstract over concrete
thinking
Understanding that virtual objects are
computer generated, and they do not need
to obey physical laws
Attention Abilities
Logic and Memory

40. Design Considerations
 Combining Real and Virtual Images
• Perceptual issues
 Interactive in Real-Time
• Interaction issues
 Registered in 3D
• Technology issues

41. AR Perceptual Issues

42. AR and Perception
 Creating the illusion that virtual images are
seamlessly part of the real world
• Must match real and virtual cues
• Depth, occlusion, lighting, shadows..

43. AR as Perception Problem
 Goal of AR to fool human senses – create
illusion that real and virtual are merged
 Depth
 Size
 Occlusion
 Shadows
 Relative motion
 Etc..

44. Perceptual Issues
 Combining multiple display modes
• Direct View, Stereo Video View, Graphics View
 Conflict between display modes
• Mismatch between depth cues

45. Perceptual Issues
 Static and Dynamic registration mismatch
 Restricted Field of View
 Mismatch of Resolution and Image clarity
 Luminance mismatch
 Contrast mismatch
 Size and distance mismatch
 Limited depth resolution
 Vertical alignment mismatches
 Viewpoint dependency mismatch

47. Depth Cues
 Pictorial: visual cues
• Occlusion, texture, relative brightness
 Kinetic: motion cues
• Relative motion parallax, motion perspective
 Physiological: motion cues
• Convergence, accommodation
 Binocular disparity
• Two different eye images

48. Use the Following Depth Cues
 Movement parallax.
 Icon/Object size (for close objects)
 Linear perspective
 To add side perspective bar.
 Overlapping
 Works if the objects are big enough
 Shades and shadows.
 Depends on the available computation

49. Provide Perspective Cue
 Eg ground plane grid

51. Depth Perception

52. Information Presentation
• Amount of information
• Clutter, complexity
• Representation of information
• Navigation cues, POI representation
• Placement of information
• Head, body, world stabilized
• View combination
• Multiple views

53. Twitter 360
 www.twitter-360.com
 iPhone application
 See geo-located tweets in real world
 Twitter.com supports geo tagging

54. Wikitude – www.mobilizy.com
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah

55. Information Filtering

56. Information Filtering

57. Outdoor AR: Limited FOV

58. Possible solutions
 Overview + Detail
 spatial separation; two views
 Focus + Context
 merges both views into one view
 Zooming
 temporal separation

59.  TU Graz – HIT Lab NZ - collaboration
 Zooming panorama
 Zooming Map
Zooming Views

60. AR Interaction Issues

61.  Interface Components
 Physical components
 Display elements
- Visual/audio
 Interaction metaphors
Physical
Elements
Display
ElementsInteraction
MetaphorInput Output
AR Design Elements

62. Physical Elements

63. AR Design Space
Reality Virtual Reality
Augmented Reality
Physical Design Virtual Design

64. Design of Objects
 Objects
 Purposely built – affordances
 “Found” – repurposed
 Existing – already at use in marketplace
 Affordance
 The quality of an object allowing an action-
relationship with an actor
 An attribute of an object that allows people to
know how to use it
- e.g. a door handle affords pulling

65. Norman on Affordances
"...the term affordance refers to the perceived
and actual properties of the thing, primarily
those fundamental properties that determine
just how the thing could possibly be used.
[...] Affordances provide strong clues to the
operations of things. Plates are for pushing.
Knobs are for turning. Slots are for inserting
things into. Balls are for throwing .. "
(Norman, The Psychology of Everyday
Things 1988, p.9)

66. Physical vs. Virtual Affordances
 Physical affordances
- Physical and material aspects of real object
 Virtual affordance
- Visual and perceived aspects of digital objects
 AR is mixture of physical and virtual
affordances
 Physical
- Tangible controllers and objects
 Virtual
- Virtual graphics and audio
-

67. Affordance Framework
William W. Gaver. 1991. Technology affordances. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (CHI '91), Scott P. Robertson, Gary M. Olson, and Judith S.
Olson (Eds.). ACM, New York, NY, USA, 79-84.

68. Affordance Led Design
 Make affordances perceivable
 Provide visual, haptic, tactile, auditory cues
 Affordance Led Usability
 Give feedback
 Provide constraints
 Use natural mapping
 Use good cognitive model

69. Example: AR Chemistry
 Tangible AR chemistry education (Fjeld)
Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible interaction
approach. Proceedings of INTERACT 2003, September 1st -5th 2003, Zurich,
Switzerland.

70. Input Devices
 Form informs function and use

71. Picking up an Atom

72. AR Interaction Metaphors

73.  Interface Components
 Physical components
 Display elements
- Visual/audio
 Interaction metaphors
Physical
Elements
Display
ElementsInteraction
MetaphorInput Output
AR Design Elements

74. Interface Design Path
1/ Prototype Demonstration
2/ Adoption of Interaction Techniques from other
interface metaphors
3/ Development of new interface metaphors
appropriate to the medium
4/ Development of formal theoretical models for
predicting and modeling user actions
Desktop WIMP
Virtual Reality
Augmented Reality

75. Interface metaphors
 Designed to be similar to a physical entity but also has own
properties
 e.g. desktop metaphor, search engine
 Exploit user’s familiar knowledge, helping them to understand
‘the unfamiliar’
 Conjures up the essence of the unfamiliar activity, enabling
users to leverage of this to understand more aspects of the
unfamiliar functionality
 People find it easier to learn and talk about what they are
doing at the computer interface in terms familiar to them

76. Example: The spreadsheet
 Analogous to ledger
sheet
 Interactive and
computational
 Easy to understand
 Greatly extending
what accountants
and others could do
www.bricklin.com/history/refcards.htm

77. Why was it so good?
 It was simple, clear, and obvious to the users how to
use the application and what it could do
 “it is just a tool to allow others to work out their
ideas and reduce the tedium of repeating the same
calculations.”
 capitalized on user’s familiarity with ledger sheets
 Got the computer to perform a range of different
calculations in response to user input

78. The Star interface

79. Benefits of interface metaphors
 Makes learning new systems easier
 Helps users understand the underlying
conceptual model
 Can be innovative and enable the realm of
computers and their applications to be made
more accessible to a greater diversity of users

80. Problems with interface metaphors
(Nielson, 1990)
 Break conventional and cultural rules
 e.g., recycle bin placed on desktop
 Can constrain designers in the way they conceptualize a problem
 Conflict with design principles
 Forces users to only understand the system in terms of the
metaphor
 Designers can inadvertently use bad existing designs and transfer
the bad parts over
 Limits designers’ imagination with new conceptual models

81. Microsoft Bob

83.  PSDoom – killing processes

84. Back to the Real World
 AR overcomes limitation of TUIs
 enhance display possibilities
 merge task/display space
 provide public and private views
 TUI + AR = Tangible AR
 Apply TUI methods to AR interface design

85. Tangible AR Design Principles
 Tangible AR Interfaces use TUI principles
 Physical controllers for moving virtual content
 Support for spatial 3D interaction techniques
 Time and space multiplexed interaction
 Support for multi-handed interaction
 Match object affordances to task requirements
 Support parallel activity with multiple objects
 Allow collaboration between multiple users