2013 Lecture4: Designing AR Interfaces


Published on

This is the COSC 426 Lecture 4 on Designing AR Interfaces. Taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury. This is part of his graduate course on Augmented Reality. Taught on August 2nd 2013

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2013 Lecture4: Designing AR Interfaces

  1. 1. COSC 426: Augmented Reality Mark Billinghurst mark.billinghurst@hitlabnz.org August 2nd 2013 Lecture 4: Designing AR Interfaces
  2. 2. Key Points from Lecture 3
  3. 3. Tracking Requirements   Augmented Reality Information Display   World Stabilized   Body Stabilized   Head Stabilized Increasing Tracking Requirements Head Stabilized Body Stabilized World Stabilized
  4. 4. Tracking Types Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking Mechanical Tracker
  5. 5. Marker Based Tracking: ARToolKit http://artoolkit.sourceforge.net/
  6. 6. Markerless Tracking Magnetic Tracker Inertial Tracker Ultrasonic Tracker Optical Tracker Marker-Based Tracking Markerless Tracking Specialized Tracking Edge-Based Tracking Template-Based Tracking Interest Point Tracking   No more Markers! Markerless Tracking
  7. 7. Natural Feature Tracking   Use Natural Cues of Real Elements   Edges   Surface Texture   Interest Points   Model or Model-Free   ++: no visual pollution Contours Features Points Surfaces
  8. 8. Combining Sensors and Vision   Sensors -  Produce noisy output (= jittering augmentations) -  Are not sufficiently accurate (= wrongly placed augmentations) -  Gives us first information on where we are in the world, and what we are looking at   Vision -  Is more accurate (= stable and correct augmentations) -  Requires choosing the correct keypoint database to track from -  Requires registering our local coordinate frame (online- generated model) to the global one (world)
  9. 9. Outdoor AR Tracking System You, Neumann, Azuma outdoor AR system (1999)
  10. 10. The Registration Problem   Virtual and Real must stay properly aligned   If not:   Breaks the illusion that the two coexist   Prevents acceptance of many serious applications
  11. 11. Sources of registration errors   Static errors   Optical distortions   Mechanical misalignments   Tracker errors   Incorrect viewing parameters   Dynamic errors   System delays (largest source of error) -  1 ms delay = 1/3 mm registration error
  12. 12. Reducing static errors   Distortion compensation   Manual adjustments   View-based or direct measurements   Camera calibration (video)
  13. 13. Reducing dynamic errors (1)   Reduce system lag   Faster components/system modules   Reduce apparent lag   Image deflection   Image warping
  14. 14. Reducing System Lag Tracking Calculate Viewpoint Simulation Render Scene Draw to Display x,y,z r,p,y Application Loop Faster Tracker Faster CPU Faster GPU Faster Display
  15. 15. Predictive Tracking Time Position Past Future Can predict up to 80 ms in future (Holloway) Now
  16. 16. AR Design
  17. 17. experiences applications tools components Building Compelling AR Experiences Tracking, Display Authoring Interaction Usability
  18. 18. 3. Discovering User Needs Mark Billinghurst
  19. 19. Interaction Design Process
  20. 20. Interaction Design is User Centered A methodology for ensuring good user experiences with products by getting feedback from users to inform the design.
  21. 21. 50 Discovering Needs • Why: Requirements definition: the stage where failure occurs most commonly Getting requirements right is crucial
  22. 22. What, How and Why? • What 1. Understand users, task, context 2. Produce a stable set of requirements • How: Data gathering activities Data analysis activities Expression as ‘requirements’ • Why: Requirements definition: failure occurs most commonly Getting requirements right is crucial Iterate
  23. 23. What Are the Requirements?   Functional   What should the system do? What has it done?   Data   What type of data?   Environment or Context of use   Physical: dusty? noisy? light? heat? humidity? ….   Social: sharing of files, of displays, work alone ..   Organizational: IT support, hierarchy, training..   Users: Who are they?   Usability: learnability, throughput, flexibility..
  24. 24. Who REALLY are your Users/Stakeholders? • Not as obvious as you think: — those who interact directly with the product — those who manage direct users — those who receive output from the product — those who make the purchasing decision — those who use competitor’s products • Three categories of user (Eason, 1987): — primary: frequent hands-on — secondary: occasional or via someone else — tertiary: affected by its introduction, or will influence its purchase
  25. 25. Who are the Stakeholders? Check-out operators CustomersManagers and owners • Suppliers • Local shop 
  26. 26. Consider the Whole User
  27. 27. Data Gathering Mark Billinghurst
  28. 28. Data Gathering Techniques (1)   Questionnaires   Looking for specific information   Qualitative and quantitative results   Good for getting data from a large, dispersed group   Interviews   Good for exploring issues, using props   Structured, unstructured or semi-structured   But are time consuming and difficult to visit everyone
  29. 29. Data Gathering Techniques (2)   Workshops or focus groups   Group interviews/activities   Good at gaining a consensus view and/or highlighting areas of conflict   Observations   Spending time with users in day to day tasks   Good for understanding task context   requires time and can result in a huge amount of data   Documentation   Procedures and rules written down in manuals
  30. 30. A day in the Life of.. Cultural Probes.. Role Playing..
  31. 31. Case Study: Equator Domestic Probes
  32. 32. 4. Design and Prototyping Mark Billinghurst
  33. 33. Interaction Design Process
  34. 34. Elaboration and Reduction   Elaborate - generate solutions. These are the opportunities   Reduce - decide on the ones worth pursuing   Repeat - elaborate and reduce again on those solutions Source: Laseau,P. (1980) Graphic Thinking for Architects & Designers. John Wiley and Sons
  35. 35. The Design Funnel   Alternate generation of ideas and convergence until resolution Modified from Pugh, S. (1990) Total design: Integrated methods for successful products engineering. Addison-Wesley. P. 75
  36. 36. Design Inspiration
  37. 37. Unmet Needs   Digital Cameras for Apple Late 80s – What would happen if we married a computer with a digital camera? Many studies of current photography use. Uncovering of unmet needs in traditional experience - not enough photos - not knowing whether photo good or not - ability to add sound - ability to organize Features determined through research and designed Iteratively.
  38. 38. Brainstorming   Best with interdisciplinary team
  39. 39. Other Products   Notice all the iPod look-alikes?
  40. 40. Tools for Effective Design  Personas  Scenarios  Storyboards  Wireframes and Mock-ups  Prototypes
  41. 41. Persona Technique •  Personas are a design tool to help visualize who you are designing for and imagine how person will use the product •  A persona is an archetype that represents the behavior and goals of a group of users •  Based on insights and observations from customer research •  Not real people, but synthesised from real user characteristics •  Bring them to life with a name, characteristics, goals, background •  Develop multiple personas
  42. 42. Gunther the Ad Guy Gunther is from Germany. He Travels extensively for work and As he is an advertising executive he needs to present concepts to clients quickly and easily. He is a person very well-versed in new technologies and wishes he had easier portable solutions for his presentations…..
  43. 43. Scenarios Usage Scenarios are narrative descriptions of how the product meets the needs of a persona Short (2 pages max) Focus on unmet needs of persona Concrete story Set of stories around essential tasks, problems... Use to test ideas
  44. 44. A business woman travels from SF to Paris on a business trip. On her way to the airport she narrowly misses a traffic delay. She avoids the jam because her Smartphone beeps and send her a warning text message on her route from the office to the airport. Upon arrival the location-sensitive Smartphone notifies the airline that she‘ll check in shortly and an airline employee finds her immediately and takes her baggage. Her display shows that her flight is on time and provides a map to her gate. On the way she downloads tourist information (maps, events) for Paris.
  45. 45. Once found her seat, she begins to review the downloaded information. She books a ticket for an opera she wants to see. Her Smartphone makes the booking using her credit card number stored in memory.The security software of the Smartphone protects her against fraud. The Smartphone stores the opera booking along with emails written on the plane.As soon as she steps off the plane, it makes the calls and sends the emails.As she leaves the airport, a map appears on the display and guides her to her hotel
  46. 46. Storyboarding Sequence of sketches showing use of system in everyday use context Concrete example Easier (faster) to grasp than text based stories Means of communication with users and system developers Sketches, not drawings... Use to test interaction and make sure design works
  47. 47. Example Storyboard
  48. 48. Example Storyboard p.2
  49. 49. Sketching Mark Billinghurst
  50. 50. Sketching is about design Sketching is not about drawing It is about design. Sketching is a tool to help you: -  express -  develop, and -  communicate design ideas Sketching is part of a process: -  idea generation, -  design elaboration -  design choices, -  engineering
  51. 51. Why is sketching useful?   Early ideation   Think through ideas   Force you to visualize how things come together   Communicate ideas to others to inspire new designs   Active brainstorming
  52. 52. Figure 41: A Sketch of a Dialogue with a Sketch The “conversation” between the sketch (right bubble) and the mind (left bubble). A sketch is created from current knowledge (top arrow). Reading, or interpreting the resulting representation (bottom arrow), creates new knowledge. The creation results from what Goldschmidt calls “seeing that” reasoning, and the extraction of new knowledge results from what she calls “seeing as.” sketch representation mind (new) knowledge Create (seeing that) Read (seeing as) 54
  53. 53. Sketchbook examples Industrial designer   ideas   variations   annotations Source: sketch page from student Industrial designer Samnang Eav Form factors for wearable computer watch
  54. 54. Sketchbook examples Idea variations Source: Nicolai Marquardt sketchbook, with permission. Four different versions of an idea
  55. 55. Sketch examples – design variations From Carloyn Snyder’s Book: Paper Prototyping (2003) Morgan Kaufmann, p350
  56. 56. Sketch examples – storyboard transitions From Carloyn Snyder’s Book Paper Prototyping (2003) Morgan Kaufmann, p11
  57. 57. Techniques: Annotations   Augment a sketch   Directly on sketch   As layer -  Tracing -  Photoshop layer   Over dynamic video
  58. 58. Technique: Annotations   Textual notes   Name and/or explain thinks   Add details   List of items   Questions/issues about design   Graphical marks   Connect notes to sketch elements   Relates sketch elements   Show dynamics of elements or interaction over time
  59. 59. Technique: Tracing   Basic Idea -  Copy/trace the fixed elements of an existing interface/screen   Technology -  Camera, photograph, tracing paper or.. -  Photoshop or equivalent (trace over image on a separate layer)   Drawing skill required -  Almost zero
  60. 60. Technique Hybrid Sketches   Mixes full and low fidelity elements   High fidelity parts:   Fixed design elements   Low fidelity parts:   Speculative components   Contrast   Makes certain parts of the sketch stand out
  61. 61. Your Turn   Design a mobile AR user interface   Viewing virtual buildings in Christchurch   Views (Should be able to swap between)   Map View   AR View   List View
  62. 62. Prototyping Mark Billinghurst
  63. 63. 68www.id-book.com What is a prototype? In interaction design it can be (among other things): •  a series of screen sketches •  a storyboard, i.e. a cartoon-like series of scenes •  a Powerpoint slide show •  a video simulating the use of a system •  a lump of wood (e.g. PalmPilot) •  a cardboard mock-up •  a piece of software with limited functionality
  64. 64. 69 www.id-book.com Why Prototype? •  Evaluation and feedback are central to interaction design •  Stakeholders can see, hold, interact with a prototype more easily than a document or a drawing •  Team members can communicate effectively •  You can test out ideas for yourself •  It encourages reflection: very important aspect of design •  Prototypes answer questions, and support designers in choosing between alternatives
  65. 65. Sketch vs. Prototype Sketch   Prototype   Invite   A)end   Suggest   Describe   Explore   Refine   Ques;on   Answer   Propose   Test   Provoke   Resolve   Tenta;ve,  non  commi)al   Specific  Depic;on   The primary differences are in the intent
  66. 66. Types of Prototypes Low Fidelity – quick and dirty, easy access materials like cardboard and paper. High Fidelity – more involved electronic versions similar in materials to final product.
  67. 67. RAPID Prototyping   Fast and inexpensive   Identifies problems before they’re coded   Elicits more and better feedback from users   Helps developers think creatively   Gets users and other stakeholders involved early   Fosters teamwork and communication   Avoids opinion wars   Helps decide design directions
  68. 68. Paper Prototyping (Low Fidelity) Quick and simple means of sketching interfaces Use office materials Easier to criticize, quick to change Creative process (develop in team) Can also use for usability test (focus on flow of interaction rather than visuals) Used a lot to test out concepts before real design begins.
  69. 69. Paper Prototyping
  70. 70. Paper Proto: Create Widgets
  71. 71. Paper Prototyping with Hardware
  72. 72. Blank Model Prototypes
  73. 73. Technique: State Transition Diagrams   Create transition diagram   key interaction steps   branch points   For each transition   Sketch the screen   Include the transitional diagram   Label the transition with what triggered the transition (typically user input of set of system responses)
  74. 74. Technique: State Transition Diagrams
  75. 75. Wireframes and Page Semantics
  76. 76. Half-Functional Prototypes More hi-fi HTML (or Powerpoint) ... Mostly precoded flow, but allows clicking and experiencing flow
  77. 77. Wireframing Tools   Pidoco (websites, mobile, desktop)   No programming   http://www.pidoco.com   WireframeSketcher   Plug-in for Eclipse   http://wireframesketcher.com/   Omnigraffle (Mac)   Digramming tool   http://www.omnigroup.com/products/omnigraffle/
  78. 78. Screen MockUp Prototypes
  79. 79. HTML Wireframes
  80. 80. Transitions
  81. 81. Technique: Make storyboards come alive   An interactive foam core and paper sketch/storyboard
  82. 82. Mobile Rapid Prototype Tools   Android: App Inventor   http://appinventor.mit.edu/
  83. 83. Interactive Role Play
  84. 84. High-fidelity prototyping •  Uses materials that you would expect to be in the final product. •  Prototype looks more like the final system than a low-fidelity version. •  For a high-fidelity software prototype common environments include Macromedia Director,Visual Basic, and Smalltalk. •  Danger that users think they have a full system…….see compromises
  85. 85. Interactive Role Play in Environment
  86. 86. Tethered Prototypes (High Fidelity) Goal: CommunicateVision HIGH FIDELITY PROTOTYE
  87. 87. Rapid Prototyping   Speed development time with quick hardware mockups   handheld device connected to PC   LCD screen, USB phone keypad, Camera   Can use PC development tools for rapid development   Flash, Visual Basic, etc
  88. 88. ‘Wizard-of-Oz’ Prototyping • The user thinks they are interacting with a computer, but a developer is responding to output rather than the system. • Usually done early in design to understand users’ expectations >Blurb blurb >Do this >Why? User
  89. 89. Typical Iterations for HW/SW Product: 1.  Paper prototype to evaluate conceptual model 2.  Interactive computer-based prototype with rough screens to evaluate feature placement 3.  Tethered prototype to evaluate button + screen interactions 4.  Real device prototyped with major features working 5.  Real device prototyped with all features working and graphic design implemented
  90. 90. AR Design Case Study
  91. 91. ITERATIVE DESIGN PROCESS Design   (redesign)   Prototype   Test  Design  with  Users  (Note   problems)   Evaluate   (Fix  Issues)  
  93. 93. BUNRATTY FOLK PARK  Irish visitor attraction run by Shannon Heritage  19th century life is recreated  Buildings from the mid-west have been relocated to the 26-land surrounding Bunratty Castle  30 buildings are set in a rural or village setting there.
  94. 94. AUGMENTED REALITY In Bunratty Folk Park:  Allows the visitor to point a camera at an exhibit, the device recognises it by it’s location and layers digital information on to the display  3-dimensional virtual objects can be positioned with real ones on display  Leads to dynamic combination of a live camera view and information
  95. 95. NAVIGATIONAL AID Smartphone Platform Most people carry mobile phones and are comfortable with them Ideal Augmented Reality Technology  Global tracking tools  Wireless communication capabilities  Location based computing  Large display for interaction
  97. 97. HUMAN CENTRED DESIGN Goal of the Navigational Aid  Easy to use, clear and understandable  Useful to visitors  Creating interaction between the visitor and the aid through the user interface Engage the visitor  It is necessary to understand the visitor to Bunratty  Identify visitor motives and goals while going through the Folk Park.
  98. 98. HUMAN CENTRED DESIGN Understanding the technology and Related Works  Literature  Similar Technologies  Electronic Tours in Museum Settings  Interactions design in Outdoor Museums Understanding the User over time  Observations  Interviews
  99. 99. HUMAN CENTRED DESIGN Findings  Most visitors do not use the map  Most visitors have mobile phones  Visitors want more information  View the Folk Park at their own pace  Information should be straight to the point  Large social interaction within groups
  100. 100. NEXT STEPS FROM RESEARCH  Define Visitors Goals  Define Functionalities of the Aid  Develop Personas – visitors who use the Aid  Develop Scenarios – how the persona uses the navigational aid in the Folk Park  Draw up Storyboards on scenarios
  101. 101. FUNCTIONALITY  View Options  Camera View  Map View  List View  Sub-Options  Places  Events  Restaurants  AR Features  Text Information  3D Objects  3D Tour Guide  3D Placement of Buildings
  102. 102. STORYBOARD
  104. 104. ITERATIVE DESIGN PROCESS Prototyping and User Testing  Low Fidelity Prototyping  Sketches  Paper Prototyping  Post-It Prototyping  PowerPoint Prototyping  High Fidelity Prototyping  Wikitude
  105. 105. INITIAL SKETCHES - Photoshop Pros: • Good for idea generation • Cheap • Concepts seem feasible Cons: • Not great feedback gained • Photoshop not fast enough for making changes
  106. 106. POST IT PROTOTYPING First  DraJ   Camera  View  with  3D   Second  DraJ   Third  DraJ   •   Selec;on  highlighted  in  blue   •   Home  bu)on  added  for  easy   naviga;on  to  main  menu  
  107. 107. POWERPOINT PROTOTYPING Benefits • Used for User Testing • Interactive • Functionalities work • Quick • Easy arrangement of slides User Testing • Participants found • 15 minute sessions screen captured • ‘Talk Allowed’ technique used • Notes taken • Post-Interview
  108. 108. WIKITUDE  Popular AR browser  Mapping  Point of Interest abilities  Multiplatform  Shows points of interest  Markers can be selected in and information pop-up appears
  109. 109. WIKITUDE User Testing  Application well received  Understandable  Participants playful with the technology
  111. 111. FINAL DESIGN CONCEPT Key Issues   Fix issues found in previous sessions   Design with guidelines in mind   Appealing to the Mental Model  Icon Design   Aesthetic Design  Colour/Font  Buttons  Look
  112. 112. VIDEO PROTOTYPE  Flexible  tool  for  capturing  the  use  of   an  interface    Elaborate  simula;on  of  how  the   naviga;onal  aid  will  work    Does  not  need  to  be  realis;c  in   every  detail    Gives  a  good  idea  of  how  the   finished  system  will  work  
  113. 113. Project List   Mobile   Hybrid Tracking for Outdoor AR   City Scale AR Visualization   Outdoor AR Authoring Tool   Outdoor AR collaborative game   AR interaction for Google Glass   Non-Mobile   AR Face Painting   AR Authoring Tool   Tangible AR puppeteer studio   Gesture based interaction with AR content
  114. 114. BuildAR Assignment   Download and install Build AR (or HIT Lab NZ)   http://www.buildar.org/   Build two AR scenes   Marker based -  at least 3 markers, 3D models, same theme   Image based -  at least 2 images, 3D model, text, image/video   Due August 16th
  115. 115. Project Assignment   Design/Related work exercise   Individual   Each person find 2 relevant papers/videos/websites   Write two page literature review   As a team - prototype design   Sketch out the user interface of application   Design the interaction flow/Screen mockups   3 minute Presentation in class August 16th