Successfully reported this slideshow.
Your SlideShare is downloading. ×

Lean User Testing Intro

Upcoming SlideShare
User testing on a diet
User testing on a diet
Loading in …3

Check these out next

1 of 47 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (16)


Similar to Lean User Testing Intro (20)


Recently uploaded (20)

Lean User Testing Intro

  1. 1. Jessica DuVerneay Information Architect, The Understanding Group - @undrstndng LEAN USER TESTING
  2. 2. AGENDA ‣ 6:30 - Agenda & Intro ‣ 6:40 - User Testing Basics ‣ 6:50 - Benefits & Limitations ‣ 7:00 - The Lean Approach ‣ 7:10 - When to Test ‣ 7:20 - What to Test ‣ 7:30 - Case Study ‣ 7:40 - Resources & Next Steps ‣ 7:50 - Q&A 2
  3. 3. INTRODUCTION ‣ Clients: Herman Miller, Thompson Reuters / Techstreet, Macy’s, Metro Health System, University of Michigan, ASAPS, Cooper Tire ‣ Talks: Taxonomy - WIAD 2013, LAUX Meetup ‣ Workshops: IA/UX Topics including Wireframes, Performance Continuums, User Testing ‣ Articles: Expert Contributor for’s blog, TUG’s site ‣ Global Producer - World Information Architecture Day (2012) JESSICA DUVERNEAY - INFORMATION ARCHITECT, THE UNDERSTANDING GROUP (TUG) 3
  6. 6. USER TESTING BASICS USER TESTING ‣ Wikipedia says: “Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system.[1] This is in contrast with usability inspection methods where experts use different methods to evaluate a user interface without involving users. Usability testing focuses on measuring a human- made product's capacity to meet its intended purpose. Examples of products that commonly benefit from usability testing are foods, consumer products, web sites or web applications, computer interfaces, documents, and devices. Usability testing measures the usability, or ease of use, of a specific object or set of objects.” ‣ It is NOT: A complete set of User Research Activities, Market Research, Statistically Relevant Research, A/B Testing, Qualitative Formal Research 6 PRO TIP: Remember, User Testing is not actually testing the users - it’s testing the product from the user’s actual perspective.
  7. 7. USER TESTING BASICS 7 Activities Deliverables SME Interviews Digital Report Card Experience Map Score Notes Owned Paid Social Mobile Access Search B A C B C B ScenarioScenarioScenario Alignment Session Heuristic Analysis Cross Channel Map Pagename PHASE 1 PHASE 2 PHASE 3 Channel Channel Channel Tactics by Channel Content Audit Flow Diagrams User Interviews Scenario Collab Design Competitive Site Map You ThemThemFeatures Them Impressions 1 2.1 2.2 4.1 4.2 4.3 3.1 2 3 4 User Co-Creation Personas Wireframes Persona Scenarios 2.0 Page Name Strategic Brief Content Inventory Concept Testing Stakeholder Needs Best Practices User Needs b a c d 2.0 Page Name Functional Spec. Analytic Report Workshops Recommendations 1.0 1.1 1.2 2.0 3.0 User Testing Roadmap Analytic Wishlist 1 2 3 Phased Roadmap 2 1 3 4 We wish we knew...
  8. 8. USER TESTING BASICS 8 The Elements of User Experience A basic duality: The Web was originally conceived as a hypertextual information space; but the development of increasingly sophisticated front- and back-end technologies has fostered its use as a remote software interface. This dual nature has led to much confusion, as user experience practitioners have attempted to adapt their terminology to cases beyond the scope of its original application. The goal of this document is to define some of these terms within their appropriate contexts, and to clarify the underlying relationships among these various elements. Jesse James Garrett Visual Design: graphic treatment of interface elements (the "look" in "look-and-feel") Information Architecture: structural design of the information space to facilitate intuitive access to content Interaction Design: development of application flows to facilitate user tasks, defining how the user interacts with site functionality Navigation Design: design of interface elements to facilitate the user's movement through the information architecture Information Design: in the Tuftean sense: designing the presentation of information to facilitate understanding Functional Specifications: "feature set": detailed descriptions of functionality the site must include in order to meet user needs User Needs: externally derived goals for the site; identified through user research, ethno/techno/psychographics, etc. Site Objectives: business, creative, or other internally derived goals for the site Content Requirements: definition of content elements required in the site in order to meet user needs Interface Design: as in traditional HCI: design of interface elements to facilitate user interaction with functionality Information Design: in the Tuftean sense: designing the presentation of information to facilitate understanding Web as software interface Web as hypertext system Visual Design: visual treatment of text, graphic page elements and navigational components Concrete Abstract time Conception Completion Functional Specifications Content Requirements Interaction Design Information Architecture Visual Design Information Design Interface Design Navigation Design Site Objectives User Needs User Needs: externally derived goals for the site; identified through user research, ethno/techno/psychographics, etc. Site Objectives: business, creative, or other internally derived goals for the site This picture is incomplete: The model outlined here does not account for secondary considerations (such as those arising during technical or content development) that may influence decisions during user experience development. Also, this model does not describe a development process, nor does it define roles within a user experience development team. Rather, it seeks to define the key considerations that go into the development of user experience on the Web today. task-oriented information-oriented 30 March 2000 © 2000 Jesse James Garrett
  12. 12. UNMODERATED BENEFITS & LIMITATIONS BENEFITS ‣ Inexpensive, Fast ‣ Can learn high level pain points, random fatal flaws, percentages of task completion, time on task, qualitative verbal opinions offered, basic analytics & demographic info.. more on that later. ‣ Easy to aggregate concrete numbers if needed in small or very large test populations ‣ “Quick and dirty” ‣ Can design to validate pretty much anything ‣ Existing tools = Easy to share across stakeholders ‣ Easy to test a pre-determined flow ‣ Great for iterative design, repeatable ‣ Anonymous comparative analysis & A/B testing ‣ Easy for one person to write, analyze and extract findings ‣ Do not need to set up testing lab ‣ No moderator bias / influence ‣ User in natural environment using preferred technologies & workflows ‣ Asynchronous allows for User Testing to be fit in with other work responsibilities ‣ Non-local / Global Reach much easier than moderated testing 12
  13. 13. UNMODERATED BENEFITS & LIMITATIONS LIMITATIONS ‣ Not as empathy driven, less intuitive ‣ Can’t learn deep opinions or user motivations, non-verbal cues, users values & personality, organic / natural use case, deep analytics, detailed statistics, eye mapping ‣ Not able to correct user ‣ Can not restate or correct poorly written test on the fly ‣ Not able to dig in / probe for subjective issues or edge cases ‣ Less non-verbal inference (eye tracking, facial expressions, etc) ‣ Depending on recruits can be biased very heavily ‣ Easy to misinterpret by external stakeholders ‣ Somewhat complicated to analyze by multiple team members ‣ Does not build relationship with customer ‣ Does not completely assure confidentiality or an NDA 13
  15. 15. THE LEAN APPROACH THE ‘MVP’ OF USER TESTING: FIT & FINISH TESTING If you test nothing else, please test key product segments & flows for fatal flaws & show stoppers before you launch. Common UX & IA Performance Metrics to consider: ‣ Task Success (Binary Approach for Lean Process) ‣ Time On Task ‣ Errors ‣ Efficiency (Effort / Number of Steps) ‣ Learnability ‣ Perceived Severity of Identified Issues 15 OVERHEARD: “Friends don’t let friends launch without user testing.” True story.
  16. 16. THE LEAN APPROACH HOW MANY USERS SHOULD WE TEST? 16 Image: Jakob Nielsen Group Roughly, 5-6 users will uncover a vast majority of the issues per scenario, task, or flow. The exact number of tests depends on the number of scenarios and demographic segments you need to test.
  18. 18. THE LEAN APPROACH 18
  19. 19. THE LEAN APPROACH HOW TO TEST ‘LEAN’: OTHER CONSIDERATIONS ‣ Create a Testing Team (1-4 people) ‣ Create a Testing Plan (Strategy, Budget, Timeline) ‣ Know the Tools Available ‣ Write Effective Tests ‣ Be Cautious About Recruiting & Incentives ‣ Analyze Data Effectively ‣ Present Your Findings (If Needed) Strategically & Clearly ‣ Iterate, Repeat, and Improve 19 PRO TIP: A poorly created & implemented test can be worse than useless. Take time to educate yourself & plan the test.
  21. 21. WHEN TO TEST TYPICAL PROJECT LIFECYCLE Discovery: Understanding organizational and business goals for the product or service. This is generally when the “What & Why” of a product is determined. User Research: Understanding end user needs and expectations for the same product or service. While the Discovery Phase established the “What & Why”, this phase is to establish the “Who & Why” Strategy & Structural Design: Creating a solid strategy & structure for the product or product improvements. This phase is when the “What, Why & Who” are translated into the strategic and researched “How”. Launch & Assessment: Building and launching the product, measuring and evaluating success. This is the continuous improvement phase when efficacy of strategic changes or initial product launches can be measured, and organizations can begin gathering ideas and creating roadmaps for further improvements or product releases. 21
  22. 22. WHEN TO TEST LAUNCH & ASSESSMENT ‣ Validate visual fit & finish just prior to launch as a lightweight QA process ‣ Validate improvement over previous product iterations ‣ Augment quantitative A/B testing to investigate user preference from a qualitative perspective ‣ Identify unnoticed usability issues or awkward product flows ‣ Show instances of a product in multiple environments (Desktop vs. Mobile) ‣ Provide an area for users to provide opinions, suggestions, and desired features lists (especially valuable in the startup space) 22
  23. 23. WHEN TO TEST DISCOVERY ‣ Establish successful existing user flows that should be protected during upcoming versions ‣ Identify show stopping usability issues that may have been missed in initial product road mapping activities ‣ Benchmark your existing product so that at the end of your project lifecycle, you have data against which to compare your improved product. ‣ Provide insight into the competitive landscape 23
  24. 24. WHEN TO TEST USER RESEARCH ‣ Provide a quick and easy way to see a product in front of multiple user types. ‣ Provide qualitative and quantitative data to flesh out user research deliverables such as personas (scenarios, tasks, demographic info, opinions & motivators). ‣ NOTE: Contrary to popular perception, this is actually the least effective phase to do user testing. User Research, when done correctly, is far more based on getting to know the user and their needs, and less on how they would use the existing product or service. 24
  25. 25. WHEN TO TEST STRATEGY & STRUCTURAL DESIGN ‣ Point out key issues with Strategic or Structural Design Direction prior to build using prototypes or sandbox environments ‣ Test and refine complex user flows ‣ Test and refine navigation structures (TreeJack is a great tool for this) ‣ Identify unanticipated usability issues that may come up after strategy and structure is approved, during design and dev phases 25
  26. 26. WHEN TO TEST MAIN POINT 26 No matter where you are in your product cycle, it is advantageous to consider lean unmoderated testing. Unmoderated testing is not just for usability only - testing can add a different value and data set at each stage of the project lifecycle. PRO TIP: Test early, test often.
  28. 28. WHAT TO TEST 1. UX / IA / USABILITY ISSUES What constitutes success for one product may not necessarily be indicative of optimal state for another product. Qualitative data is valuable, but do not ignore the insights extracted from qualitative information (opinions, ratings, exclamations, and comments) - they can guide some of the most effective product changes.  28 There are some basic measurable indicators to consider while usability testing. (Note: The indicators listed below heavily reference Measuring the User Experience by Tullis and Alberts) ‣ Task Completion: Can a user complete the task? To what percentage can they move in the correct way in the task completion funnel? What issues are impeding the completion of the task? Do users want to complete the task in the first place? ‣ Time On Task: How long does it take the user to complete the task? Does the time it takes to complete the task allow the user to lose interest or focus? Does the task take too little time to complete? ‣ Errors: Did users make avoidable errors? Did users make unavoidable errors? Were they able to recover from the errors? What would make the errors less likely to occur or easier to recover from? ‣ Efficiency: Was the system efficient in effort and number of steps it took for users to complete a task? Which steps were superfluous or confusing? Which steps can be streamlined or eliminated? Which steps need to be added? ‣ Learnability: For repeat visitors, is the system learnable? Does task success and perceived efficiency increase? Does time on task and  number of errors decrease? Do certain key actions and flows become tacit? ‣ Perceived Severity of Identified Issues: Are any of the issues noticed by the testing team not actually registered as issues to the users? Which issues are most important to achievement of key business flows? Are any of the issues deal breakers?
  29. 29. WHAT TO TEST 2. COMPETITIVE TESTING A lesser known benefits of lean unmoderated testing is the ability to show a test subject a competitor’s product without the risk of moderator bias. Learning from a competitor – which flows, content, and design patterns are successful and which should be avoided – can be particularly useful in the startup space where lean UX teams may have to make s product decisions on limited research.  QUESTIONS TO CONSIDER: ‣ Which product do users prefer? Why? ‣ Which product was easier to use? Why? ‣ What are some errors my competitor made that I can avoid during product development? ‣ What are some successful features that my competitor has that I might want to consider for product roadmapping? ‣ How do users of different demographics (Age, Gender, OS, income, etc) respond to each product? Is one better for my target market?  29
  30. 30. WHAT TO TEST 3. PREFERENCE Preference testing may provide direct user insights around: ‣ Visual design & Branding ‣ Interaction design ‣ Copy ‣ Navigation approaches ‣ Use of imagery ‣ Page layout and information hierarchy ‣ Any other contentious issues at your organization that are delaying launch.   30 PRO TIP: Don’t mistake this for A/B testing. While lean user preference testing can give insight, it’s not statistically significant.
  31. 31. WHAT TO TEST 4. VALIDATION OF FIT & FINISH ‣ One to four main user flows (keep this as simple as possible) ‣ The main conversion point of the system ‣ Overall opinions of the visual design and interaction design ‣ Any crucial UX / IA issues as previously described 31
  32. 32. WHAT TO TEST 5. OTHER USES: COPYWRITING VALIDATION, LIGHTWEIGHT MULTI PLATFORM QA, PERSONA INFO ‣ Personas: I’ve augmented other user research activities with data I’ve collected from unmoderated testing to flesh out personas as needed. ‣ QA Testing: I’ve seen unmoderated user testing implemented as a lightweight QA process – running the same test on a product in multiple desktop and mobile environments can provide invaluable insights for a small team with limited QA/ UAT capabilities. ‣ Copywriting & Language: Testing copywriting or taxonomies to some extent can be done in unmoderated testing. Again, the success of these inquiries will depend largely on the tools used & the manner in which the tests and scenarios are crafted.  32
  35. 35. SUMMARY OF ISSUES KEY QUESTIONS CASE STUDY How can we optimize our eCommerce flow to mitigate these issues? What can we learn from Lean User Testing and how can we add it into our design cycle? Well known e-commerce client facing: • Navigation Issues • Cart Abandonment • Small Average Order Size (# of Products per Conversion) • Cross Channel Confusion of Digital Properties (Information Site vs. Store) • Browse Customers = Lower Conversion On Site • Brand Perception Suboptimal Online vs. Physical World Product Interaction WELL KNOWN FURNITURE COMPANY 35
  36. 36. CASE STUDY SUMMARY: UNMODERATED USABILITY RESEARCH APPROACH 36 PRO TIP: In hindsight, I should have tested the proposed navigation with Treejack during the Structural Design Phase.
  37. 37. CASE STUDY DISCOVERY: HIGH LEVEL USABILITY RESEARCH & COMPETITIVE INSIGHT ‣ Why (Goal) 1: Identify “low hanging fruit” issues and areas for deeper research ‣ Why (Goal) 2: Competitive Analysis ‣ Who: 18 Users, 3 Tests, 1 focused on competitor ‣ When: 3 weeks quick testing ‣ How: Unmoderated User Testing, Recruits ‣ Resulted in 20 Quick Fixes and ~5 areas for significant focused future research, structural design ideas from competitive research 37
  38. 38. CASE STUDY USER RESEARCH: SPECIFIC FEATURE & PAGE USABILITY RESEARCH Lightweight Approach: • 12 users, 2 Tests, Captured through Screener • Incentive = $75 AMEX Gift Card • Done in conjunction with Moderated Tests, Existing Personas, and Analytics Work Multiple Hypotheses Tested Flows & Features: • Cart Abandonment Motivation • Wishlist Use • Listing Page Efficacy • Feature Value to User • Check Out Flow • Also Gathered Qualitative User Opinions Time = ~1 month • 1 week to plan • 1 week to implement • 1 week to Analyze 38
  39. 39. CASE STUDY RECOMMENDATIONS & OUTCOMES ‣ Multiple UX “Quick Fixes” were launched during the longer project life cycle ‣ Improved Navigation, Checkout Flow, Product Detail Page, Strategic Use of Visual Assets ‣ Road map of features & functionality (Faceted Search Results, Improved Integration of Multiple Sites) ‣ The Preliminary Numbers: - 16% Increase in order volume - Organic Traffic increased by 53% (due to improved SEO performance due to navigation changes) - 84% increase in page views, 28% decline in bounce rate from landing pages 39
  40. 40. CASE STUDY UNMODERATED USABILITY RESEARCH HELPED ‣ Provide Insight into User Behaviors & Expectations ‣ Ensure a Pervasive Customer Focus through Design Cycles ‣ Identify Initial Problem Set ‣ Point out easily implemented immediate “Quick Fixes” ‣ Narrow & Focus Future Research and Development Efforts ‣ Craft a Strategic User Research Phase ‣ Guide Successful Site Changes ‣ Establish IA / UX Road Mapping & Priority for Next Project Iterations ‣ ... All on relatively lean time & money budgets 40
  42. 42. RESOURCES 42
  43. 43. RESOURCES 43 ‣ When to Test: Incorporating User Testing Into Product Design ‣ Lessons From an Expert: What to Test ‣ How to Test: 9 Steps for Creating the Perfect User Test USERTESTING.COM BLOG
  45. 45. NEXT STEPS 45 PRO TIP: 5 Free Test Credits for participants in this class - use the code NO CODE AVAILABLE at for 5 free tests TRY IT OUT
  46. 46. NEXT STEPS 46 • can do it for you • can train you to do it yourselves • can do it with you SHAMELESS SELF PROMOTION
  47. 47. Q&A LEAN USER TESTING 47 Jessica DuVerneay - Information Architect The Understanding Group (TUG) Twitter: @undrstndng THANK YOU!