Webinar lab + ous presentation may10

605 views

Published on

How does one pair online usability testing with lab usability testing? On May 6th, 2010, UserZoom joined forces with Bentley University’s Design and Usability Center to present how online, unmoderated usability testing can be combined with lab usability testing to produce rich, actionable results.

We examined Amazon’s e-commerce grocery store and compared the results of each method to reveal the secrets and lessons we have learned.

Published in: Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
605
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors. After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors. After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors. After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  • Webinar lab + ous presentation may10

    1. 1. Webinar: Combining Lab and Online Usability Testing: Lessons Learned May 2010 Dial: 312-878-0211 Access Code: 599-198-841 Audio PIN: Shown after joining the Webinar Webinar ID: 309-304-048
    2. 2. First, the introductions… Alfonso de la Nuez Partner & Chief Marketing Officer at UserZoom Kim Oslob Research & Product Strategy Director at UserZoom Bill Albert Director of the Design & Usability Center, Bentley University Alfonso
    3. 3. First, the introductions… Alfonso <ul><ul><ul><ul><li>Leading online user experience research software company </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Develops proprietary on-demand software app for online research </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Built for & by UX & marketing professionals </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Offers a cost-effective, time-saving methodology </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Has 9 years of experience in UX research & consulting </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Has offices in Sunnyvale (CA), London (UK) & Barcelona (Spain) </li></ul></ul></ul></ul>
    4. 4. What makes us proud? Customers in 10 countries Alfonso
    5. 5. First, the introductions… <ul><li>The Design and Usability Center at Bentley University was founded in 1999 </li></ul><ul><li>Provides UX consulting services to corporate clients </li></ul><ul><li>Supports the MS program in Human Factors in Information Design </li></ul><ul><li>Focuses on user experience research and evaluation </li></ul>
    6. 6. Acknowledgements <ul><li>Special thanks to Cynthia Kamishlian and Dharmesh Mistry from Bentley University for carrying out this research </li></ul><ul><li>Based on our recent book “Beyond the Usability Lab”, co-authored with Tom Tullis and Donna Tedesco </li></ul>
    7. 7. Agenda <ul><li>Introduction to both research methods (UZ, 10 mins) </li></ul><ul><li>The power of combining methods (DUC, 10 mins) </li></ul><ul><li>Case study: Parallel studies (DUC + UZ, 20 mins) </li></ul><ul><li>Conclusion: Looking at the ‘big picture’ (UZ, 10 mins) </li></ul><ul><li>Q&A (10 mins) </li></ul>Five things we’d like to talk to you about today: Alfonso
    8. 8. Brief introduction to both research methods Usability testing in the Lab  Moderation, think-aloud… Bill
    9. 9. Brief introduction to both research methods Usability testing in the Lab  Observing, recording…
    10. 10. Brief introduction to both research methods Usability testing in the Lab  Eye Tracking…
    11. 11. Brief introduction to both research methods Usability testing in the Lab  Benefits/pros & limitations/cons <ul><li>The face-to-face  Catching the very details </li></ul><ul><li>Ability to ask questions/interact with the participant live </li></ul><ul><li>The ability to invite others to attend the sessions </li></ul><ul><li>Small sample size </li></ul><ul><li>Lack of natural environment/context </li></ul><ul><li>Cost +++ </li></ul>
    12. 12. Brief introduction to both research methods Online Usability Testing (a.k.a. Unmoderated Remote Usability Testing) Kim
    13. 13. Brief introduction to both research methods Online usability testing  At a glance… <ul><li>Hundreds of users can be tested </li></ul><ul><li>Participation in the natural context… </li></ul><ul><li>… from geographically spread locations </li></ul><ul><li>No human moderation needed </li></ul><ul><li>Our browser bar connects users with our secure servers </li></ul>
    14. 14. Brief introduction to both research methods Online usability testing  Build the study first using a software solution
    15. 15. Brief introduction to both research methods Online usability testing  Features! Power to the researcher <ul><li>High-level of flexibility </li></ul><ul><li>Advanced task validation criteria (by question, by page interception, by time...) </li></ul><ul><li>Advanced questionnaires tool (all type of questions: Choice, scales, open...) </li></ul><ul><li>Insert multimedia (images, videos, flash, applets, sounds...) </li></ul><ul><li>Manage advanced logic/branching & conditions </li></ul><ul><li>Behavioral questionnaires (by interception) </li></ul><ul><li>Randomization: tasks, answers... </li></ul><ul><li>Piping capabilities </li></ul><ul><li>Mandatory options </li></ul><ul><li>Time-out options </li></ul><ul><li>Sample quality control </li></ul><ul><li>Multi-lingual studies support (including Asian languages). </li></ul><ul><li>… and more </li></ul>
    16. 16. Brief introduction to both research methods Partners of <ul><li>Choose the user recruitment yourself </li></ul><ul><li>Panel </li></ul><ul><li>Intercept </li></ul><ul><li>Private mailing list </li></ul>
    17. 17. Brief introduction to both research methods Online usability testing  This is what it looks like…
    18. 18. Brief introduction to both research methods
    19. 19. Online usability testing  Analyzing results Brief introduction to both research methods <ul><li>Effectiveness ratios </li></ul><ul><li>Efficiency ratios (time and clicks) </li></ul><ul><li>Visual clickstream (nav paths) </li></ul><ul><li>Click heatmaps </li></ul><ul><li>Users' suggestions & feedback </li></ul><ul><li>Satisfaction & brand perception indicators </li></ul><ul><li>Cluster analysis & dendograms </li></ul><ul><li>All on a per-task level , results gathered automatically </li></ul>
    20. 20. Online usability testing  Analyzing results Brief introduction to both research methods
    21. 21. Brief introduction to both research methods Online usability testing  Benefits/pros and limitations/cons <ul><li>Quantifying usability  Objective, statistically significant data </li></ul><ul><li>Automation of results, cost-effectiveness </li></ul><ul><li>Participation in the natural context </li></ul><ul><li>Lack of face-to-face </li></ul><ul><li>Can’t invite to observe sessions </li></ul><ul><li>Must anticipate participant’s questions in advance </li></ul>
    22. 22. Agenda <ul><li>Introduction to both research methods (UZ, 5 mins) </li></ul><ul><li>The power of combining methods (DUC, 10 mins) </li></ul><ul><li>Case study: Parallel studies (DUC + UZ, 20 mins) </li></ul><ul><li>Conclusion: Looking at the ‘big picture’ (UZ, 10 mins) </li></ul><ul><li>Q&A (10 mins) </li></ul>Five things we’d like to talk to you about today: Alfonso
    23. 23. Our UX Toolkit
    24. 24. Power of Combining Methods <ul><li>Every research method has unique strengths and limitations </li></ul><ul><li>Gather new insights with each method </li></ul><ul><li>Greater confidence when observing similar findings through multiple methods </li></ul>
    25. 25. Seeing the Big Picture
    26. 26. Which one goes first? Lab first, then Online Online first, then Lab Identify/fix “low hanging fruit”, then focus on remaining tasks with large sample size Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues Generate new concepts, ideas, questions through lab testing, then test/validate online Collect video clips or more quotes of users to help bring metrics to life Validate attitudes/preferences observed in lab testing Gather all the metrics to validate design – if it tests well, then no need to bring users into the lab
    27. 27. Agenda <ul><li>Introduction to both research methods (UZ, 5 mins) </li></ul><ul><li>The power of combining methods (DUC, 10 mins) </li></ul><ul><li>Case study: Parallel studies (DUC + UZ, 20 mins) </li></ul><ul><li>Conclusion: Looking at the ‘big picture’ (UZ, 10 mins) </li></ul><ul><li>Q&A (10 mins) </li></ul>Five things we’d like to talk to you about today: Alfonso
    28. 28. Case study: Parallel studies <ul><li>We conducted 2 studies parallel. Here are the basic details: </li></ul><ul><ul><li>Website tested: Amazon.com’s Grocery and Gourmet Food Store </li></ul></ul><ul><ul><li>Lab study conducted by the DUC </li></ul></ul><ul><ul><li>Online Usability Study conducted by UZ </li></ul></ul><ul><ul><li>Date of the study: April 2010 </li></ul></ul>
    29. 29. Case study: Parallel studies <ul><li>Basic details of the studies (II) </li></ul><ul><ul><li>Objectives: </li></ul></ul><ul><ul><ul><li>What is the unique contribution of each method? </li></ul></ul></ul><ul><ul><ul><li>What do we learn by combining methods? </li></ul></ul></ul><ul><ul><li>Participants : 10 for the lab study, 100 for the online study </li></ul></ul><ul><ul><li>Study design: </li></ul></ul><ul><ul><ul><li>Initial questionnaire </li></ul></ul></ul><ul><ul><ul><li>3 tasks </li></ul></ul></ul><ul><ul><ul><li>Follow up questions </li></ul></ul></ul><ul><ul><ul><li>Final questionnaire </li></ul></ul></ul>
    30. 30. Case study: Parallel studies <ul><li>Participant Demographics from the Lab Usability Study: </li></ul><ul><ul><li>Average age 29 </li></ul></ul><ul><ul><li>Age range between 19 and 51 </li></ul></ul><ul><ul><li>6 Males, 4 Females </li></ul></ul><ul><ul><li>All had previously used Amazon.com website </li></ul></ul><ul><ul><li>None had ordered groceries on-line </li></ul></ul><ul><ul><li>Visited Amazon.com approximately every 2 weeks, on average </li></ul></ul><ul><ul><li>Previously purchased books, electronics, toys, and more. </li></ul></ul><ul><ul><li>Expected packaged, non-perishable grocery items to be sold on-line </li></ul></ul><ul><ul><li>Most said they hadn’t bought groceries on-line because they, “like the experience of picking up the produce and meat.” </li></ul></ul><ul><ul><ul><li>Expressed concern about products being fresh </li></ul></ul></ul><ul><ul><li>A few worried about exorbitant service or shipping charges making it not worthwhile. </li></ul></ul>
    31. 31. Case study: Parallel studies <ul><ul><ul><li>Task 1: </li></ul></ul></ul><ul><ul><ul><li>“ You just ran out of dishwasher detergent. See if Amazon has a pack of 6 Cascade Complete Dishwasher Detergent Powder, in the 45 ounce size.” </li></ul></ul></ul>
    32. 32. Case study: Parallel studies <ul><li>Search vs. Menu Navigation </li></ul><ul><ul><li>Approximately half of the participants wanted to immediately use SEARCH </li></ul></ul><ul><li>Category Confusion </li></ul><ul><ul><ul><li>Most likely categories were “Home & Garden”, “Grocery, Health & Beauty” </li></ul></ul></ul><ul><ul><ul><li>Chose “Grocery, Health & Beauty” through process of elimination </li></ul></ul></ul><ul><ul><ul><li>Further confusion within “Grocery, Health & Beauty” </li></ul></ul></ul><ul><ul><ul><li>“ I don’t know if ‘Grocery and Gourmet food’ will have detergent” </li></ul></ul></ul>Results of Locating Dishwashing Detergent
    33. 33. Case study: Parallel studies <ul><li>Confusion within “Grocery & Gourmet Food” menu </li></ul><ul><ul><ul><ul><li>First menu is 4 screens long </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Participants were confused where to look for dishwasher detergent </li></ul></ul></ul></ul>Results of Locating Dishwashing Detergent Dishwasher Detergent found under here
    34. 34. Case study: Parallel studies <ul><li>Nesting Confusion </li></ul><ul><ul><li>All participants were confused by path to find dishwashing detergent: </li></ul></ul><ul><ul><ul><li>Amazon.com > Grocery & Gourmet Food > All Household (page is titled ‘Health & Personal Care ‘) > All Household Supplies > All Dishwashing > Scroll to locate Cascade Dishwashing Detergent </li></ul></ul></ul><ul><ul><ul><li>Participants scrolled up and down navigation menus </li></ul></ul></ul><ul><ul><ul><li>A few gave up mid-way & used SEARCH </li></ul></ul></ul><ul><ul><ul><ul><ul><li>“ There are a lot of clicks to go through: In Household Cleaners, Kitchen Cleaners…I don’t see dishes…maybe by brand? I don’t see Cascade here. I would do SEARCH.” </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>“ That took probably 5-6 minutes. For me, that’s really long.” </li></ul></ul></ul></ul></ul>Results of Locating Dishwashing Detergent
    35. 35. Case study: Parallel studies <ul><ul><ul><li>Task 2: </li></ul></ul></ul><ul><ul><ul><li>“ You want to buy Skippy Peanut Butter. Find out if there are any discounts or coupons for it.” </li></ul></ul></ul>
    36. 36. Case study: Parallel studies <ul><li>Product Search vs. Coupon Search </li></ul><ul><li>A few looked on Amazon.com home page for Today’s Deals </li></ul><ul><li>Most looked for the product in the “Grocery” section, and then looked for a coupon </li></ul><ul><li>Some looked at “Special Offers” in the “Grocery” screen </li></ul><ul><ul><li>Special Offers led to a coupon and New and Used food, which disturbed participants. </li></ul></ul>Results of Locating Skippy Peanut Butter Coupon
    37. 37.
    38. 38. Case study: Parallel studies <ul><li>Participants that looked for the product complained about the length of the menu on the left side of the grocery page </li></ul><ul><ul><li>Peanut Butter, under “Sauces and Dips,” was four screens down plus 2 clicks. “It was tedious.” </li></ul></ul>Results of Locating Skippy Peanut Butter Coupon
    39. 39. Case study: Parallel studies <ul><li>Users had strong feelings about coupons for grocery items </li></ul><ul><li>“ I would expect they’d show me the price they have, it doesn’t matter if it is cheaper.” </li></ul><ul><li>“ If I am purchasing a big purchase, I would do it from Google. I would look for coupons [only] for big items.” </li></ul><ul><li>“ I would have preferred to see a link from the home page– specials, coupons, etc…I don’t want to hunt down each thing.” </li></ul><ul><li>“ It was not clear that a discount was available until you went to the [item] description page” </li></ul><ul><li>“ I had to look through a lot of things.&quot; </li></ul>Results of Locating Skippy Coupon
    40. 40. Case study: Parallel studies <ul><ul><ul><li>Task 3: </li></ul></ul></ul><ul><ul><ul><li>“ Find out how you would schedule the same groceries to be delivered every 3 months.” </li></ul></ul></ul>
    41. 41. Case study: Parallel studies <ul><li>All found the correct starting place: the “Grocery & Gourmet Food” page </li></ul><ul><ul><li>Most noticed the “Shop Subscribe and Save” link </li></ul></ul><ul><ul><li>Others tried to schedule each item from the item’s page </li></ul></ul><ul><ul><ul><li>“ I think you will have to do it for every item.” </li></ul></ul></ul><ul><ul><ul><li>“ I would probably go to the product and check if I can schedule it.” </li></ul></ul></ul><ul><ul><ul><li>“ [a product page] might say, “do you want this delivered?” </li></ul></ul></ul>Scheduling Groceries for Repeated Delivery
    42. 42. Case study: Parallel studies <ul><li>Participants were confused on how to schedule grocery delivery </li></ul><ul><ul><li>Most missed the delivery scheduling link at the top of the page </li></ul></ul><ul><ul><li>“ The text was small, and I wasn’t sure I was in the right place.” </li></ul></ul><ul><ul><li>“ The grayed-out part doesn’t jump out” </li></ul></ul>Scheduling Groceries for Repeated Delivery Delivery Scheduling Information Link
    43. 43. Case study: Parallel studies <ul><li>Confusing IA </li></ul><ul><li>“ Dishwasher detergent was under “Health [& Personal Care], which is not the first place I’d look.”” </li></ul><ul><li>“ There were too many categories, it confuses you…” </li></ul><ul><li>“ It is not as well organized as other parts of Amazon.” </li></ul><ul><li>“ I don’t know why ‘All Household Supplies’ is under ‘Health and Personal Care’” </li></ul><ul><li>Difficult navigation </li></ul><ul><li>“ Navigation wasn’t intuitive. The left-hand-side menu was so far down the page and not duplicated in the banner.” </li></ul><ul><li>“ The browse list was too long. I don’t want to go through 10 chocolates before going to the pasta.” </li></ul><ul><li>“ It didn’t feel intuitive.” </li></ul><ul><li>“ I didn’t like how many levels I had to go through.” </li></ul><ul><li>“ It was hard to find things. There were long lists to find food items.” </li></ul>Conclusions from the Lab Study
    44. 44. Case study: Parallel studies Highlighted findings from the Online Usability Study: Kim
    45. 45. Study Background <ul><li>- 71% Male, 29% Female </li></ul><ul><li>Age range distributed between 18–55; 88% fell between 26-45 </li></ul><ul><li>All have purchased from Amazon.com and visit Amazon once a week, month or every few months. </li></ul><ul><li>Eighty-eight percents of participants purchase groceries from in the store only, 6% online only and 6% both online and in a store. </li></ul><ul><li>Main reasons for considering purchasing groceries online is for convenience and time. </li></ul><ul><li>- April 17 th to April 26, 2010 </li></ul><ul><li>Unmoderated remote method </li></ul><ul><li>100 participants </li></ul><ul><li>Recruited through participant panel </li></ul>Profile Details
    46. 46. Tasks <ul><ul><ul><li>Locate specific item: You just ran out of dishwasher detergent. Find Cascade Complete Dishwasher Detergent Powder, 45 oz (pack of 6). Validation by reaching the correct page with the detergent (URL validation). </li></ul></ul></ul><ul><ul><ul><li>Finding a discount code: You want to buy Skippy Peanut Butter with a discount you heard about on Amazon.com. Please write down the discount code. Validation by choosing the correct discount code (multiple choice). </li></ul></ul></ul><ul><ul><ul><li>Schedule grocery delivery: Find out how often you can schedule a delivery to your home. Validation by how often you can set up a delivery (multiple choice). </li></ul></ul></ul>
    47. 47. Study Design Initial Questionnaire: How frequently do you visit Amazon.com? Where do you usually buy your groceries? Why have you considered purchasing groceries online? Post Task: Validation by url or question (Success, Error, Abandon ratios collected). Ease of Use. How intuitive links and menus were and how easy it was to start the search. Issues and problems experienced while completing tasks. Final Questionaire: Overall satisfaction Satisfaction with navigation, product search, information offered and look and feel of the site. Likelihood to purchase groceries from Amazon.com. Likelihood to recomment to a friend.
    48. 48. T1: Locating Cascade Complete Success Non-Success Participants did feel particularly lost on the site (most likely due to experience using Amazon.com. However they did not feel the menus and links were extremely intuitive and weren’t always sure were to start their search.
    49. 49. T1: Locating Cascade Complete Over half (51%) of the participants defaulted to using search indicating the menus and navigation of the site were not very intuitive. Fifteen percent started in Health and Personal while 5% looked in promotional deals. Highlights dominant path *3% insignificance removed
    50. 50. T1: Locating Cascade Complete What participants found difficult when attempting the task: Exact Quotes: “ Hard to navigate to find desired product. Too long and was getting frustrating.” “ There were a lot of results so you have to type it in exactly or spend a couple minutes searching.” “ I didn’t see a soap category and didn’t realize how the search function worked.” “ I didn’t know where to look.”
    51. 51. T2: Finding a discount code Success Non-Success Forty-seven percent of the participants thought it was very difficult to locate the discount code. They thought it was very unclear how to start searching, it was not where they expected it to be and were somewhat lost during the search process. Extremely Easy Extremely Difficult
    52. 52. T2: Finding a discount code
    53. 53. T2: Finding a discount code Error participants (55%) could not locate the code, thought it was too difficult to find the code or the peanut butter and stated: “ I was expecting the discount in the results page, or the product page. I ended up looking for it under ’specials’, but I don’t know if that’s the best place for it.” “ Too much searching and clicking.” “ Looked under special offers having firstly used search, had to notice the offer code above product within the offer description.”
    54. 54. T3: Schedule Grocery Delivery Success Non-Success Seventy-five percent of participants could not complete the task. They did not think it was clear how to start searching and felt lost during the search process. Forty percent of participants who were successful, felt it was very difficult to complete the task. Extremely Easy Extremely Difficult
    55. 55. T3: Schedule Grocery Delivery Abandon participants (38%) could not find the information, thought the menus were difficult to understand and the task was taking too long. “ It was not obvious to me that I needed to click on the link at the top right of the screen to get this information until I had spent quite a bit of time looking all over the rest of the screen.” “ I wasn’t certain where to look for the information.” “ Nothing about this process has been user friendly.”
    56. 56. Final Questionnaire Fifty-three percent of participants thought the navigation was very difficult, 39% thought it was difficult to search for a product, forty-seven percent rated the overall look and feel of the site as very poor.
    57. 57. Final Questionnaire Fifty-five percent of participants stated they would not purchase their groceries in the future from Amazon.com. Net Promoter Score (NPS) calculation: 0 to 6 = Detractors 7 to 8 = Passive 9 to 10 = Promoters Net Promoter Score (NPS) = % of Promoters - % of Detractors
    58. 58. Agenda <ul><li>Introduction to both research methods (UZ, 15 mins) </li></ul><ul><li>The power of combining methods (DUC, 10 mins) </li></ul><ul><li>Practical example: Parallel studies (DUC + UZ, 20 mins) </li></ul><ul><li>Conclusion: Looking at the ‘big picture’ (UZ, 5 mins) </li></ul><ul><li>Q&A (10 mins) </li></ul>Five things we’d like to talk to you about today: Alfonso
    59. 59. Conclusions: Looking at the ‘big picture’
    60. 60. Conclusions: Looking at the ‘big picture’ <ul><li>Main issue, of course: Gourmet store built like the book store </li></ul><ul><li>IA + navigation + labeling problems encountered </li></ul><ul><li>Both research methods prove this: </li></ul><ul><ul><li>By observing & listening at the Lab we learned a lot, but then… </li></ul></ul><ul><ul><li>… we also quantified how big the issues are through metrics (UserZoom) </li></ul></ul>
    61. 61. Agenda <ul><li>Introduction to both research methods (UZ, 5 mins) </li></ul><ul><li>The power of combining methods (DUC, 10 mins) </li></ul><ul><li>Practical example: Parallel studies (DUC + UZ, 20 mins) </li></ul><ul><li>Conclusion: Looking at the ‘big picture’ (UZ, 5 mins) </li></ul><ul><li>Q&A  Shoot! (10 mins) </li></ul>Five things we’d like to talk to you about today:
    62. 62. º USA 440 N. Wolfe Rd. Sunnyvale, CA 94085 USA Phone: +1 (408) 524 – 74 45 Contact: Alfonso de la Nuez [email_address] UK Aylesbury House, 17 – 18 Aylesbury Street, London ECIR 0DB, UK Phone: +44 (0) 20 7193 2171 Cell: +44 (0) 7900 472 920 Contact: Arthur Moan [email_address] SPAIN Av. Diagonal 419 3º 2ª 08008 Barcelona, Spain Phone: +34 93 414 7554 Cell: +34 93 368 4295 Contact: Javier Darriba [email_address] Thanks so much for your time! Oh! Don’t forget: The slides and the video of this webinar will be available in a few days in our blog www.userzoom.com

    ×