Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

World Usability Day 2009 - Remote vs Lab Usability Testing


Published on

This presentation to FedEx stakeholders at their World Usability Day expo summarizes the tradeoffs between onsite / lab-based vs. remote user testing for usability studies and provides tips for both methods.

  • Be the first to comment

  • Be the first to like this

World Usability Day 2009 - Remote vs Lab Usability Testing

  1. 1. lab vs. remote usability testing<br />presented 11/12/09 at FedEx World Usability Day Expo<br />
  2. 2. Lokion is a boutique interactive agency. The depth of our services in strategy, design and technology outweighs our size, as do our clients and partners. <br />
  3. 3. Lokion Services<br />strategy<br />user experience and usability<br />information architecture<br />business analysis<br />content strategy<br />quality assurance<br />design<br />user-sensitive design<br />branded content creation and management<br />multimedia production<br />technology<br />expert custom software development<br />systems architecture<br />ecommerce software implementation<br />
  4. 4. Client Sampling<br />AutoZone<br />Viking Range Corporation<br />FedEx<br />International Paper<br />Memphis Grizzlies<br />First Tennessee / First Horizon<br />Hilton<br />Nokia<br />Deluxe Corporation<br />Hewlett Packard<br />FedEx Institute of Technology<br />Smith and Nephew<br />ServiceMaster Clean<br />AmeriSpec<br />National Ornamental Metal Museum<br />FedEx Kinko’s<br />Accredo<br />Lightyear Capital<br />Harrah’s<br /><br />David Lusk Gallery<br />Cellular South<br />Jeppesen<br />Scripps Network<br />Stephens<br /><br />Rhodes College<br />Luminetx Corporation<br />Strategic Financial Services<br />SRA/Environmental Protection Agency<br />Cap Gemini<br />Horizon Ag<br />mbi<br />Bridges<br />
  5. 5. Strategic User Experience<br />
  6. 6. Strategy: People + Process<br />Information Architecture (IA)<br />Sitemapping<br />Use Cases<br />Wireframes<br />Prototyping<br />User Experience (UX)<br />Thinkfactory<br />Audience Research<br />Persona Creation<br />Usability Reviews<br />Benchmark Studies<br />User Testing<br />Web Analytics<br />Accessibility Audits<br />Buzzwatch<br />Business Analysis (BA)<br />Business Requirements<br />Gap Analysis<br />Process Analysis and Improvement<br />Training and Change Management<br />Content Strategy (CS)<br />Unified Strategic Plan for Content<br />Content Gathering, Creation and Editing<br />Content Style Guides<br />Content Audits<br />Content Management Systems and Training<br />Search Engine Optimization (SEO)<br />Personalization and Scenario Marketing<br />Quality Assurance (QA)<br />Test Plan<br />QA Test Cases<br />QA Management and Reporting<br />QA Test Participation<br />Regression/Smoke Testing<br />User Acceptance Testing (UAT)<br />
  7. 7. Why Lokion?<br />Passionate user experience advocates<br />Conducted 200+ usability test sessions including 28 remote test sessions<br />Facilitated 44 user test sessions for FedEx including Ship Online, Billing Online, Critical Inventory Logistics and Investor Relations<br />Lokion is 10 years old and has worked with FedEx consistently for 9 of those years. <br />Other usability testing engagements include First Tennessee / First Horizon, Service Master, Cellular South, Viking Range, SkyTel, Ducks Unlimited and Rhodes College<br />
  8. 8. Lokion and FedEx<br /> &quot;We needed to redesign FedEx® Billing Online to be more useful than ever, and more efficient than the paper invoicing process. We selected Lokion Interactive because of their knowledge of FedEx technology and processes, and their design expertise. This was definitely the right choice, as they exceeded our expectations and project goals with their Prototype development and Usability management. We saw a measurable customer satisfaction improvement with the new design.“<br />-- Derrick Reives, Sr. Marketing Specialist<br />&quot;We had a great experience working with the Lokion team on the FBO Project, our first dUX team partnership on a usability study. Lokion provided excellent support in setting up the usability sessions and insightful feedback on each of the prototype revisions. Success metrics were determined pre-usability and their Usability Report included very detailed improvement measurements and actionable recommendations. We look forward to working with them more in the future.“<br />-- Jane Fulmer, Marketing Specialist Advisor<br />&quot;We recently worked with Lokion to perform usability on an improved user experience for an existing application. The team at Lokion provided great value in recruiting the right customers, managing the project deliverables, facilitating the sessions and providing a top notch summary and detailed usability findings report. I strongly recommend their services!“<br />-- Beth Newby, Marketing Principal<br />&quot;Lokion was an effective business partner for every step of the prototype development and usability. We were able to get great information from local and remote customers without spending travel money. Their facilities and their facilitator were fabulous!“<br />-- Ann Crislip, Sr. Marketing Specialist<br />
  9. 9. Lab vs. Remote Usability Testing<br />VS<br />
  10. 10. Lab-based Usability Testing<br /> Individual representative users physically come into a lab setting and perform a series of pre-defined tasks using a web site, application or prototype.<br /> One or more observers in the same room or an adjacent room record the level of success, usability issues encountered and time it takes to complete each task.<br />
  11. 11. Usability Lab :: Typical Setup<br />
  12. 12. Participant Room<br />
  13. 13. Observation Room<br />
  14. 14. Guinea Pig Syndrome?<br />
  15. 15. “They make frequent testing easy, manage it well, and give great, timely results.”<br />Lokion’s onsite usability lab<br />— Vice President, Internet Development<br /> Online Financial Services First Tennessee<br />
  16. 16. unobtrusive test environment<br />
  17. 17. comfortable observation room<br />
  18. 18. Remote Usability Testing<br /> Individual representative users stay in their own native settings using their own computers and perform a series of pre-defined tasks using a web site, application or prototype.<br /> One or more observers connect with the user over the phone and Internet to record the level of success, usability issues encountered and time it takes to complete each task.<br />
  19. 19. Remote Testing :: Typical Setup<br />
  20. 20. Tools of the Trade<br />Integrated: Techsmith Morea / Uservue, Ethnio, Loop11, OpenVULab, TestLed<br />Automated: Chalkmark, ClickTale, Keynote WebEffective, LabsMediClickHeat, m-pathy, OpenHallway, OptimalSort, RelevantView, Simple Mouse Tracking (SMT), Treejack, UserFocus, UserZoom, Vividence, Webnographer<br />Recorded: Adobe Connect, Breeze, Camtasia, CamStudio, Co-Pilot, DataLogger, GoTo Meeting, Hello Direct Pro, LiveLook, LK Audio QuckTap, Note Pad, Skype, Trillian, WebEx, Windows Meeting Space<br />
  21. 21. Other Variations<br />Moderated / facilitated vs. automated / unattended<br />Traditional agency recruiting vs. online scheduled or live native task recruiting<br />Synchronous vs. asynchronous<br />Quantitative vs. qualitative<br />Baseline, iterative or comparative<br />Electronic vs. paper<br />High- vs. low-fidelity<br />Static, interactive, mockup or live<br />
  22. 22. Trade-offs :: Logistics and Technicalities<br />Lab<br />Remote<br />
  23. 23. Comparison :: Logistics<br />
  24. 24. Tradeoffs :: Interactions<br />
  25. 25. Comparison :: Interactions<br />
  26. 26. Emotional experience :: non-verbal cues<br />
  27. 27. Tips and Considerations<br />Both methods are valid.<br />Remind repeatedly (emails, Outlook appointments and snail mail).<br />Suggest arrival or install / connect 10 minutes prior to session.<br />Make creative use of backups.<br />Privacy agreements matter.<br />Communication helps.<br />Expect glitches (Plan B).<br />Lather, rinse, repeat…<br />
  28. 28. Tips :: Lab-based Usability Testing<br />Include a map and parking instructions.<br />It’s the little things (water, snacks, bathroom…).<br />Cash is good….<br />
  29. 29. Tips :: Remote Usability Testing<br />Confirm email and phone.<br />Include instructions and demo links.<br />Installation is a hurdle.<br />Practice muting.<br />Amazon gift certificates only need an email address to fulfill.<br />
  30. 30. The bottom line…<br />Remote:<br /><ul><li> distributed
  31. 31. contextual
  32. 32. convenient</li></ul>Lab:<br /><ul><li> controlled
  33. 33. focused
  34. 34. personal</li></ul>VS<br />There is a place for both methods. <br />They are complementary. <br />Mix and match.<br />
  35. 35. Resources :: Learn More<br />“Comparative Study of Synchronous Remote and Traditional In-Lab Usability Evaluation Methods,” by PrakaashSelvaraj, Virginia Polytechnic Institute and State University, May 19, 2004.<br />“An Empirical Comparison of Lab and Remote Usability Testing of Web Sites,” by Tom Tullis, Stan Fleischman, Michelle McNulty, Carrie Cianchette, and Marguerite Bergel, Human Interface Design Dept, Fidelity Investments.<br />“Remote Online Usability Testing: Why, How, and When to Use It,” by Dabney Gough and  Holly Phillips, Boxes and Arrows, June 9, 2003.<br />“Experience Remote Usability Testing: Examine study results on the benefits and downside of remote usability testing,” by VeldaBartek and Deane Cheatham, IBM, January 1, 2003.<br />