Your SlideShare is downloading. ×
0
Usability Boot Camp  Brian Sullivan DFW-UPA Workshop December 5, 2009
Introduction
What is Usability Testing? <ul><li>Usability testing involves: </li></ul><ul><li>You Watch Customers </li></ul><ul><li>The...
Cast of Characters:  Three Main Roles <ul><li>Facilitator:   This person oversees the entire test process—plan, test, and ...
Benefits of Usability Testing: Feedback from Real Customers <ul><li>If your website is difficult to use, customers  leave ...
Benefits of Usability Testing: Informed Design Decisions Design A Design B <ul><li>We encountered these results from a Usa...
Benefits of Usability Testing:  Mitigate Risk (Also, Rewarded) For years, companies have been performing usability testing...
Benefits of Usability Testing:  Saves Time and Money <ul><li>Randy Bias and Deborah Mayhew’s book provides hundreds of for...
Benefits of Usability Testing:  Makes Lots of Money Jared Spool The $300,000,000 Fix (per Jared Spool): In the above examp...
Benefits of Usability Testing:  Makes Lots of Money Jared Spool New Customers: New customers did  not  want to register.  ...
Benefits of Usability Testing:  Makes Lots of Money Jared Spool Existing Customers: Existing customers couldn’t remember t...
Benefits of Usability Testing:  Makes Lots of Money Jared Spool Jared Spool’s $300,000,000 Fix: Make it obvious. The resul...
Usability Methods and Tools
Two Primary Methods of Data Collection Usability Methods Table
Quantitative and Qualitative Testing
Quantitative Testing Tool (User Zoom Demo) <ul><li>Quantitative Usability Testing has grew in the last decade. </li></ul><...
Qualitative Testing Tools and Methods <ul><li>Qualitative usability testing for the web has really matured in the last 30 ...
Three Types of Usability Tests (Qualitative Perspective) Today’s workshop will focus on  summative usability testing.
Basic Usability Testing Process <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing...
A Typical Scene Behind The Glass in a Usability Test
Step 1: Planning Your Usability Test
Basic Usability Testing Process <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing...
A Simplified Version of Activities-People-Context Picture from Stephen Anderson Somebody (person) does something (activity...
Here is the Royal Caribbean International Website
Describe Activities That Might Be Done….. <ul><li>Create a group reservation </li></ul><ul><li>Plan an event (ex: wedding)...
Describe People That Might Be Using the Site….. <ul><li>Captain of the Ship </li></ul><ul><li>Group Traveler  </li></ul><u...
Describe the Context of Use….. <ul><li>Company website </li></ul><ul><li>IVR system </li></ul><ul><li>Kiosk (on the ship) ...
It Is Not Too Complicated….. <ul><li>People = Usability Participants </li></ul><ul><li>Context = Test Scenarios </li></ul>...
Activities:  What Concerns Does Your Product Team Have? <ul><li>Ask the project team to list out their overall concerns wi...
People:  Who are the Target Customers? <ul><li>Ask the project team.  They tend to focus on a  single  user.  </li></ul><u...
Context:  Where and How are People Going to Use It? <ul><li>Ask the team.  Usually, the team focuses on the delivery devic...
It Is Not Too Complicated….. Just Ask!!! <ul><li>People = Usability Participants </li></ul><ul><li>Context = Test Scenario...
With People-Context-Activities Defined, Data Can Be Collected <ul><li>Performance Data: </li></ul><ul><li>Average Task Com...
Basic Usability Testing Process—Recruting <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you a...
Number of Participants Debate
Number of Participants Debate (Sources)  Jakob Nielsen (circa 1993) Five users will uncover about 85% of the usability iss...
Number of Participants (Brian Sullivan, 2009) <ul><li>General Rules:   </li></ul><ul><li>Test with 6 customers  (per group...
Recruiting Participants <ul><li>Recruiting: </li></ul><ul><li>In-house sources </li></ul><ul><li>Recruiting agency </li></...
Selecting the Right Customers  <ul><li>User profiles: </li></ul><ul><li>Potential Customer </li></ul><ul><li>New Customer ...
Your Recruiting is Only As Good As the Screener <ul><li>Steve Krug has said this: </li></ul><ul><li>I respectfully disagre...
Basic Usability Testing Process—Write Out the Test <ul><li>Step1: Planning Your Usability Test </li></ul><ul><li>Define wh...
Scenarios and Tasks <ul><li>Some experts distinguish between tasks and scenarios: </li></ul><ul><li>Scenarios: </li></ul><...
Example of Scenario with Tasks <ul><li>Scenario #1: </li></ul><ul><li>You want to book a sailing on Royal Caribbean Intern...
Practice Writing Scenarios and Tasks <ul><li>We have two teams today. --Team Cool --Team Ninja Assassins </li></ul><ul><li...
Step 2: Conducting Your Test
Basic Usability Process—Step 2 <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing ...
Don’t Rely on Luck!!!
You Do Not Need a Usability Lab to Conduct a Test
Test Side-by-Side in a Conference Room
Test Wherever Your Customers Might Be I have done usability testing at Starbuck’s and a User Conference.
Remote Usability Testing Easier Than Ever
Morae is Portable, Affordable, and ………………Usable
Shhhhh….you don’t need to record, either
Basic Usability Process—Step 2 <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing ...
Conduct a Pilot Test <ul><li>Dry run of a complete usability test </li></ul><ul><li>Include all test material </li></ul><u...
Basic Usability Process—Step 2 <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing ...
Again, Adopt a Workable Schedule
Tips for Being a Good Moderator <ul><li>Know Your UCD Basics and Design Guidelines </li></ul><ul><li>Be a Quick Learner </...
Tips for Getting the Most from Participants <ul><li>Start with an Easy Task To Build Confidence </li></ul><ul><li>Choose t...
Team Cool Starts With Wizard of Oz Style <ul><li>Wizard of Oz Style Testing: </li></ul><ul><li>Moderator </li></ul><ul><li...
Rules for the Merry Old Land of Oz….. <ul><li>Here are The Rules: </li></ul><ul><li>Get into Groups of three. </li></ul><u...
Team Ninja Assassins Will Be in Usability Lab <ul><li>These are the Test Stations: </li></ul><ul><li>Two Waiting Participa...
After 1.5 Hours, We Switch Locations <ul><li>The Teams Switch: </li></ul><ul><li>Team Cool in Labs </li></ul><ul><li>Team ...
Step 3: Analyzing and Reporting
Basic Usability Process—Home Stretch Now!!! <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you...
Dilbert’s Boss Solves Your Customer Problem
8 Steps to Analyze and Report Your Findings <ul><li>Complete Your Notes </li></ul><ul><li>Summarize Your Findings </li></u...
Step 1—Complete Your Notes <ul><li>After each session, fill in any gaps. </li></ul><ul><li>Basically, complete the short-h...
Step 2—Summarize Findings <ul><li>After each session, create a list of the major issues. </li></ul><ul><li>Organize by Iss...
Step 3—Compile and Tabulate Data <ul><li>Spreadsheet work best here </li></ul><ul><li>Do not work on solutions, yet </li><...
Step 4—Analyze Findings <ul><li>Which tasks were simple? </li></ul><ul><li>Which tasks were problematic? </li></ul><ul><li...
Step 5—Prioritize Findings <ul><li>Severity 1 = Showstopper.  Unless corrected all users will essentially have a problem. ...
Step 6—Create Recommendations <ul><li>Start with Positive Findings </li></ul><ul><li>Remind the team of people-activities-...
Nobody reads a “BIG, HONKING” report.
Step 7—Create Final Document or Presentation <ul><li>Consider the audience </li></ul><ul><li>Some teams just want an Excel...
Step 8—Follow-up <ul><li>Always Provide Usability Next Steps </li></ul><ul><li>Usability Needs to Be Part of the  Developm...
Let’s Compile and Analyze Our Results
Conduct K-J Analysis of Wizard of Oz <ul><li>List all observations on stickies </li></ul><ul><li>Post them on a wall </li>...
Conduct On Board Analysis of Lab Testing <ul><li>Write issues on board after each participant (positive and negative) </li...
Wizard of Oz Testing on Chalkmark
Lab Testing of Four Sites (Sprouts)
Lab Testing of Four Sites (Consumer Search)
Lab Testing of Four Sites (Broccoli.com)
Lab Testing of Four Sites (Casa Ma ň ana)
Question & Answer
Upcoming SlideShare
Loading in...5
×

Usability Testing Workshop

4,258

Published on

Usability Testing Workshop heled by DFW-UPA. Spotlights methods and tools, including Wizard of OZ style and lab-based testing. Demos of UserZoom and Morae were also done.

Published in: Technology, Design
2 Comments
14 Likes
Statistics
Notes
No Downloads
Views
Total Views
4,258
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
269
Comments
2
Likes
14
Embeds 0
No embeds

No notes for slide

Transcript of "Usability Testing Workshop"

  1. 1. Usability Boot Camp Brian Sullivan DFW-UPA Workshop December 5, 2009
  2. 2. Introduction
  3. 3. What is Usability Testing? <ul><li>Usability testing involves: </li></ul><ul><li>You Watch Customers </li></ul><ul><li>They Perform Tasks </li></ul><ul><li>You Note Their Problems </li></ul><ul><li>Making Recommendations </li></ul><ul><li>Rinse and repeat (i.e. iterate) </li></ul>Jakob Nielsen’s Definition of Usability Usability is a quality attribute that assesses how easy user interfaces are to use. The word &quot;usability&quot; also refers to methods for improving ease-of-use during the design process.
  4. 4. Cast of Characters: Three Main Roles <ul><li>Facilitator: This person oversees the entire test process—plan, test, and report. </li></ul><ul><li>Participant: Actual or potential customers. Representative users (marketing, designers) should not be used. </li></ul><ul><li>Observer: Records events as they occur. Limits interaction with the customer. Does contribute to the report. </li></ul>In today’s workshop, you will play all three roles!
  5. 5. Benefits of Usability Testing: Feedback from Real Customers <ul><li>If your website is difficult to use, customers leave . </li></ul><ul><li>If they get lost in your website, customers leave . </li></ul><ul><li>If a customer can’t FIND your product, they can’t BUY it. …. Then, they leave ! </li></ul>Real feedback directly from customers that use the product, which provides data—not opinions.
  6. 6. Benefits of Usability Testing: Informed Design Decisions Design A Design B <ul><li>We encountered these results from a Usability Test: </li></ul><ul><li>Average Time to Book a Limo with Design A = 3:59 </li></ul><ul><li>Average Time to Book a Limo with Design B = 8:00 </li></ul><ul><li>12 of 12 customers successfully booked the correct limo with Design A. </li></ul><ul><li>7 of 12 customers successfully booked the correct limo with Design B. </li></ul><ul><li>All users were TRAINED and currently used Design B. </li></ul>
  7. 7. Benefits of Usability Testing: Mitigate Risk (Also, Rewarded) For years, companies have been performing usability testing. They just call it a product release. Teams that do usability testing before a release mitigate risk. Plus, they are rewarded . (ex: $300 Million Button)
  8. 8. Benefits of Usability Testing: Saves Time and Money <ul><li>Randy Bias and Deborah Mayhew’s book provides hundreds of formulas to show the return on investment for usability testing. </li></ul><ul><li>Here is one formula: </li></ul>Annual Savings in Productivity X Increase efficiency (in hours) X Average hourly wage of user X # of days per year X # of uses per day # of users Using the previous example, a switch to Design B saves over $2 million USD.
  9. 9. Benefits of Usability Testing: Makes Lots of Money Jared Spool The $300,000,000 Fix (per Jared Spool): In the above example, we have a form with two fields, two buttons, and one link. This form appeared on the bottom of a Shopping Cart. What are the problems? Hint: Current and New Customers both had problems.
  10. 10. Benefits of Usability Testing: Makes Lots of Money Jared Spool New Customers: New customers did not want to register. They resisted clicking the Register button. New customers wanted to buy what was in their shopping cart. They did not want a relationship. Most wanted to leave . As it turns out, customers are not required to register—it is optional.
  11. 11. Benefits of Usability Testing: Makes Lots of Money Jared Spool Existing Customers: Existing customers couldn’t remember their exact email and/or password. They clicked the “Forgot Password” link. Help Desk logs revealed this link was clicked about 167,000 each day! According to Spool, 75% of these people didn’t make a purchase. They did leave !
  12. 12. Benefits of Usability Testing: Makes Lots of Money Jared Spool Jared Spool’s $300,000,000 Fix: Make it obvious. The results: The number of customers purchasing went up by 45% . The extra purchases resulted in an extra $15,000,000 the first month. For the first year, the site saw an additional $300,000,000 . $ $
  13. 13. Usability Methods and Tools
  14. 14. Two Primary Methods of Data Collection Usability Methods Table
  15. 15. Quantitative and Qualitative Testing
  16. 16. Quantitative Testing Tool (User Zoom Demo) <ul><li>Quantitative Usability Testing has grew in the last decade. </li></ul><ul><li>Many tools exist for quantitative testing tools are out there. </li></ul><ul><li>Keynote Systems has been around for a long time now. </li></ul><ul><li>You may currently use or know people that use Google Analytics. </li></ul><ul><li>Chalkmark is something we will see later today. </li></ul><ul><li>Lou Capone will give us a demo of UserZoom (30-minute demo). </li></ul>Web IQ
  17. 17. Qualitative Testing Tools and Methods <ul><li>Qualitative usability testing for the web has really matured in the last 30 years. </li></ul><ul><li>Quantitative usability testing tools has really matured in the last 10 years. </li></ul><ul><li>Morae and OvoStudios both offer mature solutions for usability professionals. </li></ul><ul><li>IntuitionHQ is very new to the market—I was just made aware of them this week. </li></ul><ul><li>We will use Morae today. </li></ul>
  18. 18. Three Types of Usability Tests (Qualitative Perspective) Today’s workshop will focus on summative usability testing.
  19. 19. Basic Usability Testing Process <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul><ul><li>Step 2: Conducting Your Usability Test </li></ul><ul><li>Conduct a pilot test </li></ul><ul><li>Facilitate remaining tests </li></ul><ul><li>Step 3: Analyzing and Reporting Your Usability Test </li></ul><ul><li>Compile results </li></ul><ul><li>Make recommendations </li></ul>
  20. 20. A Typical Scene Behind The Glass in a Usability Test
  21. 21. Step 1: Planning Your Usability Test
  22. 22. Basic Usability Testing Process <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul>Time = Money: Most usability tests are less than 90 minutes. Plus, you are taking your customer’s time. Put first things first—what needs testing!!! (Activities) (People) (Context)
  23. 23. A Simplified Version of Activities-People-Context Picture from Stephen Anderson Somebody (person) does something (activity) some where or some how (context). This model is Robert Bailey’s Human Performance Model. Let’s use this model to quickly plan out a test!
  24. 24. Here is the Royal Caribbean International Website
  25. 25. Describe Activities That Might Be Done….. <ul><li>Create a group reservation </li></ul><ul><li>Plan an event (ex: wedding) </li></ul><ul><li>Update profile (ex: Passport) </li></ul><ul><li>Review Sailing Route </li></ul>Picture from Stephen Anderson <ul><li>Make a second payment </li></ul><ul><li>Move to a different ship </li></ul><ul><li>Upgrade to a new cabin </li></ul><ul><li>Buy an excursion deal </li></ul>
  26. 26. Describe People That Might Be Using the Site….. <ul><li>Captain of the Ship </li></ul><ul><li>Group Traveler </li></ul><ul><li>Senior Citizen </li></ul><ul><li>College Student </li></ul>Picture from Stephen Anderson <ul><li>Potential customer </li></ul><ul><li>Travel Agent </li></ul><ul><li>Special Needs Traveler </li></ul><ul><li>Worker (on the Ship) </li></ul>
  27. 27. Describe the Context of Use….. <ul><li>Company website </li></ul><ul><li>IVR system </li></ul><ul><li>Kiosk (on the ship) </li></ul><ul><li>Printout (ex: Itinerary) </li></ul>Picture from Stephen Anderson <ul><li>Social Media </li></ul><ul><li>Email/text messaging </li></ul><ul><li>Voice Mail </li></ul><ul><li>TV, Radio, DVR </li></ul><ul><li>Inside or outside </li></ul><ul><li>Public or private </li></ul><ul><li>Noisy or quiet </li></ul><ul><li>Legal regulations </li></ul><ul><li>Social attitudes </li></ul><ul><li>Company goals </li></ul>
  28. 28. It Is Not Too Complicated….. <ul><li>People = Usability Participants </li></ul><ul><li>Context = Test Scenarios </li></ul><ul><li>Activities = Usability Tasks </li></ul>
  29. 29. Activities: What Concerns Does Your Product Team Have? <ul><li>Ask the project team to list out their overall concerns with their product. Let’s go back to the Royal Caribbean International. </li></ul><ul><li>Overall Concerns: </li></ul><ul><li>Ease of learning </li></ul><ul><li>Ease of use </li></ul><ul><li>Ease of navigation </li></ul><ul><li>Usefulness of Help </li></ul><ul><li>Recovering for errors </li></ul><ul><li>Specific Concerns: </li></ul><ul><li>Booking Group Reservations </li></ul><ul><li>Splitting Group Payments </li></ul><ul><li>Availability of Group Cabins </li></ul>
  30. 30. People: Who are the Target Customers? <ul><li>Ask the project team. They tend to focus on a single user. </li></ul><ul><li>Frame It This Way: </li></ul><ul><li>Who are primary customers? (Define their characteristics.) </li></ul><ul><li>Is there another group? (How are they like 1 st group?) (How are thy different?) </li></ul><ul><li>Is there another group? (How are they like both groups?) (How are they different?) </li></ul>Travel Agent Event Planner
  31. 31. Context: Where and How are People Going to Use It? <ul><li>Ask the team. Usually, the team focuses on the delivery device. They will need help with the other contextual factors. </li></ul><ul><li>Frame It This Way: </li></ul><ul><li>What device(s) is it on? </li></ul><ul><li>Is inside or outside? </li></ul><ul><li>Is it public or private? </li></ul><ul><li>Is it work or leisure-related? </li></ul><ul><li>Is it noisy or quiet? </li></ul><ul><li>Any legal regulations? </li></ul>
  32. 32. It Is Not Too Complicated….. Just Ask!!! <ul><li>People = Usability Participants </li></ul><ul><li>Context = Test Scenarios </li></ul><ul><li>Activities = Usability Tasks </li></ul>
  33. 33. With People-Context-Activities Defined, Data Can Be Collected <ul><li>Performance Data: </li></ul><ul><li>Average Task Completion </li></ul><ul><li>Average Time on Task </li></ul><ul><li># of clicks to Help </li></ul><ul><li># of omitted steps </li></ul><ul><li># of errors made </li></ul><ul><li># of hints </li></ul><ul><li>Subjective Data: </li></ul><ul><li>Prefer Design A or B </li></ul><ul><li>Design Suggestions </li></ul><ul><li>Task Ease Rating </li></ul><ul><li>Satisfaction Rating </li></ul><ul><li>User Comments </li></ul><ul><li>Ranking Features </li></ul>Performance and subjective data are equally valuable. We will come back to data collection in a few minutes.
  34. 34. Basic Usability Testing Process—Recruting <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul>(Activities) (People) (Context)
  35. 35. Number of Participants Debate
  36. 36. Number of Participants Debate (Sources) Jakob Nielsen (circa 1993) Five users will uncover about 85% of the usability issues of a product. Virzi backs up this claim around the same time. Jared Spool (circa 2000) Five users revealed only 35% of the usability issues of a product. This claim is backed up by Schroeder. Laura Faulkner (2003) - Five users uncovered 85% of usability issues on average. - But, the percent ranged from 55-100% (confidence interval). - Ten users uncovered 95% with a lower bound of 82%.
  37. 37. Number of Participants (Brian Sullivan, 2009) <ul><li>General Rules: </li></ul><ul><li>Test with 6 customers (per group) —if time permits, use 8 customers. </li></ul><ul><li>More customers = more reliable results </li></ul><ul><li>More customers = fewer new results </li></ul><ul><li>For larger samples, do multiple rounds of smaller tests (rather than 1 large UT) </li></ul><ul><li>Base Your Number On: </li></ul><ul><li>Importance of project </li></ul><ul><li>Complexity of the product </li></ul><ul><li>Experience of testing team </li></ul><ul><li>Expected variety of user reactions </li></ul>
  38. 38. Recruiting Participants <ul><li>Recruiting: </li></ul><ul><li>In-house sources </li></ul><ul><li>Recruiting agency </li></ul><ul><li>Incentives: </li></ul><ul><li>Gift checks ($100 per session) </li></ul><ul><li>Food or gift cards </li></ul><ul><li>Scheduling: </li></ul><ul><li>Slot times </li></ul><ul><li>Breaks between participants </li></ul><ul><li>Backups, if needed </li></ul><ul><li>Floaters, if needed </li></ul>
  39. 39. Selecting the Right Customers <ul><li>User profiles: </li></ul><ul><li>Potential Customer </li></ul><ul><li>New Customer </li></ul><ul><li>Returning Customer </li></ul><ul><li>Primary or tertiary customers </li></ul><ul><li>Characteristics: </li></ul><ul><li>Job responsibilities (not their title) </li></ul><ul><li>Years experience </li></ul><ul><li>Expertise level (novice, intermediate, expert) </li></ul><ul><li>Specific skills (example: heart surgeons rather MDs or nurses) </li></ul><ul><li>Geographic location </li></ul>Source: David Armano
  40. 40. Your Recruiting is Only As Good As the Screener <ul><li>Steve Krug has said this: </li></ul><ul><li>I respectfully disagree. Because… </li></ul><ul><li>Your test is a waste of time , if you recruit the wrong people. </li></ul><ul><li>Your usability test is dangerous , if you act upon recommendations from a bad recruiting sample. </li></ul><ul><li>You must screen participants. </li></ul>For recruiting, grade on a curve ! Grab a neighbor.
  41. 41. Basic Usability Testing Process—Write Out the Test <ul><li>Step1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul><ul><li>People = Usability Participants </li></ul><ul><li>Context = Test Scenarios </li></ul><ul><li>Activities = Usability Tasks </li></ul>
  42. 42. Scenarios and Tasks <ul><li>Some experts distinguish between tasks and scenarios: </li></ul><ul><li>Scenarios: </li></ul><ul><li>Provide the participant with motivation and context </li></ul><ul><li>Sets the stage or tells a story </li></ul><ul><li>Increases understanding </li></ul><ul><li>Makes the situation more realistic </li></ul><ul><li>Includes tasks </li></ul><ul><li>Tasks: </li></ul><ul><li>Quick and to the point </li></ul><ul><li>Do not need context to increase understanding </li></ul>
  43. 43. Example of Scenario with Tasks <ul><li>Scenario #1: </li></ul><ul><li>You want to book a sailing on Royal Caribbean International for next June with your church group. The group is called “Saint Francis Summer 2010”. The group is selling out fast, so you want to book a cabin, which is close to an elevator because your leg hurts from a recent injury. </li></ul><ul><li>Open your browser </li></ul><ul><li>Click the link labeled “Royal Caribbean” </li></ul><ul><li>Tell me the available cabins in the “Saint Francis Summer 2010” group </li></ul><ul><li>Tell me a cabin number closest to an elevator </li></ul><ul><li>Book the cabin the best suits your needs </li></ul>
  44. 44. Practice Writing Scenarios and Tasks <ul><li>We have two teams today. --Team Cool --Team Ninja Assassins </li></ul><ul><li>Pair up with someone. </li></ul><ul><li>Write a scenario with tasks. </li></ul><ul><li>We will pass out papers of some sample sites. </li></ul><ul><li>You will use your script later today to run some tests. </li></ul>
  45. 45. Step 2: Conducting Your Test
  46. 46. Basic Usability Process—Step 2 <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul><ul><li>Step 2: Conducting Your Usability Test </li></ul><ul><li>Conduct a pilot test </li></ul><ul><li>Facilitate remaining tests </li></ul>
  47. 47. Don’t Rely on Luck!!!
  48. 48. You Do Not Need a Usability Lab to Conduct a Test
  49. 49. Test Side-by-Side in a Conference Room
  50. 50. Test Wherever Your Customers Might Be I have done usability testing at Starbuck’s and a User Conference.
  51. 51. Remote Usability Testing Easier Than Ever
  52. 52. Morae is Portable, Affordable, and ………………Usable
  53. 53. Shhhhh….you don’t need to record, either
  54. 54. Basic Usability Process—Step 2 <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul><ul><li>Step 2: Conducting Your Usability Test </li></ul><ul><li>Conduct a pilot test </li></ul><ul><li>Facilitate remaining tests </li></ul>
  55. 55. Conduct a Pilot Test <ul><li>Dry run of a complete usability test </li></ul><ul><li>Include all test material </li></ul><ul><li>Focus on the test scenarios, not findings </li></ul><ul><li>At least, one participant should be a pilot </li></ul><ul><li>Conduct before test begins (one day prior) </li></ul><ul><li>Make changes, if needed </li></ul><ul><li>May need to test, again </li></ul>
  56. 56. Basic Usability Process—Step 2 <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul><ul><li>Step 2: Conducting Your Usability Test </li></ul><ul><li>Conduct a pilot test </li></ul><ul><li>Facilitate remaining tests </li></ul>
  57. 57. Again, Adopt a Workable Schedule
  58. 58. Tips for Being a Good Moderator <ul><li>Know Your UCD Basics and Design Guidelines </li></ul><ul><li>Be a Quick Learner </li></ul><ul><li>Set an Instant Rapport with Participants </li></ul><ul><li>Have an Excellent Memory </li></ul><ul><li>Be a Good Listener (Use Active Listening) </li></ul><ul><li>Don’t Stress Out About Ambiguity </li></ul><ul><li>Be Very Flexible </li></ul><ul><li>Have a Long Attention Span </li></ul><ul><li>Show Empathy (ie Be a People Person) </li></ul><ul><li>Thing “Big Picture” </li></ul><ul><li>Be a Good Communicator </li></ul><ul><li>Be a Good Organizer and Coordinator </li></ul>
  59. 59. Tips for Getting the Most from Participants <ul><li>Start with an Easy Task To Build Confidence </li></ul><ul><li>Choose the Right Format (remote, on-site UT) </li></ul><ul><li>Sit Beside the Person Not Behind the Glass </li></ul><ul><li>Use “Think-Out-Loud” Protocol </li></ul><ul><li>Give Them Time to Think It Through </li></ul><ul><li>Offer Appropriate Encouragement </li></ul><ul><li>Lead Participants, Don’t Answer Question (Being an Enabler) </li></ul><ul><li>Don’t Act Knowledgeable (Treat Them As the Experts) </li></ul><ul><li>Don’t Get Too Involved in Data Collection </li></ul><ul><li>Don’t Jump to Conclusions </li></ul><ul><li>Don’t Solve Their Problems Immediately (ie act as a Help Desk) </li></ul>
  60. 60. Team Cool Starts With Wizard of Oz Style <ul><li>Wizard of Oz Style Testing: </li></ul><ul><li>Moderator </li></ul><ul><li>Tester </li></ul><ul><li>Wizard (Person moves screen shots) </li></ul>@suredoc (Keith Anderson) is your guide today. NOTE: Keith is NOT the WWW – Wicked Witch of the West.
  61. 61. Rules for the Merry Old Land of Oz….. <ul><li>Here are The Rules: </li></ul><ul><li>Get into Groups of three. </li></ul><ul><li>Each person picks a role. </li></ul><ul><li>Wizard does not talk. </li></ul><ul><li>Moderator analyzes Tester </li></ul><ul><li>Tester talks through the task. </li></ul><ul><li>Complete the Test. </li></ul><ul><li>Switch Roles. </li></ul><ul><li>Find two new people. </li></ul><ul><li>Repeat Steps 1-6. </li></ul><ul><li>Compile results via K-J Method (well do that later, though). </li></ul>
  62. 62. Team Ninja Assassins Will Be in Usability Lab <ul><li>These are the Test Stations: </li></ul><ul><li>Two Waiting Participants </li></ul><ul><li>One Tester in Test Cell </li></ul><ul><li>Moderator </li></ul><ul><li>Test Cell Observer </li></ul><ul><li>Camera Operator </li></ul><ul><li>TV Operator </li></ul><ul><li>Remote Viewer Logger </li></ul><ul><li>Remote Viewer Manager </li></ul><ul><li>Results Compiler (2 people) </li></ul><ul><li>Two Observers </li></ul><ul><li>Rotate between each station </li></ul>We’ll test four different sites today.
  63. 63. After 1.5 Hours, We Switch Locations <ul><li>The Teams Switch: </li></ul><ul><li>Team Cool in Labs </li></ul><ul><li>Team Ninja Assassins (Wizard of Oz) </li></ul>
  64. 64. Step 3: Analyzing and Reporting
  65. 65. Basic Usability Process—Home Stretch Now!!! <ul><li>Step 1: Planning Your Usability Test </li></ul><ul><li>Define what you are testing </li></ul><ul><li>Define which customers you are testing </li></ul><ul><li>Define what tasks you are testing </li></ul><ul><li>Recruit your customers </li></ul><ul><li>Write your usability scenarios and tasks </li></ul><ul><li>Step 2: Conducting Your Usability Test </li></ul><ul><li>Conduct a pilot test </li></ul><ul><li>Facilitate remaining tests </li></ul><ul><li>Step 3: Analyzing and Reporting Your Usability Test </li></ul><ul><li>Compile results </li></ul><ul><li>Make recommendations </li></ul>
  66. 66. Dilbert’s Boss Solves Your Customer Problem
  67. 67. 8 Steps to Analyze and Report Your Findings <ul><li>Complete Your Notes </li></ul><ul><li>Summarize Your Findings </li></ul><ul><li>Compile and Tabulate Data </li></ul><ul><li>Analyze Your Findings </li></ul><ul><li>Prioritize Findings by Severity Level </li></ul><ul><li>Create Recommendations </li></ul><ul><li>Develop a Final Document or Presentation </li></ul><ul><li>Follow-up </li></ul>
  68. 68. Step 1—Complete Your Notes <ul><li>After each session, fill in any gaps. </li></ul><ul><li>Basically, complete the short-hand used in the UT. </li></ul><ul><li>Mark any areas worth noting. </li></ul><ul><li>Use a whiteboard! It is really much easier, in the end. </li></ul>
  69. 69. Step 2—Summarize Findings <ul><li>After each session, create a list of the major issues. </li></ul><ul><li>Organize by Issue and Participant. </li></ul><ul><li>Do Not Start Solution Finding Here! </li></ul><ul><li>Below is an Example of How to Summarize: </li></ul>
  70. 70. Step 3—Compile and Tabulate Data <ul><li>Spreadsheet work best here </li></ul><ul><li>Do not work on solutions, yet </li></ul><ul><li>Try to find the real problem </li></ul>
  71. 71. Step 4—Analyze Findings <ul><li>Which tasks were simple? </li></ul><ul><li>Which tasks were problematic? </li></ul><ul><li>Which participants had issues? (Look at it by group.) </li></ul><ul><li>Speculate on solutions. </li></ul><ul><li>Use Balsamiq to sketch out recommendations. </li></ul>Sketched in Balsamiq in less than 5 minutes.
  72. 72. Step 5—Prioritize Findings <ul><li>Severity 1 = Showstopper. Unless corrected all users will essentially have a problem. </li></ul><ul><li>Severity 2 = High Impact. Significant impact to usability or performance. High customer annoyance and frustration. </li></ul><ul><li>Severity 3 = Medium Impact. Some users experience degraded performance or frustration. Help Desk calls possible. </li></ul><ul><li>Severity 4 = Low Impact. Probably will not have a significant impact to user performance. Might be cosmetic in nature. </li></ul>
  73. 73. Step 6—Create Recommendations <ul><li>Start with Positive Findings </li></ul><ul><li>Remind the team of people-activities-context </li></ul><ul><li>Be a problem solver </li></ul><ul><li>Present Issues by Severity Level (helps to prioritize) </li></ul><ul><li>It great to do a MOCKUP here </li></ul><ul><li>Tie recommendation back to USER PERSONAS, too </li></ul>
  74. 74. Nobody reads a “BIG, HONKING” report.
  75. 75. Step 7—Create Final Document or Presentation <ul><li>Consider the audience </li></ul><ul><li>Some teams just want an Excel spreadsheet </li></ul><ul><li>Provide enough detail (executive summary v. full report) </li></ul><ul><li>Create a distribution list for key stakeholders </li></ul><ul><li>Purpose (identify issues, prioritize them, make recommendations) </li></ul>
  76. 76. Step 8—Follow-up <ul><li>Always Provide Usability Next Steps </li></ul><ul><li>Usability Needs to Be Part of the Development/Design Cycle </li></ul><ul><li>Invite People to Other Studies </li></ul><ul><li>Create an Action Plan </li></ul>
  77. 77. Let’s Compile and Analyze Our Results
  78. 78. Conduct K-J Analysis of Wizard of Oz <ul><li>List all observations on stickies </li></ul><ul><li>Post them on a wall </li></ul><ul><li>Cluster into groups </li></ul><ul><li>Name the group </li></ul><ul><li>Vote on each group </li></ul><ul><li>If possible, make recommendations </li></ul>
  79. 79. Conduct On Board Analysis of Lab Testing <ul><li>Write issues on board after each participant (positive and negative) </li></ul><ul><li>Place checkmark when the issue is found by another user </li></ul><ul><li>Rank issues at the end of the test </li></ul><ul><li>If desired, we can think about solutions. </li></ul>
  80. 80. Wizard of Oz Testing on Chalkmark
  81. 81. Lab Testing of Four Sites (Sprouts)
  82. 82. Lab Testing of Four Sites (Consumer Search)
  83. 83. Lab Testing of Four Sites (Broccoli.com)
  84. 84. Lab Testing of Four Sites (Casa Ma ň ana)
  85. 85. Question & Answer
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×