“Testing with one user is 100% betterthan testing with none.”- Steve Krug
What is usability testing?Design + Technology + PsychologyImprove the user experience; competitive edgeObserving real users doing real tasks
Evaluating usability“Usability testing is the second most used evaluationmethod and the method that has the most impact onmaking products better.”-Rosebaum, et al (2000)
Types of usability studies• Formative vs. Summative• Moderated vs. Unmoderated• In Person vs. Remote
When can you uncover issues?• Idea/ concept stage• Design is in prototype stage• Design is ready to go live• Design is liveAt any point during the lifecycle of a product.
The preparation for anunmoderated usability study
General Construct1. Goals2. Prepare3. Recruit4. Set up5. Analyze6. Report
Step 1: Determining GoalsTalk to stakeholders to determine thebusiness and usability goals of the studyIt is necessary to collaborate withstakeholders to produce a high qualitystudy
Step 2: Answering the questions• Where is the product in terms of development cycle• What tasks will be covered?• Duration of the session?• Who should be your participants?• What should be your methodology?• How many participants should you have?• How should you compensate your participants?
Step 2: Preparing, cont. The more severe the problem, the more likely it is to be uncovered withinthe ﬁrst few sessions.“Approximately 80% of usability problems identiﬁed would have foundafter only ﬁve subjects.”Number of participants 3 – 65% / 5 – 80% / 9 - 95%“Virtually all of the problems judged high in severity were discovered afterrunning 5 subjects, whereas only 55% of the problems judged low inseverity were found.”Source: Virzi study
The unmoderated studyHe works for University of Minnesotaand is considering re-designing thelibrary website. He is somewhat awareabout usability but has not had the timeto dive into it. He wants to make surethat the students can perform the taskseasily and quickly (like ﬁnding a book,making the reservations for a meetingroom). Unfortunately, he has little timeand little budget. Tip: 15 minutes | 5 participants | 4 tasks | 4 post session questions
Writing scenarios and tasksHow to choose the right user and tasks :• Tasks that are typical or frequent/ critical tooperation• New or existing users
Writing scenarios and tasksChoose your words carefully!• Do not use words that are in the product• Write in plain language• What order should tasks be done?• How many tasks should you have?• What are the dependencies between the tasks?• Can I compare designs?• How can I give moderated assists?
Writing unbiased scenarios andtasksFollowing are some of the biased questions that you should avoid: Rate the degree to which the software you just used has no performanceproblemAre you interested in obtaining another degree in the next 10 years?About how many times in the last year did you use online help?____Number of timesHow often did you use Product X during the last month?(Check one answer)_____ Never_____ Rarely_____ Occasionally_____ Regularly
Finding the right participantsTip: Running a pilot | Getting refund back
Step 5: AnalysisWhat to note?• Positive• Issues• Bugs• Observations• Quotations and annotated screenshotsWhen to note?• Revisit the recordings and make notes • During the session• Have a scribe, if possible
Prioritize your ﬁndings Categorize the positive/ negative/ observationSeverity low/ medium/ highFrequency rare/ common/ very commonScope local/ global
Step 6: Reporting your ﬁndings • Purpose• Why did we test?• How did we test?• Whom did we test?• Executive Summary• Positives• Major Issues• Detailed ﬁndings• Recommendations• Observations• Next Steps
The executive summaryThere were no Urgent problems uncovered. Overall, the response was positive and participants could ﬁnd their waytowards achieving the tasks quickly.High severity ﬁndingsBulleted listMedium severity ﬁndingsBulleted listTask scoresTasks are rated from 1 (low) to 5 (high) by the moderator for eﬃciency and eﬀectiveness, based on what the participantdoes and says. Anything 3.5 or high is acceptable, but 4 or higher is the goal!Average overall score by task:[Task 1] = 3.85[Task 2] = 3.01[Task 3] = 3.55[Task 4] = 3.66When asked what their overall experience was, given everything they’d seen in the session, the average score providedby participants was 3.6.
Other unmoderated usability studytools to consider• Chalkmark• Loop11• Open Hallway• Usabilla• UserZoomhttp://groups.drupal.org/node/166059"Handbook: