1. How to Easily Uncover UsabilityIssuesDharmesh MistryUX Researcher, AcquiaJune 20, 2013Drupal.org id/ Twitter: @dcmistry
2. Welcome Dharmesh MistryUX Researcher, Acquia•  UX for Acquia, Drupal•  UXPA Boston, Board of Director•  Co-organizer: Boston Design Jam, Design 4Drupal, UX/UI Summit, UXPA Boston AnnualConference•  Speaker, Blogger, Trainer, Evangelist•  Bentley University•  Desserts and Photography
3. Agenda•  Why usability matters?•  A few basics•  Preparation for a study•  Unmoderated usability study•  Analyzing observations•  Reporting your ﬁndings•  Q & A
4. Why usability matters?
5. “Testing with one user is 100% betterthan testing with none.”- Steve Krug
6. What is usability testing?Design + Technology + PsychologyImprove the user experience; competitive edgeObserving real users doing real tasks
7. Evaluating usability“Usability testing is the second most used evaluationmethod and the method that has the most impact onmaking products better.”-Rosebaum, et al (2000)
8. Types of usability studies•  Formative vs. Summative•  Moderated vs. Unmoderated•  In Person vs. Remote
9. When can you uncover issues?•  Idea/ concept stage•  Design is in prototype stage•  Design is ready to go live•  Design is liveAt any point during the lifecycle of a product.
10. The preparation for anunmoderated usability study
11. General Construct1.  Goals2.  Prepare3.  Recruit4.  Set up5.  Analyze6.  Report
12. Step 1: Determining GoalsTalk to stakeholders to determine thebusiness and usability goals of the studyIt is necessary to collaborate withstakeholders to produce a high qualitystudy
13. Step 2: Answering the questions•  Where is the product in terms of development cycle•  What tasks will be covered?•  Duration of the session?•  Who should be your participants?•  What should be your methodology?•  How many participants should you have?•  How should you compensate your participants?
15. Step 2: Preparing, cont. The more severe the problem, the more likely it is to be uncovered withinthe ﬁrst few sessions.“Approximately 80% of usability problems identiﬁed would have foundafter only ﬁve subjects.”Number of participants 3 – 65% / 5 – 80% / 9 - 95%“Virtually all of the problems judged high in severity were discovered afterrunning 5 subjects, whereas only 55% of the problems judged low inseverity were found.”Source: Virzi study
16. Conducting an
unmoderated usability study
17. The unmoderated studyHe works for University of Minnesotaand is considering re-designing thelibrary website. He is somewhat awareabout usability but has not had the timeto dive into it. He wants to make surethat the students can perform the taskseasily and quickly (like ﬁnding a book,making the reservations for a meetingroom). Unfortunately, he has little timeand little budget. Tip: 15 minutes | 5 participants | 4 tasks | 4 post session questions
18. The unmoderated study
19. URL and Scenario
21. Post Session
22. Writing scenarios and tasksHow to choose the right user and tasks :•  Tasks that are typical or frequent/ critical tooperation•  New or existing users
23. Writing scenarios and tasksChoose your words carefully!•  Do not use words that are in the product•  Write in plain language•  What order should tasks be done?•  How many tasks should you have?•  What are the dependencies between the tasks?•  Can I compare designs?•  How can I give moderated assists?
24. Writing unbiased scenarios andtasksFollowing are some of the biased questions that you should avoid: Rate the degree to which the software you just used has no performanceproblemAre you interested in obtaining another degree in the next 10 years?About how many times in the last year did you use online help?____Number of timesHow often did you use Product X during the last month?(Check one answer)_____ Never_____ Rarely_____ Occasionally_____ Regularly
25. Finding the right participantsTip: Running a pilot | Getting refund back
26. Analyzing your observations
27. Video recordings
28. Step 5: AnalysisWhat to note?•  Positive•  Issues•  Bugs•  Observations•  Quotations and annotated screenshotsWhen to note?•  Revisit the recordings and make notes •  During the session•  Have a scribe, if possible
29. Prioritize your ﬁndings Categorize the positive/ negative/ observationSeverity low/ medium/ highFrequency rare/ common/ very commonScope local/ global
30. Step 6: Reporting your ﬁndings •  Purpose•  Why did we test?•  How did we test?•  Whom did we test?•  Executive Summary•  Positives•  Major Issues•  Detailed ﬁndings•  Recommendations•  Observations•  Next Steps
31. The executive summaryThere were no Urgent problems uncovered. Overall, the response was positive and participants could ﬁnd their waytowards achieving the tasks quickly.High severity ﬁndingsBulleted listMedium severity ﬁndingsBulleted listTask scoresTasks are rated from 1 (low) to 5 (high) by the moderator for eﬃciency and eﬀectiveness, based on what the participantdoes and says. Anything 3.5 or high is acceptable, but 4 or higher is the goal!Average overall score by task:[Task 1] = 3.85[Task 2] = 3.01[Task 3] = 3.55[Task 4] = 3.66When asked what their overall experience was, given everything they’d seen in the session, the average score providedby participants was 3.6.
32. Other unmoderated usability studytools to consider•  Chalkmark•  Loop11•  Open Hallway•  Usabilla•  UserZoomhttp://groups.drupal.org/node/166059"Handbook: