Your SlideShare is downloading. ×

Rapid usability testing


Published on

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 2. Lisa Rex @lisarex#frontendunited
  • 3. AND YOU ARE?
  • 4. AGENDAThe basicsPreparation for a studyConducting a usability studyAnalysis & debrief
  • 7. WHAT DOES YOUR INTERFACE SAY?We hope you have excellent eyesight. We hope you have excellent eyesight. We hope you have excellent eyesight.
  • 8. Yeah, hmm, we don’t know what that error message is either.
  • 9. Sorry we couldnt be bothered to test our site actually makes sense to anyone outside our organization.
  • 10. A
  • 11. B
  • 12. C
  • 13. “Testing with one user is 100% better than testing with none.” - Steve Krug
  • 14. WHAT IS RAPID USABILITY Stakeholder buy-in. Observe real users doing real tasks. Quick turnaround. No. Reports. Ever.
  • 16. DO NOT BE ALARMED Preparation will take longer than steps 2 & 3 combined
  • 17. PREP: GOALS Talk to stakeholders:“What are the business goals of the study?” “What do we need to learn?” “When do we need it?” “What’s our budget?”
  • 18. PREP: WHAT TO TESTIdeaInformation architecturePrototypeStaging siteProduction site You can test at any point during the project lifecycle!
  • 19. PREP: METHODOLOGYModerated vs unmoderatedOnline tool vs own setupRemote with screenshare vs in personOne-on-one vs many-to-one
  • 20. PREP: METHODOLOGYModerated vs unmoderatedOnline tool vs own setupRemote with screenshare vs in personOne-on-one vs many-to-one
  • 21. PREP: METHODOLOGYModerated vs unmoderatedOnline tool vs own setupRemote with screenshare vs in person
  • 22. PREP: WHO TO TEST Type of skillset Expertise New users vs. current users 5 people per study is ideal
  • 23. PREP: TESTING ENVIRONMENTHow will they access the site?Content and user account setup is needed?Do you need to set up separate sites for each? Or re-setbetween sessions? (hint: use Drush)
  • 24. PREP: STUDY OUTLINEOverviewParticipant briefingPre-session questionsScenarioLogin detailsTasks and questionsPost-session questions
  • 25. PREP: WRITING TASKSDo not use words that are in the UIPhrase things in the way people think and in plain languageWrite in cues for moderator (e.g. “Begin recording” or“Observe how they ____”)Dependencies between the tasks & task orderWrite moderated assistsAsk them to rate their experience
  • 27. PREP: THE SCREENERDevelop a screener to weedout the ‘wrong’ participantFind out when they’reavailable (use Doodle?)Set expectations withparticipants (when, howlong, compensation)
  • 28. PREP: RECRUITINGWhen: 0-2 days before you want to talk to them, ifmoderatedHow: social networks, online free ads, groups ororganizations, meetups, even family and friends“General” participant profile: can use a bar or cafeSend them to your screener, e.g.
  • 29. PREP: SCHEDULINGConfirm timeSend calendar invite(s)Remind them of expectationsThank them
  • 30. PREP: OBSERVATION FORMPriority Participants who had the issue (frequency)Issue summary (quotes!) Other things: totals,Link to screenshot estimated fix date, commentsTask name
  • 31. PREP: OBSERVERS 1 = catastrophic 2 = serious 3= important 4 = minor 5 = observation 6 = positive 7.x = resolved
  • 33. MODERATED STUDY She’s working on a site that is highly focused on a specific type of user: the Drupal developer. She needs feedback from Drupal developers... anyone else won’t provide the knowledge that is needed to improve the site.15-60 minute session | 5 participants | 3-5 tasks | 3-5 pre- and post-session questions
  • 34. CONDUCTING A MODERATED STUDYUse your study outline prepared scriptParticipant needs come above everything elseGet good information from them Gonna talk about this in detail at 13:45!
  • 36. THE RESULTSAnalyze and add to your observation spreadsheet
  • 37. TREEJACK STUDY He’s working on a Drupal distribution that is being redeveloped for Drupal 7. The previous distribution has some issues, so will users understand the new information architecture? He needs feedback from both new and current users.10 minute session | 40-400 (120 optimum) participants | 3-10 tasks | 3-5 pre- and post-session questions
  • 38. TREEJACKWhen? Anytime...tree is isolated• Where would they go to do ___?• Can they find a specific piece of content?• Do navigation labels make sense?• Are my subsections well-organized?• How does Tree A compare to Tree B?
  • 39. BUILDING THE TREEBuild the structure in a spreadsheet, one column for eachlevelInclude sample contentInclude controls within the site, if that’s what you want totest (e.g. “add comment”)Only the leaves of the tree are selectable as answers,branches are not.
  • 41. TREEJACK SETUPThink about what you want to validate, then write the tasksDecide the right answers; used to calculate the success ofyour tasksCustomize pre-session and post-session questionmessage options
  • 42. TREEJACK PROTIPSRun a few moderated trials with people unfamiliar with yoursite, to catch the WTFs ....Set your ‘right’ answers at the very end, as you will blowthem away if you rebuild the treeConsider hacky chunking of optionsDon’t allow skip tasks, but include [None of these]
  • 43. THE RESULTSWhat you’ll get is several graphs & complex visuals torepresent the paths people took, the success of each task, etc Analyze and add to your observation spreadsheet
  • 44. USERTESTING.COM STUDY He works at a university library. He isn’t a usability expert but wants to make sure that the students can perform their primary tasks easily and quickly (like finding a book, making the reservations for a meeting room). Unfortunately, he has little time and little budget. 15 minute session | 5-8 participants | 4-5 tasks | 3-4 post-session questions
  • 45. USERTESTING.COM SETUPReally flexible: show them a URL to live site or prototypeGive them detailed instructions at each stepWrite hints, just in case they missed itPre-session and post-session question and messageoptions
  • 46. Tip: Run a pilot | Ask for refund
  • 47. THE RESULTSWhat you’ll get is recordings of the tester’s browser and voice, plus the answers they fill in at the end. Analyze and add to your observation spreadsheet
  • 48. THE ANALYSIS & DEBRIEF Positive comments Have a scribe, if no other stakeholders are presentIssues/Problems/Bugs and the session isn’t recorded Observations If recording, revisit the Quotes tapes and make notesAnnotated screenshotsTake notes during the session
  • 49. ANALYSISTeam (or just you) should review observations andcondense, if possibleCategorize the positive negative/observationsSeverity: Low/Medium/ High or 1-5Frequency: Rare/ Common / Very CommonScope: Local/ Global
  • 50. EXECUTIVE SUMMARYThere were no Urgent problems uncovered. Overall, the response was positive and participants could find their way towards achieving the tasks quickly. Task scores Tasks are rated from 1 (low) to 5 (high) by the moderator for efficiency and effectiveness, based on what the participant does and says. Anything 3.5 or high is acceptable, but 4 or higher is the goal! Average overall score by task: [Task 1] = 3.85 [Task 2] = 3.01 [Task 3] = 3.55 [Task 4] = 3.66When asked what their overall experience was, given everything they’d seen in the session, the average score provided by participants was 3.6.
  • 51. GOING OVER RESULTSGet all stakeholders togetherWithin 2 days of final studyInformal discussionMake recommendations
  • 52. NEEDS MOAR RAPID• Test, debrief and present • Inventtasks and questions findings within 3 days based on participants thoughts = less prep, less• Recruit and test at an event analysis e.g. from the booth or nabbing people in the • Have feedback incorporated hallway into current or next sprint• Getdecision-makers to observe sessions
  • 53. TOOLS AND ARTICLESClickHeat, ClickTale Usertesting.comCrazyEgg UserVueEthnio VerifyappFive Second TestFeedback Army Article: Extremely Rapid Usability TestingInvision App upa_publications/jus/2009may/Loop11 pawson1.htmlMechanical TurkMorae Presentation: Rapid turnaround usability testing: not just a pipe dream http://Open Hallway (OptimalSort, turnaround-usability-testing-not-just-a-Treejack, ChalkMark) pipe-dreamUsabilla
  • 54. DRUPAL USABILITY DIRECTORY Group for Drupal Creation UX Study Plan 8 User Experience 7 User Experience Project
  • 55. Wanna volunteer in the workshop? Needed: 3 moderators 3 participants several observers for 3 short studies on
  • 56. Q&A
  • 57. Thank you! Lisa Rex @lisarex#frontendunited
  • 59. Lisa Rex @lisarex#frontendunited
  • 60. MODERATINGUse a study guideExplain the purpose of the study and what to expectGet consent to recordEncourage thinking out loudAssure them it’s a test of the software, not their skills orabilitiesExplain your role as an unbiased observer
  • 61. MODERATINGEstablish rapport, but be professionalEnsure their physical and emotional comfortBe observantBe greedy for informationWatch the timeGive assists, but only if neededThank them
  • 62. STUDY