Downloaded 21 times
Uploaded byCarol Smith
Easy & Effective Usability Testing at CodeMash 2012
The document presents insights on effective usability testing, emphasizing the importance of understanding user needs to prevent building unwanted products. It outlines various methods and practical tips for conducting usability tests, including participant recruitment, testing environments, and measurement criteria for success. The document advocates for iterative testing and analysis to enhance user satisfaction and improve product design continuously.
More Related Content
Similar to Easy & Effective Usability Testing at CodeMash 2012
Easy & Effective Usability Testing at CodeMash 2012
- 1. CodeMash 2012 January 12, 2012 EASY & EFFECTIVE USABILITY TESTING PRESENTED BY CAROL SMITH @CAROLOGIC
- 2. "The biggest waste ofall is building something no one wants" @ericries #LeanStartupMI via @MelBugai – May 19, 2011
- 3. FIND OUT WHATYOUR USERS REALLY NEED http://www.flickr.com/photos/ableman/534155207/sizes/o/in/photostream/ Page 3 http://www.flickr.com/photos/ableman/
- 4. AVOID MISTAKES http://www.flickr.com/photos/dwulff/12256075/sizes/m/in/photostream/ Page 4 http://www.flickr.com/photos/dwulff/
- 5. MINIMIZE COST &EFFORT Page 5
- 6. SAVE TIME &MONEY Page 6
- 7. GET HAPPY, SATISFIEDUSERS http://www.flickr.com/photos/cogdog/2194678510/sizes/o/in/photostream/ Page 7 http://www.flickr.com/photos/cogdog/
- 8. Create a great, usable,accessible, and relevant experience
- 9. WHAT IS USABILITY TESTING?
- 10. USABILITY TESTING •Real users doing real tasks •Not guided •Using prototypes or live products •Observed in the field Steve Krug - http://www.sensible.com/rsme.html Page 10
- 11. L O CAT I O N I S N ’ T I M P O R TA N T •Anywhere •Any Stage of dev •Anytime •Realistic environment http://www.flickr.com/photos/nzdave/491411546/sizes/o/in/photostream/ Page 11 http://www.flickr.com/photos/nzdave/
- 12. PA R TI C I PA N T S Expert Led Required • Heuristic reviews • Usability Testing • Walk-through • RITE Testing • Accessibility review • Card Sorting Garrett Goldfield . Fast and Cheap Usability Methods: Using Discount Usability Techniques to Drive Page 12 Design on Time and Under Budget. http://www.nngroup.com/events/tutorials/guerilla_usability.html
- 13. WHY DO USABILITYTESTING? Page 13
- 14.
- 15. REGULAR TESTING (Yes, this is an old idea; a great one!) Page 15
- 16. WHY REGULAR? •Team becomes: • accustomed to steady stream of qualitative insight • insight ensures quick decisions • …line up with business and user goals Page 16 Adapted from Jeff Gothelf - http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/
- 17. “Teams should stretch toget work into that day’s test and use the cadence to drive productivity.” Jeff Gothelf - http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/
- 18. BRING IT ON! •Recruitment process •Small focused tests •Regular timing or per Sprint •Same day mid-week (not Monday or Friday) Page 18
- 19. W H ATT O T E S T ? •Select 2 weeks prior to study •“Focus ruthlessly on a small number of the most important problems” – Steve Krug • Work in Progress • Websites, mobile, products, services • Prototypes: Paper to hi-fi • Concepts, rough ideas, brainstorming • Competing designs (A/B testing) • Comparative studies across market • User research Page 19
- 20. U S AB I L I T Y T E S T I N G D AY •Make team aware •Invite everyone • Watch remotely • Recurring meeting invites for stakeholders Page 20
- 21.
- 22. MEASURE ABILITY TOACHIEVE GOALS •Effectiveness •Efficiency •Learnability •Satisfaction •More… Page 22
- 23. MEASUREMENTS •Success •Time on task (not generally recommended) •% of tasks completed/not completed •Number of steps to accomplish task •Learning •Number of errors •Number of times help consulted •Satisfaction Page 23
- 24. MAKE IT REPEATABLE: PLAN AHEAD
- 25. PLANNING IS TIGHT •UX • Plan study • Space and equipment • Identify work Page 25
- 26. PRE-BOOK YOUR ROOMS •Test & Observation Rooms •Any location will do: • Conference rooms • Offices • Quiet corner of cafeteria • Remote •Time for facilitation, breaks, post-meeting • No more than 5 per day http://creativecommons.org/licenses/by-sa/2.0/ http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/ Page 26 http://www.flickr.com/photos/raphaelquinet/
- 27. RECRUITMENT •Who are your primary users? • Will they be difficult to recruit? • How expensive is their time? Page 27
- 28. C R EAT E A S C R E E N E R Guide that helps determine who will participate. •Get details: • Education • Computer/Internet use and expertise • Knowledge of topic •Get them talking • Clear Communicator • Able to express themselves verbally •People who pass the screener should closely match your user group definition Page 28
- 29. RULES OF THUMB Create a Budget •Recruiting: ~$250 per participant (varies) •Compensation: Starts at $50 - $100 and up • Internally - use swag • “Big fans” – cheaper alternatives ($15 Starbucks card) Page 29
- 30. HONORARIUMS •Make it easy – buy ahead • Gift cards from retail stores (Target, Starbucks) •More complex • Amazon Gift Cards • Product credits •Do not recommend Visa gift cards or similar Page 30
- 31. HIRE A RECRUITER •Allows you to focus. •Good recruiters: • find right participants • give regular updates • take care of directions, confirmations, incentives, etc. Page 31
- 32. DO IT YOURSELF •Internal resource who REALLY understands who you need •Create a panel (large recruits, less often) • Internal customer lists, user groups • Schedule as needed • Use cautiously •Final scheduling ALWAYS by phone • Ask questions that force them to talk • Don’t recruit non-talkers Page 32
- 33. USE SITE TORECRUIT • Pop up on your site: • http://ethnio.com Page 33
- 34. INCLUDE PEOPLE WITHDISABILITIES (PWD) “We are all only temporarily able-bodied. Accessibility is good for us all.” •Provide the Spirit of the law (Section 508, WCAG 2.0) Page 34 -@mollydotcom at #stirtrek 2011 via @carologic May 6, 2011
- 35. N U MB E R O F PA R T I C I PA N T S Controversy Abounds •As many as possible (rarely statistically significant) •Usability Testing Research (in 1990’s) • 5 from distinct sub-group of the user population will yield 80% of the findings (Nielsen, Virzi, Lewis) • Assumes expert has reviewed and found obvious issues •Recommend: • Early tests with 8 – 12 participants per user group • Iterative testing (3 per day, iterate, 3 new participants) Page 35 Barnum, Carol M. (Jan. 2003). What’s in a Number? STC Usability SIG Newsletter, Usability Interface. http://www.stcsig.org/usability/newsletter/0301-number.html Retrieved: 20080323
- 36. F O RM A L L A B A P P R O P R I AT E ? •Your team • Need them to show up • Need to be able to schedule as needed •Recommend laptop/portable lab • Can be semi-permanent • Ready when you are (minimal setup) Page 36
- 37. C R EAT E R E U S A B L E T E M P L AT E S •Screener •Scripts/Guides • Tasks (same or alternating) • Pre and Post questions •Consent Forms •Data Collection format Page 37
- 38.
- 39. REMOTE TESTING OPTIONS Moderated Un-Moderated • Researcher is present • No Researcher during study • Same effort as in-person • Minimal effort • Limited number of participants • High number of participants per per day (3-5) day • Immediate feedback • Test data compiled usually • Get to the “Why?” by asking within 2 days questions • Reduced or no ability to ask questions Page 39
- 40. R E MO T E - M O D E R AT E D •Lab setup • Robust computer • Screen sharing software •Participants camera (optional) •Internet on both ends • Speedy • Separate from phone line Page 40
- 41. R E MO T E – U N - M O D E R AT E D •No “Lab” needed – online software •Participants camera (optional) •Participant’s internet only •Recommend to complement Moderated Testing Page 41
- 42. BENEFITS OF REMOTE •No travel •Easy to set up •No special room required (advised!) •Can be conducted from almost anywhere •See their computer environment •Drawbacks: • No F2F • Missed interaction • Technology will fail Page 42
- 43. S O FT WA R E Un-moderated Moderated • Userlytics • GoToMeeting (screen • Loop 11 sharing) • UserZoom • Skype • ChalkMark • Morae • Silverback Page 43
- 44.
- 45. N O TE TA K I N G Paper? Computer? • Printed Script • People are less bothered • Quiet by typing now • Analysis may take longer • Can you pay attention? • Great for remote • Quicker Analysis Page 45
- 46. Q U IC K E R – O N B I G PA P E R Issue P1 P2 P3 Search Used Yes No No Widget 1 Used N/A Used – unsure about Task 1 Notes 3 – easy 2 – needed 3 – easy help Task 2 Notes 2 – needed 2 – easy 2 – needed help help Task 3 Notes 2 – needed 3 – easy Ran out of help time Task 4 Notes 2 – needed 3 – easy Ran out of help time Page 46
- 47. OBSERVERS •PO, PM, dev, etc. • Stakeholders must attend to approve changes • Training: Set expectations (no interference) • Take notes • Make a list of 3 most serious issues • Help operate software/recording equipment (if in use) Page 47
- 48. THEY’RE HERE… Relax, they are only human •Get them talking •Will get frustrated • Be prepared - how react • Be supportive • Guide them back to the task • Do not lead •Listen more than you talk http://www.flickr.com/photos/chuybregts/ Page 48 http://creativecommons.org/licenses/by-nc-sa/2.0/
- 49.
- 50. TRADITIONAL USABILITY TESTING •Conduct anytime •One concept tested throughout •Feedback given when the testing is done •Qualitative and quantitative study •Remote and in-person •“Start earlier than you think makes sense.” – Steve Krug Page 50
- 51. STEVE KRUG STYLE 2.5 Weeks to Testing •January 16 (Monday) • Start planning what to test and with whom • Create Screener • Start Recruit •January 23 • Determine tasks to test and create guide • Finish recruiting by Jan. 31 •February 2 • 1st Usability Testing Day •February 6th • Begin planning for Feb. 16th UT Day Page 51
- 52. STEVE KRUG (CONTINUED) •3 – 5 users per day • Recruit loosely and grade on a curve •Stakeholders watch tests – spectator sport •Debrief at end of study •Quick analysis Page 52
- 53. TWEAK, DON’T REDESIGN •Small iterative changes • Make it better now • Don’t break something else •Take something away • Reduce distractions • Don’t add – question it Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. Page 53 By Steve Krug
- 54. GUERILLA STUDY “Hallway Testing” •Very quick studies •Very early in the design process (paper or soon after) •Recruiting is basic • Not familiar with the project • Ideally representative of user group Page 54
- 55.
- 56. R A PI D I T E R AT I V E T E S T I N G & E VA L U AT I O N •RITE is qualitative, exploratory, and conceptual •Creating conversation around elements •RITE is not for: • measuring performance • validating detailed design decisions Page 56
- 57. RITE OVERVIEW •Most appropriate for: • early requirements gathering • design phase of production •Initial wireframes used to: • validate information architecture • gather feedback on proposed functionality Page 57
- 58. RITE OVERVIEW Test Prioritize Update Test 1 High 2 Medium 3 Low •Series of small usability tests •Participants attempt tasks on concept •Qualitative user feedback (actions + comments) •New version tested with next day’s participants Page 58
- 59. R I TE V S . S TA N D A R D U S A B I L I T Y T E S T I N G Topic RITE Standard Usability Testing Tested Concept Evolves Mostly static Suitable stage Must be early Anytime of project Dynamics New concepts each day One concept tested throughout Feedback Different elements Same elements throughout Recommendati Discussed throughout Received in the final ons report Data Qualitative Qualitative and quantitative Stakeholder Crucial to have daily Preferred involvement Page 59
- 60.
- 61. D A ILY R E C A P S E S S I O N S •After the last session •30 minutes •Room with a whiteboard or paper •All decision makers MUST be present • If not, wait for UX analysis Page 61
- 62. GUIDELINES •Stay on Topic •Be Constructive •Don’t get distracted by small problems •Intense focus on fixing most serious problems first Page 62
- 63. GOAL •Identify top 5 or 10 most serious issues • Top 3 from each list • Prioritize from lists • Make assignments for next piece of work • Stop Adapted from: Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Page 63 Usability Problems. By Steve Krug
- 64. DEBRIEF WITH TEAM •Quick analysis to quick decisions •Discuss: • trends seen • concerns • recommendations • prioritize changes for the next round • list lower priority changes for future iterations Page 64
- 65. MAKE USEFUL &USABLE RECOMMENDATIONS - QUICKLY
- 66. A N ALY S I S Read “between the lines” •Analysis up to ~1.5 x the facilitation time •Identify repetition and patterns •When found, continuation: • Adds cost • Delays reporting • Low probability of many new findings Page 66
- 67. T R AN S F O R M D ATA •Know what you’ve got • Sort, reorganize, review, repeat • What refutes your expectations? • Surprises? • Outliers? Page 67
- 68. DOCUMENT WITH LESSTREES •Reduce, reuse •Use email, IM, etc. •Put it on the wall • Must be easy to understand • Quickly absorbable Page 68
- 69. E M AI L O R O N E PA G E R •Your Communication Goal: • Think about audience • How will it be used? •Include: • Study’s Goal • Tasks attempted • Who observed • What planned to fix and assignment • Future research/enhancements Page 69
- 70. SHARE WHAT YOULEARN Page 70
- 71. MAKE USERS VISIBLE Information radiators • Test findings • Artifacts • Personas Page 71
- 72. T R AN S F O R M D ATA •Create useful findings and recommendations • Show screenshots of issues • Where possible show solutions They liked this! •Think about audience • How will it be used? Make this go away! Page 72
- 73. REVIEW •Plan out your goals •Recruit participants •Remote or in person? •Look for patterns •Easy-to-use findings and recommendations Page 73
- 74. DO USABILITY TESTING EARLY & OFTEN
- 75.
- 76. C O NTA C T C A R O L @carologic Email: Carol.Smith@perficient.com slideshare.net/carologic and slideshare.net/PerficientInc speakerrate.com/speakers/15585-caroljsmith Page 76
- 77. REFERENCES •Albert, Bill, Tom Tullis, and Donna Tedesco. Beyond the Usability Lab. •Gothelf , Jeff. http://blog.usabilla.com/5-effective-ways-for-usability-testing-to- play-nice-with-agile/ •Henry, S.L. and Martinson, M. Evaluating for Accessibility, Usability Testing in Diverse Situations. Tutorial, 2003 UPA Conference. •Krug, Steve. Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. •Rubin, Jeffrey and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Inc. Page 77
- 78. T O OL C O N S I D E R AT I O N S •In-person or remote? •Lab or on-site? •Prototype limitations (can it be online?, is it a document or a clickable site?) •Number of observers, number of participants? •Number of facilitators? •Logging and video editing needs (time on task, highlight video creation)? •Surveys before or after? •Eye tracking? Page 78
- 79. U S AB I L I T Y T E S T I N G S O F T WA R E •Morae •Ovo •SilverBack (Mac only) •UserWorks •Noldus •Tobii (Eye-tracker) •SMI (Eye-tracker) •SurveyMonkey Page 79
- 80. S C RE E N S H A R I N G S O F T WA R E •GoToMeeting – http://www.gotomeeting.com •Lotus Sametime Unyte – http://www.unyte.com •YuuGuu -- http://www.yuuguu.com •WebEx – http://www.webex.com •Yugma -- https://www.yugma.com/ •Trouble Shooting: CoPilot - https://www.copilot.com/ Page 80
- 81. S AT IS FA C T I O N Q U E S T I O N N A I R E S •Standard Usability Measurement Inventory (SUMI) • office/desktop software, purchase • 50 questions •Website Analysis and Measurement Inventory (WHAMMI) • Purchase • 20 questions •System Usability Scale (SUS) • Free • 10 questions Page 81