Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

3,056 views

Published on

Carol Smith presented "Faster Usability Testing in an Agile World" via webinar during the Agile UX Virtual Summit 2017 by UXPin.
This presentation covers:
- Brief intro to how the IBM Watson Design team runs continuous usability tests and integrates the UX team
- How design work fits into the Agile process via dual track development
- When to run moderated, un-moderated, remote, and in-person studies
- How to effectively communicate UX findings and recommendations

Published in: Design
  • my ­n­e­i­g­h­b­or's ­m­ot­h­er ­m­A­k­es $28 ­h­our­ly ­o­n t­h­e ­Stepium. S­h­e ­h­As ­b­e­e­n ­out ­o­f w­or­k ­f­or f­iv­e ­m­o­nt­hs ­but ­l­Ast ­m­o­nt­h ­h­er ­p­Ay­m­e­nt w­As $5000 just w­or­k­i­n­g ­o­n t­h­e ­Stepium ­f­or ­A f­ew ­h­ours. ­g­o t­o t­h­is w­e­b s­it­e ­A­n­d r­e­A­d ­m­or­e ­.............. .. HERE► https://ethstepium.stepium.com/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • tổ chức hội nghị hội thảo https://vietwindevent.vn/to-chuc-hoi-nghi-hoi-thao.html
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by UXPin

  1. 1. UXPin Agile UX Virtual Summit June 15, 2017 Carol Smith @carologic Faster Usability Testing in an Agile World
  2. 2. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Design at IBM Watson Design Thinking & Agile at IBM with Vera Rhoads 16th June 2017, 2:00-3:00 PM PDT
  3. 3. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Researcher and interaction designer on each squad • Visual designer often shared • Designers Consult on other teams as needed Watson - Squads include UX Designers
  4. 4. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Iterative • Repeatable • Incremental • Get to 80% confidence Light Design - Lean UX
  5. 5. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Just enough • Identify and solve problems • Use each study to pick up information • Think ahead (not make ahead) – Concept Cars (vision) – Know the North Star No BDUF* at Watson *Big Design Up Front
  6. 6. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Fit testing into Agile process • Never too early - whatever is ready • Regular testing Faster Usability Testing Image of sign, created by IBM’er Thomas J. Watson http://www.newyorker.com/cartoons/bob-mankoff/computer-games
  7. 7. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Representative users • Doing real tasks • Being observed • Using prototypes or live products Usability Testing is… http://creativecommons.org/licenses/by-sa/2.0/ http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/ http://www.flickr.com/photos/raphaelquinet/
  8. 8. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • There is a difference • User testing infers testing people – Can the participant figure out our design work? • Usability testing puts emphasis on what is being tested – Reduces the focus on the participant’s performance – Is the product successful? Usability NOT User Testing
  9. 9. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • All interfaces have usability problems • Limited resources to fix them • More problems than resources • Less serious problems distract • Intense focus on fixing most serious problems first True Statements Adapted from: Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems, by Steve Krug
  10. 10. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Parallel-Track Workflows (How to Think Ahead) a.k.a. Staggered Sprints
  11. 11. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Developer Track: – Focus is on production – Whole team (including UX) • UX Track (Interaction Designer Track) – Focus is on user contact Agile Design Timing: Parallel Tracks “Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy. Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/ UX
  12. 12. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Iteration 1 “Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy. Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/ UX
  13. 13. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Iteration 2 “Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy. Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
  14. 14. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 And so on… “Adapting Usability Investigations for Agile User-Centered Design” by Desirée Sy. Journal of Usability Studies, Volume 2, Issue 3 http://www.uxpajournal.org/
  15. 15. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 1) Development work – Maximize velocity to make working software – Goal: Working software 2) Discovery work – Learning velocity - fast learning and validation – Goal: Learning Reality Check – 2 Types of Work Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton http://jpattonassociates.com/dual-track-development/
  16. 16. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Dual Track Development is not Duel Track Dual Track Development is not Duel Track, May 10, 2017 By Jeff Patton http://jpattonassociates.com/dual-track-development/
  17. 17. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Between the two tracks • Essential for success • Not just hand-offs • All at once – two tracks, not two teams • All work will not move from discovery to development Constant Communication From Tutorial: Making UX Agile and Lean at UXPA 2016 by John Schrag, Carol Smith and Thyra Rauch http://uxpa2016.org/event/making-ux-agile-and-lean
  18. 18. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Pros – Space is made for discovery work • Cons – Can lead to BDUF and Waterfall – Not a solo sport - need a team Pros and Cons - Parallel/Staggered work
  19. 19. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Balancing Act • Understand users, context, etc. • Create personas, mental models, etc. • Prepare for story mapping and other sessions thoroughly Strive for UX Best Practices • Engineers need designs to develop • Research cannot continue forever Meet Production Needs Agile requires leaner methods Time box and focus on outcomes Slide from Tutorial: Making UX Agile and Lean at UXPA 2016 By John Schrag, Carol Smith and Thyra Rauch http://uxpa2016.org/event/making-ux-agile-and-lean
  20. 20. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Continuous improvement • Continue to prioritize Dev work • Start small with Discovery work in next sprint • May need a Design Spike Existing products/teams
  21. 21. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 What to Test
  22. 22. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Learn from whatever is ready… • Concepts, rough ideas, brainstorming • Work in Progress • Prototypes - make to learn – Clickable (code is optional) • Websites, mobile, products, services • Competing designs (A/B testing) • Comparative studies • Conduct user research
  23. 23. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Clickable – Easy and quick to create – Easy to change • Real Code – Risky – more time invested – Great if it’s the right solution • Not paper - participants may misunderstand Clickable or Code?
  24. 24. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 When should I test? • Early! Any Stage of process • Anywhere (conference room, remotely) • Anytime (un-moderated) • Realistic test environment Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/
  25. 25. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Testing Methods
  26. 26. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • What do we need to know? • What gets us to 80% confidence/disproves? • Bare minimum of effort? • Place higher value on repeatable formative studies Adapting Usability Testing Methods
  27. 27. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Consider budget, resources • Time – Recruiting – Facilitating – Analyzing Scope Effort
  28. 28. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Small Scope is Better • 5-7 participants uncover ~80% of major usability problems • Don’t try to be statistically significant • More participants - diminishing returns Jakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users. March 19, 2000. http://www.useit.com/alertbox/20000319.html Number of Test Participants Participants are representative of specific user group and are doing representative tasks More participants take more time, cost more money - significantly diminishing returns.
  29. 29. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Moderated – In Person – Remote (Web Meeting) • Un-moderated (online tools) Testing Options
  30. 30. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • 3 + participants (tie breaker) • Short sessions – Moderated: 30 minutes to 1 hour each – Unmoderated: Under 20 minutes Make it Quick
  31. 31. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Pros – Get to “why” by asking follow-up questions – Work with them longer – Save or end session as appropriate • Cons – Harder to recruit – Takes longer to plan Moderated
  32. 32. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Pros – Great for quick answers - complements moderated testing – Easy recruiting • Cons – Recruiting very relaxed – Professional participants • Participants camera and internet only Unmoderated
  33. 33. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Remote Testing Options Moderated Un-Moderated Researcher is present Researcher not needed during study Same effort as in-person Minimal effort Limited number of participants per day (3-6) High number of participants per day Immediate feedback Test data compiled usually within 2 days Get to the “Why?” by asking questions Reduced or no ability to ask questions
  34. 34. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • No travel • Easy to set up • Conducted from almost anywhere • Option to see participants actual environment • Drawback: Not as high quality as in-person Benefits of Remote
  35. 35. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Robust computer • Screen sharing software • Fast Internet on both ends – May require separate phone line if no VOIP • Participants camera – People are becoming more comfortable – I still do not use Remote – Moderated Requirements
  36. 36. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 More on Remote Testing • Getting the most out of remote research and testing – http://interactions.acm.org/archive/view/march-april- 2017/getting-the-most-out-of-remote-research-and-testing – ACM Interactions Magazine by Carol Smith Getting the most out of remote research and testing. By Carol J. Smith. 2017. Interactions, 24, 2 (February 2017), 82-84. DOI: https://doi.org/10.1145/3038225 http://interactions.acm.org/archive/view/march-april-2017/getting-the-most-out-of- remote-research-and-testing
  37. 37. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • In-person (office or participant location) or remote? • Prototype/organization limitations (Online? Clickable?) • Number of participants, observers? • Logging and video editing (time on task, highlights)? • Surveys before or after? • Eye tracking? Software Tool Considerations
  38. 38. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Software Tool Considerations Meeting Software ModeratedUn-Moderated Complex Simple
  39. 39. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Nice – Are your users local? – Opt for large screen TVs/Cameras vs. two way glass • Recommend laptop/portable lab – Can be semi-permanent – Ready when you are (minimal setup) Formal Lab?
  40. 40. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Make it Repeatable
  41. 41. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 “Teams should stretch to get work into that day’s test and use the cadence to drive productivity.” - Jeff Gothelf http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice- with-agile/
  42. 42. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Regular and Early Testing • Reduces risk and chance of major issues later • Priority needs and problems get priority • Don’t sweat the small stuff
  43. 43. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Small focused tests • Reduced recruitment time • Once per Sprint • Same day(s) mid-week (avoid Mondays and Fridays) Bring it On!
  44. 44. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Team becomes: – accustomed to steady stream of qualitative insight – insight encourages quick decisions…line up with business and user goals Why Regular? Adapted from Jeff Gothelf - http://blog.usabilla.com/5- effective-ways-for-usability-testing-to-play-nice-with-agile/
  45. 45. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Problems we will find while usability testing • Research to inform personas • Design/UX debt • Advocate to get prioritized (UX rep should be in planning) Design Stories
  46. 46. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Learn from whatever is ready… • Concepts, rough ideas, brainstorming • Work in Progress • Prototypes - make to learn – Clickable (code is optional) • Websites, mobile, products, services • Competing designs (A/B testing) • Comparative studies • Conduct user research
  47. 47. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Managing Testing
  48. 48. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Select what to test and/or additional work • Create guide • Determine recruit and automate where possible Planning is Tight
  49. 49. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Screener – Technology use/experience – Knowledge of topic • Guides/Scripts • Consent Forms • Data Collection Create Reusable Templates
  50. 50. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Test & Observation Rooms – Use repeat calendar feature • Any location will do – Conference rooms – Offices – Quiet corner of cafeteria Pre-Book Rooms Quarterly
  51. 51. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Recruiting
  52. 52. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Hire a Recruiter • Allows you to focus • Good recruiters: – find right participants. – give regular updates. – take care of directions, confirmations, incentives, etc.
  53. 53. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Email, phone, etc. • Create a panel (large recruits, less often) – Customer lists, user groups – Schedule as needed – Use cautiously • Internal resource who REALLY understands who you need Do it Yourself
  54. 54. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Use Site to Recruit Ethnio: https://ethn.io/
  55. 55. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • “We are all only temporarily able-bodied. Accessibility is good for us all.” -@mollydotcom at #stirtrek 2011 • Know the law, build for the spirit of accessibility – U.S. Section 508 - https://www.section508.gov/content/learn – WCAG 2.0 - https://www.w3.org/WAI/intro/wcag.php Include people with disabilities
  56. 56. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Keep it easy – In Person: Gift cards from retail stores – Remote: Online Gift Cards • More complex – Product credits – Swag (don’t bother) Honorariums
  57. 57. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Make team aware • Invite everyone – Can watch remotely – Recurring meeting invites for stakeholders Usability Testing Day!
  58. 58. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Observing Studies
  59. 59. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Stakeholders and decision makers – PO, PM, Dev, etc. • Greet participants (in person sessions) • Take notes and share them – Create a template if helpful • Help operate software/recording equipment Team Members Are Active Observers
  60. 60. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Pro’s – Gain empathy for users – Understand struggles and what works – Quick decisions can be made • Cons – Time is limited – Not all sessions are great (but most are!) Observers
  61. 61. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Notetaking • Keep it simple • Paper or Computer – Ask participant if bothered by typing – Consider if you can pay attention (turn off all distractions) Issue P1 P2 P3 Search Used Yes No No Widget 1 Used N/A Used – unsure about Task 1 Notes 3 – easy 2 – needed help 3 – easy Task 2 Notes 2 – needed help 2 – easy 2 – needed help Task 3 Notes 2 – needed help 3 – easy Ran out of time Task 4 Notes 2 – needed help 3 – easy Ran out of time
  62. 62. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Analysis, Debrief and Useful & Usable Recommendations
  63. 63. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Occurs without full analysis when stakeholders have watched tests • If not: Quick analysis to quick decisions Debrief with Team
  64. 64. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • All decision makers MUST be present • Stay on Topic • Be Constructive • Don’t get distracted by small problems • Intense focus on fixing most serious problems first Guidelines for Debrief
  65. 65. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Identify top 5 or 10 most serious issues – Top 3 from each observer’s list – Prioritize from lists – Commit resources for next sprint – Stop Goal Adapted from: Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems, by Steve Krug
  66. 66. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Excel - Find Patterns in Data Learn more: I’m Sorry. I Can’t. Don’t Hate Me: The Post-It Breakup by Kyle Soucy at IA Summit 2017 https://www.slideshare.net/usableinterface/im-sorry-i- cant-dont-hate-me-the-postit-breakup
  67. 67. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Look for patterns • Read “between the lines” • Know what you’ve got – Sort, reorganize, review, repeat – What refutes your expectations? – Surprises? – Outliers? Transform Data
  68. 68. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Don’t be a tree killer Image of stack of paper: https://commons.wikimedia.org/wiki/File:Stack_of_Copy_Paper.jpg X Marks the Spot https://openclipart.org/detail/193050/x-marks-the-spot
  69. 69. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Bulleted list – Who will use information and how? – Share via common tools (wiki, Box, etc.) • Include – Goal of study – High level findings and recommendations – Link to guide and analysis (tasks attempted, actual performance, etc.) Just enough
  70. 70. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Identify stories related to findings • Add findings/evidence/recommendations in comments on related stories in repo • Tangential findings/evidence/recommendations - also note in comments Tie to Stories in Repo
  71. 71. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Small iterative changes – Make it better now – Don’t break something else • Take something away – Reduce distractions – Don’t just add – question it Tweak, Don’t Redesign Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems, by Steve Krug
  72. 72. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Faster Usability Testing • Fit testing into Agile process • Never too early – whatever is ready • Regular testing https://www.flickr.com/photos/fireatwillrva/5823369990
  73. 73. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Contact Carol Smith in/CarolJSmith @Carologic slideshare.net/carologic
  74. 74. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Resources, References, etc.
  75. 75. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Recommended Readings 7 5
  76. 76. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 • Hire great people • Balance skills and experience • Enable to work to their strengths • Encourage to grow, attend webinars, conferences, etc. • Make time for 1:1’s Towards Building Teams
  77. 77. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Encourage Information Radiators • User information • Customer journeys • Photos from research • Usability metrics
  78. 78. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 Agile Frustrations via Spotify Scaling Agile at Spotify via Slideshare of Vlad Mysla http://www.slideshare.net/vmysla/scrum-at-spotify?qid=2345c3ad-7e68-4383-9673- 9e715ff47a75&v=default&b=&from_search=14 squad member
  79. 79. Adapting Usability Investigations for Agile User-Centered Design by Desirée Sy Journal of Usability Studies, Volume 2, Issue 3 (the most-cited paper in JUS) http://www.uxpajournal.org/
  80. 80. IBM Watson / UXPin Agile UX Virtual Summit / @carologic / June 15, 2017 References and Resources Albert, Bill, Tom Tullis, and Donna Tedesco. Beyond the Usability Lab. Gothelf , Jeff. http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/ Henry, S.L. and Martinson, M. Evaluating for Accessibility, Usability Testing in Diverse Situations. Tutorial, 2003 UPA Conference. Krug, Steve. Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. Patton, Jeff (with Peter Economy). User Story Mapping, Discover the whole story, build the right product. Rubin, Jeffrey and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Inc. Usability.gov W3C Web Accessibility Initiative - http://www.w3.org/WAI/ UXPA – User Experience Professionals Association - http://www.uxpa.org

×