Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

User Testing: Adapt to Fit Your Needs

463 views

Published on

EffectiveUI's Ari Weissman (Lead Experience Architect) and Lys Maitland (Senior Experience Planner) spoke at Denver Startup Week 2016. Discussion description:

Test early, test often.

It’s a mantra that’s been proven successful time and again when it comes to innovation and design. So why aren’t you doing it? In the start-up world, when everything is moving so quickly, it can be easy to overlook or postpone collecting feedback from real people because of cost, time, or lack of preparation. Don’t let those things stop you. Valid data can be captured cheaply, quickly, and with half-finished products and strategies.

This talk will cover:

What is user testing and why is it important
How to plan for user testing
What are ways to make testing cheaper
What are ways to make testing quicker
How to test with different fidelities of concept and design
How to collect data more frequently
Opportunities for getting the whole team engaged
What to do with the insights/outcomes of research

Published in: Data & Analytics
  • Be the first to comment

User Testing: Adapt to Fit Your Needs

  1. 1. Sponsors
  2. 2. User Testing: Adapt to Fit Your Needs #DenStartupWeek #EffectiveUI
  3. 3. Agenda 1. Intro 2. What is evaluative research? 3. How To 4. Engaging the whole team 5. What to do next
  4. 4. Ari Weissman Lead Experience Architect888.310.5327 • 10+ years global experience • Responsible for all things UX, from research through experience design • Project lead for TimeWarner Cable, AMEX, FreemanCo, and Securian @ EffectiveUI
  5. 5. Lys Maitland Senior Experience Planner888.310.5327 • 15+ years global experience • Responsible for all things UX, from research through experience design • Research lead for Davita, Scottrade, NextGear, Securian, CenturyLink @ EffectiveUI
  6. 6. Insight-driven digital product company
  7. 7. Locations 2162 Market Street
 Denver, CO 80205 DENVER 274 North Goodman St, Unit B264
 Rochester, NY 14607 ROCHESTER 85 Broad Street, 18th fl.
 New York, NY 10004 NEW YORK CITY
  8. 8. Our clients
  9. 9. What is evaluative research? [a.k.a. user testing]
  10. 10. Generative v. Evaluative Generative research Helps define the problem. The goal of generative research is to look to the activities, environments, interactions, objects and people to find opportunities for solutions and innovation. These solutions could be new products or experiences or they could be an update or improvement to an existing one. Evaluative research Evaluates an existing design (in prototype form or in final form). The goal of evaluative research is to test the existing solution to see if it meets people’s needs, is easy to access and use, and is enjoyable. This type of research should be conducted throughout the development lifecycle, from early concept design (think rough sketches or prototypes) to the final site, app, or product.
  11. 11. DEFINE DISCOVER DESIGN DEVELOP DEPLOY EVALUATIVE RESEARCH Field Research Iterative Concept Testing Usability Testing Define Problem Problem Approach Gather Data Observational Research Benchmark Current Site Distinguish Insights Analytics Assessment Create 
 Concepts Concept Test Explore Solutions Build Prototypes Test & Refine Collaborate On Options Discover Gaps & Problems in Experience Analytics Strategy Implementation/ Change Management Continuous Measurement Test & Refine
  12. 12. Some examples of evaluative research Testing an IA or design learnability CLICK TEST Testing flows and content USABILITY TESTING Testing and measuring current state for redesign BENCHMARKING Closed card sort to validate an IA CARD SORTING Iterative testing of flows and concepts, some generative research ITERATIVE DESIGN TESTING
  13. 13. Benefits of User Testing 1 Validate assumptions 4 Provide justification for additional effort 2 Decide between options 5 Reveal how users think about the problems your solution solves 3 Discover hidden issues 6 Removes opinion from the design process
  14. 14. HowTo
  15. 15. How To 1. Define the goals 2. Define the audience 3. What to test 4. $ -> $$$ 5. Debrief 6. Analysis 7. Outcomes/Deliverables
  16. 16. What are your goals? 888.310.5327 Some questions to ask: - Why are you testing and what will you do with the data? - What do you need to know / learn / validate? Why?
  17. 17. Who is your audience? 888.310.5327 Some questions to ask: - Who will use of the thing you’re building? - Are there different groups of users? If so, which have the highest priority? - Where are they?
  18. 18. Recruiting Screener • Script used to recruit • Intro to the study • Questions to find participants who demonstrate desired behaviors • Offer an incentive Schedule 1 hour interviews with at least 30 minutes between sessions Recruit • 5-7 participants per segment • 5 participants finds 85% of usability issues 1 2 3
  19. 19. Participants are good to the extent they represent your target. If your participants don't match your target, your study will be useless. You can learn valuable things by asking the right people the wrong questions. If you are talking to the wrong people it doesn't matter what you ask. Bad participants undermine everything you’re trying to do. — Erika Hall, author of Just Enough Research “ ”
  20. 20. Run the first session as a pilot to work out the kinks and refine the moderators guide. Schedule an hour after the first to re-tool as needed. PRO TIP
  21. 21. What can I test? ANYTHING - CONCEPT - HAND SKETCHED MESSY - HAND SKETCHED NEAT - WIREFRAMES - VISUAL DESIGN - CLICK-THROUGH PROTOTYPE - CODED PROTOTYPE - FINAL PRODUCT
  22. 22. The quick and dirty
  23. 23. Why quick and dirty? - NO TIME - NO MONEY - NO FACILITIES - TESTING SOMETHING SMALL
  24. 24. Quick and dirty Location Anywhere! Logistics Find a place Find some people Ask some questions Write down answers Costs Minimal to none Fidelity to test Any 1 2 3 4
  25. 25. Middle of the road
  26. 26. Why middle of the road? - NEED SOME DOCUMENTATION TO SHARE WITH TEAM OR SAVE - TOO MUCH CONTENT FOR QUICK AND DIRTY - WANT TO HAVE OBSERVERS - DESIRED USERS ARE NOT READILY AVAILABLE
  27. 27. Middle of the road Location Preferably in context, though may be remote In a quiet room at a table Logistics Schedule a location and time for one hour Recruit participants and consider incentive Prep discussion guide Create data capture sheet Audio/Video recording Costs Location Recruiting and incentives Screen sharing/ Recording tool Fidelity to test Typically wireframe or above 1 2 3 4
  28. 28. Formal
  29. 29. Why formal testing? - REQUIRES FORMAL DOCUMENTATION - DESIRED USERS ARE NOT READILY AVAILABLE - STRICT DISCUSSION GUIDE REQUIREMENTS - LOTS OF THINGS TO TEST - NEED TO INCLUDE NOTETAKERS AND OBSERVERS - MULTIPLE DEVICES OR COMPLEX TECHNOLOGY REQUIREMENTS
  30. 30. Formal Location Quiet room with a table Separate location for observers Quality phone if remote Logistics Schedule a location and time for two hours Recruit participants and distribute incentive Prep/share/revise discussion guide Create data capture sheet Audio/Video recording Costs Location Recruiting and incentives Screen sharing/ Recording tool Snacks and beverages Transcriptions Fidelity to test Typically wireframe or above 1 2 3 4
  31. 31. Facilitation Do you have 3 more hours?
  32. 32. Facilitation: The highlights Asking questions Ground your questions • Will you How have you…? • In general In this specific instance…? • Tell me how Show me how…? Ask why? A lot! Rhythm • Ask • Listen • Probe • Validate 3 Types of tasks • Specific verb based tasks (do, send, complete) • Scavenger hunt (find, discover) • Interview (tell me about) 1 2 3
  33. 33. So what I heard was, you do X and Y because of A and B, but a little bit of 123 would make it better. Is that right? PRO TIP
  34. 34. The Debrief 888.310.5327 Debrief after EVERY session It’s a quick assessment of a session: - Summary of the participant - What went well - What needs improvement - Insights or surprises - Opportunities
  35. 35. Analysis and Synthesis Trends Look for trends across all participants or within a participant segment
 Good and Bad Document what doesn't work AND what does work
 Prioritize Choose the most important findings to share and focus on solving
 Think big picture Is the issue identified a symptom of a bigger problem? Quotes Identify quotes that capture the essence of your insights Solutions What solutions were identified by participants or observers?
 Future Opportunities What do the outcomes suggest would be a good next step?
 Centralize and share All data in one place for access and redundancy. Shit happens.
  36. 36. Examples of analysis OUTPUTS
  37. 37. Spreadsheet 39
  38. 38. Post-its 40
  39. 39. Live
  40. 40. Examples of evaluative research results OUTPUTS
  41. 41. $ quick and dirty
  42. 42. $ quick and dirty
  43. 43. $$ middle of the road
  44. 44. $$$$ formal
  45. 45. Engaging theWholeTeam
  46. 46. Engage your team at every step - CHOOSING GOALS AND AUDIENCE - REVIEWING DISCUSSION GUIDE - OBSERVING AND NOTE TAKING - DEBRIEFING - PROBING DURING ANALYSIS - PRIORITIZING RESULTS - DETERMINING NEXT STEPS
  47. 47. What to do with outcomes
  48. 48. What to do with outcomes 1 4 2 5 3 6 Share your findings (email, present, print, posterize) Create a roadmap with priorities Identify opportunities Support ideation Stay engaged! Be available to answer questions. Make recommendations 7 Test Again!
  49. 49. Questions? Vragen Preguntas 問題 Des questions ‫שאלות‬ вопросов คำถาม frågor
  50. 50. Thank you. Ari + Lys P.S. We’re hiring! http://www.effectiveui.com/company/careers/
  51. 51. www.linkedin.com/company/effectiveui LINKEDIN https://twitter.com/effectiveui TWITTER www.facebook.com/InsideEffectiveUI FACEBOOK www.instagram.com/effectiveui INSTAGRAM https://medium.com/@effectiveui MEDIUM Stay connected with EffectiveUI
  52. 52. https://abookapart.com/products/just-enough-research JUST ENOUGH RESEARCH BY ERIKA HALL http://rosenfeldmedia.com/books/interviewing-users INTERVIEWING USERS BY STEVE PORTIGAL https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users WHY YOU ONLY NEED TO TEST WITH 5 USERS BY JAKOB NIELSEN Mentioned References

×