Don’t Fear the Small Numbers: Why a Few Design Research Participants Can Provide Big Insights
•Download as PPTX, PDF•
0 likes•101 views
Don't Fear the Small Numbers - HXR 2016 - Dan Berlin and Susan Mercer
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Don’t Fear the Small Numbers: Why a Few Design Research Participants Can Provide Big Insights
Similar to Don’t Fear the Small Numbers: Why a Few Design Research Participants Can Provide Big Insights (20)
More from Mad*Pow
More from Mad*Pow (20)
Recently uploaded
Recently uploaded (20)
Don’t Fear the Small Numbers: Why a Few Design Research Participants Can Provide Big Insights
- 1. #HxR2016 Don’t Fear the Small Numbers Why a Few Design Research Participants Can Provide Big Insights Dan Berlin VP, Experience Research, Mad*Pow dberlin@madpow.com @banderlin Susan Mercer Experience Research Director, Mad*Pow smercer@madpow.com @susanamercer
- 2. #HxR2016 Topics Prevalent requests for large numbers of participants How small numbers provide big insights Determining the “right” number of participants
- 4. #HxR2016 Design Research isn’t… Market Research
- 5. #HxR2016
- 6. #HxR2016 But we’re not in the Mad Men days any more
- 7. #HxR2016 Good design follows an iterative process
- 9. #HxR2016
- 13. #HxR2016 Five to eight users for usability finds 80-90% of problems Proportion of Usability Problems Found by Number of Participants
- 15. #HxR2016 Find 85% of the problems, update design, repeat
- 17. #HxR2016 UX Research is Typically Directional
- 18. #HxR2016 We conduct hundreds of interviews each year
- 19. #HxR2016 Trends emerge after a few participants
- 20. #HxR2016 Outliers provide incremental data during qualitative research
- 21. #HxR2016 Usability studies are qualitative Positive: Users were able to find the “Buy Now” button. 3 out of 8 participants mentioned that the picture had no relevance to them. Consider: replacing this with an infographic or another picture that may draw in users. 6 out of 8 participants mentioned that the text in the buttons was hard to read. Consider: increasing the contrast of the text to ease readability.
- 22. #HxR2016 Priority Finding Recommendation Medium Participants said that the text was hard to read Increase the contrast of the text High Participants were unable to locate the product; they said they expected it under Products & Services Move XYZ product to the Product & Services area High Participants were unable to discern the top from the bottom of the vial Include a visual cue as to which is the top and bottom of the vial Medium Participants were able to figure out how to use the insulin pump, but said that the instructions were very hard to follow Add detail throughout the instruction manual Typical Usability Study Results
- 23. #HxR2016 The “Right” Number of Participants
- 24. #HxR2016 Keep Design RFPs Open-Ended
- 26. #HxR2016 Source: http://howiechang.com/post/3045331253 Segment Users by Needs Transactional Informational
- 27. #HxR2016 Project Scope Influences Number of Participants
- 28. #HxR2016 Segment Larger Projects Application Features Navigation Content
- 29. #HxR2016 • 12-16 participants per project segment & user group Discovery Interviews • 8-10 participants per project segment Usability Studies Very General Rules of Thumb
- 30. #HxR2016 Why? To Maximize ROI on UX Research
- 31. #HxR2016 Thank you! Dan Berlin @banderlin dberlin@madpow.com Susan Mercer @susanamercer smercer@madpow.com
Editor's Notes
- We’ll start with two things that Design Research is not.
- Marketing is great for it’s traditional uses: determining who to sell the product to and at what price point When determining what people will use and how much they will pay for something, statistically significant data makes sense – you want those large numbers as “proof” for very important company decisions This is still very important, so still use your 4 Ps or 4 As, the ARF, and Ogilvy’s book (DON”T ASSUME PEOPLE KNOW THESE) 4 P’s: Product, price, promotion, and place 4 A’s: Acceptability, affordability, accessibility, & awareness ARF: Advertising Research Foundation
- We’re not designing print advertisements or TV ads that are a one-shot deal We’re (typically) designing digital interfaces that can and should easily change This has facilitated iterative design: where we design something, learn from users, update the design, and repeat as necessary Even if it isn’t a digital interface – rapid physical prototyping will allow similar design processes soon enough (if not already) Design research is a very new thing and people often confuse it with market research Hence the desire for large numbers of participants when familiar with more traditional marketing research, not design research
- Iterative processes allow designers to check-in with users to ensure that the design meets their expectations We can course correct as we learn from/about users Also allows us to see if user wants and needs align with business goals Companies can easily release an early beta to get real-world feedback Can also continue to iterate after release – making incremental improvements
- To make a vast understatement: just as with market research, clinical trials are extremely important Yes, you definitely should have very statistically significant data when determining the efficacy and safety of a treatment But again, this is not typical for design research – we don’t need to “prove” that an interface is intuitive Naturally, the exception is when you’re conducting usability research for FDA approval of a device But that’s an final summative study; there should be iterative research along the way with smaller numbers of participants.
- We are talking about design research here today – research that informs the design of interfaces, workflows, devices, and other experiences We’re not determining who will buy the product and at what price – that’s for marketing to determine We’re also not determining the efficacy of a medical device or treatment – that’s for the clinicians to determine Design Research focuses on: How people complete specific tasks today What interfaces, devices and services they use today What challenges they have and why How these interfaces, devices, and services can be improved Design Research activities are mostly Qualitative
- Problem: The design of this injection pen required patients to push the two halves together and then press the button to inject. However, patients with small hands couldn’t do both things at once with one hand Patient Solution: Use a rubber band to push the two halves together and use their hand to press the button We would never have seen the use of the rubber band unless we went to the person’s home
- Things we have learned from studies in the OR: Nurses having to change a battery that’s at the top of a stack Medical device reps often stand in the OR to troubleshoot and advise live
- In the 90’s, Bob Virzi found that 5-8 participants will generally find 85% of your usability problems. More complex now, but it’s still a good rule of thumb. Minimum of 5 participants per demographic, if they truly represent different interaction/information needs. Typically, 12-18 participants per usability study, depending on the number of demographics. There are typically diminishing returns above 10 participants, which can waste precious project time and budget
- There are myriad UX design research methods, some of which may be familiar, some may be new Part of our jobs, as UX researchers, is to determine the right tool to answer a particular question during a certain project phase at the right price More studies are needed as to how many participants are needed for the different methods But, for 95% of these methods, under 100 or even under 50 participants will yield plenty of useful data
- The nice thing about having all of these methods is that we can be nimble Remember the iterative design process? Each iteration is different and we seek to learn different things: Usefulness Usability Does the navigation make sense? Which design is more appealing? Or engenders a particular emotional response? Again, for each of these questions, the trends typically emerge after just a few participants
- UX research is not definitive – we’re not “proving” anything. We’re giving the design direction based on user input. When deciding what product to sell or if a treatment is effective, yes, you need research that proves it But not with design research; we take direction from users, design, check-in with users, and so on It’s a collaborative, participatory process
- At Mad*Pow, we conduct hundreds of interviews each year across a wide array of domain These are typically conducted in sets of 8-24 participants Smaller numbers when we just want to “check-in” with users; larger numbers when looking at a broader scope and set of user needs There are also the studies where we DO have 30-70 participants: Wide user-base with different needs – Example: Working with a University about a research grant tracking database used by 100s of different users in different departments for different types of gratns Deep product functionality FDA approval
- In conducting all of these interviews, we typically find that high-level trends emerge after just a few participants Whether it’s something tactical, for example: most participants clicking the wrong path during a usability study Or strategic: most participants expressing the same needs for an injection pen diary So in just a few days of interviews, we can give a design project actionable insights that allow designs to create something that aligns with user expectations and needs
- As we conduct additional interviews during a study, there are tidbits of good information that we gather that aren’t part of the overall trends There’s always that one participant who finds a different way to complete a task, or someone that goes off on a tangent the proves to be insightful While this can be useful, the ROI on adding participants to get incremental data is quite low
- This is an example of what a usability report looks like. Researchers will point out prioritized opportunities for improvement. Reporting on the positive findings is important so we don’t break what is already working.
- The overall goal is to report on the study findings and the associated recommendations. Always report a recommendation for each finding. Ensure the findings are prioritized, a H/M/L scale is typically sufficient.
- Focus on your goals and the decisions that you need to make based on the findings Let the professionals (researchers) recommend research methods and participant numbers Provide budget ranges and target users – this helps us scope projects Every project is different and conversations make for a successful project – not RFPs with subsequent Q&A documents
- We’re not aiming for statistically significant data – it’s not worth the cost. We’re not proving anything, we’re getting design direction. Personas are documents that capture a “typical” user’s wants and needs for an interface, design, product, etc. If you feel that a demographic needs to be part of the user audience specification, ask yourself “Why?” What BEHAVIOR or ATTITUDE is this demographic suggesting to you? Use that behavior/attitude in your user description, instead of the demographic.
- Similar to Maslow’s hierarchy of needs, this is the UX hierarchy of needs, first posited by Stephen Anderson, a UX thought-leader At the base is functionality/usefulness – this is where you need to segment your users for research recruiting What are the different transactions and information needs that your users have? Confirm these assumptions with users early in the project and adjust from there If these needs are very similar, you don’t need that many participants; If these are very different – you will need more participants
- This is why the scope of a project is also important when determining the number of participants to interview Are you looking to update a microsite with limited functionality? You probably don’t need that many participants. Are you designing an EMR? You definitely need a lot of participants – and a variety of research methods
- Chunk larger projects into bite-sized pieces and conduct multiple studies There may be overlap between projects, but that’s an opportunity to confirm assumptions The project as a whole may have a lot of participants, but each individual study can have just a few to move the project along
- As a vast generalization, these are numbers that tend to work and have proven to be efficient Discovery interviews tend to be more open-ended and we want to cast a wider net at the beginning of a project Usability studies and the like tend to be more tactical: Can the user accomplish the primary task on the website? Tactical findings take fewer participants to uncover
- When you keep in mind that design research is neither market research nor a clinical trial and that small numbers of participants can advance the project you: Get direction from users throughout the project Save project time and money – participant recruitment can be quite expensive, depending on the demographic Produce a product that better aligns with target users’ expectations