Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Research Traps: 7 ways of thinking that keep you from doing great customer research

1,960 views

Published on

Presented at UPA 2009 on June 11, 2009.

Published in: Technology, Business
  • Be the first to comment

Research Traps: 7 ways of thinking that keep you from doing great customer research

  1. 1. Research Traps: 7 ways of thinking that keep you from doing great customer research Wendy Castleman Principal User Research Scientist To be presented at the UPA Conference in Portland, OR in June 2009
  2. 2. Who am I? Experience Research
  3. 3. RESEARCH TRAPS
  4. 4. Mental Shortcuts
  5. 5. Mental shortcuts help us • Make quicker decisions to take action faster…
  6. 6. Those shortcuts become research traps
  7. 7. Awareness
  8. 8. Habit False Illusory Consensus Correlation Recency Congruency Availability Confirmation
  9. 9. Meet Elsie
  10. 10. What do you think happened? I lived here I moved here maps.google.com
  11. 11. Habit We tend to do things the way we usually do
  12. 12. New project? Sure, I can run a usability test!
  13. 13. Habit Trap Can lead us to do the wrong research Your favorite research method may not be the best way to learn what you need to know
  14. 14. The best way to find that out would be a site visit…
  15. 15. How to Avoid the Habit Trap • Look at every project as unique • Consider what you need to learn • Identify the best method
  16. 16. • If Yes = guess most people would agree • If No = guess most people wouldn’t agree
  17. 17. False Consensus We tend to attribute our beliefs, thoughts and behaviors to others
  18. 18. Hi guys! Wanna be in a study?
  19. 19. False Consensus Trap Test with the wrong participants Other people may not think like you…
  20. 20. Junior League of Palo Alto
  21. 21. Avoiding the False Consensus Trap • Focus on the customer –spend time watching and talking • Test with people who aren’t like you
  22. 22. What is the rule? 2, 4, 6, 8, ___ Hypotheses: Each value must be 2 higher than the one before. How do you test this hypothesis? Actual Rule: Each value must be any number bigger than the one before.
  23. 23. Congruence Bias We jump to conclusions by only looking at one approach
  24. 24. Try out our new idea for an iPhone application!
  25. 25. Congruence Trap insufficiently inform the design Only trying one solution may miss a better one
  26. 26. Try out each of these iPhone applications!
  27. 27. Avoiding the Congruence Trap • Test several different solutions • Test out what shouldn’t work
  28. 28. Which card do you turn over? T6L 4 Hypothesis: The back of every “6” is an “L”. Did you say “T”?
  29. 29. Confirmation Error Bias We have a tendency to search for data to confirm expectations
  30. 30. As I suspected! 25% of users can’t do that task!
  31. 31. Confirmation Trap Get an incomplete picture of the data Seeking to prove our ideas can lead to missed surprises
  32. 32. Quicken used by small business
  33. 33. Avoiding the Confirmation Trap • Look for surprises, instead of what you expect • Test out what shouldn’t work • Consider independent evaluation
  34. 34. In American English are there more: Words that begin with the letter “K”? Or Words where “K” is the third letter? There are 3x as many words with “K” in the 3rd position
  35. 35. Availability Heuristic We have a tendency to put too much weight on what comes easily to mind
  36. 36. Availability Trap in Research …a lot of people had trouble with that task…
  37. 37. Availability Trap Draw inaccurate conclusions What comes to mind easiest may not be the most important or most frequent finding.
  38. 38. Hmm… I didn’t realize that happened so often…
  39. 39. Avoiding the Availability Trap • Gather key usability metrics –(task success, specific error counts, time) • Don’t rely on your memory • Look at all of the data –Encourage others to do the same
  40. 40. Last 3 movies Prior 3 movies
  41. 41. Recency Bias We tend to put too much weight on what we saw most recently
  42. 42. Recency Bias in Research Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Click “Done” Click Click “Done” Click Click “Cancel” “Continue” “Continue” Most people clicked “Continue”
  43. 43. Recency Trap Draw inaccurate conclusions What you saw most recently may not be the most important or most frequent finding.
  44. 44. Hmm… I didn’t realize that happened so often…
  45. 45. How to Avoid the Recency Trap • Gather key usability metrics –(task success, specific error counts, time) • Don’t rely on your memory • Look at all of the data
  46. 46. Happiness Weight
  47. 47. Illusory Correlation The tendency to find patterns where none exist
  48. 48. This is the third guy who uses a laptop in his living room. Maybe all men use laptops in their living rooms
  49. 49. Illusory Correlation Trap Draw inaccurate conclusions Things that co-occur may not be related.
  50. 50. That’s the fourth man who has bought this version. I need to find out how many men buy this…
  51. 51. Avoiding the Illusory Correlation Trap • Recognize the limitations of your research methods • Verify magnitude estimations and correlations with large-scale quantitative studies
  52. 52. Habit False Illusory Consensus Correlation Recency Congruency Availability Confirmation
  53. 53. Ways to avoid the traps… Planning 1. Look at every project as unique 2. Consider what you need to learn 3. Identify the best method 4. Test with people who aren’t like you 5. Test several different solutions 6. Test out what shouldn’t work
  54. 54. Ways to avoid the traps… Conducting 1. Look for surprises, instead of what you expect 2. Gather key usability metrics 3. Consider independent evaluation
  55. 55. Ways to avoid the traps… Analyzing 1. Don’t rely on your memory 2. Look at all of the data 3. Recognize the limitations of your research methods 4. Verify magnitude estimations and correlations with large-scale quantitative studies
  56. 56. http://Deepunderstandings.blogspot.com QUESTIONS? Email me Wendy_castleman@intuit.com

×