Research Traps:    7 ways of thinking that keep you from doing great customer research Wendy Castleman Principal User Rese...
Who am I?
RESEARCH TRAPS
Mental Shortcuts
Mental shortcuts help us <ul><li>Make quicker decisions to take action faster… </li></ul>
Those shortcuts become research traps
Awareness
<ul><li>7 </li></ul>
 
Meet Elsie
What do you think happened? I lived here I moved here
Habit   We tend to do things the way we usually do
New project?  Sure, I can run  a usability test!
Habit Trap   Can lead us to do the wrong research   Your favorite research method may not be the best way to learn what yo...
The best way  to find that out  would be a site visit…
How to Avoid the Habit Trap <ul><li>Look at every project as unique </li></ul><ul><li>Consider what you need to learn  </l...
<ul><li>If Yes = guess most people  would  agree </li></ul><ul><li>If No = guess most people  wouldn’t  agree </li></ul>Ea...
<ul><li>False Consensus </li></ul><ul><li>We tend to attribute our beliefs, thoughts and behaviors to others  </li></ul>
Hi guys!  Wanna be in a study?
<ul><li>False Consensus Trap </li></ul><ul><li>Test with the wrong participants  </li></ul>Other people may not think like...
Junior League of Palo Alto
Avoiding the False Consensus Trap <ul><li>Focus on the customer </li></ul><ul><ul><li>spend time watching and talking </li...
What is the rule? 2, 4, 6, 8, ___ Hypotheses:  Each value must be 2 higher than the one before. How do you test this hypot...
<ul><li>Congruence Bias  </li></ul><ul><li>   </li></ul>We jump to conclusions by  only looking at one approach
Try out our new idea  for an iPhone application!
<ul><li>Congruence Trap  </li></ul><ul><li>   </li></ul>Only trying one solution may miss a better one insufficiently info...
Try out each of these  iPhone applications!
Avoiding the Congruence Trap <ul><li>Test several different solutions </li></ul><ul><li>Test out what shouldn’t work </li>...
Which card do you turn over? <ul><li>Hypothesis:  </li></ul><ul><li>The back of every “6” is an “L”. </li></ul>T 6 L 4 Did...
<ul><li>Confirmation Error Bias  </li></ul><ul><li>We have a tendency to search  for data to confirm expectations </li></ul>
As I suspected!  25% of users  can’t do that task!
<ul><li>Confirmation Trap </li></ul><ul><li>Get an incomplete picture of the data </li></ul>Seeking to prove our ideas can...
Quicken used by small business
Avoiding the Confirmation Trap <ul><li>Look for surprises, instead of what you expect </li></ul><ul><li>Test out what shou...
In American English are there more: There are 3x as many words with “K” in the 3 rd  position
<ul><li>Availability Heuristic </li></ul><ul><li>We have a tendency to put too much weight   on what comes easily to mind ...
Availability Trap in Research … a lot of people had trouble with that task…
<ul><li>Availability Trap </li></ul><ul><li>Draw inaccurate conclusions </li></ul>What comes to mind easiest may not be th...
Hmm… I didn’t realize that happened so often…
Avoiding the Availability Trap <ul><li>Gather key usability metrics  </li></ul><ul><ul><li>(task success, specific error c...
<ul><ul><ul><li>Last 3 movies </li></ul></ul></ul><ul><ul><ul><li>Prior 3 movies </li></ul></ul></ul>
<ul><li>Recency Bias </li></ul>We tend to put too much weight on what we saw most recently
Recency Bias in Research Click  “ Continue” Participant 5 Click  “ Continue” Participant 4 Click “Done” Participant 3 Clic...
<ul><li>Recency Trap </li></ul>What you saw most recently may not be the most important or most frequent finding.  Draw in...
Hmm… I didn’t realize that happened so often…
How to Avoid the Recency Trap <ul><li>Gather key usability metrics  </li></ul><ul><ul><li>(task success, specific error co...
Weight
<ul><li>Illusory Correlation </li></ul>The tendency to find patterns where none exist
This is the third guy who uses a laptop in his living room.  Maybe all men  use laptops  in their living rooms
<ul><li>Illusory Correlation Trap </li></ul>Things that co-occur may not be related.  Draw inaccurate conclusions
That’s the fourth man  who has bought this version. I need to find out how  many men buy this…
Avoiding the Illusory Correlation Trap <ul><li>Recognize the limitations of your research methods </li></ul><ul><li>Verify...
 
 
Ways to avoid the traps… <ul><li>Planning </li></ul><ul><ul><li>Look at every project as unique </li></ul></ul><ul><ul><li...
Ways to avoid the traps… <ul><li>Conducting </li></ul><ul><ul><li>Look for surprises, instead of what you expect </li></ul...
Ways to avoid the traps… <ul><li>Analyzing </li></ul><ul><ul><li>Don’t rely on your memory </li></ul></ul><ul><ul><li>Look...
QUESTIONS? [email_address]
Upcoming SlideShare
Loading in …5
×

Research Traps: 7 ways of thinking that keep you from doing great customer research

3,286 views

Published on

Presentation given at the joint Silicon Valley UPA/ BayCHI Usability Engineering BOF on September 16, 2008 in Mountain View, CA.

The slides are from a talk about common traps that professional usability researchers make. It also includes suggestions for how to avoid the traps.

Published in: Business, Technology
1 Comment
8 Likes
Statistics
Notes
No Downloads
Views
Total views
3,286
On SlideShare
0
From Embeds
0
Number of Embeds
49
Actions
Shares
0
Downloads
0
Comments
1
Likes
8
Embeds 0
No embeds

No notes for slide
  • Research Traps: 7 ways of thinking that keep you from doing great customer research

    1. 1. Research Traps:   7 ways of thinking that keep you from doing great customer research Wendy Castleman Principal User Research Scientist To be presented at the UPA/BayCHI Usability BOF, September 16, 2008
    2. 2. Who am I?
    3. 3. RESEARCH TRAPS
    4. 4. Mental Shortcuts
    5. 5. Mental shortcuts help us <ul><li>Make quicker decisions to take action faster… </li></ul>
    6. 6. Those shortcuts become research traps
    7. 7. Awareness
    8. 8. <ul><li>7 </li></ul>
    9. 10. Meet Elsie
    10. 11. What do you think happened? I lived here I moved here
    11. 12. Habit We tend to do things the way we usually do
    12. 13. New project? Sure, I can run a usability test!
    13. 14. Habit Trap Can lead us to do the wrong research Your favorite research method may not be the best way to learn what you need to know
    14. 15. The best way to find that out would be a site visit…
    15. 16. How to Avoid the Habit Trap <ul><li>Look at every project as unique </li></ul><ul><li>Consider what you need to learn </li></ul><ul><li>Identify the best method </li></ul>
    16. 17. <ul><li>If Yes = guess most people would agree </li></ul><ul><li>If No = guess most people wouldn’t agree </li></ul>Eat at Joe’s
    17. 18. <ul><li>False Consensus </li></ul><ul><li>We tend to attribute our beliefs, thoughts and behaviors to others </li></ul>
    18. 19. Hi guys! Wanna be in a study?
    19. 20. <ul><li>False Consensus Trap </li></ul><ul><li>Test with the wrong participants </li></ul>Other people may not think like you…
    20. 21. Junior League of Palo Alto
    21. 22. Avoiding the False Consensus Trap <ul><li>Focus on the customer </li></ul><ul><ul><li>spend time watching and talking </li></ul></ul><ul><li>Test with people who aren’t like you </li></ul>
    22. 23. What is the rule? 2, 4, 6, 8, ___ Hypotheses: Each value must be 2 higher than the one before. How do you test this hypothesis? Actual Rule: Each value must be any number bigger than the one before.
    23. 24. <ul><li>Congruence Bias </li></ul><ul><li>   </li></ul>We jump to conclusions by only looking at one approach
    24. 25. Try out our new idea for an iPhone application!
    25. 26. <ul><li>Congruence Trap </li></ul><ul><li>   </li></ul>Only trying one solution may miss a better one insufficiently inform the design
    26. 27. Try out each of these iPhone applications!
    27. 28. Avoiding the Congruence Trap <ul><li>Test several different solutions </li></ul><ul><li>Test out what shouldn’t work </li></ul>
    28. 29. Which card do you turn over? <ul><li>Hypothesis: </li></ul><ul><li>The back of every “6” is an “L”. </li></ul>T 6 L 4 Did you say “T”?
    29. 30. <ul><li>Confirmation Error Bias </li></ul><ul><li>We have a tendency to search for data to confirm expectations </li></ul>
    30. 31. As I suspected! 25% of users can’t do that task!
    31. 32. <ul><li>Confirmation Trap </li></ul><ul><li>Get an incomplete picture of the data </li></ul>Seeking to prove our ideas can lead to missed surprises
    32. 33. Quicken used by small business
    33. 34. Avoiding the Confirmation Trap <ul><li>Look for surprises, instead of what you expect </li></ul><ul><li>Test out what shouldn’t work </li></ul><ul><li>Consider independent evaluation </li></ul>
    34. 35. In American English are there more: There are 3x as many words with “K” in the 3 rd position
    35. 36. <ul><li>Availability Heuristic </li></ul><ul><li>We have a tendency to put too much weight on what comes easily to mind </li></ul>
    36. 37. Availability Trap in Research … a lot of people had trouble with that task…
    37. 38. <ul><li>Availability Trap </li></ul><ul><li>Draw inaccurate conclusions </li></ul>What comes to mind easiest may not be the most important or most frequent finding.
    38. 39. Hmm… I didn’t realize that happened so often…
    39. 40. Avoiding the Availability Trap <ul><li>Gather key usability metrics </li></ul><ul><ul><li>(task success, specific error counts, time) </li></ul></ul><ul><li>Don’t rely on your memory </li></ul><ul><li>Look at all of the data </li></ul>
    40. 41. <ul><ul><ul><li>Last 3 movies </li></ul></ul></ul><ul><ul><ul><li>Prior 3 movies </li></ul></ul></ul>
    41. 42. <ul><li>Recency Bias </li></ul>We tend to put too much weight on what we saw most recently
    42. 43. Recency Bias in Research Click “ Continue” Participant 5 Click “ Continue” Participant 4 Click “Done” Participant 3 Click “Done” Participant 1 Click “ Cancel” Participant 2 Most people clicked “Continue”
    43. 44. <ul><li>Recency Trap </li></ul>What you saw most recently may not be the most important or most frequent finding. Draw inaccurate conclusions
    44. 45. Hmm… I didn’t realize that happened so often…
    45. 46. How to Avoid the Recency Trap <ul><li>Gather key usability metrics </li></ul><ul><ul><li>(task success, specific error counts, time) </li></ul></ul><ul><li>Don’t rely on your memory </li></ul><ul><li>Look at all of the data </li></ul>
    46. 47. Weight
    47. 48. <ul><li>Illusory Correlation </li></ul>The tendency to find patterns where none exist
    48. 49. This is the third guy who uses a laptop in his living room. Maybe all men use laptops in their living rooms
    49. 50. <ul><li>Illusory Correlation Trap </li></ul>Things that co-occur may not be related. Draw inaccurate conclusions
    50. 51. That’s the fourth man who has bought this version. I need to find out how many men buy this…
    51. 52. Avoiding the Illusory Correlation Trap <ul><li>Recognize the limitations of your research methods </li></ul><ul><li>Verify magnitude estimations and correlations with large-scale quantitative studies </li></ul>
    52. 55. Ways to avoid the traps… <ul><li>Planning </li></ul><ul><ul><li>Look at every project as unique </li></ul></ul><ul><ul><li>Consider what you need to learn </li></ul></ul><ul><ul><li>Identify the best method </li></ul></ul><ul><ul><li>Test with people who aren’t like you </li></ul></ul><ul><ul><li>Test several different solutions </li></ul></ul><ul><ul><li>Test out what shouldn’t work </li></ul></ul>
    53. 56. Ways to avoid the traps… <ul><li>Conducting </li></ul><ul><ul><li>Look for surprises, instead of what you expect </li></ul></ul><ul><ul><li>Gather key usability metrics </li></ul></ul><ul><ul><li>Consider independent evaluation </li></ul></ul>
    54. 57. Ways to avoid the traps… <ul><li>Analyzing </li></ul><ul><ul><li>Don’t rely on your memory </li></ul></ul><ul><ul><li>Look at all of the data </li></ul></ul><ul><ul><li>Recognize the limitations of your research methods </li></ul></ul><ul><ul><li>Verify magnitude estimations and correlations with large-scale quantitative studies </li></ul></ul>
    55. 58. QUESTIONS? [email_address]

    ×