Research Traps: 7 ways of thinking that keep you from doing great customer research Wendy Castleman Principal User Research Scientist To be presented at the UPA/BayCHI Usability BOF, September 16, 2008

Who am I?

RESEARCH TRAPS

Mental Shortcuts

Mental shortcuts help us Make quicker decisions to take action faster…

Make quicker decisions to take action faster…

Those shortcuts become research traps

Awareness

7

7

Meet Elsie

What do you think happened? I lived here I moved here

Habit We tend to do things the way we usually do

New project? Sure, I can run a usability test!

Habit Trap Can lead us to do the wrong research Your favorite research method may not be the best way to learn what you need to know

The best way to find that out would be a site visit…

How to Avoid the Habit Trap Look at every project as unique Consider what you need to learn Identify the best method

Look at every project as unique

Consider what you need to learn

Identify the best method

If Yes = guess most people would agree If No = guess most people wouldn’t agree Eat at Joe’s

If Yes = guess most people would agree

If No = guess most people wouldn’t agree

False Consensus We tend to attribute our beliefs, thoughts and behaviors to others

False Consensus

We tend to attribute our beliefs, thoughts and behaviors to others

Hi guys! Wanna be in a study?

False Consensus Trap Test with the wrong participants Other people may not think like you…

False Consensus Trap

Test with the wrong participants

Junior League of Palo Alto

Avoiding the False Consensus Trap Focus on the customer spend time watching and talking Test with people who aren’t like you

Focus on the customer

spend time watching and talking

Test with people who aren’t like you

What is the rule? 2, 4, 6, 8, ___ Hypotheses: Each value must be 2 higher than the one before. How do you test this hypothesis? Actual Rule: Each value must be any number bigger than the one before.

Congruence Bias We jump to conclusions by only looking at one approach

Congruence Bias

Try out our new idea for an iPhone application!

Congruence Trap Only trying one solution may miss a better one insufficiently inform the design

Congruence Trap

Try out each of these iPhone applications!

Avoiding the Congruence Trap Test several different solutions Test out what shouldn’t work

Test several different solutions

Test out what shouldn’t work

Which card do you turn over? Hypothesis: The back of every “6” is an “L”. T 6 L 4 Did you say “T”?

Hypothesis:

The back of every “6” is an “L”.

Confirmation Error Bias We have a tendency to search for data to confirm expectations

Confirmation Error Bias

We have a tendency to search for data to confirm expectations

As I suspected! 25% of users can’t do that task!

Confirmation Trap Get an incomplete picture of the data Seeking to prove our ideas can lead to missed surprises

Confirmation Trap

Get an incomplete picture of the data

Quicken used by small business

Avoiding the Confirmation Trap Look for surprises, instead of what you expect Test out what shouldn’t work Consider independent evaluation

Look for surprises, instead of what you expect

Test out what shouldn’t work

Consider independent evaluation

In American English are there more: There are 3x as many words with “K” in the 3 rd position

Availability Heuristic We have a tendency to put too much weight on what comes easily to mind

Availability Heuristic

We have a tendency to put too much weight on what comes easily to mind

Availability Trap in Research … a lot of people had trouble with that task…

Availability Trap Draw inaccurate conclusions What comes to mind easiest may not be the most important or most frequent finding.

Availability Trap

Draw inaccurate conclusions

Hmm… I didn’t realize that happened so often…

Avoiding the Availability Trap Gather key usability metrics (task success, specific error counts, time) Don’t rely on your memory Look at all of the data

Gather key usability metrics

(task success, specific error counts, time)

Don’t rely on your memory

Look at all of the data

Last 3 movies Prior 3 movies

Last 3 movies

Prior 3 movies

Recency Bias We tend to put too much weight on what we saw most recently

Recency Bias

Recency Bias in Research Click “ Continue” Participant 5 Click “ Continue” Participant 4 Click “Done” Participant 3 Click “Done” Participant 1 Click “ Cancel” Participant 2 Most people clicked “Continue”

Recency Trap What you saw most recently may not be the most important or most frequent finding. Draw inaccurate conclusions

Recency Trap

Hmm… I didn’t realize that happened so often…

How to Avoid the Recency Trap Gather key usability metrics (task success, specific error counts, time) Don’t rely on your memory Look at all of the data

Gather key usability metrics

(task success, specific error counts, time)

Don’t rely on your memory

Look at all of the data

Weight

Illusory Correlation The tendency to find patterns where none exist

Illusory Correlation

This is the third guy who uses a laptop in his living room. Maybe all men use laptops in their living rooms

Illusory Correlation Trap Things that co-occur may not be related. Draw inaccurate conclusions

Illusory Correlation Trap

That’s the fourth man who has bought this version. I need to find out how many men buy this…

Avoiding the Illusory Correlation Trap Recognize the limitations of your research methods Verify magnitude estimations and correlations with large-scale quantitative studies

Recognize the limitations of your research methods

Verify magnitude estimations and correlations with large-scale quantitative studies

Ways to avoid the traps… Planning Look at every project as unique Consider what you need to learn Identify the best method Test with people who aren’t like you Test several different solutions Test out what shouldn’t work

Planning

Look at every project as unique

Consider what you need to learn

Identify the best method

Test with people who aren’t like you

Test several different solutions

Test out what shouldn’t work

Ways to avoid the traps… Conducting Look for surprises, instead of what you expect Gather key usability metrics Consider independent evaluation

Conducting

Look for surprises, instead of what you expect

Gather key usability metrics

Consider independent evaluation

Ways to avoid the traps… Analyzing Don’t rely on your memory Look at all of the data Recognize the limitations of your research methods Verify magnitude estimations and correlations with large-scale quantitative studies

Analyzing

Don’t rely on your memory

Look at all of the data

Recognize the limitations of your research methods

Verify magnitude estimations and correlations with large-scale quantitative studies

QUESTIONS? [email_address]