How confronting cognitive biases can help us do better user research. Often it is non-researchers doing UX research with constrains of time/resources and quantity is prioritized over quality. But recognizing some of the key cognitive biases can help us get more out of less. This presentation talks about biases, gives examples and suggests ways to avoid these biases.

1. Beware Bias
Path to better UX research
Nidhi Khanwalkar
23rd March

2. A man and his son are in a terrible accident.The
father dies and the son is rushed to the hospital in
critical care.The surgeon looks at the boy and
exclaims "I can't operate on this boy, he's my son!"
How could this be?

3. This is us… a few thousand years ago..

5. What does this have to do with UX
research?

6. BIAS
1. Confirmation BIAS
2. Serial-Position & Peak BIAS
3. Availability & Selection BIAS
4. Anchoring BIAS
5. Observer Expectancy BIAS

7. BIAS
1. Confirmation BIAS
2. Serial-Position & Peak BIAS
3. Availability & Selection BIAS
4. Anchoring BIAS
5. Observer Expectancy BIAS
WHY: Framing
|
WHO: Participants
|
HOW: Questions
|
WHAT: Analysis
|
SOWHAT: Insight
UX Research

8. “We took your feedback and have simplified the Home Page.
Do you like it now?”
“How useful was today’s session”?
Tell me about your experience
with today’s session?

9. Confirmation BIAS
The tendency to interpret new information so that it becomes compatible with
our existing theories, beliefs and convictions.

10. “Students will use this device”
“Please tell us what are 2
things that attracted you
to the product?”
70
33
27
17
13
10
Hybrid/detachable/conv
ertible/laptop & tablet
Having touchscreen
function
User-
friendly/comfortable to…
Size of product
Speed/performance
Good picture
quality/resolution
What was your experience with the quality
of the screen display:
60% satisfied
What will you replace with this device:
• 40% say nothing
• 40% say tablet
• 20% laptop
not use

12. WHY: Framing
WHO: Participants
HOW: Questions
WHAT: Analysis
SOWHAT: Insight
Reframe/question your hypothesis
Get feedback from other people
List your assumption
Wider & representative mix of participants
Check your emotions, watch your body
language. Keep a poker face!!
Non leading, neutral questions
Do not look for data only to prove a hypothesis
Balanced scales
Be open to iterate

14. SERIAL-POSITION EFFECT
We pay more attention to the earlier and
later parts of long lists.
PEAK END RULE BIAS
We tend to judge an experience more on
how we felt at its most intense peak/point
rather than on its average

15. A usability test with 10 users
First time user
OR
Advanced user
First time user
OR
Advanced user
1 2 3 4 5 6 7 8 9 10

16. Card Sorting

17. A usability test with 10 users
Extremely articulate
Came in a bad mood
Thrashed your design
JUST MORE MEMORABLE!!
1 2 3 4 5 6 7 8 9 10

18. Often overemphasis on a certain
experience at the start or end
Quit too soon
Prioritizations, selection from a
list can be biased
Recounts only the most available
or memorable experience
Researcher User

19. WHY: Framing
WHO: Participants
HOW: Questions
WHAT: Analysis
SOWHAT: Insight
What evidence is good enough?
Run the test for a prescribed time
Randomize Q&A sequence
Consider all evidence equally – do not overemphasis a
certain experience
Triangulate with other methods/ data

21. Not equal representation
(some intended exceptions)

22. Information USED
• Recent
• Frequent
• Extreme/peak
• Vivid
• Negative (loss)
ALL the Information
AVAILABILITY BIAS
Examples of things that come readily to mind are
more representative than is actually the case
When the researcher decides who is
going to be studied
SELECTION BIAS
AvailableRefuse
Not available
Filtered
Studied
Who we RECRUIT
Participants recruitment
Who we should talk to?

23. HOWWE RECRUIT PARTICIPANTS
Based on who they are
Early user
Beta User
Signed up for
deals, offers
Specific ageOn social media

24. Log in at a specific time Community
Provide
feedback
HOWWE RECRUIT PARTICIPANTS
Based on their activity

25. Can’t make it
to research
Not online
HOWWE RECRUIT PARTICIPANTS
Based on who is available or not available

26. WHY: Framing
WHO: Participants
HOW: Questions
WHAT: Analysis
SOWHAT: Insight
What do I know about my customer/ user
Identify & recruit for missing groups
Recruit at different places, channels
Random Sampling
Short lists
Do not overemphasis a certain experience
Only communicate whats top of mind

27. “How much would you be willing to pay for a yearly
subscription to our premium service?
- Less than 20$ /year
- 21-50$ /year
- 51-75$ /year
- 76-100$ /year
- Over 100$ /year
- Less than 50$ /year
- 50-75$ /year
- 76-100$ /year
- 100-150$ /year
- Over 150$ / year
“How much would you be willing
to pay for a yearly subscription to
our premium service?”
Rs.

28. 1. How many hours do you spend
on the app?
2. Are you aware of XYZ
features/ offers/ services?
1. Are you aware of XYZ
features/ offers/ services?
2. How many hours do you spend
on the app?
False
POSITIVE

30. Anchoring BIAS
Order of Information
Scales
Sequence of Questions
People tend to focus on a single, initial piece of information, which
influences how they estimate value and make subsequent decisions.

31. WHO: Participants
HOW: Questions
WHAT: Analysis
SOWHAT: Insight
Randomize participants & stimulus
Be neutral and use balanced scales.
Ask Open ended questions or use appropriate ranges
Randomize Q&A sequence
Probe with “W”: what, why, how
Make an analysis plan
Be Collaborative
Write customer stories

32. OBERVER EXPECTANCY/ HAWTHORNE EFFECT
The alteration of behavior by the subjects of a study due to their awareness of
being observed.
Dance like no one is watching !!
Acquiescence bias or friendliness
bias: Reciprocity, Authority,
Fatigue
Social desirability bias or social
acceptability bias

33. Talk less, don’t interrupt Listen more
Ask less, fewer Qs Observe more

34. HOW: Questions
SOWHAT: Insight
Warm up: ask your big “question”
then Shut up!!
Probe with “W”: what, why, how
Write customer stories
Smaller experiments, sequential
recycling or use of story board,
paper prototypes

35. WHY: Framing
What do already know? Am I focusing on
right problem
WHO: Participants
Am I speaking to the right people
HOW: Questions
Am I asking the right questions
WHAT: Analysis
Am I analyzing to fit for a hypothesis
SOWHAT: Insight
What did I learn about the customer/ user
Reframe/question your hypothesis/assumptions, get feedback
What evidence is good enough?
What do I know about my customer/ user
|
Wider & representative mix of participants
Identify & recruit for missing groups at different places, channels
Random Sampling
Randomize participants & stimulus
Run the test for a prescribed time
|
Keep a poker face!! Ask non-leading, neutral questions
Short lists
Ask open ended Qs, Randomize Q&A sequence. Use appropriate
ranges, be neutral, use balanced scales
Talk less listen more . Probe with “W”: what, why, how
|
Look for data only to prove a hypothesis
Consider all evidence equally
Do not overemphasis a certain experience
Make an analysis plan & be Collaborative
|
Be open to iterate
Write customer stories
Only communicate whats top of mind
Triangulate with other methods/ data

36. PracticeAware Identify

37. https://twitter.com/nidhik
Nidhi Khanwalkar
https://www.linkedin.com/in/nidhikhanwalkar/