Research Design
Gaining deeper insight
Colleen Roller
Author & Practitioner
Areas of research
 Behavioral decision theory
 The study of how people judge, decide and behave
 Cognitive psychology
...
No neutral design
 Preferences, judgments, and
decisions are remarkably sensitive to
the context in which they are made.
No neutral design
No neutral design
No neutral design
This presentation
 Uses research findings to inform
research design
Clever Hans
The power of the subconscious
Rosenthal/Jacobsen 1968
‘When we expect certain behaviors of others,
we are likely to act in ways that make the
expected behavior more likely to o...
Confirmation bias
 The tendency to seek out and assign more
weight to evidence that confirms one’s
hypothesis, and ignore...
So what?
 We may be influencing participants –
and research findings – in ways
we’re not even aware of.
FRAMING
What is framing?
Framing
‘I’ve got great news. You managed
to avoid a salary decrease.’
Frame the study
 The purpose of this study is to test
the design, not you.
Child custody
Parent A
Average income
Average health
Average working hours
Reasonable rapport with the child
Relatively st...
Child custody
Parent A
Average income
Average health
Average working hours
Reasonable rapport with the child
Relatively st...
Positive/negative
 What did you like/dislike…
 How useful, or not,…
 How confident, or not…
 Why would you continue/st...
How we ask the question
 Positive/negative
 Breadth/depth – broad/narrow
 Did you notice anything different
about this process…
 How does this process compare to
how you currently do this?
 What information do you need in
order to feel confident about
doing x…
 What would make you feel
confident about doing ...
 How often do you use the x
website…
 Tell me about how and when you
use the x website…
Counterintuitive frame
 What do you think of the brand?
 What does the brand think of you?
‘A small question can never
hold a large answer.’
- Gerald Zaltman (How Customers Think)
Assumptions
 Be careful that your assumptions
aren’t built into the question
 Turn your assumptions into questions
Rating scales
 How successful have you been in life,
so far?
0 1 2 3 4 5 6 7 8 9 10
Not satisfied Very satisfied
-5 -4 -3...
Scholarships
 Two scholarships are available.
 You prefer a scholarship with less waiting
time and more money.
 Which o...
Graphical scale
 Figure 1: wait time salient – preferred A
 Figure 2: $ amount salient – preferred B
Sun, Li, Bonini 2010
Graphical comparison
0 10 20 30 40
Toothpaste B
Toothpaste A
People (per 5000) with serious gum disease
E.R. Stone et al. ...
Graphical comparison
Toothpaste A
30 out of 5000
Toothpaste B
15 out of 5000
E.R. Stone et al. 2003
So what?
 How does the frame itself influence the
meaning that people take away?
ORDERING OF QUESTIONS
Survey design
 Please rate your overall satisfaction.
 How many times have you had
repairs?
German students
 How happy are you these days?
 How many dates did you have last month?
Strack et al, 1988
Reverse order
 How happy are you these days?
 How many dates did you have last month?
 How many dates did you have last...
‘The sequential arrangement of information often
creates the very meaning of that information.
The order of presentation d...
So what?
 In what ways are you creating
(unintended) meaning?
 What are you measuring?
THE POWER OF WORDS
Different meanings
 Gambling v. gaming
 Liberal v. progressive
 Liquor v. spirits
 Used v. pre-owned
 Etc.
An auto accident
 At what speed did car A contact car B?
 At what speed did car A smash into car B?
Loftus/Palmer 1974
3...
Power of words
 Make contact
 Smash into
 Crash into
 Bump into
 Run into
 Collide with
So what?
 How are your words impacting
 user perception – and consequently, research
findings?
 how your business partn...
Delivering findings
 Users were…
 Aggravated…
 Annoyed…
 Confused…
 Disappointed…
 Frustrated…
 Irritated…
 Surpri...
Neutral
 Users…
 Said…
 Expected…
 Thought…
 Anticipated…
IN SUMMARY
Research design
‘The function of a research design is to ensure
that the evidence obtained enables us to answer
the initia...
The design matters
There is no neutral design
‘All research methods involve
compromises with reality.’
- Gerald Zaltman (How Customers Think)
The solution
 Be aware
 Control what we can
 How we ask questions
 Ordering of questions
 Choice of words
 Assumptio...
 UXmatters.com
 UX magazine (uxmag.com)
Colleen Roller
Upcoming SlideShare
Loading in …5
×

Insights on Research Design

600
-1

Published on

Colleen Roller's presentation from the UXPA Boston 2014 conference.

Published in: Design, Technology, Education
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
600
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
23
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Insights on Research Design

  1. 1. Research Design Gaining deeper insight Colleen Roller Author & Practitioner
  2. 2. Areas of research  Behavioral decision theory  The study of how people judge, decide and behave  Cognitive psychology  The study of how people perceive, think, remember and learn  Social psychology  The study of how people are influenced and affected by other people
  3. 3. No neutral design  Preferences, judgments, and decisions are remarkably sensitive to the context in which they are made.
  4. 4. No neutral design
  5. 5. No neutral design
  6. 6. No neutral design
  7. 7. This presentation  Uses research findings to inform research design
  8. 8. Clever Hans
  9. 9. The power of the subconscious Rosenthal/Jacobsen 1968
  10. 10. ‘When we expect certain behaviors of others, we are likely to act in ways that make the expected behavior more likely to occur.’ - Rosenthal and Babad
  11. 11. Confirmation bias  The tendency to seek out and assign more weight to evidence that confirms one’s hypothesis, and ignore or underweight evidence that could disconfirm it.
  12. 12. So what?  We may be influencing participants – and research findings – in ways we’re not even aware of.
  13. 13. FRAMING
  14. 14. What is framing?
  15. 15. Framing ‘I’ve got great news. You managed to avoid a salary decrease.’
  16. 16. Frame the study  The purpose of this study is to test the design, not you.
  17. 17. Child custody Parent A Average income Average health Average working hours Reasonable rapport with the child Relatively stable social life Shafir, Simonson, Tversky, 1993  Which parent should get sole custody? Award – 64% Parent B Above-average income Minor health problems Lots of work-related travel Very close relationship with the child Extremely active social life
  18. 18. Child custody Parent A Average income Average health Average working hours Reasonable rapport with the child Relatively stable social life Shafir, Simonson, Tversky, 1993  Which parent should not get sole custody? Award – 64% Deny – 55% Parent B Above-average income Minor health problems Lots of work-related travel Very close relationship with the child Extremely active social life
  19. 19. Positive/negative  What did you like/dislike…  How useful, or not,…  How confident, or not…  Why would you continue/stop using…
  20. 20. How we ask the question  Positive/negative  Breadth/depth – broad/narrow
  21. 21.  Did you notice anything different about this process…  How does this process compare to how you currently do this?
  22. 22.  What information do you need in order to feel confident about doing x…  What would make you feel confident about doing x…
  23. 23.  How often do you use the x website…  Tell me about how and when you use the x website…
  24. 24. Counterintuitive frame  What do you think of the brand?  What does the brand think of you?
  25. 25. ‘A small question can never hold a large answer.’ - Gerald Zaltman (How Customers Think)
  26. 26. Assumptions  Be careful that your assumptions aren’t built into the question  Turn your assumptions into questions
  27. 27. Rating scales  How successful have you been in life, so far? 0 1 2 3 4 5 6 7 8 9 10 Not satisfied Very satisfied -5 -4 -3 -2 -1 0 1 2 3 4 5 Not satisfied Very satisfied 34% 13% Schwartz et al, 1991
  28. 28. Scholarships  Two scholarships are available.  You prefer a scholarship with less waiting time and more money.  Which one would you pick? Sun, Li, Bonini 2010
  29. 29. Graphical scale  Figure 1: wait time salient – preferred A  Figure 2: $ amount salient – preferred B Sun, Li, Bonini 2010
  30. 30. Graphical comparison 0 10 20 30 40 Toothpaste B Toothpaste A People (per 5000) with serious gum disease E.R. Stone et al. 2003
  31. 31. Graphical comparison Toothpaste A 30 out of 5000 Toothpaste B 15 out of 5000 E.R. Stone et al. 2003
  32. 32. So what?  How does the frame itself influence the meaning that people take away?
  33. 33. ORDERING OF QUESTIONS
  34. 34. Survey design  Please rate your overall satisfaction.  How many times have you had repairs?
  35. 35. German students  How happy are you these days?  How many dates did you have last month? Strack et al, 1988
  36. 36. Reverse order  How happy are you these days?  How many dates did you have last month?  How many dates did you have last month?  How happy are you these days? Strack et al, 1988
  37. 37. ‘The sequential arrangement of information often creates the very meaning of that information. The order of presentation determines the reaction.’ - Dr. Frank Luntz (Words That Work)
  38. 38. So what?  In what ways are you creating (unintended) meaning?  What are you measuring?
  39. 39. THE POWER OF WORDS
  40. 40. Different meanings  Gambling v. gaming  Liberal v. progressive  Liquor v. spirits  Used v. pre-owned  Etc.
  41. 41. An auto accident  At what speed did car A contact car B?  At what speed did car A smash into car B? Loftus/Palmer 1974 31.8 mph 40.5 mph Actual: 12 mph
  42. 42. Power of words  Make contact  Smash into  Crash into  Bump into  Run into  Collide with
  43. 43. So what?  How are your words impacting  user perception – and consequently, research findings?  how your business partners perceive the findings?
  44. 44. Delivering findings  Users were…  Aggravated…  Annoyed…  Confused…  Disappointed…  Frustrated…  Irritated…  Surprised…
  45. 45. Neutral  Users…  Said…  Expected…  Thought…  Anticipated…
  46. 46. IN SUMMARY
  47. 47. Research design ‘The function of a research design is to ensure that the evidence obtained enables us to answer the initial question as unambiguously as possible.’
  48. 48. The design matters There is no neutral design
  49. 49. ‘All research methods involve compromises with reality.’ - Gerald Zaltman (How Customers Think)
  50. 50. The solution  Be aware  Control what we can  How we ask questions  Ordering of questions  Choice of words  Assumptions and their effect  Use of ‘scale’ – charts, graphs, rating scales  Comparison – graphs, ratings
  51. 51.  UXmatters.com  UX magazine (uxmag.com) Colleen Roller

×