SlideShare a Scribd company logo
Pete Cape, SSI, UK
April Lecture Series 2014
Questionnaire Design – A Masterclass
Pete Cape
Global Knowledge Director, SSI
2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
Pete Cape, SSI, UK
April Lecture Series 2014
| 2 |
A questionnaire might have all of these….
• Open questions
• Yes/No questions
• Single coded questions with multiple answers
• Multi-coded questions with multiple answers
• Items to be ranked
• Items to be rated
• Items to be chosen between
Pete Cape, SSI, UK
April Lecture Series 2014
| 3 |
A questionnaire might have all of these….
• Open questions
• Yes/No questions
• Single coded questions with multiple answers
• Multi-coded questions with multiple answers
• Items to be ranked
• Items to be rated
• Items to be chosen between
• But this is such a boring way of looking at it….
Pete Cape, SSI, UK
April Lecture Series 2014
| 4 |
A questionnaire might have all of these….
• Open questions
• Yes/No questions
• Single coded questions with multiple answers
• Multi-coded questions with multiple answers
• Items to be ranked
• Items to be rated
• Items to be chosen between
• But this is such a boring way of looking at it….
• Let’s start with some principles….
Pete Cape, SSI, UK
April Lecture Series 2014
| 5 |
Things to remember about respondents….
Pete Cape, SSI, UK
April Lecture Series 2014
| 6 |
Things to remember about respondents….
• They know they are in an experiment
Pete Cape, SSI, UK
April Lecture Series 2014
| 7 |
Things to remember about respondents….
• They know they are in an experiment
• Everything they see, hear or read is meant deliberately, and has
meaning
Pete Cape, SSI, UK
April Lecture Series 2014
| 8 |
Things to remember about respondents….
• They know they are in an experiment
• Everything they see, hear or read is meant deliberately, and has
meaning
• They want to please
Pete Cape, SSI, UK
April Lecture Series 2014
| 9 |
Things to remember about respondents….
• They know they are in an experiment
• Everything they see, hear or read is meant deliberately, and has
meaning
• They want to please
• They think you know what you are doing…
Pete Cape, SSI, UK
April Lecture Series 2014
Pete Cape, SSI, UK
April Lecture Series 2014
Pete Cape, SSI, UK
April Lecture Series 2014
| 12 |
Things to remember about questionnaires….
• Everything has meaning
Pete Cape, SSI, UK
April Lecture Series 2014
| 13 |
Things to remember about questionnaires….
• Everything has meaning
• You are supposed to know what that meaning is
Pete Cape, SSI, UK
April Lecture Series 2014
| 14 |
Questionnaires do a lot of work
• Motivate the respondent
Pete Cape, SSI, UK
April Lecture Series 2014
| 15 |
Questionnaires do a lot of work
• Motivate the respondent to provide
— complete answers
— accurate answers
Pete Cape, SSI, UK
April Lecture Series 2014
| 16 |
Questionnaires do a lot of work
• Motivate the respondent to provide
— complete answers
— accurate answers
• Communicate to the respondent
Pete Cape, SSI, UK
April Lecture Series 2014
| 17 |
Questionnaires do a lot of work
• Motivate the respondent to provide
— complete answers
— accurate answers
• Communicate to the respondent
• Help the respondent work out their answers
Pete Cape, SSI, UK
April Lecture Series 2014
| 18 |
Questionnaires do a lot of work
• Motivate the respondent to provide
— complete answers
— accurate answers
• Communicate to the respondent
• Help the respondent work out their answers
• Make the respondents task easy
Pete Cape, SSI, UK
April Lecture Series 2014
| 19 |
Is this the simplest question of all?
• Yes
• No
Pete Cape, SSI, UK
April Lecture Series 2014
| 20 |
Respondents want to please
Do you care to
disagree with
me….?
I am a scientist…..
• Acquiescence bias
Pete Cape, SSI, UK
April Lecture Series 2014
Acquiescence bias
• Some respondents are simply agreeable, and indicate
agreement out of politeness.
Saris, Krosnick and Shaeffe
Pete Cape, SSI, UK
April Lecture Series 2014
Acquiescence bias
• Some respondents are simply agreeable, and indicate
agreement out of politeness.
• Other respondents expect that the researchers agree with the
listed items and defer to their judgment.
Saris, Krosnick and Shaeffe
Pete Cape, SSI, UK
April Lecture Series 2014
Acquiescence bias
• Some respondents are simply agreeable, and indicate
agreement out of politeness.
• Other respondents expect that the researchers agree with the
listed items and defer to their judgment.
• Most respondents engage in survey satisficing and find that
agreeing takes less effort than carefully weighing each
optional level of disagreement and agreement.
Saris, Krosnick and Shaeffe
Pete Cape, SSI, UK
April Lecture Series 2014
Affirmation vs Multicode
Pete Cape, SSI, UK
April Lecture Series 2014
Affirmation vs Multicode
Pete Cape, SSI, UK
April Lecture Series 2014
Overcoming acquiescence bias
• Find them through traps and exclude
Pete Cape, SSI, UK
April Lecture Series 2014
Overcoming acquiescence bias
• Find them through traps and exclude
• Follow up question “how sure?”
• If not 100% sure then recode as “no”
Pete Cape, SSI, UK
April Lecture Series 2014
Overcoming acquiescence bias
• Find them through traps and exclude
• Follow up question “how sure?”
• If not 100% sure then recode as “no”
• Avoid the question type and give alternatives
— “chose the one of these that best describes you”
— “to what extent do you…”
Pete Cape, SSI, UK
April Lecture Series 2014
Affirmation vs Multicode
Pete Cape, SSI, UK
April Lecture Series 2014
Affirmation vs Multicode
why so big…?
Pete Cape, SSI, UK
April Lecture Series 2014
Social Desirability bias
• Wanting to look good
Pete Cape, SSI, UK
April Lecture Series 2014
Social Desirability bias
• Wanting to look good
— To society
— To other individuals
— To myself
Pete Cape, SSI, UK
April Lecture Series 2014
Some Normal Everyday Things…that I keep to myself..
Chart Title
Pete Cape, SSI, UK
April Lecture Series 2014
Some Normal Everyday Things…I want you to think I do..
Chart Title
Pete Cape, SSI, UK
April Lecture Series 2014
20% bought a book last week?
| 35 |
Pete Cape, SSI, UK
April Lecture Series 2014
Avoiding Social Desirability
• Is very hard
• Try to make respondents feel “normal”
Pete Cape, SSI, UK
April Lecture Series 2014
Avoiding Social Desirability
• Is very hard
• Try to make respondents feel “normal”
This third section is about social attitudes and values. There is a
lot of talk about discrimination and racial prejudice, and different
people have different views on the size of such problems, their
causes and what should be done about them. There are no right
or wrong answers, it is your honest opinions we are looking for. If
you do not wish to answer any of these questions please use the
option “prefer not to say”.
Pete Cape, SSI, UK
April Lecture Series 2014
Questionnaire Design and Bias
• Questionnaire design has to overcome respondents’
psychological biases
Pete Cape, SSI, UK
April Lecture Series 2014
Questionnaire Design and Bias
• Questionnaire design has to overcome respondents’
psychological biases
• And also experimenter bias
Pete Cape, SSI, UK
April Lecture Series 2014
Questionnaire Design and Bias
• Questionnaire design has to overcome respondents’
psychological biases
• And also experimenter bias
• i.e. you and your client….
Pete Cape, SSI, UK
April Lecture Series 2014
Experimenter biases
• It is easy to suggest the answer…
Pete Cape, SSI, UK
April Lecture Series 2014
Question suggests the answer
Which carbonated soft drinks, like Coca-Cola or Pepsi
for example, do you drink regularly?”
Pete Cape, SSI, UK
April Lecture Series 2014
Question suggests the answer
Which carbonated soft drinks, like Coca-Cola or Pepsi
for example, do you drink regularly?”
Raises answers for drinks
“like” Pepsi and Coke
i.e. Brown or cola drinks
Pete Cape, SSI, UK
April Lecture Series 2014
Question suggests the answer
Which carbonated soft drinks, like Coca-Cola or Pepsi
for example, do you drink regularly?”
Raises answers for drinks
“like” Pepsi and Coke
i.e. Brown or cola drinks
Lowers answers for non-
brown drinks
e.g. Fanta, 7-Up
Pete Cape, SSI, UK
April Lecture Series 2014
Question suggests the answer
Which carbonated soft drinks, like Coca-Cola or Pepsi
for example, do you drink regularly?”
Raises answers for drinks
“like” Pepsi and Coke
i.e. Brown or cola drinks
Lowers answers for non-
brown drinks
e.g. Fanta, 7-Up
Problem is in use of jargon term: “carbonated soft drinks”
Pete Cape, SSI, UK
April Lecture Series 2014
Question suggests “my” answer
Testing perfumes on animals involves causing them
pain. Do you agree with testing perfumes on animals?
Pete Cape, SSI, UK
April Lecture Series 2014
Question suggests “my” answer
Testing perfumes on animals involves causing them
pain. Do you agree with testing perfumes on animals?
“involves causing them pain” is an opinion
Pete Cape, SSI, UK
April Lecture Series 2014
Avoiding Experimenter Bias
• Is easier
Pete Cape, SSI, UK
April Lecture Series 2014
Avoiding Experimenter Bias
• Is easier
• Think neutral
Pete Cape, SSI, UK
April Lecture Series 2014
Avoiding Experimenter Bias
• Is easier
• Think neutral
• Don’t think your view is the world view
Pete Cape, SSI, UK
April Lecture Series 2014
Questionnaire as a measuring tool
• We rarely measure physical things
Pete Cape, SSI, UK
April Lecture Series 2014
Questionnaire as a measuring tool
• We rarely measure physical things
• But we do measure frequency:
— How often do you x, y, z?
Pete Cape, SSI, UK
April Lecture Series 2014
Questionnaire as a measuring tool
• We rarely measure physical things
• But we do measure frequency:
— How often do you x, y, z?
• And we do measure attitudes:
— How much do you think x,y,z?
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• As an relative concept in a fixed time period
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• As a relative concept in a fixed time period
• Define:
— The activity
• What does it mean to “go shopping”?
• What does it mean to “surf the internet?”
— The time period
• Is it the relevant one?
• What is an “average week”?
— The measurement scale
• Always, often, sometimes, never…..
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• As an absolute number in a fixed time period
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• As an absolute number in a fixed time period
• Define:
— The activity
• What does it mean to “go shopping”?
• What does it mean to “surf the internet?”
— The time period
• Is it the relevant one?
• What is an “average week”?
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• As an absolute number in a fixed time period
How many times did you take a bath in the past year?
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• As an absolute number in a fixed time period
How many times did you take a bath in the past year?
too longwhat does
this mean?
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• Precision and correct framing
How many times, if at all, did you take a bath in the past week? By
taking a bath we mean actually using a bath tub and filling it with
water and sitting in it. Please exclude any showers you may have taken
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• Precision and correct framing
How many times, if at all, did you take a bath in the past week? By
taking a bath we mean actually using a bath tub and filling it with
water and sitting in it. Please exclude any showers you may have taken
OMG, I only take showers,
he will think I am so dirty
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• Precision and correct framing, and overcoming social desirability bias
Some people only take baths, some only take showers, some take both.
This part of the survey is only concerned with bath taking.
How many times, if at all, did you take a bath in the past week? By
taking a bath we mean actually using a bath tub and filling it with
water and sitting in it. Please exclude any showers you may have taken
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• Precision and correct framing, and overcoming social desirability bias, and to
make life easy
Some people only take baths, some only take showers, some take both.
This part of the survey is only concerned with bath taking.
How many times, if at all, did you take a bath in the past week? By
taking a bath we mean actually using a bath tub and filling it with
water and sitting in it. Please exclude any showers you may have taken
0 / only took showers
1 – 2
3 – 4
5 – 6
7 or more
Can’t remember
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring frequency
• Precision and correct framing, and overcoming social desirability bias, and to
make life easy
Some people only take baths, some only take showers, some take both.
This part of the survey is only concerned with bath taking.
How many times, if at all, did you take a bath in the past week? By
taking a bath we mean actually using a bath tub and filling it with
water and sitting in it. Please exclude any showers you may have taken
0 / only took showers
1 – 2
3 – 4
5 – 6
7 or more
Can’t remember
Does this have any meaning?
Pete Cape, SSI, UK
April Lecture Series 2014
The meaning of ranges
Pete Cape, SSI, UK
April Lecture Series 2014
The meaning of ranges
• Assumption of normal distribution
Pete Cape, SSI, UK
April Lecture Series 2014
The meaning of ranges
• Assumption of normal distribution
Pete Cape, SSI, UK
April Lecture Series 2014
Manipulating numbers
Answer list A Answer list B
Pete Cape, SSI, UK
April Lecture Series 2014
Manipulating numbers
Answer list A Answer list B
Pete Cape, SSI, UK
April Lecture Series 2014
Manipulating numbers
Answer list A Answer list B
Pete Cape, SSI, UK
April Lecture Series 2014
Manipulating numbers
Answer list A
• The central banding should contain the mean/median
• But we don’t know it, it’s what we’re trying to find out!
Answer list B
Pete Cape, SSI, UK
April Lecture Series 2014
Collecting real numbers
• Just ask for it
• Respondent knows it and will give it
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring attitudes
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring attitudes
Fill in the words that correspond to the scale points for “Likely”
0 = not at all likely
1 = ?
2 = ?
3 = ?
4 = ?
5 = ?
6 = ?
7 = ?
8 = ?
9 = ?
10 = extremely likely
?
?
Exercise
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring attitudes
Fill in the words that correspond to the scale points for “Likely”
0 = not at all likely
1 = ?
2 = ?
3 = ?
4 = ?
5 = ?
6 = ?
7 = ?
8 = ?
9 = ?
10 = extremely likely
Exercise
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring attitudes
Fill in the words that correspond to the scale points for “Likely”
0 = not at all likely
1 = ?
2 = ?
3 = ?
4 = ?
5 = ?
6 = ?
7 = ?
8 = ?
9 = ?
10 = extremely likely
Exercise
Exercise outcomes:
a) No-one will be able to do it
b) Some people will write:
5 = “neither likely not unlikely”
5 = “some degree of likelihood”
Pete Cape, SSI, UK
April Lecture Series 2014
Measuring attitudes
Fill in the words that correspond to the scale points for “Likely”
0 = not at all likely
1 = ?
2 = ?
3 = ?
4 = ?
5 = ?
6 = ?
7 = ?
8 = ?
9 = ?
10 = extremely likely
Exercise
Exercise outcomes:
a) No-one will be able to do it
b) Some people will write:
5 = “neither likely nor unlikely”
5 = “some degree of likelihood”
Pete Cape, SSI, UK
April Lecture Series 2014
Two men go into a bar….
Pete Cape, SSI, UK
April Lecture Series 2014
Two men go into a bar….
I say, what do
you think of
the new VW
Golf?
Pete Cape, SSI, UK
April Lecture Series 2014
Two men go into a bar….
I say, what do
you think of
the new VW
Golf?
I think
it’s a 6…
Pete Cape, SSI, UK
April Lecture Series 2014
Two men go into a bar….
Attitudes are expressed
in words not numbers
I say, what do
you think of
the new VW
Golf?
I think
it’s a 6…
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers are universal
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers are universal
• Numeric scales are not!
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers are universal
• Numeric scales are not!
• It all starts in school
— 0 = you weren’t even there!
— 1-4 = you might as well not have been there…
— 5-6 = you are going to have to work way harder in the future
— 7-8 = okay, could do better
— 9 = good
— 10 = very good
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers are universal
• Numeric scales are not!
• It all starts in school
— 0 = you weren’t even there!
— 1-4 = you might as well not have been there…
— 5-6 = you are going to have to work way harder in the future
— 7-8 = okay, could do better
— 9 = good
— 10 = very good
• And this is for cultures that score 0 - 10, does everyone?
Pete Cape, SSI, UK
April Lecture Series 2014
German School System
• 1 (sehr gut, very good) is the best possible grade and is given for
outstanding performance
• 2 (gut, good) is the next-highest and is given for performance that
meets the standard completely and is above-average
• 3 (befriedigend, satisfactory) indicates "average" performance.
• 4 (ausreichend, sufficient) is the lowest passing grade and is given if
the standard has been met but with a number of notable errors.
• 5 (mangelhaft, deficient) is the higher of two failing grades and is
given if the standard has not been met but the basics have been
understood.
• 6 (ungenügend, insufficient) is the lowest possible grade and is given
if the standard has not been met and the basics have not been
understood.
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers have meaning
0 1 2 3 4 5 6 7 8 9 10
Schwartz, 1996
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers have meaning
0 1 2 3 4 5 6 7 8 9 10
34%
Schwartz, 1996
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers have meaning
0 1 2 3 4 5 6 7 8 9 10
-5 -4 -3 -2 -1 0 1 2 3 4 5
34%
Schwartz, 1996
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers have meaning
0 1 2 3 4 5 6 7 8 9 10
-5 -4 -3 -2 -1 0 1 2 3 4 5
34%
13%?
Schwartz, 1996
Pete Cape, SSI, UK
April Lecture Series 2014
Numbers have meaning
0 1 2 3 4 5 6 7 8 9 10
-5 -4 -3 -2 -1 0 1 2 3 4 5
34%
13%
Schwartz, 1996
Pete Cape, SSI, UK
April Lecture Series 2014
This what scales are trying to achieve
Pete Cape, SSI, UK
April Lecture Series 2014
This what scales are trying to achieve
Pete Cape, SSI, UK
April Lecture Series 2014
• Turn to academia for the answer
How long should a scale be?
Pete Cape, SSI, UK
April Lecture Series 2014
• Turn to academia for the answer
• Is not always shared 
• But is somewhere between 5 and 9
points
How long should a scale be?
Pete Cape, SSI, UK
April Lecture Series 2014
• Turn to academia for the answer
• Is not always shared 
• But is somewhere between 5 and 9
points
• Jon Krosnick says:
• Unipolar: 5 points
• Bipolar: 7 points
How long should a scale be?
Pete Cape, SSI, UK
April Lecture Series 2014
• Turn to academia for the answer
• Is not always shared 
• But is somewhere between 5 and 9
points
• Jon Krosnick says:
• Unipolar: 5 points
• Bipolar: 7 points
• Why?
• Maximises Reliability and Validity
How long should a scale be?
Pete Cape, SSI, UK
April Lecture Series 2014
What is reliability and validity?
• Reliability
— Measures the same thing in the same way every time
Pete Cape, SSI, UK
April Lecture Series 2014
What is reliability and validity?
• Reliability
— Measures the same thing in the same way every time
• Validity
— Measures what you think it measures
Pete Cape, SSI, UK
April Lecture Series 2014
What was Unipolar and Bipolar?
“Unipolar”
One direction from zero to lots
he middle and surfeit at the ends
Pete Cape, SSI, UK
April Lecture Series 2014
What was Unipolar and Bipolar?
“Unipolar”
One direction from zero to lots
“Bipolar”
Opposites with indifference in the middle
he middle and surfeit at the ends
Pete Cape, SSI, UK
April Lecture Series 2014
Scales
Unipolar:
• Extremely x
• Very x
• Somewhat x
• Slightly x
• Not at all x
(an) academic conclusion
Pete Cape, SSI, UK
April Lecture Series 2014
Scales
Unipolar:
• Extremely x
• Very x
• Somewhat x
• Slightly x
• Not at all x
Bipolar:
• Extremely x
• Very x
• Somewhat x
• Neither x nor un-x
• Somewhat un-x
• Very un-x
• Extremely un-x
(an) academic conclusion
Pete Cape, SSI, UK
April Lecture Series 2014
Scales
Unipolar:
• Extremely x
• Very x
• Somewhat x
• Slightly x
• Not at all x
Bipolar:
• Extremely x
• Very x
• Somewhat x
• Neither x nor un-x
• Somewhat un-x
• Very un-x
• Extremely un-x
And this is what I recommend when asked....
btw: you can’t mix bi- and unipolar scales
(an) academic conclusion
Pete Cape, SSI, UK
April Lecture Series 2014
Scales
• Design issues
• Display issues
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put “don’t know”
• Visual and Conceptual midpoint
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put “don’t know”
• Visual and Conceptual midpoint
Conceptual midpoint
Visual
midpoint
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put “don’t know”
• Visual and Conceptual midpoint
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put “don’t know”
• Visual and Conceptual midpoint
Visual
midpoint
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put “don’t know”
• Visual and Conceptual midpoint
• Skews data towards “disagree”
Conceptual midpoint
Visual
midpoint
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put “don’t know”
• Visual and Conceptual midpoint
Conceptual midpoint Visual
midpoint
Pete Cape, SSI, UK
April Lecture Series 2014
What order to present?
Disagree
Strongly
Disagree
Slightly
Neither Agree
nor Disagree
Agree
Slightly
Agree
Strongly
Pete Cape, SSI, UK
April Lecture Series 2014
What order to present?
Disagree
Strongly
Disagree
Slightly
Neither Agree
nor Disagree
Agree
Slightly
Agree
Strongly
Order bias
Pete Cape, SSI, UK
April Lecture Series 2014
What order to present?
Disagree
Strongly
Disagree
Slightly
Neither Agree
nor Disagree
Agree
Slightly
Agree
Strongly
Order bias Acquiescence bias
Pete Cape, SSI, UK
April Lecture Series 2014
What order to present?
Disagree
Strongly
Disagree
Slightly
Neither Agree
nor Disagree
Agree
Slightly
Agree
Strongly
Order bias Acquiescence bias
Central tendency
Pete Cape, SSI, UK
April Lecture Series 2014
What order to present?
Disagree
Strongly
Disagree
Slightly
Neither Agree
nor Disagree
Agree
Slightly
Agree
Strongly
Order bias Acquiescence bias
Central tendency
Agree
Strongly Agree Slightly
Neither Agree
nor Disagree
Disagree
Slightly
Disagree
Strongly
Central tendency
Order bias
Acquiescence bias
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put the answer box
• Next to answers
• 22% failure rate on matching gender to panel record!
Pete Cape, SSI, UK
April Lecture Series 2014
Where to put the answer box
• Next to answers
• 22% failure rate on matching gender to panel record!
• Left right processing
• 98% of that: Males coded as Females
Pete Cape, SSI, UK
April Lecture Series 2014
Scales
• Maximises reliability and validity
• Minimises bias
• Prevents order error
• Not just “the way we always do it…”
Pete Cape, SSI, UK
April Lecture Series 2014
Scales
• Maximises reliability and validity
• Minimises bias
• Prevents order error
• Not just “the way we always do it…”
• But isn’t it old-fashioned?
Pete Cape, SSI, UK
April Lecture Series 2014
Sliders with visual elements
Pete Cape, SSI, UK
April Lecture Series 2014
129125
77
25
4
0
50
100
150
200
250
Different styles of slider scale
Pete Cape, SSI, UK
April Lecture Series 2014
Different styles of slider scale
114
8892
17
1
0
50
100
150
200
250
Pete Cape, SSI, UK
April Lecture Series 2014
Different styles of slider scale
97
21
27
8
1
0
50
100
150
200
250
Pete Cape, SSI, UK
April Lecture Series 2014
4
10
124
4751
0
50
100
150
200
250
Different styles of slider scale
Pete Cape, SSI, UK
April Lecture Series 2014
Different distributions, same means?
UK USA Germany China
traditional - raw data 2.27 2.32 2.50 3.79
traditional - rescored 2.60 2.28 2.50 3.87
slider with all labels 2.56 2.64 2.74 3.89
slider with all tick marks 2.33 2.35 2.46 3.61
slider with no tick marks 2.32 2.39 2.29 3.62
slider with no tick marks, score showing 2.34 2.44 2.33 3.56
Pete Cape, SSI, UK
April Lecture Series 2014
Grids and Matrices
• Q: Do we have these in telephone research?
• Q: In Face to Face?
Pete Cape, SSI, UK
April Lecture Series 2014
Grids and Matrices
• Q: Do we have these in telephone research?
• Q: In Face to Face?
• A: No
• Q: Where did we have them?
Pete Cape, SSI, UK
April Lecture Series 2014
Grids and Matrices
• Q: Do we have these in telephone research?
• Q: In Face to Face?
• A: No
• Q: Where did we have them?
• A: postal research
Pete Cape, SSI, UK
April Lecture Series 2014
Item-in-a-series
Pete Cape, SSI, UK
April Lecture Series 2014
Item-in-a-series
“When one has to ask a series of
questions that use the same answer
categories it is convenient to combine
them into a [grid/matrix format].
Combining them into this ‘item-in-a-
series’, with a common introduction
that defines the general question and
response format, eliminates
considerable redundancy with regard
to stating questions, it also saves
considerable space”
Pete Cape, SSI, UK
April Lecture Series 2014
Item-in-a-series
“but do it carefully”
Pete Cape, SSI, UK
April Lecture Series 2014
Item-in-a-series
“but do it carefully”
• Items are now in a comparative
framework
• Visual rendering makes them a unit
“if the sponsor wants individuals to
contemplate each item separately, it is
advisable to present each of them as
individual items”
Pete Cape, SSI, UK
April Lecture Series 2014
Item-in-a-series
Should these should be the same?
Pete Cape, SSI, UK
April Lecture Series 2014
Item-in-a-series
Shouldn’t these have been the same?
Pete Cape, SSI, UK
April Lecture Series 2014
| 136 |
What else do grids do?
• Encourage you to go too fast
Pete Cape, SSI, UK
April Lecture Series 2014
Speed of grid completion
•What sort of processing is being done at this speed?
| 137 |
Pete Cape, SSI, UK
April Lecture Series 2014
| 138 |
What else do grids do?
• Encourage you to go too fast
• Process the underlying latent construct, not the items
— Straightlining
Pete Cape, SSI, UK
April Lecture Series 2014
| 139 |
What else do grids do?
• Encourage you to go too fast
• Process the underlying latent construct, not the items
— Straightlining
• Miss the subtleties
— Heuristic answering
Pete Cape, SSI, UK
April Lecture Series 2014
Direct instruction at item 39/40 – pass/fail
| 140 |
Pete Cape, SSI, UK
April Lecture Series 2014
Which heuristic was used?
| 141 |
answer at trap
Pete Cape, SSI, UK
April Lecture Series 2014
Which heuristic was used?
| 142 |
answer at trap
Pete Cape, SSI, UK
April Lecture Series 2014
And those left over?
| 143 |
Pete Cape, SSI, UK
April Lecture Series 2014
And those left over?
| 144 |
Pete Cape, SSI, UK
April Lecture Series 2014
| 145 |
Anything else wrong with grids?
• Yes, we use the wrong scale for the construct
Pete Cape, SSI, UK
April Lecture Series 2014
Beware of the agree-disagree question
• Make your scale answers the appropriate ones
To what extent to you agree or disagree that this
training course is extremely useful?
Agree extremely
Very much agree
Somewhat agree
Neither agree nor disagree
Somewhat disagree
Very much disagree
Disagree extremely
Pete Cape, SSI, UK
April Lecture Series 2014
2 stage process
1. Decide how useful you find this
Pete Cape, SSI, UK
April Lecture Series 2014
2 stage process
1. Decide how useful you find this
2. Translate into agree-disagree, hopefully:
Extremely useful = Agree extremely
Very useful = Very much agree
Somewhat useful = Somewhat agree
Neither useful not disuseful = Neither agree nor disagree
Somewhat disuseful = Somewhat disagree
Very disuseful = Very much disagree
Extremely disuseful = Disagree extremely
Pete Cape, SSI, UK
April Lecture Series 2014
2 stage process
1. Decide how useful you find this
2. Translate into agree-disagree, hopefully:
• Disuseful?
Extremely useful = Agree extremely
Very useful = Very much agree
Somewhat useful = Somewhat agree
Neither useful not disuseful = Neither agree nor disagree
Somewhat disuseful = Somewhat disagree
Very disuseful = Very much disagree
Extremely disuseful = Disagree extremely
Pete Cape, SSI, UK
April Lecture Series 2014
2 stage process
• Reality (maybe):
Extremely useful = Agree extremely
Very useful = Very much agree
Somewhat useful = Somewhat agree
Slightly useful = Somewhat agree
Not at all useful = Neither/nor, Somewhat disagree,Very
much disagree,Disagree extremely
Pete Cape, SSI, UK
April Lecture Series 2014
How to measure attitudes/constructs
• Work out the underlying construct
— i.e. what are you measuring
Pete Cape, SSI, UK
April Lecture Series 2014
How to measure attitudes/constructs
• Work out the underlying construct
— i.e. what are you measuring
• Decide how it is measured
— What words
— Unipolar or Bipolar
Pete Cape, SSI, UK
April Lecture Series 2014
How to measure attitudes/constructs
• Work out the underlying construct
— i.e. what are you measuring
• Decide how it is measured
— What words
— Unipolar or Bipolar
• Check your measurement gaps are equal
• Ask question..
Pete Cape, SSI, UK
April Lecture Series 2014
How to measure attitudes/constructs
• Work out the underlying construct
— i.e. what are you measuring
• Decide how it is measured
— What words
— Unipolar or Bipolar
• Check your measurement gaps are equal
• Ask question..
• More work for you, less work for respondent
• And more valid and reliable
Pete Cape, SSI, UK
April Lecture Series 2014
Construct Specific Scale
How useful, if at all, did you find this training course?
Extremely useful
Very useful
Somewhat useful
Slightly useful
Not at all useful
• As you read the question your answer is forming…..
?
Pete Cape, SSI, UK
April Lecture Series 2014
Construct Specific Scale
How useful, if at all, did you find this training course?
Extremely useful
Very useful
Somewhat useful
Slightly useful
Not at all useful
• As you read the question your answer is forming…..
• Order effects?
Pete Cape, SSI, UK
April Lecture Series 2014
Construct Specific Scale
Extremely useful
Very useful
Somewhat useful
Slightly useful
Not at all useful
Pete Cape, SSI, UK
April Lecture Series 2014
Construct Specific Scale
Extremely useful
Very useful
Somewhat useful
Slightly useful
Not at all useful
Expectation: up = good
(Tourangeau)
Pete Cape, SSI, UK
April Lecture Series 2014
Construct Specific Scale
Extremely useful
Very useful
Somewhat useful
Slightly useful
Not at all useful
Expectation: up = good
(Tourangeau)
Answer speed = faster
(Christian et al)
No order bias found
Pete Cape, SSI, UK
April Lecture Series 2014
Manipulating people by accident
• Because we want to make it more fun
Pete Cape, SSI, UK
April Lecture Series 2014
Please select your most preferred holiday type…
Summer Beach Winter
City Break Adventure
Pete Cape, SSI, UK
April Lecture Series 2014
Please select your most preferred holiday type…
Summer Beach Winter
City Break Adventure
Pete Cape, SSI, UK
April Lecture Series 2014
Please select your most preferred holiday type…
Summer Beach Winter
City Break Adventure
Pete Cape, SSI, UK
April Lecture Series 2014
The results
58%
60%
36%
0%
20%
40%
60%
80%
100%
“Summer Beach”
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation
Self-Determination Theory (Deci & Ryan)
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation
• people are active ‘organisms’
• having tendencies toward psychological growth
and development
• who strive to master ongoing challenges
• and to integrate their experiences into a
coherent sense of self.
• the social context can either support or thwart
the natural tendencies
Self-Determination Theory (Deci & Ryan)
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation under Self-Determination Theory
A continuum that relates to the task in
the social context, not a person
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation under Self-Determination Theory
A continuum that relates to the task in
the social context, not a person
Regulation
Introjection
Identification
Integration
to obtain an externally imposed reward
pressure to avoid guilt or to enhance self-esteem
conscious valuing of activity
assimilated to self
task is not done
for it’s own reward
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation under Self-Determination Theory
A continuum that relates to the task in
the social context, not a person
Regulation
Introjection
Identification
Integration
to obtain an externally imposed reward
pressure to avoid guilt or to enhance self-esteem
conscious valuing of activity
assimilated to self
task is not done
for it’s own reward
Where do most market research
respondents fall?
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation under Self-Determination Theory
A continuum that relates to the task in
the social context, not a person
Regulation
Introjection
Identification
Integration
to obtain an externally imposed reward
pressure to avoid guilt or to enhance self-esteem
conscious valuing of activity
assimilated to self
task is not done
for it’s own reward
Where do most market research
respondents fall?
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation under Self-Determination Theory
Task Outcomes
Regulation
Introjection
Identification
Integration
little interest, value or effort
more effort, anxiety, poor coping with failure
enjoyment, good coping skills
greater enjoyment, competency
task is not done properly
interest, enjoyment
highly competent
Where would we like respondents
to be?
Pete Cape, SSI, UK
April Lecture Series 2014
Motivation under Self-Determination Theory
Task Outcomes
Regulation
Introjection
Identification
Integration
little interest, value or effort
more effort, anxiety, poor coping with failure
enjoyment, good coping skills
greater enjoyment, competency
task is not done properly
interest, enjoyment
highly competent
Where would we like respondents
to be?
Pete Cape, SSI, UK
April Lecture Series 2014
Moving people along the motivation continuum
• Foster feelings of:
— Autonomy – you are free to do this or not as you choose
— Competence – you are good at this
— Relatedness – people like you do this
— Value – what you are doing has meaning
Pete Cape, SSI, UK
April Lecture Series 2014
Moving people along the motivation continuum
• Foster feelings of:
— Autonomy – you are free to do this or not as you choose
— Competence – you are good at this
— Relatedness – people like you do this
— Value – what you are doing has meaning
• Perhaps this was what the interviewer used to do?
• This is what Gamification helps do
Pete Cape, SSI, UK
April Lecture Series 2014
Motivating respondents
Thank you very much for choosing to do this important survey.
As one of our most expert survey takers we have selected you to
help on this survey.
People like you all over the country are taking part in this survey.
Your answers and those of everyone else will help improve the
services our client provides to customers like you.
Please press the > button when you are ready to continue with
your survey.
Pete Cape, SSI, UK
April Lecture Series 2014
Motivating respondents
Thank you very much for choosing to do this important survey.
As one of our most expert survey takers we have selected you to
help on this survey.
People like you all over the country are taking part in this survey.
Your answers and those of everyone else will help improve the
services our client provides to customers like you.
Please press the > button when you are ready to continue with
your survey.
• No mention of rewards
• No threats
Pete Cape, SSI, UK
April Lecture Series 2014
Summary
• The questionnaire is a mixture of art and science
• The art of writing well, of motivating respondents
• The science of writing precisely, avoiding bias
• Everything has meaning
• You are supposed to know what it means
• There are better ways of asking questions
• There are better ways of collecting answers
• You might as well do the best you can
Pete Cape, SSI, UK
April Lecture Series 2014
Summary
• Questionnaire writing
• (and understanding why it is important)
• Sets us apart from others
• And has real value
Pete Cape, SSI, UK
April Lecture Series 2014
Summary - Principles
• Respondents know they are in an experiment
• They think everything they see, hear or read is deliberate, and
has meaning
• They want to please
• Questionnaire motivates the respondent to provide
— complete answers
— accurate answers
• It communicates to the respondent
• It helps the respondent work out their answers
• It makes the respondents task easy
Pete Cape, SSI, UK
April Lecture Series 2014
And guess what?
• If it was easy do you think there
would be textbooks on it?
Pete Cape, SSI, UK
April Lecture Series 2014
Thank you!
Pete Cape
Global Knowledge Director, SSI
2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
Pete Cape, SSI, UK
April Lecture Series 2014
The Sponsors for this Event
If you are interested in sponsoring a future NewMR event
Email Michele.Poynter@TheFuturePlace.com
2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
Pete Cape, SSI, UK
April Lecture Series 2014
Q & A
Ray Poynter
The Future Place
Pete Cape
SSI
2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
Pete Cape, SSI, UK
April Lecture Series 2014
| 186 |
surveysampling.com
info@surveysampling.com
pete.cape@surveysampling.com

More Related Content

What's hot

Ielts speaking part 1
Ielts speaking part 1Ielts speaking part 1
Ielts speaking part 1
st0rmchazer
 
NNSTOY A Framework of Fun:Learning Through and From Games
NNSTOY A Framework of Fun:Learning Through and From Games NNSTOY A Framework of Fun:Learning Through and From Games
NNSTOY A Framework of Fun:Learning Through and From Games
Curtis Chandler
 
Unclassroom elementary 2014
Unclassroom elementary 2014Unclassroom elementary 2014
Unclassroom elementary 2014
Curtis Chandler
 
Names four corners intro
Names four corners intro Names four corners intro
Names four corners intro
Fishkiller
 
Paired Choice
Paired ChoicePaired Choice
Paired Choice
ebenimzo
 
Personal Preference
Personal PreferencePersonal Preference
Personal Preference
ebenimzo
 
Involve me and I'll understand: engaging students in research-led teaching
Involve me and I'll understand: engaging students in research-led teachingInvolve me and I'll understand: engaging students in research-led teaching
Involve me and I'll understand: engaging students in research-led teaching
Gary Wood
 
Suggest and Offer
Suggest and OfferSuggest and Offer
Suggest and Offer
Yudha Bhaskoro
 
Back To School Night Presentation 2009
Back To School Night Presentation 2009Back To School Night Presentation 2009
Back To School Night Presentation 2009
jtiggs
 
Ielts Speaking Test1
Ielts Speaking Test1Ielts Speaking Test1
Ielts Speaking Test1
ethanzhang
 
Top 10 discussion starters for children
Top 10 discussion starters for childrenTop 10 discussion starters for children
Top 10 discussion starters for children
Jackie Bolen
 
Technology and education
Technology and educationTechnology and education
Technology and education
mpaul01
 
Technology and education
Technology and educationTechnology and education
Technology and education
mpaul01
 
Minarets High PBE Sophomores 2014 Template - Matt Powers
Minarets High PBE Sophomores 2014 Template - Matt PowersMinarets High PBE Sophomores 2014 Template - Matt Powers
Minarets High PBE Sophomores 2014 Template - Matt Powers
Matt Powers
 

What's hot (14)

Ielts speaking part 1
Ielts speaking part 1Ielts speaking part 1
Ielts speaking part 1
 
NNSTOY A Framework of Fun:Learning Through and From Games
NNSTOY A Framework of Fun:Learning Through and From Games NNSTOY A Framework of Fun:Learning Through and From Games
NNSTOY A Framework of Fun:Learning Through and From Games
 
Unclassroom elementary 2014
Unclassroom elementary 2014Unclassroom elementary 2014
Unclassroom elementary 2014
 
Names four corners intro
Names four corners intro Names four corners intro
Names four corners intro
 
Paired Choice
Paired ChoicePaired Choice
Paired Choice
 
Personal Preference
Personal PreferencePersonal Preference
Personal Preference
 
Involve me and I'll understand: engaging students in research-led teaching
Involve me and I'll understand: engaging students in research-led teachingInvolve me and I'll understand: engaging students in research-led teaching
Involve me and I'll understand: engaging students in research-led teaching
 
Suggest and Offer
Suggest and OfferSuggest and Offer
Suggest and Offer
 
Back To School Night Presentation 2009
Back To School Night Presentation 2009Back To School Night Presentation 2009
Back To School Night Presentation 2009
 
Ielts Speaking Test1
Ielts Speaking Test1Ielts Speaking Test1
Ielts Speaking Test1
 
Top 10 discussion starters for children
Top 10 discussion starters for childrenTop 10 discussion starters for children
Top 10 discussion starters for children
 
Technology and education
Technology and educationTechnology and education
Technology and education
 
Technology and education
Technology and educationTechnology and education
Technology and education
 
Minarets High PBE Sophomores 2014 Template - Matt Powers
Minarets High PBE Sophomores 2014 Template - Matt PowersMinarets High PBE Sophomores 2014 Template - Matt Powers
Minarets High PBE Sophomores 2014 Template - Matt Powers
 

Similar to Pete cape april lecture series - 2014

Interviewing ppt
Interviewing pptInterviewing ppt
Interviewing ppt
jhiggins11
 
Peter cape training day - 2013
Peter cape   training day - 2013Peter cape   training day - 2013
Peter cape training day - 2013
Ray Poynter
 
What I wish I knew in my first year of testing
What I wish I knew in my first year of testingWhat I wish I knew in my first year of testing
What I wish I knew in my first year of testing
💻 Nicola Lindgren (née Owen)
 
Young researchers upload version
Young researchers   upload versionYoung researchers   upload version
Young researchers upload version
katiecdec
 
Young Researchers
Young ResearchersYoung Researchers
Young Researchers
katiecdec
 
Improving Whole Class Inquiry Participation
Improving Whole Class Inquiry ParticipationImproving Whole Class Inquiry Participation
Improving Whole Class Inquiry Participation
Joan Gallagher
 
The Art of Finding Your Story: NetSquared Vancouver 2013-08-14
The Art of Finding Your Story: NetSquared Vancouver 2013-08-14The Art of Finding Your Story: NetSquared Vancouver 2013-08-14
The Art of Finding Your Story: NetSquared Vancouver 2013-08-14
NetSquared Vancouver
 
mark [Autosaved].ppt
mark [Autosaved].pptmark [Autosaved].ppt
mark [Autosaved].ppt
AndrewPruett3
 
Tips on Conducting Interviews
Tips on Conducting InterviewsTips on Conducting Interviews
Tips on Conducting Interviews
David Sheets
 
Interviewing101
Interviewing101Interviewing101
Interviewing101
jhiggins11
 
Evaluation question 3
Evaluation question 3Evaluation question 3
Evaluation question 3
bethan1234
 
QA Analysis.pptx
QA Analysis.pptxQA Analysis.pptx
Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018
Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018
Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018
Joe O'Donnell
 
Evaluation question 3
Evaluation question 3Evaluation question 3
Evaluation question 3
bethan1234
 
Question Everything: The Power of Asking Questions
Question Everything: The Power of Asking QuestionsQuestion Everything: The Power of Asking Questions
Question Everything: The Power of Asking Questions
Diane Windingland
 
Vsb an intro to afl.nov.2010
Vsb   an intro to afl.nov.2010Vsb   an intro to afl.nov.2010
Vsb an intro to afl.nov.2010
Faye Brownlie
 
mark2.ppt
mark2.pptmark2.ppt
mark2.ppt
AndrewPruett3
 
Talis Aspire User Group Australia Moving Forward
Talis Aspire User Group Australia Moving ForwardTalis Aspire User Group Australia Moving Forward
Talis Aspire User Group Australia Moving Forward
Talis
 
NC Paraprofessional 2019 - How To Ask Better Questions
NC Paraprofessional 2019 - How To Ask Better QuestionsNC Paraprofessional 2019 - How To Ask Better Questions
NC Paraprofessional 2019 - How To Ask Better Questions
Amanda McLellan
 
QualitativeLecture
QualitativeLectureQualitativeLecture

Similar to Pete cape april lecture series - 2014 (20)

Interviewing ppt
Interviewing pptInterviewing ppt
Interviewing ppt
 
Peter cape training day - 2013
Peter cape   training day - 2013Peter cape   training day - 2013
Peter cape training day - 2013
 
What I wish I knew in my first year of testing
What I wish I knew in my first year of testingWhat I wish I knew in my first year of testing
What I wish I knew in my first year of testing
 
Young researchers upload version
Young researchers   upload versionYoung researchers   upload version
Young researchers upload version
 
Young Researchers
Young ResearchersYoung Researchers
Young Researchers
 
Improving Whole Class Inquiry Participation
Improving Whole Class Inquiry ParticipationImproving Whole Class Inquiry Participation
Improving Whole Class Inquiry Participation
 
The Art of Finding Your Story: NetSquared Vancouver 2013-08-14
The Art of Finding Your Story: NetSquared Vancouver 2013-08-14The Art of Finding Your Story: NetSquared Vancouver 2013-08-14
The Art of Finding Your Story: NetSquared Vancouver 2013-08-14
 
mark [Autosaved].ppt
mark [Autosaved].pptmark [Autosaved].ppt
mark [Autosaved].ppt
 
Tips on Conducting Interviews
Tips on Conducting InterviewsTips on Conducting Interviews
Tips on Conducting Interviews
 
Interviewing101
Interviewing101Interviewing101
Interviewing101
 
Evaluation question 3
Evaluation question 3Evaluation question 3
Evaluation question 3
 
QA Analysis.pptx
QA Analysis.pptxQA Analysis.pptx
QA Analysis.pptx
 
Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018
Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018
Skills@work Presentation Skills Workshop - O'Connells - 5th March 2018
 
Evaluation question 3
Evaluation question 3Evaluation question 3
Evaluation question 3
 
Question Everything: The Power of Asking Questions
Question Everything: The Power of Asking QuestionsQuestion Everything: The Power of Asking Questions
Question Everything: The Power of Asking Questions
 
Vsb an intro to afl.nov.2010
Vsb   an intro to afl.nov.2010Vsb   an intro to afl.nov.2010
Vsb an intro to afl.nov.2010
 
mark2.ppt
mark2.pptmark2.ppt
mark2.ppt
 
Talis Aspire User Group Australia Moving Forward
Talis Aspire User Group Australia Moving ForwardTalis Aspire User Group Australia Moving Forward
Talis Aspire User Group Australia Moving Forward
 
NC Paraprofessional 2019 - How To Ask Better Questions
NC Paraprofessional 2019 - How To Ask Better QuestionsNC Paraprofessional 2019 - How To Ask Better Questions
NC Paraprofessional 2019 - How To Ask Better Questions
 
QualitativeLecture
QualitativeLectureQualitativeLecture
QualitativeLecture
 

More from Ray Poynter

The State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and FindingsThe State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and Findings
Ray Poynter
 
ResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis toolResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis tool
Ray Poynter
 
AI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from YasnaAI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from Yasna
Ray Poynter
 
Artificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So FarArtificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So Far
Ray Poynter
 
State of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMRState of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMR
Ray Poynter
 
Sudden Death of Beliefs
Sudden Death of BeliefsSudden Death of Beliefs
Sudden Death of Beliefs
Ray Poynter
 
Uncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden NarrativesUncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden Narratives
Ray Poynter
 
Narrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at MondelēzNarrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at Mondelēz
Ray Poynter
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
Ray Poynter
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
Ray Poynter
 
The State of Insights – September 2023
The State of Insights – September 2023The State of Insights – September 2023
The State of Insights – September 2023
Ray Poynter
 
Research Thinking in the age of AI
Research Thinking in the age of AIResearch Thinking in the age of AI
Research Thinking in the age of AI
Ray Poynter
 
How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?
Ray Poynter
 
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
Ray Poynter
 
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
Ray Poynter
 
Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys
Using Generative AI to Assess the Quality of Open-Ended Responses in SurveysUsing Generative AI to Assess the Quality of Open-Ended Responses in Surveys
Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys
Ray Poynter
 
Exploring the future of verbatim coding with ChatGPT
Exploring the future of verbatim coding with ChatGPTExploring the future of verbatim coding with ChatGPT
Exploring the future of verbatim coding with ChatGPT
Ray Poynter
 
Using Generative AI to bring Qualitative Capabilities to Quantitative Surveys
Using Generative AI to bring Qualitative Capabilities to Quantitative SurveysUsing Generative AI to bring Qualitative Capabilities to Quantitative Surveys
Using Generative AI to bring Qualitative Capabilities to Quantitative Surveys
Ray Poynter
 
How AI / ChatGPT Drives Business Growth
How AI / ChatGPT Drives Business GrowthHow AI / ChatGPT Drives Business Growth
How AI / ChatGPT Drives Business Growth
Ray Poynter
 
Tech for tech’s sake? Learnings from experiments with AI in consumer research
Tech for tech’s sake? Learnings from experiments with AI in consumer researchTech for tech’s sake? Learnings from experiments with AI in consumer research
Tech for tech’s sake? Learnings from experiments with AI in consumer research
Ray Poynter
 

More from Ray Poynter (20)

The State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and FindingsThe State of AI in Insights and Research 2024: Results and Findings
The State of AI in Insights and Research 2024: Results and Findings
 
ResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis toolResearchWiseAI - an artificial intelligence driven research data analysis tool
ResearchWiseAI - an artificial intelligence driven research data analysis tool
 
AI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from YasnaAI-powered interviewing: Best practices from Yasna
AI-powered interviewing: Best practices from Yasna
 
Artificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So FarArtificial Intelligence and Qual: The Story So Far
Artificial Intelligence and Qual: The Story So Far
 
State of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMRState of Research Insights in Q1, 2024 from NewMR
State of Research Insights in Q1, 2024 from NewMR
 
Sudden Death of Beliefs
Sudden Death of BeliefsSudden Death of Beliefs
Sudden Death of Beliefs
 
Uncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden NarrativesUncovering Consumers’ Hidden Narratives
Uncovering Consumers’ Hidden Narratives
 
Narrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at MondelēzNarrative Exploration of New Categories at Mondelēz
Narrative Exploration of New Categories at Mondelēz
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
 
The Future in Focus
The Future in FocusThe Future in Focus
The Future in Focus
 
The State of Insights – September 2023
The State of Insights – September 2023The State of Insights – September 2023
The State of Insights – September 2023
 
Research Thinking in the age of AI
Research Thinking in the age of AIResearch Thinking in the age of AI
Research Thinking in the age of AI
 
How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?How might AI impact Research and Insights over the next two years?
How might AI impact Research and Insights over the next two years?
 
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
From Words to Wisdom: Unleashing the Potential of Language Models for Human-C...
 
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
ChatGPT for Social Media Listening: practical application with YouScan’s Insi...
 
Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys
Using Generative AI to Assess the Quality of Open-Ended Responses in SurveysUsing Generative AI to Assess the Quality of Open-Ended Responses in Surveys
Using Generative AI to Assess the Quality of Open-Ended Responses in Surveys
 
Exploring the future of verbatim coding with ChatGPT
Exploring the future of verbatim coding with ChatGPTExploring the future of verbatim coding with ChatGPT
Exploring the future of verbatim coding with ChatGPT
 
Using Generative AI to bring Qualitative Capabilities to Quantitative Surveys
Using Generative AI to bring Qualitative Capabilities to Quantitative SurveysUsing Generative AI to bring Qualitative Capabilities to Quantitative Surveys
Using Generative AI to bring Qualitative Capabilities to Quantitative Surveys
 
How AI / ChatGPT Drives Business Growth
How AI / ChatGPT Drives Business GrowthHow AI / ChatGPT Drives Business Growth
How AI / ChatGPT Drives Business Growth
 
Tech for tech’s sake? Learnings from experiments with AI in consumer research
Tech for tech’s sake? Learnings from experiments with AI in consumer researchTech for tech’s sake? Learnings from experiments with AI in consumer research
Tech for tech’s sake? Learnings from experiments with AI in consumer research
 

Recently uploaded

Top 10 AI Trends to Watch in 2024 with Intelisync
Top 10 AI Trends to Watch in 2024 with IntelisyncTop 10 AI Trends to Watch in 2024 with Intelisync
Top 10 AI Trends to Watch in 2024 with Intelisync
nehapardhi711
 
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdfHow to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
SimpleMoneyMaker
 
3 Best “Add to Calendar” Link Generator Tools (2024)
3 Best  “Add to Calendar”  Link Generator Tools (2024)3 Best  “Add to Calendar”  Link Generator Tools (2024)
3 Best “Add to Calendar” Link Generator Tools (2024)
Y
 
How to Generate Add to Calendar Link using Cal.et
How to Generate Add to Calendar Link using Cal.etHow to Generate Add to Calendar Link using Cal.et
How to Generate Add to Calendar Link using Cal.et
Y
 
在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样
在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样
在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样
5ys5mvlp
 
Title: Making Money the Easy Way: A Quick Guide to Generating Income
Title: Making Money the Easy Way: A Quick Guide to Generating IncomeTitle: Making Money the Easy Way: A Quick Guide to Generating Income
Title: Making Money the Easy Way: A Quick Guide to Generating Income
WilliamZinsmeister
 
Top Strategies for Building High-Quality Backlinks in 2024 PPT.pdf
Top Strategies for Building High-Quality Backlinks in 2024 PPT.pdfTop Strategies for Building High-Quality Backlinks in 2024 PPT.pdf
Top Strategies for Building High-Quality Backlinks in 2024 PPT.pdf
1Solutions Pvt. Ltd.
 
在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样
在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样
在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样
5ys5mvlp
 
Advertising and Promotion of whisper by Sakthi Sundar
Advertising and Promotion of whisper by Sakthi SundarAdvertising and Promotion of whisper by Sakthi Sundar
Advertising and Promotion of whisper by Sakthi Sundar
sakthisundar2001
 
2024 case study-SHEIN Analysis-Final.pdf
2024 case study-SHEIN Analysis-Final.pdf2024 case study-SHEIN Analysis-Final.pdf
2024 case study-SHEIN Analysis-Final.pdf
Sun.Lee
 
SEO in the AI Era - Trust, Quality and Content Discovery - Andy Crestodina
SEO in the AI Era - Trust, Quality and Content Discovery - Andy CrestodinaSEO in the AI Era - Trust, Quality and Content Discovery - Andy Crestodina
SEO in the AI Era - Trust, Quality and Content Discovery - Andy Crestodina
DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions
 
Viapulsa - Content Strategist Assignment
Viapulsa - Content Strategist AssignmentViapulsa - Content Strategist Assignment
Viapulsa - Content Strategist Assignment
Amal Agung Cahyadi
 
Why bridging the gap between PR and SEO is the only way forward for PR Profes...
Why bridging the gap between PR and SEO is the only way forward for PR Profes...Why bridging the gap between PR and SEO is the only way forward for PR Profes...
Why bridging the gap between PR and SEO is the only way forward for PR Profes...
Isa Lavs
 
Embark on style journeys Indian clothing store denver guide.pptx
Embark on style journeys Indian clothing store denver guide.pptxEmbark on style journeys Indian clothing store denver guide.pptx
Embark on style journeys Indian clothing store denver guide.pptx
Omnama Fashions
 
2025 Adventur ehomes Product Guide released Jun 1 2024
2025 Adventur ehomes Product Guide released Jun 1 20242025 Adventur ehomes Product Guide released Jun 1 2024
2025 Adventur ehomes Product Guide released Jun 1 2024
JimWilliams206540
 
Customer relationship management notes for Bcom students
Customer relationship management notes for Bcom studentsCustomer relationship management notes for Bcom students
Customer relationship management notes for Bcom students
PratibhaShelake1
 
Evaluating the Effectiveness of Women-Focused Marketing
Evaluating the Effectiveness of Women-Focused MarketingEvaluating the Effectiveness of Women-Focused Marketing
Evaluating the Effectiveness of Women-Focused Marketing
HighViz PR
 
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
Jorge Calmett
 
SEO and Google Ads Fundamentals Gokce Yesilbas
SEO and Google Ads Fundamentals Gokce YesilbasSEO and Google Ads Fundamentals Gokce Yesilbas
SEO and Google Ads Fundamentals Gokce Yesilbas
gokceyesilbas
 
一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理
一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理
一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理
dqvc9xf1
 

Recently uploaded (20)

Top 10 AI Trends to Watch in 2024 with Intelisync
Top 10 AI Trends to Watch in 2024 with IntelisyncTop 10 AI Trends to Watch in 2024 with Intelisync
Top 10 AI Trends to Watch in 2024 with Intelisync
 
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdfHow to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
How to Start Affiliate Marketing with ChatGPT- A Step-by-Step Guide (1).pdf
 
3 Best “Add to Calendar” Link Generator Tools (2024)
3 Best  “Add to Calendar”  Link Generator Tools (2024)3 Best  “Add to Calendar”  Link Generator Tools (2024)
3 Best “Add to Calendar” Link Generator Tools (2024)
 
How to Generate Add to Calendar Link using Cal.et
How to Generate Add to Calendar Link using Cal.etHow to Generate Add to Calendar Link using Cal.et
How to Generate Add to Calendar Link using Cal.et
 
在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样
在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样
在线办理(worc毕业证书)伍斯特大学毕业证成绩单一模一样
 
Title: Making Money the Easy Way: A Quick Guide to Generating Income
Title: Making Money the Easy Way: A Quick Guide to Generating IncomeTitle: Making Money the Easy Way: A Quick Guide to Generating Income
Title: Making Money the Easy Way: A Quick Guide to Generating Income
 
Top Strategies for Building High-Quality Backlinks in 2024 PPT.pdf
Top Strategies for Building High-Quality Backlinks in 2024 PPT.pdfTop Strategies for Building High-Quality Backlinks in 2024 PPT.pdf
Top Strategies for Building High-Quality Backlinks in 2024 PPT.pdf
 
在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样
在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样
在线办理(英国UWS毕业证书)西苏格兰大学毕业证学位证一模一样
 
Advertising and Promotion of whisper by Sakthi Sundar
Advertising and Promotion of whisper by Sakthi SundarAdvertising and Promotion of whisper by Sakthi Sundar
Advertising and Promotion of whisper by Sakthi Sundar
 
2024 case study-SHEIN Analysis-Final.pdf
2024 case study-SHEIN Analysis-Final.pdf2024 case study-SHEIN Analysis-Final.pdf
2024 case study-SHEIN Analysis-Final.pdf
 
SEO in the AI Era - Trust, Quality and Content Discovery - Andy Crestodina
SEO in the AI Era - Trust, Quality and Content Discovery - Andy CrestodinaSEO in the AI Era - Trust, Quality and Content Discovery - Andy Crestodina
SEO in the AI Era - Trust, Quality and Content Discovery - Andy Crestodina
 
Viapulsa - Content Strategist Assignment
Viapulsa - Content Strategist AssignmentViapulsa - Content Strategist Assignment
Viapulsa - Content Strategist Assignment
 
Why bridging the gap between PR and SEO is the only way forward for PR Profes...
Why bridging the gap between PR and SEO is the only way forward for PR Profes...Why bridging the gap between PR and SEO is the only way forward for PR Profes...
Why bridging the gap between PR and SEO is the only way forward for PR Profes...
 
Embark on style journeys Indian clothing store denver guide.pptx
Embark on style journeys Indian clothing store denver guide.pptxEmbark on style journeys Indian clothing store denver guide.pptx
Embark on style journeys Indian clothing store denver guide.pptx
 
2025 Adventur ehomes Product Guide released Jun 1 2024
2025 Adventur ehomes Product Guide released Jun 1 20242025 Adventur ehomes Product Guide released Jun 1 2024
2025 Adventur ehomes Product Guide released Jun 1 2024
 
Customer relationship management notes for Bcom students
Customer relationship management notes for Bcom studentsCustomer relationship management notes for Bcom students
Customer relationship management notes for Bcom students
 
Evaluating the Effectiveness of Women-Focused Marketing
Evaluating the Effectiveness of Women-Focused MarketingEvaluating the Effectiveness of Women-Focused Marketing
Evaluating the Effectiveness of Women-Focused Marketing
 
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
01 Field+Guide+to+Human-Centered+Design_IDEOorg_English GUIA COMPLETA DETALLA...
 
SEO and Google Ads Fundamentals Gokce Yesilbas
SEO and Google Ads Fundamentals Gokce YesilbasSEO and Google Ads Fundamentals Gokce Yesilbas
SEO and Google Ads Fundamentals Gokce Yesilbas
 
一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理
一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理
一比一原版英国南安普顿大学毕业证(Soton学位证)如何办理
 

Pete cape april lecture series - 2014

  • 1. Pete Cape, SSI, UK April Lecture Series 2014 Questionnaire Design – A Masterclass Pete Cape Global Knowledge Director, SSI 2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
  • 2. Pete Cape, SSI, UK April Lecture Series 2014 | 2 | A questionnaire might have all of these…. • Open questions • Yes/No questions • Single coded questions with multiple answers • Multi-coded questions with multiple answers • Items to be ranked • Items to be rated • Items to be chosen between
  • 3. Pete Cape, SSI, UK April Lecture Series 2014 | 3 | A questionnaire might have all of these…. • Open questions • Yes/No questions • Single coded questions with multiple answers • Multi-coded questions with multiple answers • Items to be ranked • Items to be rated • Items to be chosen between • But this is such a boring way of looking at it….
  • 4. Pete Cape, SSI, UK April Lecture Series 2014 | 4 | A questionnaire might have all of these…. • Open questions • Yes/No questions • Single coded questions with multiple answers • Multi-coded questions with multiple answers • Items to be ranked • Items to be rated • Items to be chosen between • But this is such a boring way of looking at it…. • Let’s start with some principles….
  • 5. Pete Cape, SSI, UK April Lecture Series 2014 | 5 | Things to remember about respondents….
  • 6. Pete Cape, SSI, UK April Lecture Series 2014 | 6 | Things to remember about respondents…. • They know they are in an experiment
  • 7. Pete Cape, SSI, UK April Lecture Series 2014 | 7 | Things to remember about respondents…. • They know they are in an experiment • Everything they see, hear or read is meant deliberately, and has meaning
  • 8. Pete Cape, SSI, UK April Lecture Series 2014 | 8 | Things to remember about respondents…. • They know they are in an experiment • Everything they see, hear or read is meant deliberately, and has meaning • They want to please
  • 9. Pete Cape, SSI, UK April Lecture Series 2014 | 9 | Things to remember about respondents…. • They know they are in an experiment • Everything they see, hear or read is meant deliberately, and has meaning • They want to please • They think you know what you are doing…
  • 10. Pete Cape, SSI, UK April Lecture Series 2014
  • 11. Pete Cape, SSI, UK April Lecture Series 2014
  • 12. Pete Cape, SSI, UK April Lecture Series 2014 | 12 | Things to remember about questionnaires…. • Everything has meaning
  • 13. Pete Cape, SSI, UK April Lecture Series 2014 | 13 | Things to remember about questionnaires…. • Everything has meaning • You are supposed to know what that meaning is
  • 14. Pete Cape, SSI, UK April Lecture Series 2014 | 14 | Questionnaires do a lot of work • Motivate the respondent
  • 15. Pete Cape, SSI, UK April Lecture Series 2014 | 15 | Questionnaires do a lot of work • Motivate the respondent to provide — complete answers — accurate answers
  • 16. Pete Cape, SSI, UK April Lecture Series 2014 | 16 | Questionnaires do a lot of work • Motivate the respondent to provide — complete answers — accurate answers • Communicate to the respondent
  • 17. Pete Cape, SSI, UK April Lecture Series 2014 | 17 | Questionnaires do a lot of work • Motivate the respondent to provide — complete answers — accurate answers • Communicate to the respondent • Help the respondent work out their answers
  • 18. Pete Cape, SSI, UK April Lecture Series 2014 | 18 | Questionnaires do a lot of work • Motivate the respondent to provide — complete answers — accurate answers • Communicate to the respondent • Help the respondent work out their answers • Make the respondents task easy
  • 19. Pete Cape, SSI, UK April Lecture Series 2014 | 19 | Is this the simplest question of all? • Yes • No
  • 20. Pete Cape, SSI, UK April Lecture Series 2014 | 20 | Respondents want to please Do you care to disagree with me….? I am a scientist….. • Acquiescence bias
  • 21. Pete Cape, SSI, UK April Lecture Series 2014 Acquiescence bias • Some respondents are simply agreeable, and indicate agreement out of politeness. Saris, Krosnick and Shaeffe
  • 22. Pete Cape, SSI, UK April Lecture Series 2014 Acquiescence bias • Some respondents are simply agreeable, and indicate agreement out of politeness. • Other respondents expect that the researchers agree with the listed items and defer to their judgment. Saris, Krosnick and Shaeffe
  • 23. Pete Cape, SSI, UK April Lecture Series 2014 Acquiescence bias • Some respondents are simply agreeable, and indicate agreement out of politeness. • Other respondents expect that the researchers agree with the listed items and defer to their judgment. • Most respondents engage in survey satisficing and find that agreeing takes less effort than carefully weighing each optional level of disagreement and agreement. Saris, Krosnick and Shaeffe
  • 24. Pete Cape, SSI, UK April Lecture Series 2014 Affirmation vs Multicode
  • 25. Pete Cape, SSI, UK April Lecture Series 2014 Affirmation vs Multicode
  • 26. Pete Cape, SSI, UK April Lecture Series 2014 Overcoming acquiescence bias • Find them through traps and exclude
  • 27. Pete Cape, SSI, UK April Lecture Series 2014 Overcoming acquiescence bias • Find them through traps and exclude • Follow up question “how sure?” • If not 100% sure then recode as “no”
  • 28. Pete Cape, SSI, UK April Lecture Series 2014 Overcoming acquiescence bias • Find them through traps and exclude • Follow up question “how sure?” • If not 100% sure then recode as “no” • Avoid the question type and give alternatives — “chose the one of these that best describes you” — “to what extent do you…”
  • 29. Pete Cape, SSI, UK April Lecture Series 2014 Affirmation vs Multicode
  • 30. Pete Cape, SSI, UK April Lecture Series 2014 Affirmation vs Multicode why so big…?
  • 31. Pete Cape, SSI, UK April Lecture Series 2014 Social Desirability bias • Wanting to look good
  • 32. Pete Cape, SSI, UK April Lecture Series 2014 Social Desirability bias • Wanting to look good — To society — To other individuals — To myself
  • 33. Pete Cape, SSI, UK April Lecture Series 2014 Some Normal Everyday Things…that I keep to myself.. Chart Title
  • 34. Pete Cape, SSI, UK April Lecture Series 2014 Some Normal Everyday Things…I want you to think I do.. Chart Title
  • 35. Pete Cape, SSI, UK April Lecture Series 2014 20% bought a book last week? | 35 |
  • 36. Pete Cape, SSI, UK April Lecture Series 2014 Avoiding Social Desirability • Is very hard • Try to make respondents feel “normal”
  • 37. Pete Cape, SSI, UK April Lecture Series 2014 Avoiding Social Desirability • Is very hard • Try to make respondents feel “normal” This third section is about social attitudes and values. There is a lot of talk about discrimination and racial prejudice, and different people have different views on the size of such problems, their causes and what should be done about them. There are no right or wrong answers, it is your honest opinions we are looking for. If you do not wish to answer any of these questions please use the option “prefer not to say”.
  • 38. Pete Cape, SSI, UK April Lecture Series 2014 Questionnaire Design and Bias • Questionnaire design has to overcome respondents’ psychological biases
  • 39. Pete Cape, SSI, UK April Lecture Series 2014 Questionnaire Design and Bias • Questionnaire design has to overcome respondents’ psychological biases • And also experimenter bias
  • 40. Pete Cape, SSI, UK April Lecture Series 2014 Questionnaire Design and Bias • Questionnaire design has to overcome respondents’ psychological biases • And also experimenter bias • i.e. you and your client….
  • 41. Pete Cape, SSI, UK April Lecture Series 2014 Experimenter biases • It is easy to suggest the answer…
  • 42. Pete Cape, SSI, UK April Lecture Series 2014 Question suggests the answer Which carbonated soft drinks, like Coca-Cola or Pepsi for example, do you drink regularly?”
  • 43. Pete Cape, SSI, UK April Lecture Series 2014 Question suggests the answer Which carbonated soft drinks, like Coca-Cola or Pepsi for example, do you drink regularly?” Raises answers for drinks “like” Pepsi and Coke i.e. Brown or cola drinks
  • 44. Pete Cape, SSI, UK April Lecture Series 2014 Question suggests the answer Which carbonated soft drinks, like Coca-Cola or Pepsi for example, do you drink regularly?” Raises answers for drinks “like” Pepsi and Coke i.e. Brown or cola drinks Lowers answers for non- brown drinks e.g. Fanta, 7-Up
  • 45. Pete Cape, SSI, UK April Lecture Series 2014 Question suggests the answer Which carbonated soft drinks, like Coca-Cola or Pepsi for example, do you drink regularly?” Raises answers for drinks “like” Pepsi and Coke i.e. Brown or cola drinks Lowers answers for non- brown drinks e.g. Fanta, 7-Up Problem is in use of jargon term: “carbonated soft drinks”
  • 46. Pete Cape, SSI, UK April Lecture Series 2014 Question suggests “my” answer Testing perfumes on animals involves causing them pain. Do you agree with testing perfumes on animals?
  • 47. Pete Cape, SSI, UK April Lecture Series 2014 Question suggests “my” answer Testing perfumes on animals involves causing them pain. Do you agree with testing perfumes on animals? “involves causing them pain” is an opinion
  • 48. Pete Cape, SSI, UK April Lecture Series 2014 Avoiding Experimenter Bias • Is easier
  • 49. Pete Cape, SSI, UK April Lecture Series 2014 Avoiding Experimenter Bias • Is easier • Think neutral
  • 50. Pete Cape, SSI, UK April Lecture Series 2014 Avoiding Experimenter Bias • Is easier • Think neutral • Don’t think your view is the world view
  • 51. Pete Cape, SSI, UK April Lecture Series 2014 Questionnaire as a measuring tool • We rarely measure physical things
  • 52. Pete Cape, SSI, UK April Lecture Series 2014 Questionnaire as a measuring tool • We rarely measure physical things • But we do measure frequency: — How often do you x, y, z?
  • 53. Pete Cape, SSI, UK April Lecture Series 2014 Questionnaire as a measuring tool • We rarely measure physical things • But we do measure frequency: — How often do you x, y, z? • And we do measure attitudes: — How much do you think x,y,z?
  • 54. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • As an relative concept in a fixed time period
  • 55. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • As a relative concept in a fixed time period • Define: — The activity • What does it mean to “go shopping”? • What does it mean to “surf the internet?” — The time period • Is it the relevant one? • What is an “average week”? — The measurement scale • Always, often, sometimes, never…..
  • 56. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • As an absolute number in a fixed time period
  • 57. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • As an absolute number in a fixed time period • Define: — The activity • What does it mean to “go shopping”? • What does it mean to “surf the internet?” — The time period • Is it the relevant one? • What is an “average week”?
  • 58. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • As an absolute number in a fixed time period How many times did you take a bath in the past year?
  • 59. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • As an absolute number in a fixed time period How many times did you take a bath in the past year? too longwhat does this mean?
  • 60. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • Precision and correct framing How many times, if at all, did you take a bath in the past week? By taking a bath we mean actually using a bath tub and filling it with water and sitting in it. Please exclude any showers you may have taken
  • 61. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • Precision and correct framing How many times, if at all, did you take a bath in the past week? By taking a bath we mean actually using a bath tub and filling it with water and sitting in it. Please exclude any showers you may have taken OMG, I only take showers, he will think I am so dirty
  • 62. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • Precision and correct framing, and overcoming social desirability bias Some people only take baths, some only take showers, some take both. This part of the survey is only concerned with bath taking. How many times, if at all, did you take a bath in the past week? By taking a bath we mean actually using a bath tub and filling it with water and sitting in it. Please exclude any showers you may have taken
  • 63. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • Precision and correct framing, and overcoming social desirability bias, and to make life easy Some people only take baths, some only take showers, some take both. This part of the survey is only concerned with bath taking. How many times, if at all, did you take a bath in the past week? By taking a bath we mean actually using a bath tub and filling it with water and sitting in it. Please exclude any showers you may have taken 0 / only took showers 1 – 2 3 – 4 5 – 6 7 or more Can’t remember
  • 64. Pete Cape, SSI, UK April Lecture Series 2014 Measuring frequency • Precision and correct framing, and overcoming social desirability bias, and to make life easy Some people only take baths, some only take showers, some take both. This part of the survey is only concerned with bath taking. How many times, if at all, did you take a bath in the past week? By taking a bath we mean actually using a bath tub and filling it with water and sitting in it. Please exclude any showers you may have taken 0 / only took showers 1 – 2 3 – 4 5 – 6 7 or more Can’t remember Does this have any meaning?
  • 65. Pete Cape, SSI, UK April Lecture Series 2014 The meaning of ranges
  • 66. Pete Cape, SSI, UK April Lecture Series 2014 The meaning of ranges • Assumption of normal distribution
  • 67. Pete Cape, SSI, UK April Lecture Series 2014 The meaning of ranges • Assumption of normal distribution
  • 68. Pete Cape, SSI, UK April Lecture Series 2014 Manipulating numbers Answer list A Answer list B
  • 69. Pete Cape, SSI, UK April Lecture Series 2014 Manipulating numbers Answer list A Answer list B
  • 70. Pete Cape, SSI, UK April Lecture Series 2014 Manipulating numbers Answer list A Answer list B
  • 71. Pete Cape, SSI, UK April Lecture Series 2014 Manipulating numbers Answer list A • The central banding should contain the mean/median • But we don’t know it, it’s what we’re trying to find out! Answer list B
  • 72. Pete Cape, SSI, UK April Lecture Series 2014 Collecting real numbers • Just ask for it • Respondent knows it and will give it
  • 73. Pete Cape, SSI, UK April Lecture Series 2014 Measuring attitudes
  • 74. Pete Cape, SSI, UK April Lecture Series 2014 Measuring attitudes Fill in the words that correspond to the scale points for “Likely” 0 = not at all likely 1 = ? 2 = ? 3 = ? 4 = ? 5 = ? 6 = ? 7 = ? 8 = ? 9 = ? 10 = extremely likely ? ? Exercise
  • 75. Pete Cape, SSI, UK April Lecture Series 2014 Measuring attitudes Fill in the words that correspond to the scale points for “Likely” 0 = not at all likely 1 = ? 2 = ? 3 = ? 4 = ? 5 = ? 6 = ? 7 = ? 8 = ? 9 = ? 10 = extremely likely Exercise
  • 76. Pete Cape, SSI, UK April Lecture Series 2014 Measuring attitudes Fill in the words that correspond to the scale points for “Likely” 0 = not at all likely 1 = ? 2 = ? 3 = ? 4 = ? 5 = ? 6 = ? 7 = ? 8 = ? 9 = ? 10 = extremely likely Exercise Exercise outcomes: a) No-one will be able to do it b) Some people will write: 5 = “neither likely not unlikely” 5 = “some degree of likelihood”
  • 77. Pete Cape, SSI, UK April Lecture Series 2014 Measuring attitudes Fill in the words that correspond to the scale points for “Likely” 0 = not at all likely 1 = ? 2 = ? 3 = ? 4 = ? 5 = ? 6 = ? 7 = ? 8 = ? 9 = ? 10 = extremely likely Exercise Exercise outcomes: a) No-one will be able to do it b) Some people will write: 5 = “neither likely nor unlikely” 5 = “some degree of likelihood”
  • 78. Pete Cape, SSI, UK April Lecture Series 2014 Two men go into a bar….
  • 79. Pete Cape, SSI, UK April Lecture Series 2014 Two men go into a bar…. I say, what do you think of the new VW Golf?
  • 80. Pete Cape, SSI, UK April Lecture Series 2014 Two men go into a bar…. I say, what do you think of the new VW Golf? I think it’s a 6…
  • 81. Pete Cape, SSI, UK April Lecture Series 2014 Two men go into a bar…. Attitudes are expressed in words not numbers I say, what do you think of the new VW Golf? I think it’s a 6…
  • 82. Pete Cape, SSI, UK April Lecture Series 2014 Numbers are universal
  • 83. Pete Cape, SSI, UK April Lecture Series 2014 Numbers are universal • Numeric scales are not!
  • 84. Pete Cape, SSI, UK April Lecture Series 2014 Numbers are universal • Numeric scales are not! • It all starts in school — 0 = you weren’t even there! — 1-4 = you might as well not have been there… — 5-6 = you are going to have to work way harder in the future — 7-8 = okay, could do better — 9 = good — 10 = very good
  • 85. Pete Cape, SSI, UK April Lecture Series 2014 Numbers are universal • Numeric scales are not! • It all starts in school — 0 = you weren’t even there! — 1-4 = you might as well not have been there… — 5-6 = you are going to have to work way harder in the future — 7-8 = okay, could do better — 9 = good — 10 = very good • And this is for cultures that score 0 - 10, does everyone?
  • 86. Pete Cape, SSI, UK April Lecture Series 2014 German School System • 1 (sehr gut, very good) is the best possible grade and is given for outstanding performance • 2 (gut, good) is the next-highest and is given for performance that meets the standard completely and is above-average • 3 (befriedigend, satisfactory) indicates "average" performance. • 4 (ausreichend, sufficient) is the lowest passing grade and is given if the standard has been met but with a number of notable errors. • 5 (mangelhaft, deficient) is the higher of two failing grades and is given if the standard has not been met but the basics have been understood. • 6 (ungenügend, insufficient) is the lowest possible grade and is given if the standard has not been met and the basics have not been understood.
  • 87. Pete Cape, SSI, UK April Lecture Series 2014 Numbers have meaning 0 1 2 3 4 5 6 7 8 9 10 Schwartz, 1996
  • 88. Pete Cape, SSI, UK April Lecture Series 2014 Numbers have meaning 0 1 2 3 4 5 6 7 8 9 10 34% Schwartz, 1996
  • 89. Pete Cape, SSI, UK April Lecture Series 2014 Numbers have meaning 0 1 2 3 4 5 6 7 8 9 10 -5 -4 -3 -2 -1 0 1 2 3 4 5 34% Schwartz, 1996
  • 90. Pete Cape, SSI, UK April Lecture Series 2014 Numbers have meaning 0 1 2 3 4 5 6 7 8 9 10 -5 -4 -3 -2 -1 0 1 2 3 4 5 34% 13%? Schwartz, 1996
  • 91. Pete Cape, SSI, UK April Lecture Series 2014 Numbers have meaning 0 1 2 3 4 5 6 7 8 9 10 -5 -4 -3 -2 -1 0 1 2 3 4 5 34% 13% Schwartz, 1996
  • 92. Pete Cape, SSI, UK April Lecture Series 2014 This what scales are trying to achieve
  • 93. Pete Cape, SSI, UK April Lecture Series 2014 This what scales are trying to achieve
  • 94. Pete Cape, SSI, UK April Lecture Series 2014 • Turn to academia for the answer How long should a scale be?
  • 95. Pete Cape, SSI, UK April Lecture Series 2014 • Turn to academia for the answer • Is not always shared  • But is somewhere between 5 and 9 points How long should a scale be?
  • 96. Pete Cape, SSI, UK April Lecture Series 2014 • Turn to academia for the answer • Is not always shared  • But is somewhere between 5 and 9 points • Jon Krosnick says: • Unipolar: 5 points • Bipolar: 7 points How long should a scale be?
  • 97. Pete Cape, SSI, UK April Lecture Series 2014 • Turn to academia for the answer • Is not always shared  • But is somewhere between 5 and 9 points • Jon Krosnick says: • Unipolar: 5 points • Bipolar: 7 points • Why? • Maximises Reliability and Validity How long should a scale be?
  • 98. Pete Cape, SSI, UK April Lecture Series 2014 What is reliability and validity? • Reliability — Measures the same thing in the same way every time
  • 99. Pete Cape, SSI, UK April Lecture Series 2014 What is reliability and validity? • Reliability — Measures the same thing in the same way every time • Validity — Measures what you think it measures
  • 100. Pete Cape, SSI, UK April Lecture Series 2014 What was Unipolar and Bipolar? “Unipolar” One direction from zero to lots he middle and surfeit at the ends
  • 101. Pete Cape, SSI, UK April Lecture Series 2014 What was Unipolar and Bipolar? “Unipolar” One direction from zero to lots “Bipolar” Opposites with indifference in the middle he middle and surfeit at the ends
  • 102. Pete Cape, SSI, UK April Lecture Series 2014 Scales Unipolar: • Extremely x • Very x • Somewhat x • Slightly x • Not at all x (an) academic conclusion
  • 103. Pete Cape, SSI, UK April Lecture Series 2014 Scales Unipolar: • Extremely x • Very x • Somewhat x • Slightly x • Not at all x Bipolar: • Extremely x • Very x • Somewhat x • Neither x nor un-x • Somewhat un-x • Very un-x • Extremely un-x (an) academic conclusion
  • 104. Pete Cape, SSI, UK April Lecture Series 2014 Scales Unipolar: • Extremely x • Very x • Somewhat x • Slightly x • Not at all x Bipolar: • Extremely x • Very x • Somewhat x • Neither x nor un-x • Somewhat un-x • Very un-x • Extremely un-x And this is what I recommend when asked.... btw: you can’t mix bi- and unipolar scales (an) academic conclusion
  • 105. Pete Cape, SSI, UK April Lecture Series 2014 Scales • Design issues • Display issues
  • 106. Pete Cape, SSI, UK April Lecture Series 2014 Where to put “don’t know” • Visual and Conceptual midpoint
  • 107. Pete Cape, SSI, UK April Lecture Series 2014 Where to put “don’t know” • Visual and Conceptual midpoint Conceptual midpoint Visual midpoint
  • 108. Pete Cape, SSI, UK April Lecture Series 2014 Where to put “don’t know” • Visual and Conceptual midpoint
  • 109. Pete Cape, SSI, UK April Lecture Series 2014 Where to put “don’t know” • Visual and Conceptual midpoint Visual midpoint
  • 110. Pete Cape, SSI, UK April Lecture Series 2014 Where to put “don’t know” • Visual and Conceptual midpoint • Skews data towards “disagree” Conceptual midpoint Visual midpoint
  • 111. Pete Cape, SSI, UK April Lecture Series 2014 Where to put “don’t know” • Visual and Conceptual midpoint Conceptual midpoint Visual midpoint
  • 112. Pete Cape, SSI, UK April Lecture Series 2014 What order to present? Disagree Strongly Disagree Slightly Neither Agree nor Disagree Agree Slightly Agree Strongly
  • 113. Pete Cape, SSI, UK April Lecture Series 2014 What order to present? Disagree Strongly Disagree Slightly Neither Agree nor Disagree Agree Slightly Agree Strongly Order bias
  • 114. Pete Cape, SSI, UK April Lecture Series 2014 What order to present? Disagree Strongly Disagree Slightly Neither Agree nor Disagree Agree Slightly Agree Strongly Order bias Acquiescence bias
  • 115. Pete Cape, SSI, UK April Lecture Series 2014 What order to present? Disagree Strongly Disagree Slightly Neither Agree nor Disagree Agree Slightly Agree Strongly Order bias Acquiescence bias Central tendency
  • 116. Pete Cape, SSI, UK April Lecture Series 2014 What order to present? Disagree Strongly Disagree Slightly Neither Agree nor Disagree Agree Slightly Agree Strongly Order bias Acquiescence bias Central tendency Agree Strongly Agree Slightly Neither Agree nor Disagree Disagree Slightly Disagree Strongly Central tendency Order bias Acquiescence bias
  • 117. Pete Cape, SSI, UK April Lecture Series 2014 Where to put the answer box • Next to answers • 22% failure rate on matching gender to panel record!
  • 118. Pete Cape, SSI, UK April Lecture Series 2014 Where to put the answer box • Next to answers • 22% failure rate on matching gender to panel record! • Left right processing • 98% of that: Males coded as Females
  • 119. Pete Cape, SSI, UK April Lecture Series 2014 Scales • Maximises reliability and validity • Minimises bias • Prevents order error • Not just “the way we always do it…”
  • 120. Pete Cape, SSI, UK April Lecture Series 2014 Scales • Maximises reliability and validity • Minimises bias • Prevents order error • Not just “the way we always do it…” • But isn’t it old-fashioned?
  • 121. Pete Cape, SSI, UK April Lecture Series 2014 Sliders with visual elements
  • 122. Pete Cape, SSI, UK April Lecture Series 2014 129125 77 25 4 0 50 100 150 200 250 Different styles of slider scale
  • 123. Pete Cape, SSI, UK April Lecture Series 2014 Different styles of slider scale 114 8892 17 1 0 50 100 150 200 250
  • 124. Pete Cape, SSI, UK April Lecture Series 2014 Different styles of slider scale 97 21 27 8 1 0 50 100 150 200 250
  • 125. Pete Cape, SSI, UK April Lecture Series 2014 4 10 124 4751 0 50 100 150 200 250 Different styles of slider scale
  • 126. Pete Cape, SSI, UK April Lecture Series 2014 Different distributions, same means? UK USA Germany China traditional - raw data 2.27 2.32 2.50 3.79 traditional - rescored 2.60 2.28 2.50 3.87 slider with all labels 2.56 2.64 2.74 3.89 slider with all tick marks 2.33 2.35 2.46 3.61 slider with no tick marks 2.32 2.39 2.29 3.62 slider with no tick marks, score showing 2.34 2.44 2.33 3.56
  • 127. Pete Cape, SSI, UK April Lecture Series 2014 Grids and Matrices • Q: Do we have these in telephone research? • Q: In Face to Face?
  • 128. Pete Cape, SSI, UK April Lecture Series 2014 Grids and Matrices • Q: Do we have these in telephone research? • Q: In Face to Face? • A: No • Q: Where did we have them?
  • 129. Pete Cape, SSI, UK April Lecture Series 2014 Grids and Matrices • Q: Do we have these in telephone research? • Q: In Face to Face? • A: No • Q: Where did we have them? • A: postal research
  • 130. Pete Cape, SSI, UK April Lecture Series 2014 Item-in-a-series
  • 131. Pete Cape, SSI, UK April Lecture Series 2014 Item-in-a-series “When one has to ask a series of questions that use the same answer categories it is convenient to combine them into a [grid/matrix format]. Combining them into this ‘item-in-a- series’, with a common introduction that defines the general question and response format, eliminates considerable redundancy with regard to stating questions, it also saves considerable space”
  • 132. Pete Cape, SSI, UK April Lecture Series 2014 Item-in-a-series “but do it carefully”
  • 133. Pete Cape, SSI, UK April Lecture Series 2014 Item-in-a-series “but do it carefully” • Items are now in a comparative framework • Visual rendering makes them a unit “if the sponsor wants individuals to contemplate each item separately, it is advisable to present each of them as individual items”
  • 134. Pete Cape, SSI, UK April Lecture Series 2014 Item-in-a-series Should these should be the same?
  • 135. Pete Cape, SSI, UK April Lecture Series 2014 Item-in-a-series Shouldn’t these have been the same?
  • 136. Pete Cape, SSI, UK April Lecture Series 2014 | 136 | What else do grids do? • Encourage you to go too fast
  • 137. Pete Cape, SSI, UK April Lecture Series 2014 Speed of grid completion •What sort of processing is being done at this speed? | 137 |
  • 138. Pete Cape, SSI, UK April Lecture Series 2014 | 138 | What else do grids do? • Encourage you to go too fast • Process the underlying latent construct, not the items — Straightlining
  • 139. Pete Cape, SSI, UK April Lecture Series 2014 | 139 | What else do grids do? • Encourage you to go too fast • Process the underlying latent construct, not the items — Straightlining • Miss the subtleties — Heuristic answering
  • 140. Pete Cape, SSI, UK April Lecture Series 2014 Direct instruction at item 39/40 – pass/fail | 140 |
  • 141. Pete Cape, SSI, UK April Lecture Series 2014 Which heuristic was used? | 141 | answer at trap
  • 142. Pete Cape, SSI, UK April Lecture Series 2014 Which heuristic was used? | 142 | answer at trap
  • 143. Pete Cape, SSI, UK April Lecture Series 2014 And those left over? | 143 |
  • 144. Pete Cape, SSI, UK April Lecture Series 2014 And those left over? | 144 |
  • 145. Pete Cape, SSI, UK April Lecture Series 2014 | 145 | Anything else wrong with grids? • Yes, we use the wrong scale for the construct
  • 146. Pete Cape, SSI, UK April Lecture Series 2014 Beware of the agree-disagree question • Make your scale answers the appropriate ones To what extent to you agree or disagree that this training course is extremely useful? Agree extremely Very much agree Somewhat agree Neither agree nor disagree Somewhat disagree Very much disagree Disagree extremely
  • 147. Pete Cape, SSI, UK April Lecture Series 2014 2 stage process 1. Decide how useful you find this
  • 148. Pete Cape, SSI, UK April Lecture Series 2014 2 stage process 1. Decide how useful you find this 2. Translate into agree-disagree, hopefully: Extremely useful = Agree extremely Very useful = Very much agree Somewhat useful = Somewhat agree Neither useful not disuseful = Neither agree nor disagree Somewhat disuseful = Somewhat disagree Very disuseful = Very much disagree Extremely disuseful = Disagree extremely
  • 149. Pete Cape, SSI, UK April Lecture Series 2014 2 stage process 1. Decide how useful you find this 2. Translate into agree-disagree, hopefully: • Disuseful? Extremely useful = Agree extremely Very useful = Very much agree Somewhat useful = Somewhat agree Neither useful not disuseful = Neither agree nor disagree Somewhat disuseful = Somewhat disagree Very disuseful = Very much disagree Extremely disuseful = Disagree extremely
  • 150. Pete Cape, SSI, UK April Lecture Series 2014 2 stage process • Reality (maybe): Extremely useful = Agree extremely Very useful = Very much agree Somewhat useful = Somewhat agree Slightly useful = Somewhat agree Not at all useful = Neither/nor, Somewhat disagree,Very much disagree,Disagree extremely
  • 151. Pete Cape, SSI, UK April Lecture Series 2014 How to measure attitudes/constructs • Work out the underlying construct — i.e. what are you measuring
  • 152. Pete Cape, SSI, UK April Lecture Series 2014 How to measure attitudes/constructs • Work out the underlying construct — i.e. what are you measuring • Decide how it is measured — What words — Unipolar or Bipolar
  • 153. Pete Cape, SSI, UK April Lecture Series 2014 How to measure attitudes/constructs • Work out the underlying construct — i.e. what are you measuring • Decide how it is measured — What words — Unipolar or Bipolar • Check your measurement gaps are equal • Ask question..
  • 154. Pete Cape, SSI, UK April Lecture Series 2014 How to measure attitudes/constructs • Work out the underlying construct — i.e. what are you measuring • Decide how it is measured — What words — Unipolar or Bipolar • Check your measurement gaps are equal • Ask question.. • More work for you, less work for respondent • And more valid and reliable
  • 155. Pete Cape, SSI, UK April Lecture Series 2014 Construct Specific Scale How useful, if at all, did you find this training course? Extremely useful Very useful Somewhat useful Slightly useful Not at all useful • As you read the question your answer is forming….. ?
  • 156. Pete Cape, SSI, UK April Lecture Series 2014 Construct Specific Scale How useful, if at all, did you find this training course? Extremely useful Very useful Somewhat useful Slightly useful Not at all useful • As you read the question your answer is forming….. • Order effects?
  • 157. Pete Cape, SSI, UK April Lecture Series 2014 Construct Specific Scale Extremely useful Very useful Somewhat useful Slightly useful Not at all useful
  • 158. Pete Cape, SSI, UK April Lecture Series 2014 Construct Specific Scale Extremely useful Very useful Somewhat useful Slightly useful Not at all useful Expectation: up = good (Tourangeau)
  • 159. Pete Cape, SSI, UK April Lecture Series 2014 Construct Specific Scale Extremely useful Very useful Somewhat useful Slightly useful Not at all useful Expectation: up = good (Tourangeau) Answer speed = faster (Christian et al) No order bias found
  • 160. Pete Cape, SSI, UK April Lecture Series 2014 Manipulating people by accident • Because we want to make it more fun
  • 161. Pete Cape, SSI, UK April Lecture Series 2014 Please select your most preferred holiday type… Summer Beach Winter City Break Adventure
  • 162. Pete Cape, SSI, UK April Lecture Series 2014 Please select your most preferred holiday type… Summer Beach Winter City Break Adventure
  • 163. Pete Cape, SSI, UK April Lecture Series 2014 Please select your most preferred holiday type… Summer Beach Winter City Break Adventure
  • 164. Pete Cape, SSI, UK April Lecture Series 2014 The results 58% 60% 36% 0% 20% 40% 60% 80% 100% “Summer Beach”
  • 165. Pete Cape, SSI, UK April Lecture Series 2014 Motivation
  • 166. Pete Cape, SSI, UK April Lecture Series 2014 Motivation Self-Determination Theory (Deci & Ryan)
  • 167. Pete Cape, SSI, UK April Lecture Series 2014 Motivation • people are active ‘organisms’ • having tendencies toward psychological growth and development • who strive to master ongoing challenges • and to integrate their experiences into a coherent sense of self. • the social context can either support or thwart the natural tendencies Self-Determination Theory (Deci & Ryan)
  • 168. Pete Cape, SSI, UK April Lecture Series 2014 Motivation under Self-Determination Theory A continuum that relates to the task in the social context, not a person
  • 169. Pete Cape, SSI, UK April Lecture Series 2014 Motivation under Self-Determination Theory A continuum that relates to the task in the social context, not a person Regulation Introjection Identification Integration to obtain an externally imposed reward pressure to avoid guilt or to enhance self-esteem conscious valuing of activity assimilated to self task is not done for it’s own reward
  • 170. Pete Cape, SSI, UK April Lecture Series 2014 Motivation under Self-Determination Theory A continuum that relates to the task in the social context, not a person Regulation Introjection Identification Integration to obtain an externally imposed reward pressure to avoid guilt or to enhance self-esteem conscious valuing of activity assimilated to self task is not done for it’s own reward Where do most market research respondents fall?
  • 171. Pete Cape, SSI, UK April Lecture Series 2014 Motivation under Self-Determination Theory A continuum that relates to the task in the social context, not a person Regulation Introjection Identification Integration to obtain an externally imposed reward pressure to avoid guilt or to enhance self-esteem conscious valuing of activity assimilated to self task is not done for it’s own reward Where do most market research respondents fall?
  • 172. Pete Cape, SSI, UK April Lecture Series 2014 Motivation under Self-Determination Theory Task Outcomes Regulation Introjection Identification Integration little interest, value or effort more effort, anxiety, poor coping with failure enjoyment, good coping skills greater enjoyment, competency task is not done properly interest, enjoyment highly competent Where would we like respondents to be?
  • 173. Pete Cape, SSI, UK April Lecture Series 2014 Motivation under Self-Determination Theory Task Outcomes Regulation Introjection Identification Integration little interest, value or effort more effort, anxiety, poor coping with failure enjoyment, good coping skills greater enjoyment, competency task is not done properly interest, enjoyment highly competent Where would we like respondents to be?
  • 174. Pete Cape, SSI, UK April Lecture Series 2014 Moving people along the motivation continuum • Foster feelings of: — Autonomy – you are free to do this or not as you choose — Competence – you are good at this — Relatedness – people like you do this — Value – what you are doing has meaning
  • 175. Pete Cape, SSI, UK April Lecture Series 2014 Moving people along the motivation continuum • Foster feelings of: — Autonomy – you are free to do this or not as you choose — Competence – you are good at this — Relatedness – people like you do this — Value – what you are doing has meaning • Perhaps this was what the interviewer used to do? • This is what Gamification helps do
  • 176. Pete Cape, SSI, UK April Lecture Series 2014 Motivating respondents Thank you very much for choosing to do this important survey. As one of our most expert survey takers we have selected you to help on this survey. People like you all over the country are taking part in this survey. Your answers and those of everyone else will help improve the services our client provides to customers like you. Please press the > button when you are ready to continue with your survey.
  • 177. Pete Cape, SSI, UK April Lecture Series 2014 Motivating respondents Thank you very much for choosing to do this important survey. As one of our most expert survey takers we have selected you to help on this survey. People like you all over the country are taking part in this survey. Your answers and those of everyone else will help improve the services our client provides to customers like you. Please press the > button when you are ready to continue with your survey. • No mention of rewards • No threats
  • 178. Pete Cape, SSI, UK April Lecture Series 2014 Summary • The questionnaire is a mixture of art and science • The art of writing well, of motivating respondents • The science of writing precisely, avoiding bias • Everything has meaning • You are supposed to know what it means • There are better ways of asking questions • There are better ways of collecting answers • You might as well do the best you can
  • 179. Pete Cape, SSI, UK April Lecture Series 2014 Summary • Questionnaire writing • (and understanding why it is important) • Sets us apart from others • And has real value
  • 180. Pete Cape, SSI, UK April Lecture Series 2014 Summary - Principles • Respondents know they are in an experiment • They think everything they see, hear or read is deliberate, and has meaning • They want to please • Questionnaire motivates the respondent to provide — complete answers — accurate answers • It communicates to the respondent • It helps the respondent work out their answers • It makes the respondents task easy
  • 181. Pete Cape, SSI, UK April Lecture Series 2014 And guess what? • If it was easy do you think there would be textbooks on it?
  • 182. Pete Cape, SSI, UK April Lecture Series 2014 Thank you! Pete Cape Global Knowledge Director, SSI 2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
  • 183. Pete Cape, SSI, UK April Lecture Series 2014 The Sponsors for this Event If you are interested in sponsoring a future NewMR event Email Michele.Poynter@TheFuturePlace.com 2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
  • 184. Pete Cape, SSI, UK April Lecture Series 2014 Q & A Ray Poynter The Future Place Pete Cape SSI 2014 Platinum Sponsor April Series Sponsor Lecture Sponsor
  • 185. Pete Cape, SSI, UK April Lecture Series 2014 | 186 | surveysampling.com info@surveysampling.com pete.cape@surveysampling.com