2013 CASRO Online Research Conference Presentation by: Melanie Courtright, Kartik Pashupati and Annie Pettit

1. Battle of the Scales
Examining Respondent Scale Usage
across 10 countries
By Melanie Courtright, Kartik Pashupati,
and Annie Pettit

2. Background
2

3. Battle of the Scales
Background
Scales are a cornerstone of market research
They’re how we determine that:
• 49% of people like Coca-Cola and 42% of people
like Pepsi
• Men like watching sports more than
women do
• How citizens feel about the government
• Canadians like Shania Twain more than
Brits… or do they??
3

4. Battle of the Scales
Background
4

5. Battle of the Scales
So Many Client Questions…
 Is there a difference in the reliability of
attitudinal scales when using 4-point, 5-
point, 7-point, and 10-point scales?
 Does excluding a neutral point impact the
answers?
 Does labeling each point vs. only the end
points produce different results?
 Can we replicate and extend the results of previous research on the
impact of cultural factors on response styles?
 Do scales with greater variance (e.g., 7-point and 10-point scales)
reduce Extreme Response Style (ERS) compared to lesser variance
(e.g., 4-point and 5-point scales)?
 Conversely, do scales with greater variance (e.g., 7-point and 10-point
scales) produce a greater incidence of Medium Response Style (MRS)
compared to lesser variance (e.g., 4-point and 5-point scales)?
5

6. Battle of the Scales
Research Plan
6

7. Battle of the Scales
Research Plan
 7-minute attitudinal survey
 Globally relevant topics
 Mix of positive and negative wording
 Scales with published measures of reliability
 Include behavioral statements that should correlate to
attitudinal questions
 Simultaneously field across ten countries
 Age and gender quota sampling by country
 Same sample source throughout
 Research Now’s proprietary Valued Opinions Panel (VOP)
 First study to
 Simultaneously compare the effect of multiple response
options (4-, 5-, 7- and 10-point scales)
 Using a large, census-balanced, multi-country sample
7

8. Battle of the Scales
Research Plan
 Fieldwork conducted December 2012 and January 2013
 All five scale options tested in each of ten countries
Sample 4 5 point 5 point 7 10
TOTAL
Sizes point LABELED UNLABELED point point
Brazil 494 255 253 486 488 1976
China 496 256 251 501 499 2003
France 497 252 249 497 499 1994
Germany 496 249 250 499 499 1993
India 489 243 245 494 490 1961
Japan 497 251 249 497 498 1992
Mexico 496 249 250 493 497 1985
Russia 497 248 249 495 493 1982
UK 506 250 249 499 501 2005
US 498 251 249 497 500 1995
TOTAL 4966 2504 2494 4958 964 19,886
8

9. Battle of the Scales
Analysis Plan
Three indices:
 Extreme Response Style Index (ERSI)
– Respondents who answered either extreme of scale were
assigned a score of 1. Otherwise, they were assigned zero.
 Acquiescence Response Style Index (ARSI)
– Respondents who strongly agreed with an item were
assigned a score of 1. Otherwise, they were assigned zero.
 Medium Response Style Index (MRSI)
– On scales with an odd number of options (i.e., 5 point, 7
point), respondents who answered exactly in the middle
were assigned a medium response score of 1. Otherwise,
they were assigned zero.
Possible summary values ranged from 0.0 to 1.
9

10. Battle of the Scales
Use of Scale Results
Hypothesis 1: There will be no meaningful differences in
ERS, MRS or ARS indices between male and female
respondents.
Male Female
ERS .43 .43
ARS .32 .33
MRS .28 .28
Result 1 Confirmed: Men and women do not differ in
their response patterns.
10

11. Battle of the Scales
Use of Scale Results
Hypothesis 2: There will be significant but not systematic
differences in ERS, ARS and MRS Indices across the
different age groups.
.50
.46
.44 .44
.45 .43 .43
.41 .41 .41 .41
 ERS (Extreme)
.40
.35 .33 .33
.32 .32 .32
.31
.30
.28 .29 .29 .29
.29
.30 .28 .28 .28
.27 .27
.25  ARS (Acquiescent)
.25
▲MRS (Medium)
.20
15-17 18-24 25-34 35-44 45-54 55-64 65-74 75-84 85+
Results 2 partially confirmed: There were significant
differences but the differences were systematic. ERS and
MRS gradually increased with age until 55-64, and then
declined. The pattern for ARS (yea-saying) was reversed.
11

12. Battle of the Scales
Use of Scale Results
Hypothesis 3: There will be significant differences in ERS,
ARS, and MRS across the different countries.
.60
.53 .53
.48
 MRS (Acquiescent)
.50 .45
.41 .41 .40 .40
.37
▌ERS (Extreme)
.40
.31
.29 .28 .28 .29 .29
.27 .26 .26 .27
.30
.33
.20
.10
.00
Results 3 confirmed: Respondents from Brazil and Mexico
have the highest tendency to give extreme responses.
Respondents from Japan have a significantly lower ERS
Index, and significantly higher MRS (and ARS).
12

13. Battle of the Scales
Use of Scale Results
Hypothesis 4: Individualism (Hofstede) will correlate positively
with ERS and negatively with MRS. Individualism will equate
to stronger, and therefore more extreme, opinions.
.55
.49
.50
 ERS (Extreme)
.45 .41
.39
.38
.40
.35
.31  ARS (Acquiescent)
.29 .28
.30 .27
▲MRS (Medium)
.25
.20
Low Medium High
Individualism
Results 4 not confirmed: Differences could not be attributed
to individualism. Analysis actually showed a negative
correlation with ERS.
13

14. Battle of the Scales
Use of Scale Results
Hypothesis 5: Respondents in countries that are higher in
masculinity (Hofstede) would exhibit higher ERS.
.50
.46
.45
.41 .42
 ERS (Extreme)
.40
.34
.35
.32
.31
.30  ARS (Acquiescent)
.30 .28 .28
▲MRS (Medium)
.25
Low Medium High
Masculinity
Results 5 not confirmed: While masculinity did affect the
differences, the results were not in the expected direction.
Countries with lower masculinity demonstrated higher ERS
indices.
14

15. Battle of the Scales
Use of Scale Results
15

16. Battle of the Scales
Number of Scale Points Results
Hypothesis 6: The number of scale points and scale labeling
will affect ERS, MRS, and ARS.
.50
.45 .45
.45 .42
.41 .41
.40  ERS (Extreme)
.34 .33
.35 .33 .32
.33
.32
.31 .31
.30 .27
 ARS (Acquiescent)
.25
.20
▲MRS (Medium)
.20
.15
4 point 5 point 5 point 7 point 10 point
labeled unlabeled
Results 6 confirmed: 7- and 10-point scales saw fewer
medium responses. ERS, MRS, and ARS were all lower for
the 5-point labeled scale versus the unlabeled scale.
16

17. Battle of the Scales
Number of Scale Points Results
Hypothesis 7: The number of scale points and scale
labeling will have an impact on scale reliability.
Cronbach's alpha
# of # of 5 point
Total Reverse 4 5 point unlabel 7 10
Scale Items Items point labeled ed point point
Health Environment Sensitivity 8 1 0.8 0.8 0.8 0.8 0.8
Personal Health Responsibility 8 2 0.7 0.6 0.6 0.6 0.7
Motorcycle Helmet Mandate 2 1 0.7 0.7 0.7 0.7 0.7
Attitude toward helping others
4 0 0.9 0.85 0.9 0.9 0.9
(AHO)
Material Values Scale (MVS 9) 9 2 0.8 0.75 0.8 0.8 0.8
Attitude toward Advertising in
7 4 0.7 0.7 0.7 0.7 0.7
General (AAG)
Online privacy concern 2 0 0.5 0.5 0.5 0.6 0.55
Lie acceptability scale 8 4 0.8 0.8 0.8 0.8 0.8
Results 7 Not confirmed: There is no significant variation in
the reliability of scales by number nor labeling of scale
points.
Note: For ease of reading, alphas were rounded. See the paper for precise values.
17

18. Battle of the Scales
Summary Findings
 ERS, MRS, and ARS do not differ by gender, but do differ
by age
 Response styles vary by country
– India, Mexico, Russia and Brazil are similar
– The US and UK are similar, as are France and Germany.
– Respondents from Japan are unique in terms of lower extreme and
higher medium response styles
 Reasons for country differences are not yet isolated
– Need more research and
– Need scale norms that are available on a multi-country basis.
 Varying the number of response options
– Does affect MRS
– Does NOT impact scale reliability or ERS
 Scale labeling did not impact scale reliability, but did
impact ERS, MRS, and ARS
18

19. Battle of the Scales
Closing Thoughts
READ THE PAPER!
19

20. Battle of the Scales
Closing Thoughts
20

21. Battle of the Scales
Closing Thoughts
What about mobile?
If number of
options and
labeling do
impact results
and screen space
is a luxury on
mobile devices…
21

22. Battle of the Scales
Closing Thoughts
What about social media? Hideous
Disgusting
Abhor
If number of Crap
options and Yuck
Dumb
labeling do Huh
impact results, Dunno
Whatevs
what is the Good
complementary Nice
Cool
impact on textual Awesome
data… Wicked
Bomb
22

23. Battle of the Scales
More to Come!
23