Casro 2013: Battle of the Scales

314 views

Published on

Examination of 4,5,7,and 10 point scales in terms of extreme responding, acquiescent responding, and midpoint responding as well as a comparison of labelled and unlabeled points

Published in: Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
314
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Casro 2013: Battle of the Scales

  1. 1. Battle of the ScalesExamining Respondent Scale Usageacross 10 countriesBy Melanie Courtright, Kartik Pashupati,and Annie Pettit
  2. 2. Background 2
  3. 3. Battle of the ScalesBackgroundScales are a cornerstone of market researchThey’re how we determine that:• 49% of people like Coca-Cola and 42% of people like Pepsi • Men like watching sports more than women do• How citizens feel about the government • Canadians like Shania Twain more than Brits… or do they?? 3
  4. 4. Battle of the ScalesBackground 4
  5. 5. Battle of the ScalesSo Many Client Questions… Is there a difference in the reliability of attitudinal scales when using 4-point, 5- point, 7-point, and 10-point scales? Does excluding a neutral point impact the answers? Does labeling each point vs. only the end points produce different results? Can we replicate and extend the results of previous research on the impact of cultural factors on response styles? Do scales with greater variance (e.g., 7-point and 10-point scales) reduce Extreme Response Style (ERS) compared to lesser variance (e.g., 4-point and 5-point scales)? Conversely, do scales with greater variance (e.g., 7-point and 10-point scales) produce a greater incidence of Medium Response Style (MRS) compared to lesser variance (e.g., 4-point and 5-point scales)? 5
  6. 6. Battle of the ScalesResearch Plan 6
  7. 7. Battle of the ScalesResearch Plan 7-minute attitudinal survey Globally relevant topics Mix of positive and negative wording Scales with published measures of reliability Include behavioral statements that should correlate to attitudinal questions Simultaneously field across ten countries  Age and gender quota sampling by country Same sample source throughout  Research Now’s proprietary Valued Opinions Panel (VOP) First study to  Simultaneously compare the effect of multiple response options (4-, 5-, 7- and 10-point scales)  Using a large, census-balanced, multi-country sample 7
  8. 8. Battle of the Scales Research Plan  Fieldwork conducted December 2012 and January 2013  All five scale options tested in each of ten countries Sample 4 5 point 5 point 7 10 TOTAL Sizes point LABELED UNLABELED point point Brazil 500 250 250 500 500 2,000 China 500 250 250 500 500 2,000 France 500 250 250 500 500 2,000Germany 500 250 250 500 500 2,000 India 500 250 250 500 500 2,000 Japan 500 250 250 500 500 2,000 Mexico 500 250 250 500 500 2,000 Russia 500 250 250 500 500 2,000 UK 500 250 250 500 500 2,000 US 500 250 250 500 500 2,000 TOTAL 5,000 2,500 2,500 5,000 5,000 19,886 Note: For ease of reading, sample sizes have been rounded up or down by no more than 11. 8
  9. 9. Battle of the ScalesAnalysis Plan Three indices:  Extreme Response Style Index (ERSI) – Respondents who answered either extreme of scale were assigned a score of 1. Otherwise, they were assigned zero.  Acquiescence Response Style Index (ARSI) – Respondents who strongly agreed with an item were assigned a score of 1. Otherwise, they were assigned zero.  Medium Response Style Index (MRSI) – On scales with an odd number of options (i.e., 5 point, 7 point), respondents who answered exactly in the middle were assigned a medium response score of 1. Otherwise, they were assigned zero. Possible summary values ranged from 0.0 to 1. 9
  10. 10. Battle of the ScalesUse of Scale ResultsHypothesis 1: There will be no meaningful differences inERS, MRS or ARS indices between male and femalerespondents. Male Female ERS .43 .43 ARS .32 .33 MRS .28 .28Result 1 Confirmed: Men and women do not differ intheir response patterns. 10
  11. 11. Battle of the ScalesUse of Scale ResultsHypothesis 2: There will be significant but not systematicdifferences in ERS, ARS and MRS Indices across thedifferent age groups. .50 .46 .44 .44 .45 .43 .43 .41 .41 .41 .41  ERS (Extreme) .40 .33 .33 .35 .32 .32 .32 .31 .30 .28 .29 .29 .29 .29 .30 .28 .28 .28 .27 .27 .25  ARS (Acquiescent) .25 MRS (Medium) .20 15-17 18-24 25-34 35-44 45-54 55-64 65-74 75-84 85+Results 2 partially confirmed: There were significantdifferences but the differences were systematic. ERS andMRS gradually increased with age until 55-64, and thendeclined. The pattern for ARS (yea-saying) was reversed. 11
  12. 12. Battle of the ScalesUse of Scale ResultsHypothesis 3: There will be significant differences inERS, ARS, and MRS across the different countries..60 .53 .53.50 .48  MRS (Acquiescent) .45.40 .41 .41 .40 .40 .37 ▌ERS (Extreme) .31 .27 .29 .28 .28 .29 .29 .27 .26 .26.30 .33.20.10.00Results 3 confirmed: Respondents from Brazil and Mexicohave the highest tendency to give extreme responses.Respondents from Japan have a significantly lower ERSIndex, and significantly higher MRS (and ARS). 12
  13. 13. Battle of the ScalesUse of Scale ResultsHypothesis 4: Individualism (Hofstede) will correlate positivelywith ERS and negatively with MRS. Individualism will equateto stronger, and therefore more extreme, opinions..55 .49.50  ERS (Extreme).45 .41 .39 .38.40.35 .31  ARS (Acquiescent) .29 .28.30 .27 MRS (Medium).25.20 Low Medium High Individualism Results 4 not confirmed: Differences could not be attributed to individualism. Analysis actually showed a negative correlation with ERS. 13
  14. 14. Battle of the ScalesUse of Scale ResultsHypothesis 5: Respondents in countries that are higher inmasculinity (Hofstede) would exhibit higher ERS..50 .46.45 .42 .41  ERS (Extreme).40 .34.35 .32 .31.30 .28 .28 .30  ARS (Acquiescent).25 MRS (Medium) Low Medium High MasculinityResults 5 not confirmed: While masculinity did affect thedifferences, the results were not in the expected direction.Countries with lower masculinity demonstrated higher ERSindices. 14
  15. 15. Battle of the ScalesUse of Scale Results 15
  16. 16. Battle of the ScalesNumber of Scale Points ResultsHypothesis 6: The number of scale points and scale labelingwill affect ERS, MRS, and ARS..50 .45 .45.45 .42 .41 .41.40  ERS (Extreme) .34 .33 .33 .33.35 .32 .31 .32 .31.30 .27  ARS (Acquiescent).25 .20.20 MRS (Medium).154 point 5 point 5 point 7 point 10 point labeled unlabeledResults 6 confirmed: 7- and 10-point scales saw fewermedium responses. ERS, MRS, and ARS were all lower forthe 5-point labeled scale versus the unlabeled scale. 16
  17. 17. Battle of the Scales Number of Scale Points Results Hypothesis 7: The number of scale points and scale labeling will have an impact on scale reliability. Cronbachs alpha # of # of 5 point Total Reverse 4 5 point unlabel 7 10 Scale Items Items point labeled ed point pointHealth Environment Sensitivity 8 1 0.8 0.8 0.8 0.8 0.8Personal Health Responsibility 8 2 0.7 0.6 0.6 0.6 0.7Motorcycle Helmet Mandate 2 1 0.7 0.7 0.7 0.7 0.7Attitude toward helping others 4 0 0.9 0.85 0.9 0.9 0.9(AHO)Material Values Scale (MVS 9) 9 2 0.8 0.75 0.8 0.8 0.8Attitude toward Advertising in 7 4 0.7 0.7 0.7 0.7 0.7General (AAG)Online privacy concern 2 0 0.5 0.5 0.5 0.6 0.55Lie acceptability scale 8 4 0.8 0.8 0.8 0.8 0.8 Results 7 Not confirmed: There is no significant variation in the reliability of scales by number nor labeling of scale points. Note: For ease of reading, alphas were rounded. See the paper for precise values. 17
  18. 18. Battle of the ScalesSummary Findings ERS, MRS, and ARS do not differ by gender, but do differ by age Response styles vary by country – India, Mexico, Russia and Brazil are similar – The US and UK are similar, as are France and Germany. – Respondents from Japan are unique in terms of lower extreme and higher medium response styles Reasons for country differences are not yet isolated – Need more research and – Need scale norms that are available on a multi-country basis. Varying the number of response options – Does affect MRS – Does NOT impact scale reliability or ERS Scale labeling did not impact scale reliability, but did impact ERS, MRS, and ARS 18
  19. 19. Battle of the ScalesClosing Thoughts READ THE PAPER! 19
  20. 20. Battle of the ScalesClosing Thoughts 20
  21. 21. Battle of the ScalesClosing ThoughtsWhat about mobile?If number ofoptions andlabeling doimpact resultsand screen spaceis a luxury onmobile devices… 21
  22. 22. Battle of the ScalesClosing ThoughtsWhat about social media? Hideous Disgusting AbhorIf number of Crapoptions and Yuck Dumblabeling do Huhimpact results, Dunno Whatevswhat is the Goodcomplementary Nice Coolimpact on textual Awesomedata… Wicked Bomb 22
  23. 23. Battle of the ScalesMore to Come! 23

×