Your SlideShare is downloading. ×
Battle of the Scales: Examining Respondent Scale Usage across 10 countries
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Battle of the Scales: Examining Respondent Scale Usage across 10 countries

615
views

Published on

2013 CASRO Online Research Conference Presentation by: Melanie Courtright, Kartik Pashupati and Annie Pettit

2013 CASRO Online Research Conference Presentation by: Melanie Courtright, Kartik Pashupati and Annie Pettit

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
615
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Battle of the ScalesExamining Respondent Scale Usageacross 10 countriesBy Melanie Courtright, Kartik Pashupati,and Annie Pettit
  • 2. Background 2
  • 3. Battle of the ScalesBackgroundScales are a cornerstone of market researchThey’re how we determine that:• 49% of people like Coca-Cola and 42% of people like Pepsi • Men like watching sports more than women do• How citizens feel about the government • Canadians like Shania Twain more than Brits… or do they?? 3
  • 4. Battle of the ScalesBackground 4
  • 5. Battle of the ScalesSo Many Client Questions… Is there a difference in the reliability of attitudinal scales when using 4-point, 5- point, 7-point, and 10-point scales? Does excluding a neutral point impact the answers? Does labeling each point vs. only the end points produce different results? Can we replicate and extend the results of previous research on the impact of cultural factors on response styles? Do scales with greater variance (e.g., 7-point and 10-point scales) reduce Extreme Response Style (ERS) compared to lesser variance (e.g., 4-point and 5-point scales)? Conversely, do scales with greater variance (e.g., 7-point and 10-point scales) produce a greater incidence of Medium Response Style (MRS) compared to lesser variance (e.g., 4-point and 5-point scales)? 5
  • 6. Battle of the ScalesResearch Plan 6
  • 7. Battle of the ScalesResearch Plan 7-minute attitudinal survey Globally relevant topics Mix of positive and negative wording Scales with published measures of reliability Include behavioral statements that should correlate to attitudinal questions Simultaneously field across ten countries  Age and gender quota sampling by country Same sample source throughout  Research Now’s proprietary Valued Opinions Panel (VOP) First study to  Simultaneously compare the effect of multiple response options (4-, 5-, 7- and 10-point scales)  Using a large, census-balanced, multi-country sample 7
  • 8. Battle of the Scales Research Plan  Fieldwork conducted December 2012 and January 2013  All five scale options tested in each of ten countries Sample 4 5 point 5 point 7 10 TOTAL Sizes point LABELED UNLABELED point point Brazil 494 255 253 486 488 1976 China 496 256 251 501 499 2003 France 497 252 249 497 499 1994Germany 496 249 250 499 499 1993 India 489 243 245 494 490 1961 Japan 497 251 249 497 498 1992 Mexico 496 249 250 493 497 1985 Russia 497 248 249 495 493 1982 UK 506 250 249 499 501 2005 US 498 251 249 497 500 1995 TOTAL 4966 2504 2494 4958 964 19,886 8
  • 9. Battle of the ScalesAnalysis Plan Three indices:  Extreme Response Style Index (ERSI) – Respondents who answered either extreme of scale were assigned a score of 1. Otherwise, they were assigned zero.  Acquiescence Response Style Index (ARSI) – Respondents who strongly agreed with an item were assigned a score of 1. Otherwise, they were assigned zero.  Medium Response Style Index (MRSI) – On scales with an odd number of options (i.e., 5 point, 7 point), respondents who answered exactly in the middle were assigned a medium response score of 1. Otherwise, they were assigned zero. Possible summary values ranged from 0.0 to 1. 9
  • 10. Battle of the ScalesUse of Scale ResultsHypothesis 1: There will be no meaningful differences inERS, MRS or ARS indices between male and femalerespondents. Male Female ERS .43 .43 ARS .32 .33 MRS .28 .28Result 1 Confirmed: Men and women do not differ intheir response patterns. 10
  • 11. Battle of the ScalesUse of Scale ResultsHypothesis 2: There will be significant but not systematicdifferences in ERS, ARS and MRS Indices across thedifferent age groups. .50 .46 .44 .44 .45 .43 .43 .41 .41 .41 .41  ERS (Extreme) .40 .35 .33 .33 .32 .32 .32 .31 .30 .28 .29 .29 .29 .29 .30 .28 .28 .28 .27 .27 .25  ARS (Acquiescent) .25 ▲MRS (Medium) .20 15-17 18-24 25-34 35-44 45-54 55-64 65-74 75-84 85+Results 2 partially confirmed: There were significantdifferences but the differences were systematic. ERS andMRS gradually increased with age until 55-64, and thendeclined. The pattern for ARS (yea-saying) was reversed. 11
  • 12. Battle of the ScalesUse of Scale ResultsHypothesis 3: There will be significant differences in ERS,ARS, and MRS across the different countries..60 .53 .53 .48  MRS (Acquiescent).50 .45 .41 .41 .40 .40 .37 ▌ERS (Extreme).40 .31 .29 .28 .28 .29 .29 .27 .26 .26 .27.30 .33.20.10.00 Results 3 confirmed: Respondents from Brazil and Mexico have the highest tendency to give extreme responses. Respondents from Japan have a significantly lower ERS Index, and significantly higher MRS (and ARS). 12
  • 13. Battle of the ScalesUse of Scale ResultsHypothesis 4: Individualism (Hofstede) will correlate positivelywith ERS and negatively with MRS. Individualism will equateto stronger, and therefore more extreme, opinions..55 .49.50  ERS (Extreme).45 .41 .39 .38.40.35 .31  ARS (Acquiescent) .29 .28.30 .27 ▲MRS (Medium).25.20 Low Medium High Individualism Results 4 not confirmed: Differences could not be attributed to individualism. Analysis actually showed a negative correlation with ERS. 13
  • 14. Battle of the ScalesUse of Scale ResultsHypothesis 5: Respondents in countries that are higher inmasculinity (Hofstede) would exhibit higher ERS..50 .46.45 .41 .42  ERS (Extreme).40 .34.35 .32 .31 .30  ARS (Acquiescent).30 .28 .28 ▲MRS (Medium).25 Low Medium High MasculinityResults 5 not confirmed: While masculinity did affect thedifferences, the results were not in the expected direction.Countries with lower masculinity demonstrated higher ERSindices. 14
  • 15. Battle of the ScalesUse of Scale Results 15
  • 16. Battle of the ScalesNumber of Scale Points ResultsHypothesis 6: The number of scale points and scale labelingwill affect ERS, MRS, and ARS..50 .45 .45.45 .42 .41 .41.40  ERS (Extreme) .34 .33.35 .33 .32 .33 .32 .31 .31.30 .27  ARS (Acquiescent).25 .20 ▲MRS (Medium).20.154 point 5 point 5 point 7 point 10 point labeled unlabeledResults 6 confirmed: 7- and 10-point scales saw fewermedium responses. ERS, MRS, and ARS were all lower forthe 5-point labeled scale versus the unlabeled scale. 16
  • 17. Battle of the Scales Number of Scale Points Results Hypothesis 7: The number of scale points and scale labeling will have an impact on scale reliability. Cronbachs alpha # of # of 5 point Total Reverse 4 5 point unlabel 7 10 Scale Items Items point labeled ed point pointHealth Environment Sensitivity 8 1 0.8 0.8 0.8 0.8 0.8Personal Health Responsibility 8 2 0.7 0.6 0.6 0.6 0.7Motorcycle Helmet Mandate 2 1 0.7 0.7 0.7 0.7 0.7Attitude toward helping others 4 0 0.9 0.85 0.9 0.9 0.9(AHO)Material Values Scale (MVS 9) 9 2 0.8 0.75 0.8 0.8 0.8Attitude toward Advertising in 7 4 0.7 0.7 0.7 0.7 0.7General (AAG)Online privacy concern 2 0 0.5 0.5 0.5 0.6 0.55Lie acceptability scale 8 4 0.8 0.8 0.8 0.8 0.8 Results 7 Not confirmed: There is no significant variation in the reliability of scales by number nor labeling of scale points. Note: For ease of reading, alphas were rounded. See the paper for precise values. 17
  • 18. Battle of the ScalesSummary Findings ERS, MRS, and ARS do not differ by gender, but do differ by age Response styles vary by country – India, Mexico, Russia and Brazil are similar – The US and UK are similar, as are France and Germany. – Respondents from Japan are unique in terms of lower extreme and higher medium response styles Reasons for country differences are not yet isolated – Need more research and – Need scale norms that are available on a multi-country basis. Varying the number of response options – Does affect MRS – Does NOT impact scale reliability or ERS Scale labeling did not impact scale reliability, but did impact ERS, MRS, and ARS 18
  • 19. Battle of the ScalesClosing Thoughts READ THE PAPER! 19
  • 20. Battle of the ScalesClosing Thoughts 20
  • 21. Battle of the ScalesClosing ThoughtsWhat about mobile?If number ofoptions andlabeling doimpact resultsand screen spaceis a luxury onmobile devices… 21
  • 22. Battle of the ScalesClosing ThoughtsWhat about social media? Hideous Disgusting AbhorIf number of Crapoptions and Yuck Dumblabeling do Huhimpact results, Dunno Whatevswhat is the Goodcomplementary Nice Coolimpact on textual Awesomedata… Wicked Bomb 22
  • 23. Battle of the ScalesMore to Come! 23