TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
Research association presentation section 3
1. Battle Of The Scales
49%
42%
Scales are a cornerstone of market research
2. Measurement Scales & Response Styles
A person’s tendency to systematically respond to
questionnaire items in a certain way regardless of content
Paulhus
1991
ARS
MRS
ERS
3. Two studies to
examine scale
usage across
two modes of
data collection
Scope Of This Paper
Mobile
surveys
Web
surveys
4. Overall Research Plan
Web and Mobile
Include behavioral
statements that
should correlate to
attitudinal questions
Globally
relevant topics
7-minute
attitudinal survey
Mix of positive &
negative wording
Scales with published
measures of reliability
Fielded in multiple
countries to analyze
cross-cultural
differences
5. Battle of the Scales
Analysis Plan for Studies 1 & 2
Extreme
Response
Style Index
(ERSI)
Acquiescence
Response
Style Index
(ARSI)
Medium
Response
Style Index
(MRSI)
1 1 1
Indices were
averaged for
respondents
Value of each
index ranges
from a
minimum of 0
to a maximum
of 1.0.
6. Overall Research Plan
Web & Mobile Surveys
Used only 5-point Likert
scales for all constructs
Used only 5-point Likert
scales for all constructs
Slider scales on native app
Slider scales on mobile web
browser
Standard grid questions on
mobile web browser
Fielded in 10 countries
4-point
scales
5-point
unlabeled
scales
5-point
labeled
scales
7-point
scales
10-point
scales
7. Research Questions & Hypotheses: Study 2
Differences in Response Styles Across Countries, Genders & Ages
Acquiescent Midpoint Extreme
H1: There will be no
differences between
male & female
respondents with
respect to
ERS & MRS
H2: ERS & MRS will
differ across age
categories, but the
relationship between
age & ERS or MRS
will not be monotonic
H3: There will be
significant differences
across countries with
respect to ERS, MRS
& ARS
8. Results 1: Study 2
Impact of Gender on Response Styles
Hypothesis 1: There will be no meaningful differences in ERS, MRS or ARS indices between
male & female respondents
.43 .32 .28
.43 .33 .28
ERSI ARSI MRSI
9. Results 2: Study 2
Impact of Age on Response Styles
15-17 18-24 25-34 35-44
50
45
30
25
20
0
45-54 55-64 65-74
35
Hypothesis 2: ERS & MRS will differ across age categories,
but the relationship between age & ERS or MRS will not be monotonic
40
75-84 85+
MRS (Medium) ARS (Acquiescent) ERS (Extreme)
10. Results 3: Study 2
Differences in Response Styles Across Countries
60
30
20
10
0
40
H3: There will be significant differences across countries with respect to ERS, MRS & ARS
50
ERS (Extreme) ARS (Acquiescent)
11. Results 4: Study 2
Impact of Response Scales on Response Styles
4 Point 5 Point Labelled
50
45
30
25
20
0
5 Point Unlabelled 7 Point
35
RQ3: Does the use of 4-point, 5-point, 7-point & 10-point scales make any difference to
the level of extreme, acquiescent or midpoint responding?
40
10 Point
MRS (Medium) ARS (Acquiescent) ERS (Extreme)
12. Scope Of This Paper
What is the effect of using 3 alternative data collection options
on response styles in mobile surveys?
Compare differences in responses to 5-point Likert scales
across 3 data collection options:
Standard grid questions
on mobile web browser
Slider scales on mobile
web browser
Slider scales on
native app
13. Results 1: Study 1
Differences in Extreme Response Style Across Modes
Health
environment
sensitivity
Personal
health
responsibility
Attitude
toward
helping others
Material
values
scale
70
60
40
30
20
0
Extreme response style
(ERS) index
Attitude towards
advertising in
general (AAG)
Online
piracy
concern
Lie
tolerance
scale
50
Online web labeled Online web unlabeled Mobile native app
Mobile web grid Mobile web slider
14. Results 2: Study 1
Health
environment
sensitivity
Personal
health
responsibility
Attitude
toward
helping others
Material
values
scale
50
30
20
10
0
Midpoint Response Style
(MRS) Index
Differences in Midpoint Response Style Across Modes
Attitude towards
advertising in
general (AAG)
Online
piracy
concern
Lie
tolerance
scale
40
Online web labeled Online web unlabeled Mobile native app
Mobile web grid Mobile web slider
15. Results 3: Study 1
Health
environment
sensitivity
Personal
health
responsibility
Attitude
toward
helping others
Material
values
scale
80
60
40
20
0
Acquiescence Response
Style (ARS) Index
Differences in Acquiescence Response Style Across Modes
Attitude towards
advertising in
general (AAG)
Online
piracy
concern
Lie
tolerance
scale
Online web labeled Online web unlabeled Mobile native app
Mobile web grid Mobile web slider
In survey research we rely on scales to help us understand people’s respoonses.We use scales to understand how people perceive brandsTo understand their interestsTo understand their intentionsTo understand their attitudesUnderstanding the differences in opinions from one individual to another is central to market research.It is through the use of scales that we attempt gauge these differences.
It is agreed that when responding - surroundings, culture, or even the actual nature of the content of the researchers question may cause a difference in they way a respond.Another factor that is not often considered is that people do have the tendency to respond to questions, in different ways, irrespective of the content.This is particularly the case when people respond through using rating scales.Academic DelroyPaulhus defines this phenomenon as ‘a persons tendency to systematically respond to questionnaire items in a certain way regardless of content’Over the years many different response patterns have been identified and reported on.We are going to focus on three main response patterns:Extreme Response style (ERS)Acquiescence Response Style (ARS)Mid-Point Response Style (MRS)The Extreme Response Style is characterised by respondents consistently responding at either end of the scale- for a given series of statements respondents will give a consistently high positive, or highly negative responseThe Acquiescence Response Style (ARS) is a subset of the ERS but identified by a consistently positive scoreThe Mid Point response Style (MRS)- funnily enough refers to the style whereby respondents consistently give a score that sits in the middle of the scale
Our analysis approach came up with an index for each - essentially measuring the propensity to follow a particular response style within a set of scaled responses.Today we are going to investigate to the presence of these response styles using a scaled response across two different mode: web and mobile surveys
We wanted to ensure that the results delivered we robust and provided a good and accurate representation of the effects that different scales have on response styles.In order to do this the WEB study was based on the following factorsThe topics were globally relevant to take into account differences from market to marketThere was a obvious mix of positive and negative wording in the questionnaireThe scales that we used were published with measures of reliabilityWe included behavioural statements that would correlate to the attitudinal questions
Firstly we looked at Web surveys:We designed an online study which was carried out in 10 countries. Sample size of N=2000 in each countryThe survey was 7 minutes in lengthWe look at 4, 5, 7 and 10 point scales AND labled vs unlabled 5 point scales
For the web study we had 3 hypothesis to test and one additional research question to look at:We anticipated that there would be no difference between male and female respondents with regards to ERS and MRSERS and MRS will differ across age categories but the relationship between age and ERS and MRS will not be monotonic We also believed that there would be differences across countries with respoect the ERS, MRS and ARSFinally the study also gave us an opportunity to look at the impact that the different length of each scale would have on response styles.
Looking at the response style indicies for Male and female, we saw no real difference. This to us gave strength to our initial hypothesis that there would be no meaningful difference in ERS, MRS or ARS incidies between males and female. Just so you are aware the indexes are worked out in the following manner.Extreme Response Style IndexIf respondents answered on either extreme of a scale, i.e. if they strongly agreed or strongly disagreed with an item (e.g., 1or 5 on a 5-point scale, 1 or 7 on a 7-point scale) they were assigned an extreme response score of 1, otherwise theywere assigned a score of zero. Acquiescence Response Style IndexIf respondents strongly agreed with an item (e.g., 5 on a 5-point scale, 7 on a 7-point scale) they were assigned anacquiescence response score of 1, otherwise they were assigned a score of zero. All acquiescence response scores withina group were averaged to yield an ARS Index (ARSI), with possible values ranging from 0.0 to 1.0.Midpoint Response Style IndexOn odd-numbered scales (i.e., 5-point labeled and unlabeled, and 7-point scales), if respondents answered exactly in themiddle (i.e., 3 on a 5-point scale, or 4 on a 7-point scale) they were assigned an Midpoint response score of 1, otherwisethey were assigned a score of zero. All midpoint response scores within a group were averaged to yield an MRS Index(MRSI), with possible values ranging from 0.0 to 1.0.
Looking at the age bands now.We saw some small differences in the response style indices. In general extreme responding and mid pont responding seemed to increase with age up to the 55-64 age bracket before declining.Although the differences are very small it does provide us with some backing to suggest that response styles do differ with age.
As you know our third hypothesis is concerned with the difference in response styles across different countries.Whilst it may not be overly surprising to some our data did show some differences market on market.When we look at respondents from Brazil and Mexico, for example we see that they have the greatest tendency to give extreme responses.On the other hand respondents in Japan were proved more likely to give a mid point score.
Finally looking at web surveys we wanted to look at the impact of the number of response options available.Did the response style differ according to the length of the scale?The Extreme response and acquiescent styles are fairly consistent across the different scale lengths though they do dip slightly for the 5 point labeled scaleThe Mid point response style is slightly more interesting. It also dips for a five point labelled scale (clearly the labels are having an impact on all response style); however the tendency to select a mid point also drops with the larger scales.
For the mobile element of this study:We compared the differences in responses across 3 data collection optionsStandard grid questions on a mobile web browserSlider scales on a mobile web browserSlider scales on a native appWe interview N=1500 in the USWe used 5 point Likert (agreement) scales for all statements And looked at a variety of different statement typesHealth environment sensitivityPersonal health responsibilityAttitude towards helping othersMaterial valuesAttitude towards advertisingOnline privacy concernsLie acceptability scaleAdditionally we included statements that were reversed scaled. This means that certain statements were worded in a positive fashion whilst some were worded with a negative slant
Although there are significant differences in ERSI between the different measurement modes, the differences are not attributable to a single mode of measurement. The mobile native app produced the highest levels of extreme responding for 4 of the scales but the ERSI was only significantly and meaningfully higher for 2 of the scales.The only observable pattern is that the labeled version of the online web survey generally produced lower extreme responding than the unlabeled version.
The results of the multi-mode comparison for midpoint responding style (MRS) are presented in this graph. There is at least one s trend evident here, namely, that the mobile web slider consistently produces the lowest MRSI for all of the scales – although the difference is statistically significant only for 3 out of 6 scales.
When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data indicates that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
