Practical guidance, rooted in empirical research and theory, on how to demonstrate the advantages of quality mobile research to clients. Mobile phone surveys were tested using the practical framework for assessment developed by survey methodologists and statisticians. Survey quality includes all influences on the accuracy of survey estimates as well as factors such as relevance and timeliness. The conclusions drawn about the strengths and weaknesses of mobile phone surveys will lead to practical guidance on how to demonstrate the advantages of mobile research to clients.

1. Mobile Phone Surveys from a Total Survey
Quality Perspective
Peter Lynn
Institute for Social and Economic Research
University of Essex, UK
Mobile Research Conference
19 April 2011
© Peter Lynn

2. 1.
Accuracy of Survey Estimates (Variance and Bias)

3. 2.
Accuracy of Survey Estimates (Sources)
Mean Square Error
Variance Bias
Errors of Non-observation Observational Errors
Errors of Non-observation Observational Errors
Coverage Sampling Non-response Interviewer Respondent Instrument Coder
Coverage Sampling Non-response Interviewer Respondent Instrument Coder

4. 3.
Total Survey Quality
Relevance
Accuracy
Timeliness
Accessibility
Interpretability
Coherence
(Statistics Canada, 2009)

5. 4.
Role of Mobile Phone Surveys
Focus here on interviewing, not self-completion
i.e. respondent is listening and talking, not reading and keying
Stand-alone mode or within multi-mode
Focus is on accuracy of survey estimates

6. 5.
Coverage Error
Initial interest was in mobile phone ownership as a problem:
Worsening coverage of landline frames (Finland, Slovenia etc in 1990s,
USA in 2000s; need for dual-frame approaches
Then some interest in improved coverage for phone surveys:
Sub-Saharan Africa, Eastern Europe
Now, opportunities for improving coverage in mixed-mode surveys:
Germany, UK, etc
(Blumberg and Luke 2007, Kuusela et al 2008, Tortora et al, 2008, Gallup Europe 2009,
Häder & Häder 2009)

7. 6.
Coverage: UK Example
Since 1980s: Frames that only permit F2F have provided 98 – 99%
coverage. Coverage biased, but more important for some survey topics
(social exclusion) than others (purchasing)
But, F2F fieldwork has been becoming more expensive, more difficult, and
response rates falling
Fixed phone coverage reached 95 – 96% in 1990s, but falling since. Again,
coverage biased
Meanwhile, internet penetration grew rapidly

8. 7.
UK General Population Surveys
None of fixed phones, mobile phones, internet provide good coverage
But around 99% of households have at least one of those three.
Points towards mixed-mode and/ or mixed-frame surveys to achieve good
coverage without reliance on F2F

9. 8.
Role of Mobile Phones: Coverage
General Population Surveys: Permits high coverage rates when combined
with other frames/modes
Some Specialist Surveys: Adequate coverage rates as sole mode
Dual-Frame Issues: Properties of frames constantly changing, so
efficiencies change. Conflict between accurate estimation of selection
probabilities and over-loading the interview with seemingly meaningless
questions

10. 9.
Sampling Error
Main feature affected if mobile phone numbers used as sampling elements
is selection probabilities
Personal Surveys:
- Mobiles as single mode can help to reduce sampling error, through
closer correspondence of persons to mobiles
- Mobiles in multi-mode/frame less efficient (less close correspondence
of persons to phone numbers) but still tends to reduce sampling error
compared to landline-only surveys
Household Surveys:
- Mobiles tend to increase sampling error

11. 10.
Role of Mobile Phones: Sampling
Only relevant when mobile phone numbers used as sampling elements
Effect is generally positive (more accuracy) for surveys of individuals,
rather than households
Many uses of mobiles do not involve sampling mobile phone numbers –
e.g. if number is obtained from a sample selected in another way

12. 11.
Non-Response
Early evidence that individuals less likely to participate in a survey if
approached by mobile rather than fixed phone
This now seems far less clear
Contact is quicker and contact rates higher with mobiles
But marginal effect typically more important than substitute effect
This can be positive or negative depending on the survey protocols
Likely to be positive when used as an adjunct to other modes/devices, e.g.
follow up to non-respondents
Can also be used as a mode for alerts/reminders
(Fuchs 2000, Lau 2004, Vehovar et al 2004, Bosnjak et al 2010, Lynn 2011)

13. 12.
Measurement
Measurement differences between fixed and mobile phones are small
Those differences that have been detected point in the direction of less
social desirability bias with mobiles
This may be due to a) respondent having greater ability to move to
somewhere more private; b) removed risk of a family member ‘listening
in’ on another line
There may be other measurement advantages, such as ability to obtain
reactions promptly and on-location
(Kennedy 2010, Lynn & Kaminska 2011)

14. 13.
Summary Effect of Mobiles on Accuracy
Coverage: Can be positive for mixed-mode surveys
Sampling: Either positive or neutral
Non-response: Can be positive for mixed-mode surveys with appropriate
protocols; may be negative if used as a substitute mode
Measurement: Either positive or neutral

15. 14.
A Good News Story
Scientific survey literature often views mobile phones as a problem to be
overcome
Reality is that they bring many potential benefits, even in terms of
traditional statistical concepts of accuracy of estimates
And even for government and academic social surveys
But these benefits accrue only if mobiles are incorporated appropriately
into survey design and survey protocols
We maybe need better dissemination of good survey practice for mobile
phone interviews

16. Mobile Phone Surveys from a Total Survey
Quality Perspective
Peter Lynn
Institute for Social and Economic Research
University of Essex, UK
plynn@essex.ac.uk
www.iser.essex.ac.uk/people/plynn