Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Mobile Phone Surveys from a Total Survey          Quality Perspective                        Peter Lynn       Institute fo...
1.Accuracy of Survey Estimates (Variance and Bias)
2.       Accuracy of Survey Estimates (Sources)                                                                           ...
3.Total Survey QualityRelevanceAccuracyTimelinessAccessibilityInterpretabilityCoherence(Statistics Canada, 2009)
4.Role of Mobile Phone SurveysFocus here on interviewing, not self-completioni.e. respondent is listening and talking, not...
5.Coverage ErrorInitial interest was in mobile phone ownership as a problem:Worsening coverage of landline frames (Finland...
6.Coverage: UK ExampleSince 1980s: Frames that only permit F2F have provided 98 – 99%coverage. Coverage biased, but more i...
7.UK General Population SurveysNone of fixed phones, mobile phones, internet provide good coverageBut around 99% of househ...
8.Role of Mobile Phones: CoverageGeneral Population Surveys: Permits high coverage rates when combinedwith other frames/mo...
9.Sampling ErrorMain feature affected if mobile phone numbers used as sampling elementsis selection probabilitiesPersonal ...
10.Role of Mobile Phones: SamplingOnly relevant when mobile phone numbers used as sampling elementsEffect is generally pos...
11.Non-ResponseEarly evidence that individuals less likely to participate in a survey ifapproached by mobile rather than f...
12.MeasurementMeasurement differences between fixed and mobile phones are smallThose differences that have been detected p...
13.Summary Effect of Mobiles on AccuracyCoverage: Can be positive for mixed-mode surveysSampling: Either positive or neutr...
14.A Good News StoryScientific survey literature often views mobile phones as a problem to beovercomeReality is that they ...
Mobile Phone Surveys from a Total Survey          Quality Perspective                        Peter Lynn       Institute fo...
Upcoming SlideShare
Loading in …5
×

'From a Total Survey Quality perspective' - University of Essex (Mobile Research Conference 2011)

1,424 views

Published on

Practical guidance, rooted in empirical research and theory, on how to demonstrate the advantages of quality mobile research to clients. Mobile phone surveys were tested using the practical framework for assessment developed by survey methodologists and statisticians. Survey quality includes all influences on the accuracy of survey estimates as well as factors such as relevance and timeliness. The conclusions drawn about the strengths and weaknesses of mobile phone surveys will lead to practical guidance on how to demonstrate the advantages of mobile research to clients.

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

'From a Total Survey Quality perspective' - University of Essex (Mobile Research Conference 2011)

  1. 1. Mobile Phone Surveys from a Total Survey Quality Perspective Peter Lynn Institute for Social and Economic Research University of Essex, UK Mobile Research Conference 19 April 2011 © Peter Lynn
  2. 2. 1.Accuracy of Survey Estimates (Variance and Bias)
  3. 3. 2. Accuracy of Survey Estimates (Sources) Mean Square Error Variance Bias Errors of Non-observation Observational Errors Errors of Non-observation Observational ErrorsCoverage Sampling Non-response Interviewer Respondent Instrument Coder Coverage Sampling Non-response Interviewer Respondent Instrument Coder
  4. 4. 3.Total Survey QualityRelevanceAccuracyTimelinessAccessibilityInterpretabilityCoherence(Statistics Canada, 2009)
  5. 5. 4.Role of Mobile Phone SurveysFocus here on interviewing, not self-completioni.e. respondent is listening and talking, not reading and keyingStand-alone mode or within multi-modeFocus is on accuracy of survey estimates
  6. 6. 5.Coverage ErrorInitial interest was in mobile phone ownership as a problem:Worsening coverage of landline frames (Finland, Slovenia etc in 1990s,USA in 2000s; need for dual-frame approachesThen some interest in improved coverage for phone surveys:Sub-Saharan Africa, Eastern EuropeNow, opportunities for improving coverage in mixed-mode surveys:Germany, UK, etc(Blumberg and Luke 2007, Kuusela et al 2008, Tortora et al, 2008, Gallup Europe 2009,Häder & Häder 2009)
  7. 7. 6.Coverage: UK ExampleSince 1980s: Frames that only permit F2F have provided 98 – 99%coverage. Coverage biased, but more important for some survey topics(social exclusion) than others (purchasing)But, F2F fieldwork has been becoming more expensive, more difficult, andresponse rates fallingFixed phone coverage reached 95 – 96% in 1990s, but falling since. Again,coverage biasedMeanwhile, internet penetration grew rapidly
  8. 8. 7.UK General Population SurveysNone of fixed phones, mobile phones, internet provide good coverageBut around 99% of households have at least one of those three.Points towards mixed-mode and/ or mixed-frame surveys to achieve goodcoverage without reliance on F2F
  9. 9. 8.Role of Mobile Phones: CoverageGeneral Population Surveys: Permits high coverage rates when combinedwith other frames/modesSome Specialist Surveys: Adequate coverage rates as sole modeDual-Frame Issues: Properties of frames constantly changing, soefficiencies change. Conflict between accurate estimation of selectionprobabilities and over-loading the interview with seemingly meaninglessquestions
  10. 10. 9.Sampling ErrorMain feature affected if mobile phone numbers used as sampling elementsis selection probabilitiesPersonal Surveys: - Mobiles as single mode can help to reduce sampling error, through closer correspondence of persons to mobiles - Mobiles in multi-mode/frame less efficient (less close correspondence of persons to phone numbers) but still tends to reduce sampling error compared to landline-only surveysHousehold Surveys: - Mobiles tend to increase sampling error
  11. 11. 10.Role of Mobile Phones: SamplingOnly relevant when mobile phone numbers used as sampling elementsEffect is generally positive (more accuracy) for surveys of individuals,rather than householdsMany uses of mobiles do not involve sampling mobile phone numbers –e.g. if number is obtained from a sample selected in another way
  12. 12. 11.Non-ResponseEarly evidence that individuals less likely to participate in a survey ifapproached by mobile rather than fixed phoneThis now seems far less clearContact is quicker and contact rates higher with mobilesBut marginal effect typically more important than substitute effectThis can be positive or negative depending on the survey protocolsLikely to be positive when used as an adjunct to other modes/devices, e.g.follow up to non-respondentsCan also be used as a mode for alerts/reminders(Fuchs 2000, Lau 2004, Vehovar et al 2004, Bosnjak et al 2010, Lynn 2011)
  13. 13. 12.MeasurementMeasurement differences between fixed and mobile phones are smallThose differences that have been detected point in the direction of lesssocial desirability bias with mobilesThis may be due to a) respondent having greater ability to move tosomewhere more private; b) removed risk of a family member ‘listeningin’ on another lineThere may be other measurement advantages, such as ability to obtainreactions promptly and on-location(Kennedy 2010, Lynn & Kaminska 2011)
  14. 14. 13.Summary Effect of Mobiles on AccuracyCoverage: Can be positive for mixed-mode surveysSampling: Either positive or neutralNon-response: Can be positive for mixed-mode surveys with appropriateprotocols; may be negative if used as a substitute modeMeasurement: Either positive or neutral
  15. 15. 14.A Good News StoryScientific survey literature often views mobile phones as a problem to beovercomeReality is that they bring many potential benefits, even in terms oftraditional statistical concepts of accuracy of estimatesAnd even for government and academic social surveysBut these benefits accrue only if mobiles are incorporated appropriatelyinto survey design and survey protocolsWe maybe need better dissemination of good survey practice for mobilephone interviews
  16. 16. Mobile Phone Surveys from a Total Survey Quality Perspective Peter Lynn Institute for Social and Economic Research University of Essex, UK plynn@essex.ac.uk www.iser.essex.ac.uk/people/plynn

×