SlideShare a Scribd company logo
Introduction to Survey Data
Quality
Olga Maslovskaya
University of Southampton
Survey Data
• Vast amounts of survey data are collected for many
purposes, including governmental information, public
opinion and election surveys, advertising and market
research as well as scientific research
• Survey data underlie many public policy and
business decisions
• Good quality data reduces the risk of poor policies
and decisions and is of crucial importance
Survey challenges
• Budgets are severely constrained (survey costs)
• Pressures on providing timely data are greater in the
digital age
• Public interest in participating in surveys is declining
and now at all time low (response rates)
• When cooperation obtained from reluctant
respondents, responses may be less accurate
• New modes of data collection introduce new
concerns for data quality (errors)
Definition
• Quality can be defined simply as “fitness for
use”
• Quality is a requirement for survey data to be as
accurate as necessary to achieve their intended
purposes, be available at the time it is needed
(timely), and be accessible to those for whom the
survey was conducted.
Biemer and Lyberg (2003)
Total Survey Quality (TSQ)
Total Survey Quality (TSQ)
TSQ – survey quality is more than its accuracy or statistical
dimension. It also includes among other factors producing
results that fit the needs of the survey users and providing
results that users will have confidence in. Usability of results
is of crucial importance.
Statistical
Dimension
Non-statistical
Dimension
TSQ: Quality Dimensions –Statistical
• Accuracy of estimates is the difference between
the estimate and the true parameter value
• Accuracy is the larger concept of TSQ
X = T + e
Observed
item
True value Error
Variance
(random error)
Bias (systematic
error)
TSQ: Quality Dimensions – Non-statistical
• Relevance - product is relevant and meets user needs
• Timeliness and punctuality – in disseminating results
(most important user needs)
• Accessibility and clarity – of the information
• Comparability - reliable comparisons across space and
time are often crucial; cross-national comparisons
• Coherence - single source – elementary concepts can
be combined in more complex ways; different sources –
based on common definitions, classifications and
methodological standards
• Completeness - data are rich enough to satisfy the
analysis objectives
TSQ: Quality Dimensions – Non-statistical
• Credibility – credible methodology
• Interpretability – documentation is clear
• Richness of detail - data are rich enough to
satisfy the analysis objectives
• Level of confidentiality protection
• Cost – data give good value for money
Total Survey Error (TSE)
• TSE concept was developed by Robert Groves
(1989) in book on Survey Errors and Survey Costs
• Survey estimates are derived from complex survey
data, published estimates may differ from their true
parameter values due to survey errors
• Total Survey Error is the difference between a
population mean, total, or other population
parameter and the estimate of the parameter based
on the sample survey (or census) (Biemer and
Lyberg, 2003)
TSE
TSE= sampling errors + non-sampling errors
Sources of Sampling Error
Sampling errors – can be computed for probability
samples and are due to selecting a sample instead of the
entire population
Sources:
• Sampling scheme
• Sample size
• Estimator choice
Components of Non-sampling Error
Non-sampling errors (including measurement error – cannot be
formally estimated but can be improved by interviewing
procedures and question wordings etc.) - are errors due to
mistakes or system deficiencies, also from incomplete responses
to the survey or its questions, etc.
1. Specification error
2. Frame error
3. Nonresponse error
4. Measurement error
5. Data processing error
6. Modelling/Estimation error
Other Important Factors
A number of additional factors can impact survey
data quality. Four of the more important:
• the length of time the survey was fielded,
• the use of incentives,
• the reputation of the organisation conducting the
survey
• mode of data collection
Actors affecting data quality
• Respondents (respondents’ effects on data quality):
e.g., response styles, satisficing (less efforts to
provide optimal responses)
• Interviewers (interviewers’ effects on data quality):
e.g., fabrication, ability to elicit interest and
commitment to survey in respondents, duration of
interview, duplication of responses apart from say
demographic
• Supervisors and survey research organisations
(supervisors’ effects on data quality), e.g. sampling
design, training of field workers
Data quality monitoring strategies
• Continuous quality improvement (CQI)
– Special cause variation – errors made by individual coders
– Common cause variation – errors due to the process itself
• Responsive design (Groves and Heeringa, 2006)
• Adaptive design – real time control of costs and
errors – (Schouten et al. 2013)
• Adaptive Total Design (Biemer, 2010) – adaptive
design strategy that combines ides of CQI and the
TSE paradigm to reduce costs and error across
multiple survey processes
• Six Sigma – set of principles and strategies for
improving any process
Data Quality in Practice
• No instance where a total survey quality (TSQ)
measure has ever been calculated or combined
single measure of quality taking all dimensions into
account
• Cost-benefit trade-offs to minimise different errors
depending on survey aims
• Quality reports or quality declarations have been
used where information on each dimension is
provided
• Data quality guides are meant to alert the data user
to potential sources of bias that might be present
Conclusions (1)
• Data quality is a multi-dimensional concept
• Single score or measure of data quality is not available
• Cost-benefit trade-offs to minimize different errors depending
on survey aims
• Quality frameworks are developed and adopted
• Broad range of relevant data quality indicators and information
are available with data
• The chances of users misusing the data or misinterpreting
published statistics is reduced if they understand better the
strengths and limitations of the data.
• New technologies require fresh considerations of data quality
issues in new types of surveys
Conclusions (2)
High quality of the survey data brings
– improvement in the quality of surveys themselves
– improvement of the quality of research and of policy
and financial decisions that are based on the survey
data
References
• Biemer (2010) Total survey error: Design, implementation, and evaluation. Public
Opinion Quarterly, 74(5): 817-848.
• Biemer (2016) Total Survey Error Paradigm: Theory and Practice. In The Sage
handbook of survey methodology by Wolf, Joye, Smith and Fu. London: SAGE
publications.
• Biemer and Lyberg (2003) Introduction to survey quality. New York: John Wiley & Sons.
• Groves and Heeringa (2006) Responsive design for household surveys: Tools for
actively controlling survey errors and costs. Journal of the Royal Statistical Society
Series A, 169 (3): 439-457.
• Lynn (2004) Editorial: Measuring and communicating survey quality. Journal of the Royal
Statistical Society Series A, 167 (4): 575-578.
• Lyberg and Weisberg (2016) The SAGE handbook of survey methodology. London:
SAGE publications.
• Schouten et al. (2013) Optimizing quality of response through adaptive survey designs.
Survey Methodology, 39 (1): 29-39.
• Weisberg (2005) The total survey error approach. Chicago: University of Chicago Press.

More Related Content

What's hot

difference between the qualitative and quantitative researcher, variables, co...
difference between the qualitative and quantitative researcher, variables, co...difference between the qualitative and quantitative researcher, variables, co...
difference between the qualitative and quantitative researcher, variables, co...
laraib asif
 
Two phase sampling
Two phase samplingTwo phase sampling
Two phase sampling
Kavitha Cingam
 
QUALITATIVE RESEARCH
QUALITATIVE RESEARCHQUALITATIVE RESEARCH
QUALITATIVE RESEARCH
Dr. DANIYAL MUSHTAQ
 
Non sampling error
Non sampling errorNon sampling error
Non sampling error
Mrinmoy Bharadwaz
 
Sampling and Sample Types
Sampling  and Sample TypesSampling  and Sample Types
Sampling and Sample Types
Dr. Sunil Kumar
 
Sample size
Sample sizeSample size
Sample sizezubis
 
Quota and snowball
Quota and snowballQuota and snowball
Quota and snowball
Kenisha Liyanage
 
Data Analysis, Presentation and Interpretation of Data
Data Analysis, Presentation and Interpretation of DataData Analysis, Presentation and Interpretation of Data
Data Analysis, Presentation and Interpretation of DataRoqui Malijan
 
Non sampling error
Non sampling errorNon sampling error
Non sampling error
wahengbam bigyananda
 
Lesson 7 methods of data collection
Lesson 7 methods of data collectionLesson 7 methods of data collection
Lesson 7 methods of data collection
Dr. P.B.Dharmasena
 
Normality tests
Normality testsNormality tests
Normality tests
Dr Lipilekha Patnaik
 
Sample Size Estimation
Sample Size EstimationSample Size Estimation
Sample Size Estimation
Nayyar Kazmi
 
Quantitative, qualitive and mixed research designs
Quantitative, qualitive and mixed research designsQuantitative, qualitive and mixed research designs
Quantitative, qualitive and mixed research designs
Aras Bozkurt
 
Quantitative, qualitative, and mixed method approaches
Quantitative, qualitative, and mixed method approachesQuantitative, qualitative, and mixed method approaches
Quantitative, qualitative, and mixed method approaches
muryantinarima
 
Training on Develop Mobile Data Collection Solutions using Kobo Toolbox
Training on Develop Mobile Data Collection Solutions using Kobo ToolboxTraining on Develop Mobile Data Collection Solutions using Kobo Toolbox
Training on Develop Mobile Data Collection Solutions using Kobo Toolbox
Md. Bulbul Islam
 
Collecting Data Technique
Collecting Data TechniqueCollecting Data Technique
Collecting Data TechniqueAzam Ghaffar
 
Sensitivity, specificity, positive and negative predictive
Sensitivity, specificity, positive and negative predictiveSensitivity, specificity, positive and negative predictive
Sensitivity, specificity, positive and negative predictive
Musthafa Peedikayil
 
null and alternative hypothesis.pptx
null and alternative hypothesis.pptxnull and alternative hypothesis.pptx
null and alternative hypothesis.pptx
CherrylPaderSagun
 

What's hot (20)

difference between the qualitative and quantitative researcher, variables, co...
difference between the qualitative and quantitative researcher, variables, co...difference between the qualitative and quantitative researcher, variables, co...
difference between the qualitative and quantitative researcher, variables, co...
 
Two phase sampling
Two phase samplingTwo phase sampling
Two phase sampling
 
Research question
Research questionResearch question
Research question
 
SAMPLE SIZE, CONSENT, STATISTICS
SAMPLE SIZE, CONSENT, STATISTICSSAMPLE SIZE, CONSENT, STATISTICS
SAMPLE SIZE, CONSENT, STATISTICS
 
QUALITATIVE RESEARCH
QUALITATIVE RESEARCHQUALITATIVE RESEARCH
QUALITATIVE RESEARCH
 
Non sampling error
Non sampling errorNon sampling error
Non sampling error
 
Sampling and Sample Types
Sampling  and Sample TypesSampling  and Sample Types
Sampling and Sample Types
 
Sample size
Sample sizeSample size
Sample size
 
Quota and snowball
Quota and snowballQuota and snowball
Quota and snowball
 
Data Analysis, Presentation and Interpretation of Data
Data Analysis, Presentation and Interpretation of DataData Analysis, Presentation and Interpretation of Data
Data Analysis, Presentation and Interpretation of Data
 
Non sampling error
Non sampling errorNon sampling error
Non sampling error
 
Lesson 7 methods of data collection
Lesson 7 methods of data collectionLesson 7 methods of data collection
Lesson 7 methods of data collection
 
Normality tests
Normality testsNormality tests
Normality tests
 
Sample Size Estimation
Sample Size EstimationSample Size Estimation
Sample Size Estimation
 
Quantitative, qualitive and mixed research designs
Quantitative, qualitive and mixed research designsQuantitative, qualitive and mixed research designs
Quantitative, qualitive and mixed research designs
 
Quantitative, qualitative, and mixed method approaches
Quantitative, qualitative, and mixed method approachesQuantitative, qualitative, and mixed method approaches
Quantitative, qualitative, and mixed method approaches
 
Training on Develop Mobile Data Collection Solutions using Kobo Toolbox
Training on Develop Mobile Data Collection Solutions using Kobo ToolboxTraining on Develop Mobile Data Collection Solutions using Kobo Toolbox
Training on Develop Mobile Data Collection Solutions using Kobo Toolbox
 
Collecting Data Technique
Collecting Data TechniqueCollecting Data Technique
Collecting Data Technique
 
Sensitivity, specificity, positive and negative predictive
Sensitivity, specificity, positive and negative predictiveSensitivity, specificity, positive and negative predictive
Sensitivity, specificity, positive and negative predictive
 
null and alternative hypothesis.pptx
null and alternative hypothesis.pptxnull and alternative hypothesis.pptx
null and alternative hypothesis.pptx
 

Similar to Introduction to Survey Data Quality

Data quality: total survey error
Data quality: total survey errorData quality: total survey error
Data quality: total survey error
University of Southampton
 
Module-7-Descriptive Research-survey.pdf
Module-7-Descriptive Research-survey.pdfModule-7-Descriptive Research-survey.pdf
Module-7-Descriptive Research-survey.pdf
Vikramjit Singh
 
MethodsofDataCollection.pdf
MethodsofDataCollection.pdfMethodsofDataCollection.pdf
MethodsofDataCollection.pdf
ssuser9878d0
 
MethodsofDataCollection.pdf
MethodsofDataCollection.pdfMethodsofDataCollection.pdf
MethodsofDataCollection.pdf
MohdTaufiqIshak
 
Descriptive research-survey
Descriptive research-surveyDescriptive research-survey
Descriptive research-survey
Vikramjit Singh
 
Survey
SurveySurvey
RESEARCH APPROACHES AND DESIGNS.pptx
RESEARCH APPROACHES AND DESIGNS.pptxRESEARCH APPROACHES AND DESIGNS.pptx
RESEARCH APPROACHES AND DESIGNS.pptx
PRADEEP ABOTHU
 
COMMUNITY NEED ASSESSMENT.pptx
COMMUNITY NEED ASSESSMENT.pptxCOMMUNITY NEED ASSESSMENT.pptx
COMMUNITY NEED ASSESSMENT.pptx
GhaffarAhmed9
 
Data collection methods RSS6 2014
Data collection methods RSS6 2014Data collection methods RSS6 2014
Data collection methods RSS6 2014RSS6
 
Evaluating Systems Change
Evaluating Systems ChangeEvaluating Systems Change
Evaluating Systems Change
Noel Hatch
 
Chapter Eight Quantitative Methods
Chapter Eight Quantitative MethodsChapter Eight Quantitative Methods
Chapter Eight Quantitative Methods
International advisers
 
Quantitative search and_qualitative_research by mubarak
Quantitative search and_qualitative_research by mubarakQuantitative search and_qualitative_research by mubarak
Quantitative search and_qualitative_research by mubarak
Hafiza Abas
 
Evaluation of Health IT Implementation (March 20, 2019)
Evaluation of Health IT Implementation (March 20, 2019)Evaluation of Health IT Implementation (March 20, 2019)
Evaluation of Health IT Implementation (March 20, 2019)
Nawanan Theera-Ampornpunt
 
Evaluation of Health IT Implementation
Evaluation of Health IT ImplementationEvaluation of Health IT Implementation
Evaluation of Health IT Implementation
Nawanan Theera-Ampornpunt
 
Evaluation of Health IT Implementation (February 17, 2021)
Evaluation of Health IT Implementation (February 17, 2021)Evaluation of Health IT Implementation (February 17, 2021)
Evaluation of Health IT Implementation (February 17, 2021)
Nawanan Theera-Ampornpunt
 
Practical Research 1 about quantitative and qualitative methods
Practical Research 1 about quantitative and qualitative methodsPractical Research 1 about quantitative and qualitative methods
Practical Research 1 about quantitative and qualitative methods
AndoJoshua
 
wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...
wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...
wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...
Adikesavaperumal
 
MARKET RESEARCH WEEK LESSONN PLAN 5.pptx
MARKET RESEARCH WEEK LESSONN PLAN 5.pptxMARKET RESEARCH WEEK LESSONN PLAN 5.pptx
MARKET RESEARCH WEEK LESSONN PLAN 5.pptx
PreciousChanaiwa
 
Educ 210-research-design
Educ 210-research-designEduc 210-research-design
Educ 210-research-design
BernadetteSLomeda
 

Similar to Introduction to Survey Data Quality (20)

Data quality: total survey error
Data quality: total survey errorData quality: total survey error
Data quality: total survey error
 
Module-7-Descriptive Research-survey.pdf
Module-7-Descriptive Research-survey.pdfModule-7-Descriptive Research-survey.pdf
Module-7-Descriptive Research-survey.pdf
 
MethodsofDataCollection.pdf
MethodsofDataCollection.pdfMethodsofDataCollection.pdf
MethodsofDataCollection.pdf
 
MethodsofDataCollection.pdf
MethodsofDataCollection.pdfMethodsofDataCollection.pdf
MethodsofDataCollection.pdf
 
Descriptive research-survey
Descriptive research-surveyDescriptive research-survey
Descriptive research-survey
 
Questionnaire2002
Questionnaire2002Questionnaire2002
Questionnaire2002
 
Survey
SurveySurvey
Survey
 
RESEARCH APPROACHES AND DESIGNS.pptx
RESEARCH APPROACHES AND DESIGNS.pptxRESEARCH APPROACHES AND DESIGNS.pptx
RESEARCH APPROACHES AND DESIGNS.pptx
 
COMMUNITY NEED ASSESSMENT.pptx
COMMUNITY NEED ASSESSMENT.pptxCOMMUNITY NEED ASSESSMENT.pptx
COMMUNITY NEED ASSESSMENT.pptx
 
Data collection methods RSS6 2014
Data collection methods RSS6 2014Data collection methods RSS6 2014
Data collection methods RSS6 2014
 
Evaluating Systems Change
Evaluating Systems ChangeEvaluating Systems Change
Evaluating Systems Change
 
Chapter Eight Quantitative Methods
Chapter Eight Quantitative MethodsChapter Eight Quantitative Methods
Chapter Eight Quantitative Methods
 
Quantitative search and_qualitative_research by mubarak
Quantitative search and_qualitative_research by mubarakQuantitative search and_qualitative_research by mubarak
Quantitative search and_qualitative_research by mubarak
 
Evaluation of Health IT Implementation (March 20, 2019)
Evaluation of Health IT Implementation (March 20, 2019)Evaluation of Health IT Implementation (March 20, 2019)
Evaluation of Health IT Implementation (March 20, 2019)
 
Evaluation of Health IT Implementation
Evaluation of Health IT ImplementationEvaluation of Health IT Implementation
Evaluation of Health IT Implementation
 
Evaluation of Health IT Implementation (February 17, 2021)
Evaluation of Health IT Implementation (February 17, 2021)Evaluation of Health IT Implementation (February 17, 2021)
Evaluation of Health IT Implementation (February 17, 2021)
 
Practical Research 1 about quantitative and qualitative methods
Practical Research 1 about quantitative and qualitative methodsPractical Research 1 about quantitative and qualitative methods
Practical Research 1 about quantitative and qualitative methods
 
wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...
wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...
wepik-exploring-effective-strategies-for-data-collection-navigating-the-chall...
 
MARKET RESEARCH WEEK LESSONN PLAN 5.pptx
MARKET RESEARCH WEEK LESSONN PLAN 5.pptxMARKET RESEARCH WEEK LESSONN PLAN 5.pptx
MARKET RESEARCH WEEK LESSONN PLAN 5.pptx
 
Educ 210-research-design
Educ 210-research-designEduc 210-research-design
Educ 210-research-design
 

More from University of Southampton

Generating SPSS training materials in StatJR
Generating SPSS training materials in StatJRGenerating SPSS training materials in StatJR
Generating SPSS training materials in StatJR
University of Southampton
 
Introduction to the Stat-JR software package
Introduction to the Stat-JR software packageIntroduction to the Stat-JR software package
Introduction to the Stat-JR software package
University of Southampton
 
Multi level modelling- random coefficient models | Ian Brunton-Smith
Multi level modelling- random coefficient models | Ian Brunton-SmithMulti level modelling- random coefficient models | Ian Brunton-Smith
Multi level modelling- random coefficient models | Ian Brunton-Smith
University of Southampton
 
Multi level modelling - random intercept models | Ian Brunton Smith
Multi level modelling - random intercept models | Ian Brunton SmithMulti level modelling - random intercept models | Ian Brunton Smith
Multi level modelling - random intercept models | Ian Brunton Smith
University of Southampton
 
Introduction to multilevel modelling | Ian Brunton-Smith
Introduction to multilevel modelling | Ian Brunton-SmithIntroduction to multilevel modelling | Ian Brunton-Smith
Introduction to multilevel modelling | Ian Brunton-Smith
University of Southampton
 
Biosocial research:How to use biological data in social science research?
Biosocial research:How to use biological data in social science research?Biosocial research:How to use biological data in social science research?
Biosocial research:How to use biological data in social science research?
University of Southampton
 
Integrating biological and social research data - Michaela Benzeval
Integrating biological and social research data - Michaela BenzevalIntegrating biological and social research data - Michaela Benzeval
Integrating biological and social research data - Michaela Benzeval
University of Southampton
 
Teaching research methods: pedagogy hooks
Teaching research methods: pedagogy hooksTeaching research methods: pedagogy hooks
Teaching research methods: pedagogy hooks
University of Southampton
 
Teaching research methods: pedagogy of methods learning
Teaching research methods: pedagogy of methods learningTeaching research methods: pedagogy of methods learning
Teaching research methods: pedagogy of methods learning
University of Southampton
 
better off living with parents
better off living with parentsbetter off living with parents
better off living with parents
University of Southampton
 
Multilevel models:random coefficient models
Multilevel models:random coefficient modelsMultilevel models:random coefficient models
Multilevel models:random coefficient models
University of Southampton
 
Multilevel models:random intercept models
Multilevel models:random intercept modelsMultilevel models:random intercept models
Multilevel models:random intercept models
University of Southampton
 
Introduction to multilevel modelling
Introduction to multilevel modellingIntroduction to multilevel modelling
Introduction to multilevel modelling
University of Southampton
 
How to write about research methods
How to write about research methodsHow to write about research methods
How to write about research methods
University of Southampton
 
How to write about research methods
How to write about research methodsHow to write about research methods
How to write about research methods
University of Southampton
 
Introduction to spatial interaction modelling
Introduction to spatial interaction modellingIntroduction to spatial interaction modelling
Introduction to spatial interaction modelling
University of Southampton
 
Cognitive interviewing
Cognitive interviewingCognitive interviewing
Cognitive interviewing
University of Southampton
 
Survey questions and measurement error
Survey questions and measurement errorSurvey questions and measurement error
Survey questions and measurement error
University of Southampton
 
Participatory performative and mobile methods
Participatory performative and mobile methodsParticipatory performative and mobile methods
Participatory performative and mobile methods
University of Southampton
 
Participatory theatre as a social research methods
Participatory theatre as a social research methodsParticipatory theatre as a social research methods
Participatory theatre as a social research methods
University of Southampton
 

More from University of Southampton (20)

Generating SPSS training materials in StatJR
Generating SPSS training materials in StatJRGenerating SPSS training materials in StatJR
Generating SPSS training materials in StatJR
 
Introduction to the Stat-JR software package
Introduction to the Stat-JR software packageIntroduction to the Stat-JR software package
Introduction to the Stat-JR software package
 
Multi level modelling- random coefficient models | Ian Brunton-Smith
Multi level modelling- random coefficient models | Ian Brunton-SmithMulti level modelling- random coefficient models | Ian Brunton-Smith
Multi level modelling- random coefficient models | Ian Brunton-Smith
 
Multi level modelling - random intercept models | Ian Brunton Smith
Multi level modelling - random intercept models | Ian Brunton SmithMulti level modelling - random intercept models | Ian Brunton Smith
Multi level modelling - random intercept models | Ian Brunton Smith
 
Introduction to multilevel modelling | Ian Brunton-Smith
Introduction to multilevel modelling | Ian Brunton-SmithIntroduction to multilevel modelling | Ian Brunton-Smith
Introduction to multilevel modelling | Ian Brunton-Smith
 
Biosocial research:How to use biological data in social science research?
Biosocial research:How to use biological data in social science research?Biosocial research:How to use biological data in social science research?
Biosocial research:How to use biological data in social science research?
 
Integrating biological and social research data - Michaela Benzeval
Integrating biological and social research data - Michaela BenzevalIntegrating biological and social research data - Michaela Benzeval
Integrating biological and social research data - Michaela Benzeval
 
Teaching research methods: pedagogy hooks
Teaching research methods: pedagogy hooksTeaching research methods: pedagogy hooks
Teaching research methods: pedagogy hooks
 
Teaching research methods: pedagogy of methods learning
Teaching research methods: pedagogy of methods learningTeaching research methods: pedagogy of methods learning
Teaching research methods: pedagogy of methods learning
 
better off living with parents
better off living with parentsbetter off living with parents
better off living with parents
 
Multilevel models:random coefficient models
Multilevel models:random coefficient modelsMultilevel models:random coefficient models
Multilevel models:random coefficient models
 
Multilevel models:random intercept models
Multilevel models:random intercept modelsMultilevel models:random intercept models
Multilevel models:random intercept models
 
Introduction to multilevel modelling
Introduction to multilevel modellingIntroduction to multilevel modelling
Introduction to multilevel modelling
 
How to write about research methods
How to write about research methodsHow to write about research methods
How to write about research methods
 
How to write about research methods
How to write about research methodsHow to write about research methods
How to write about research methods
 
Introduction to spatial interaction modelling
Introduction to spatial interaction modellingIntroduction to spatial interaction modelling
Introduction to spatial interaction modelling
 
Cognitive interviewing
Cognitive interviewingCognitive interviewing
Cognitive interviewing
 
Survey questions and measurement error
Survey questions and measurement errorSurvey questions and measurement error
Survey questions and measurement error
 
Participatory performative and mobile methods
Participatory performative and mobile methodsParticipatory performative and mobile methods
Participatory performative and mobile methods
 
Participatory theatre as a social research methods
Participatory theatre as a social research methodsParticipatory theatre as a social research methods
Participatory theatre as a social research methods
 

Recently uploaded

TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
EugeneSaldivar
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Thiyagu K
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
TechSoup
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
siemaillard
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
Celine George
 
A Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptxA Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptx
thanhdowork
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
Special education needs
 
Marketing internship report file for MBA
Marketing internship report file for MBAMarketing internship report file for MBA
Marketing internship report file for MBA
gb193092
 
Best Digital Marketing Institute In NOIDA
Best Digital Marketing Institute In NOIDABest Digital Marketing Institute In NOIDA
Best Digital Marketing Institute In NOIDA
deeptiverma2406
 
Group Presentation 2 Economics.Ariana Buscigliopptx
Group Presentation 2 Economics.Ariana BuscigliopptxGroup Presentation 2 Economics.Ariana Buscigliopptx
Group Presentation 2 Economics.Ariana Buscigliopptx
ArianaBusciglio
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
Jisc
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
Jisc
 
"Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe..."Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe...
SACHIN R KONDAGURI
 
Multithreading_in_C++ - std::thread, race condition
Multithreading_in_C++ - std::thread, race conditionMultithreading_in_C++ - std::thread, race condition
Multithreading_in_C++ - std::thread, race condition
Mohammed Sikander
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
Jean Carlos Nunes Paixão
 
S1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptxS1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptx
tarandeep35
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
DhatriParmar
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
Balvir Singh
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
Tamralipta Mahavidyalaya
 
Azure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHatAzure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHat
Scholarhat
 

Recently uploaded (20)

TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...TESDA TM1 REVIEWER  FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
TESDA TM1 REVIEWER FOR NATIONAL ASSESSMENT WRITTEN AND ORAL QUESTIONS WITH A...
 
Unit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdfUnit 2- Research Aptitude (UGC NET Paper I).pdf
Unit 2- Research Aptitude (UGC NET Paper I).pdf
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
 
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 
Model Attribute Check Company Auto Property
Model Attribute  Check Company Auto PropertyModel Attribute  Check Company Auto Property
Model Attribute Check Company Auto Property
 
A Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptxA Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptx
 
special B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdfspecial B.ed 2nd year old paper_20240531.pdf
special B.ed 2nd year old paper_20240531.pdf
 
Marketing internship report file for MBA
Marketing internship report file for MBAMarketing internship report file for MBA
Marketing internship report file for MBA
 
Best Digital Marketing Institute In NOIDA
Best Digital Marketing Institute In NOIDABest Digital Marketing Institute In NOIDA
Best Digital Marketing Institute In NOIDA
 
Group Presentation 2 Economics.Ariana Buscigliopptx
Group Presentation 2 Economics.Ariana BuscigliopptxGroup Presentation 2 Economics.Ariana Buscigliopptx
Group Presentation 2 Economics.Ariana Buscigliopptx
 
The approach at University of Liverpool.pptx
The approach at University of Liverpool.pptxThe approach at University of Liverpool.pptx
The approach at University of Liverpool.pptx
 
How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...How libraries can support authors with open access requirements for UKRI fund...
How libraries can support authors with open access requirements for UKRI fund...
 
"Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe..."Protectable subject matters, Protection in biotechnology, Protection of othe...
"Protectable subject matters, Protection in biotechnology, Protection of othe...
 
Multithreading_in_C++ - std::thread, race condition
Multithreading_in_C++ - std::thread, race conditionMultithreading_in_C++ - std::thread, race condition
Multithreading_in_C++ - std::thread, race condition
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
 
S1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptxS1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptx
 
The Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptxThe Accursed House by Émile Gaboriau.pptx
The Accursed House by Émile Gaboriau.pptx
 
Operation Blue Star - Saka Neela Tara
Operation Blue Star   -  Saka Neela TaraOperation Blue Star   -  Saka Neela Tara
Operation Blue Star - Saka Neela Tara
 
Home assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdfHome assignment II on Spectroscopy 2024 Answers.pdf
Home assignment II on Spectroscopy 2024 Answers.pdf
 
Azure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHatAzure Interview Questions and Answers PDF By ScholarHat
Azure Interview Questions and Answers PDF By ScholarHat
 

Introduction to Survey Data Quality

  • 1. Introduction to Survey Data Quality Olga Maslovskaya University of Southampton
  • 2. Survey Data • Vast amounts of survey data are collected for many purposes, including governmental information, public opinion and election surveys, advertising and market research as well as scientific research • Survey data underlie many public policy and business decisions • Good quality data reduces the risk of poor policies and decisions and is of crucial importance
  • 3. Survey challenges • Budgets are severely constrained (survey costs) • Pressures on providing timely data are greater in the digital age • Public interest in participating in surveys is declining and now at all time low (response rates) • When cooperation obtained from reluctant respondents, responses may be less accurate • New modes of data collection introduce new concerns for data quality (errors)
  • 4. Definition • Quality can be defined simply as “fitness for use” • Quality is a requirement for survey data to be as accurate as necessary to achieve their intended purposes, be available at the time it is needed (timely), and be accessible to those for whom the survey was conducted. Biemer and Lyberg (2003)
  • 5. Total Survey Quality (TSQ) Total Survey Quality (TSQ) TSQ – survey quality is more than its accuracy or statistical dimension. It also includes among other factors producing results that fit the needs of the survey users and providing results that users will have confidence in. Usability of results is of crucial importance. Statistical Dimension Non-statistical Dimension
  • 6. TSQ: Quality Dimensions –Statistical • Accuracy of estimates is the difference between the estimate and the true parameter value • Accuracy is the larger concept of TSQ X = T + e Observed item True value Error Variance (random error) Bias (systematic error)
  • 7. TSQ: Quality Dimensions – Non-statistical • Relevance - product is relevant and meets user needs • Timeliness and punctuality – in disseminating results (most important user needs) • Accessibility and clarity – of the information • Comparability - reliable comparisons across space and time are often crucial; cross-national comparisons • Coherence - single source – elementary concepts can be combined in more complex ways; different sources – based on common definitions, classifications and methodological standards • Completeness - data are rich enough to satisfy the analysis objectives
  • 8. TSQ: Quality Dimensions – Non-statistical • Credibility – credible methodology • Interpretability – documentation is clear • Richness of detail - data are rich enough to satisfy the analysis objectives • Level of confidentiality protection • Cost – data give good value for money
  • 9. Total Survey Error (TSE) • TSE concept was developed by Robert Groves (1989) in book on Survey Errors and Survey Costs • Survey estimates are derived from complex survey data, published estimates may differ from their true parameter values due to survey errors • Total Survey Error is the difference between a population mean, total, or other population parameter and the estimate of the parameter based on the sample survey (or census) (Biemer and Lyberg, 2003)
  • 10. TSE TSE= sampling errors + non-sampling errors
  • 11. Sources of Sampling Error Sampling errors – can be computed for probability samples and are due to selecting a sample instead of the entire population Sources: • Sampling scheme • Sample size • Estimator choice
  • 12. Components of Non-sampling Error Non-sampling errors (including measurement error – cannot be formally estimated but can be improved by interviewing procedures and question wordings etc.) - are errors due to mistakes or system deficiencies, also from incomplete responses to the survey or its questions, etc. 1. Specification error 2. Frame error 3. Nonresponse error 4. Measurement error 5. Data processing error 6. Modelling/Estimation error
  • 13. Other Important Factors A number of additional factors can impact survey data quality. Four of the more important: • the length of time the survey was fielded, • the use of incentives, • the reputation of the organisation conducting the survey • mode of data collection
  • 14. Actors affecting data quality • Respondents (respondents’ effects on data quality): e.g., response styles, satisficing (less efforts to provide optimal responses) • Interviewers (interviewers’ effects on data quality): e.g., fabrication, ability to elicit interest and commitment to survey in respondents, duration of interview, duplication of responses apart from say demographic • Supervisors and survey research organisations (supervisors’ effects on data quality), e.g. sampling design, training of field workers
  • 15. Data quality monitoring strategies • Continuous quality improvement (CQI) – Special cause variation – errors made by individual coders – Common cause variation – errors due to the process itself • Responsive design (Groves and Heeringa, 2006) • Adaptive design – real time control of costs and errors – (Schouten et al. 2013) • Adaptive Total Design (Biemer, 2010) – adaptive design strategy that combines ides of CQI and the TSE paradigm to reduce costs and error across multiple survey processes • Six Sigma – set of principles and strategies for improving any process
  • 16. Data Quality in Practice • No instance where a total survey quality (TSQ) measure has ever been calculated or combined single measure of quality taking all dimensions into account • Cost-benefit trade-offs to minimise different errors depending on survey aims • Quality reports or quality declarations have been used where information on each dimension is provided • Data quality guides are meant to alert the data user to potential sources of bias that might be present
  • 17. Conclusions (1) • Data quality is a multi-dimensional concept • Single score or measure of data quality is not available • Cost-benefit trade-offs to minimize different errors depending on survey aims • Quality frameworks are developed and adopted • Broad range of relevant data quality indicators and information are available with data • The chances of users misusing the data or misinterpreting published statistics is reduced if they understand better the strengths and limitations of the data. • New technologies require fresh considerations of data quality issues in new types of surveys
  • 18. Conclusions (2) High quality of the survey data brings – improvement in the quality of surveys themselves – improvement of the quality of research and of policy and financial decisions that are based on the survey data
  • 19. References • Biemer (2010) Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly, 74(5): 817-848. • Biemer (2016) Total Survey Error Paradigm: Theory and Practice. In The Sage handbook of survey methodology by Wolf, Joye, Smith and Fu. London: SAGE publications. • Biemer and Lyberg (2003) Introduction to survey quality. New York: John Wiley & Sons. • Groves and Heeringa (2006) Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society Series A, 169 (3): 439-457. • Lynn (2004) Editorial: Measuring and communicating survey quality. Journal of the Royal Statistical Society Series A, 167 (4): 575-578. • Lyberg and Weisberg (2016) The SAGE handbook of survey methodology. London: SAGE publications. • Schouten et al. (2013) Optimizing quality of response through adaptive survey designs. Survey Methodology, 39 (1): 29-39. • Weisberg (2005) The total survey error approach. Chicago: University of Chicago Press.

Editor's Notes

  1. So data quality is crucial
  2. All these require reconsideration of existing data quality frameworks. New aspects should be taken into account Opt-in panels Big Data Difficult to convince clients that traditional probability surveys are the best and worth extra costs
  3. (Eurostat, Statistics Canada and Statistics Sweden) Statistics Canada: Relevance Accuracy Timeliness Accessibility Interpretability Coherence Statistics Sweden: Content Accuracy Timeliness Comparability/coherence Availability/clarity
  4. Bias – mean of errors is not equal to 0, does not cancel out; variance – mean of error is equal to 0, does cancel out Accuracy is The larger concept of Total Survey Quality (TSQ) Broader that accuracy definition is needed as users are not just interested in the accuracy of the estimates provided. Accuracy is the cornestone of quality, since without it, sruvey data are of little use. If the data are erroneous, it does not help much if relevance, timeliness, accessibility, comparability, coherence and completeness are sufficient.
  5. Relevance of statistical concept (product is relevant and meets user needs) Timeliness and punctuality in disseminating results (one of the most important user needs) Accessibility and clarity of the information Comparability (reliable comparisons across space and time are often crucial; cross-national comparisons) Coherence (single source – elementary concepts can be combined in more complex ways; different sources – based on common definitions, classifications and methodological standards) Completeness (data are rich enough to satisfy the analysis objectives) Credibility (credible methodology) Interpretability – documentation is clear Richness of detail Level of confidentiality protection Cost – data give good value for money
  6. Credibility (credible methodology) Interpretability – documentation is clear Richness of detail Level of confidentiality protection Cost – data give good value for money
  7. Simple random sampling is often neither possible nor cost-effective. Stratifying the sample can reduce the sampling error, clustering the sample can reduce costs but would increase the sampling error.
  8. In many cases non-sampling error can be much more damaging than sampling error to estimates from surveys
  9. Errors can be systematic or random and correlated or uncorrelated. Uncorrelated (e.g., interviewer mistakenly records a “yes” answer as a “no” Correlated (when interviewers take multiple interviewers and when cluster sampling is used – correlated errors increase the variance of estimates due to an effective sample size that is smaller than the intended one and thereby make it more difficult to achieve statistically significant results) Measurement errors pose a serious limitation to the validity and usefulness of the information collected via survey. Having excellent samples representative of the target population, having high response rates, having complete data, etc. does us little good if our measurement instruments evoke responses that are fraught with error. Measurement error is distinct from other survey errors and it is error that occurs when the recorded or observed value is different from the true value of the variable. Reliability and validity are important in measurement error. Reliability is “agreement between two efforts to measure the same thing, using maximally similar methods” How was the survey administered (e.g. in person, by telephone, online, multiple modes, etc.)? (sensitive questions) Were the questions well constructed, clear, and not leading or otherwise biasing? (satisficing) What steps, if any, were taken to ensure that respondents were providing truthful  answers to the questions, and were any respondents removed from the final dataset (e.g., identifying speeders, satisficers, multiple completions)? (in-survey behaviour)
  10. How long was the survey in the field and how much effort was put to ensuring a good response? What incentives, if any, were respondents offered to encourage participation? (monetary incentives could bias the survey responses towards lower income groups; some respondetns could rush through) What is the record of accomplishment of the organization that conducted the survey? (organisation with long and successful records can inspire confidence)
  11. For face to face and telephone interviews
  12. potentially modify the survey design based on the analysis of paradata collected from relevant processes CQI is methods emphasize imprivement in the underlying process rather than screening the product. Initital quality improvements efforts should focus on eliminating the special cause erros since thay re usually responsible for most of the errors in a process. Reducing the common cause errors will require changing the process in such a way that the error rate can be lowered. Approach have been adopted to control costs and errors in surveys. Uses number of quality management tools. Continuous quality improvement (CQI) uses number of standard quality management tools: worksflow diagram, cause and effect (or fishbone) diagram, Pareto histograms, statistical process control methods and various production efficiency metrics Responsive design (Groves and Heeringa, 2006) – strategy that includes some of the ideas and concepts and approaches of CQI while providing several innovative strategies that use paradata as well as survey data to monitor nonresponse bias and follow up efficiency and effectiveness Adaptive design – real time control of costs and errors – Schouten et al. 2013 – strategy for tailoring key features of the survey design for different types of sample members maximizing response rates and reduce nonresponse selectivity – appropriate when substantial prior information about sample units is available Adaptive Total Design (Biemer, 2010) – adaptive design strategy that combines ides of CQI and the TSE paradigm to reduce costs and error across multiple survey processes including frame consturction, sampling, data collection and data processing Six Sigma – set of principles and strategies for improving any process
  13. Report that provides comprehensive picture of the quality of a survey, addressing each potential source of error Supplemental to regular survey documentation and should be based on information that is available in many different forms such as survey methodology reports, user manuals on how to use microdata files, and technical reports providing details about specifics.
  14. Cost-benefit trade-offs are needed to decide which errors to minimize Quality frameworks were developed and adopted and provided statistics producers with clear description of how certain dimensions of quality can be measured and why it might be important to do so. The survey community needs to find ways of ensuring that as broad a range as possible of relevant indicators and information is made available routinely (Lynn 2004) The chances of users misusing the data or misinterpreting published statistics will be reduced if they understand better the strengths and limitations of the data. The publication of data quality measures itself represent an improvement in the quality of a survey