This document discusses using complementary evaluation methods alongside cognitive techniques to pre-test establishment surveys. It outlines several proposed complementary methods, including respondent advisory panels, pilot studies, designed experiments, record keeping studies, response analysis surveys, applied ethnography, and vignettes. Each method has different strengths, such as accessing records, observing the response process, being cost-effective, or allowing quick fielding. The conclusion is that cognitive testing alone is not always sufficient and using various complementary methods can help uncover different types of problems to improve surveys.

1. Considering Complementary Evaluation Methods To Augment Cognitive Techniques During Pre-testing of Establishment Surveys
Kenneth M. Pick, Alfred D. Tuttle, Diane K. Willimack
U.S. Census Bureau
1. Background 4. Findings from Cognitive Testing Reports and Staff Interviews 6. Select Methodologies
 Pre-testing of establishment Results From Research Proposed Complementary Pre-testing Method  Record Keeping Studies are
surveys at the U.S. Census onsite reviews of an
Bureau predominantly uses a Unclear Constructs Respondent Advisory Panel establishment’s record keeping
mix of cognitive interview  Survey sponsors often had unclear objectives with  Panel of profiled, volunteer respondents from practices. This qualitative
techniques. regard to the construct they were trying to measure. establishment surveys. methodology is used to inform
 Limited resources are diverted from testing questions to  Questions posed to respondents to see if data requests the feasibility of a survey’s data
 Cognitive techniques are exploratory interviews to aid development of questions. are feasible prior to pre-testing. requests prior to questionnaire
effective in uncovering
development.
problems related to survey Question-Order Effects Pilot Study
concepts and comprehension  Due to small, qualitative sample sizes and a focus on  Survey launched to small, random sub-sample.  Response Analysis Surveys are
of survey items. specific-question constructs, staff were sometimes  Statistical analyses conducted to find potential data conducted with respondents
unable to detect potential impacts of question-order quality issues related to question-order effects before shortly after they complete a
 However, some problems
effects on data quality. full-launching the survey. survey using a structured
uncovered cannot be evaluated
protocol. They are useful in
through cognitive testing alone
Problematic Wording/Formatting Issues Designed Experiment learning about an
– other complementary
evaluation methodologies may  Problems with wording and/or formatting issues were  Two or more forms of the questionnaire testing different establishment’s record-keeping
uncovered through cognitive testing, but unresolved. wording/formatting launched to random sub-samples. practices and discrepancies
prove useful.
 Decisions to move forward with specific wording and/or  Statistical analyses conducted looking for outliers and between survey definitions and
formatting were often a judgment call by the survey statistically significant differences between treatments. answers.
2. Research Question methodologist(s).  Vignettes are ambiguous
narratives/mock records
Difficult to Access Records Applied Ethnography
 What complementary presented to respondents using
 Hypothetical probing is common as retrieval of records  Establishment visits by survey methodologist(s).
evaluation methods can be familiar terminology/formats.
is labor intensive.  Conduct in-person interviews with respondent(s).
used in concert with cognitive They are used to examine
 A disparity frequently exists between the information a  Observe/Participate in record retrieval process and
testing to pre-test underlying constructs and the
company says they can provide vs. what they actually coordination of response to an establishment survey.
establishment surveys? response process.
provide on a survey.
3. Methodologies Used 5. Considering the Strengths of Complementary Pre-Testing Methods 7. Conclusions
 Review of reports produced by Strengths (List Not Exhaustive)  Cognitive testing of
the Response Improvement Pre-testing Methodologies Quantitative Access to Observe Parts of Cost Quick establishment surveys are not
Research Staff (RIRS), Data Records Response Process Effective Fielding always effective by themselves in
Economic Programs, from resolving problems related to
2005-2010 (n=27). Applied Ethnography X X item constructs, question-order
effects, problematic wording/
 Unstructured interviews with Cognitive Interviews X X X formatting issues and accessing
current and former RIRS staff
records.
(n=7) probing on unresolved Designed Experiment X
issues identified in the staff’s  Different methodologies have
reports (including cognitive Pilot Study X X different strengths – and reveal
interviews, respondent different problems within a
Record Keeping Study X
debriefings and usability questionnaire.
testing). Response Analysis Survey (RAS) X X X X  Use of complementary pre-
 Literature review of pre-testing testing methods along with
and research methods for Respondent Advisory Panel X X X X X cognitive interviews would
evaluating survey provide a more comprehensive
Respondent Debriefings X X X X
questionnaires. evaluation of establishment
Subject-matter Expert Review X X survey questionnaires.
This report is released to inform interested parties of research and to
Vignettes X X X encourage discussion. Any views expressed on methodological issues are
those of the authors and not necessarily those of the U.S. Census Bureau.