Designed                  to engageonline research                  What is the impact of survey design                  o...
has collected SurveyScore data for                                Table 1: Survey Complexity (High to Low Engagement)more ...
We measured the consistency of                                                                                            ...
by removing bad respondents and                                                                                designing s...
Upcoming SlideShare
Loading in...5

Quirk's Article: Designed to Engage


Published on

Quirk's article examining the impact that survey design has on the level of engagement shown by the respondents taking the survey.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Quirk's Article: Designed to Engage

  1. 1. Designed to engageonline research What is the impact of survey design on respondent engagement? T he impact of questionable online survey respondents on data quality is well-documented. Previous research-on-research by our firm, MarketTools Inc., has shown that fake, duplicate or unengaged respondents compromise data quality. But what about the By Nallan Suresh and Michael Conklin ic ly design of the survey, which may affect all respondents, with both good and bad intentions? n n MarketTools conducted a comprehensive study that examines the ro O effect of survey design on data quality and found that, in order to ensure ct n the quality of research data, researchers must not only remove “bad” respondents from their samples, they must also design surveys that keep the good respondents engaged. le tio Are interrelated E u Experienced researchers have long assumed that survey design, respon- dent engagement and data quality are interrelated. For example, it r ib seems obvious that long and complex questionnaires will increase the o r likelihood of undesirable behaviors such as speeding and survey aban- donment, and that data quality will suffer if there is a high percentage of F st unengaged respondents in the survey sample. As we sought to understand and quantify the effect of survey i design on respondent engagement and data quality, we used our firm’s TrueSample SurveyScore measurements from over 1,500 surveys and D 800,000 responses to conduct a two-phase research study. Phase 1 of our research evaluated whether survey design influences the way respondents perceive a survey and how they behave while answering survey ques- tions. Phase 2 of our research examined the effect design variables and engagement measures have on the quality of response data. Simply put, we sought to determine whether “good” respondents driven “bad” by poorly designed and complex surveys could lead to reduced data quality. If so, we can help researchers to optimize their survey design to improve overall data quality. Editor’s note: Nallan Suresh is senior snapshot Show the impact director, panel analytics, at San Francisco research firm MarketTools TrueSample SurveyScore is designed When researchers effectively to be an objective measure of survey Inc. He can be reached at nallan. use the many facets of survey engagement and help show research- Michael Conklin is chief methodologist in the design at their disposal, they ers the impact that survey design Minneapolis office of MarketTools make great strides toward has on engagement. It is a function of both experiential variables, such Inc. He can be reached at michael. enhancing the respondent’s as respondents’ rating of the survey To view experience and the quality of taking experience, and behavioral this article online, enter article ID the data they provide. variables, such as survey abandonment 20100704 at and speeding. To date, MarketTools © 2010 Quirk’s Marketing Research Review ( Reprinted with permission from the July 2010 issue. This document is for Web posting and electronic distribution only. Any editing or alteration is a violation of copyright.
  2. 2. has collected SurveyScore data for Table 1: Survey Complexity (High to Low Engagement)more than 10,000 surveys, with over Moderate Complexity Medium Complexity High Complexity2.6 million completes. These surveys Design Attributes SurveyScore = 35 SurveyScore = 9 SurveyScore = 4span product categories (such as food Survey length (min) 9 16 17and beverage, financial, technology, Total survey pages 38 39 43entertainment, health and beauty,health care and travel) and research Total number of questions 40 41 45methods (such as concept screening, Avg. number of rows/matrix 4 13 13line and package optimization, and Avg. number of columns/matrix 5 6 6attitude and usage studies). Total number of matrix questions 8 8 8 Our team sought to determinewhether certain survey design vari- or partial rates. While survey length MarketTools fielded three surveysables could reliably predict the proved to be generally predictive of with varying levels of complexity,composite engagement measure of most respondent engagement mea- categorized as moderate, medium andrespondent behavior and percep- sures, there was wide variation in high. We analyzed 1,000 completestion that comprises TrueSample the design variables that were most for each survey. The experimentalSurveyScore. We built a model to influential in driving various mea- surveys had the same series of ques-predict engagement using survey sures of engagement. For example, tions about demographics, productsdesign variables and the TrueSample for the survey rating measure, one of purchased, etc., but differed based onSurveyScore database as inputs. the most predictive design variables the number of products respondentsPredictability is an indication that was the elapsed time per page of the said they purchased. The level of ic lysurvey design impacts engagement survey. For the speeding measure, complexity increased as more prod-in a consistent way, implying that however, elapsed time per page was ucts were chosen and more brand n nwe could recommend adjustments to not even in the top five most impor- attribute questions were displayed. In ro Othe design variables that would mini- tant design variables. the moderate category, respondents ct nmize adverse effects on engagement. Thus, adjusting just one parameter were asked one question per product.Specifically, we modeled the impact may not be sufficient to elicit desir- In the medium-complexity category,of more than 20 survey design vari- able behavior from respondents, nor respondents received 17 brand attri- le tioables (independent variables) that are will it singlehandedly improve their bute questions per product. In thewithin the control of survey design- perception of the survey-taking expe- high-complexity category, respon- E uers - such as survey length, and total rience. Instead, the findings reveal that dents were asked 17 questions forword count - on several respondent engagement is driven by a complex every product chosen, plus additional r ibengagement measures (dependent interaction among design variables. open-ended questions. o rvariables) reflecting the respondents’ This means that simple survey We computed and compared theperception of the survey and behavior design guidelines or rules are inad- SurveyScore for the three surveys. F stduring the survey. equate for motivating the desired Predictably, it dropped precipitously respondent engagement. There is no with the higher complexity levels. iClear indication axiom that applies in all cases, such The medium- and high-complexityThe research revealed that a multivar- as, “Surveys that require more than surveys received an extremely low Diate model that captures the complex 20 minutes result in poor respondent score, as shown in Table 1.interaction among design variables is engagement.” In fact, our researchers Next, we conducted a seriesable to predict overall engagement, uncovered several examples of long of statistical tests to evaluate thecomprised of both experiential and surveys that had a higher-than-normal effect of respondent engagementbehavioral variables. The fact that the survey rating as well as a lower-than- on data quality. By conducting dif-impact of these variables is predictable normal partial rate, which would run ferent analyses, we were able toprovides a clear indication that survey contrary to what one would expect if examine data quality from variousdesign directly influences respon- length alone were a deciding variable. angles for a more comprehensivedent perception and behavior, i.e., Conversely, we found examples of review. Specifically, we investigatedengagement, in a consistent way. This short surveys that had a lower-than- the following.means that survey designers do have normal survey rating because of the Will unengaging surveys:some degree of control in improving design of other variables.engagement. This also means that the • Increase the odds of sample bias?SurveyScore can be predicted prior An effect on quality • Make respondents more apt toto deploying a survey to help guide With the impact of survey design answer the same question incon-design modifications. on respondent engagement estab- sistently? We uncovered another inter- lished, the research team endeavored • Make respondents more prone toesting finding when we examined to determine whether engagement random answer choices?the influence of particular survey had an effect on data quality. The • Make respondents more likelydesign elements on specific aspects TrueSample SurveyScore database to provide inconsistent answerof engagement, such as survey rating allowed us to test this hypothesis. choices? To purchase paper reprints of this article, contact Edward Kane at FosteReprints at 866-879-9144 x131 or
  3. 3. We measured the consistency of the responses to questions that were repeated in separate sections of the survey, and we found that recall discrepancies increased as the SurveyScore dropped - proof that more complicated surveys lead to inconsistent and unreliable responses and lower data quality. We then measured the consis- tency of responses across all possible question pairs to develop an incon- sistency metric. This metric enabled us to determine if a given selection was random or closer to the expected response. The more unusual this pairing was - meaning the likelihood of its occurrence was low given the incidence of all the other options for these questions - the higher the departure from the expected value ic ly and the higher the inconsistency metric. Our finding was that incon- n n sistency increased as the SurveyScore ro O dropped, contributing to lower over- ct n all data quality for the more complex surveys (Figure 2). Finally, we sought to determine le tio if surveys with a low SurveyScore caused respondents to lose focus E u and provide inconsistent or unpre- dictable responses. To measure the r ib choice predictability of each of the o r surveys, we used a discrete choice model (DCM) exercise (Figure 3). F st Specifically, we tried to predict respondents’ product selections on i• Make respondents tend to select Figure 1 shows that as the number two tasks based on their selections “none” as an answer choice? of products selected increased - on seven other tasks (DCM sections D thereby increasing the number of were identical across all surveys). We examined whether a high questions to be answered - the par- We asked, for example, that respon-abandonment rate could cause bias tial or abandonment rate grew for dents select the one product theyin completed responses and thereby the more complicated surveys. would prefer to buy from each page,reduce overall data quality. In other As shown in the graph on the if any, and based on their answerswords, as the surveys became more right of Figure 1, of those respondents to previous questions, we tried tocomplicated and their SurveyScore who did not abandon the survey, the predict their response. The respon-dropped, did the makeup of the percentage who selected five products dents could also choose “none” as arespondents change and create the was much lower for the medium- and response, indicating that they wouldpotential for biased data? high-complexity surveys than it was choose none of the products. The answer was yes. As illustrated for the moderate survey. So, while the During this exercise, we noticedin the diagram in Figure 1, respon- actual data had a higher percentage that the accuracy of the predictiondents who completed the medium- or of respondents that had purchased (when the selection of “none” washigh-complexity surveys were more five products, many of these did not also included) was 75-79 percent fortolerant of the increased question load make it through the survey, result- all surveys, a relatively high predic-(the more products they selected, the ing in sample bias. tion rate. However, the model formore questions they were asked), Our research also tested whether the medium- and high-complexityleading to bias in those groups com- the respondents’ ability to answer surveys gave a much greater empha-pared to the group of respondents the same questions consistently sis to the “none” selection, meaningwho completed the moderate during a single survey was a func- that the respondents for these sur-survey. The graph on the left of tion of the survey’s complexity. veys tended to select no product, as © 2010 Quirk’s Marketing Research Review ( Reprinted with permission from the July 2010 issue. This document is for Web posting and electronic distribution only. Any editing or alteration is a violation of copyright.
  4. 4. by removing bad respondents and designing surveys that keep good respondents engaged. Research pro- fessionals now have evidence that survey design not only influences whether respondents abandon a survey but also impacts the data for those who complete it. The ability to predict the effect of various survey design variables on respondent engagement will help survey designers maximize engage- ment to increase the reliability of their data. Researchers no longer have to assume that a long survey will jeopardize the quality of the results, since we have shown that it is pos-opposed to one of the available prod- of selecting a lower unit price over sible to compensate for the adverseucts. Once we removed the “none” a higher one. The net result: surveys effects of certain design variables byoption from our model, the predic- with a low SurveyScore translated adjusting others. By using engage-tion accuracy dropped significantly to lower predictability and thus to ment measurement and prediction ic lyfor the high-complexity survey. In lower data quality. tools, researchers can know thataddition, the lower-scoring surveys survey design affects data quality, can n nhad more violations in price selec- Take responsibility measure engagement to help improve ro Otion order, meaning the respondents Our conclusion? Researchers must survey design and optimize design to enhance the reliability of results. | Q ct ntended to violate the expected order take responsibility for data quality le tio E u r ib o r F st D i