Visually Integrative Representation of User Types in Surveys (Ricardo Carvalho & Joseph Luchman & Michael Paraloglou & Vanessa Patterson & Ron Vega)

  • 204 views
Uploaded on

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
204
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Why Do Respondents Skip Questions in Surveys: A Visually Integrative Representation of User Types Ricardo Carvalho Joseph Luchman Michael Paraloglou Vanessa Patterson Ron Vega1
  • 2. Outline • Background • Our Research • Our Findings • Our Conclusions & Recommendations • Future Research2
  • 3. Background3
  • 4. Background • DoD Youth Poll December 2011 survey – Mailed to 50,000 youth ages 16 to 24 with no prior or current military experience through stratified, probability-based sampling – Address-based sample drawn from list frame estimated to cover 92% of target population – Standard mailing methodology (Dillman 2007) – Scantron survey; double-entered and verified – Up-front and contingent monetary incentives upon completion – Response Rate 3: 17%, Contact Rate 2: 92%; n= 7,210 • Note: although our study was specific to completing a paper survey, much of the theory (and more importantly, the tool) can be applied to other survey modes and experiences4
  • 5. Background5
  • 6. Background • The Issue # Refusals to Q29 and Q30 – Amount of refusals per item was very small 1-18 Refusals (one to many but not all items) (<1.5%) up to Q29 19 Refusals (all items) – But at Q30 and thereafter, it increases to 6% 6% 5% about 5-6% 5% 5% 4% • Behavior of certain users changes in consistent manner 3% 2% • Can we understand the user’s 1% 1% experience and behavior? 0% Q29 Q306
  • 7. Our Research7
  • 8. Our Research • We noticed 370 respondents whose behavior seemed fundamentally different – Are these different “user types”? – Or was there a usability issue with the survey (“troublesome areas”)? – How can we identify the “user type” or a “troublesome area”? Does this kind of information tell us what to change in the experience? No…8
  • 9. Our Research • The behavior we noticed is characteristic of “satisficing” (Simon 1957) – Economic phenomenon: “satisfying” and “sacrificing” – We exercise an acceptable level of effort to achieve a satisfactory but less than optimal outcome – Example: driving around for the cheapest gas price – There is substantial literature written on this topic and how it applies to surveys (Krosnick 1991) – Behavior points to this phenomenon, but very difficult to be certain • The focus of our presentation is NOT on exploring this behavior but on understanding and visualizing different user types – How does the survey experience impact users? – Are there usability issues we can notice or isolate? – Can we build a tool to help improve the overall user experience and hence obtain more complete and accurate information?9
  • 10. Our Findings10
  • 11. Our Findings • What we did: – Examined only the 370 respondents who refused all of Q30 – Determined if unique user types existed through mixture modeling – Wrote code to visually map these user types and their refusals for the remainder of the survey – Marked page breaks and “grid” questions in this map • What we found: – 3 distinct user types • The Quitters • The Returners • The Completers – Map allows us to easily identify these user types – Map also allows us to easily identify “troublesome areas”11
  • 12. Our Findings Black line = pages The 370 Respondents Orange line = grid question Colored squares = question was ANSWERED
  • 13. Our Findings: The Quitters Black line = pages Orange line = grid question Last page of survey Colored squares = question was ANSWERED Demographics • Engagement clearly breaks off and users flip to back of survey • Paper survey immediately presents users with workload • Completes Demographic questions (“essential” and easy items) for token of appreciation
  • 14. Our Findings: The Returners Black line = pages Orange line = grid question Colored squares = question was ANSWERED Grid questions • Engagement terminates after long second grid question (Q30) but returns • Selectively respond to “taskful” questions (i.e., grid questions) to minimize effort
  • 15. Our Findings: The Completers Black line = pages Orange line = grid question Colored squares = question was ANSWERED • Most conscientious and engaged group • Engagement terminates for only Q30 • Occasional refusals Only a few questions are left unanswered by this user type
  • 16. Our Findings: Profiling the User Types Other AsianHispanic Asians show more Completers Black Whites show almost all of the Returners White16
  • 17. Our Findings: Profiling the User Types • Mixture model: seeks homogenous distributions within data based on number of questions refused after Q30 • Predictive model based on census block sociodemographic data linked to respondent scores – Exploratory predictive model (i.e., empirically driven) Hispanic, Native Hawaiian or Pacific Islander Civilian Population in Labor Force Employed; Age 16Overall Population Median Household Income Population and up Population with less than 9th Grade Education; Age Population in Labor Force Unemployed; Age 16 andPopulation aged 16-17 Hispanic, Other Population 25 and up up Population with some High School Education; AgePopulation aged 18-20 Hispanic Population Population not in Labor Force; Age 16 and up 25 and up Population with High School Education; Age 25 and Percent of Population in Labor Force Unemployed;Population aged 21-24 Population in Nursing Home up Age 16 and up Population in other Institutionalized Group Population with some College Education; Age 25 Population employed in Private, for Profit; Age 16 andMedian Age Quarters and up up Population Employed in Private, not-for Profit; Age 16Non-Hispanic, White Population Population in College Dorms Population with Associates Degree; Age 25 and up and up Population Employed in Local Government; Age 16Non-Hispanic, Black Population Population in Military Barracks Population with Bachelors Degree; Age 25 and up and up Population in Non-Institutionalized Group Population Employed in State Government; Age 16Non-Hispanic, American Indian Population Population with Masters Degree; Age 25 and up Quarters and up Population Employed in Federal Government; Age 16Non-Hispanic, Asian Population Average Household Size Population with Professional Degree; Age 25 and up and upNon-Hispanic, Native Hawaiian or Pacific Islander Average Household Size – Non-Family Household Population with Doctorate Degree; Age 25 and up Population Self-Employed; Age 16 and upPopulationNon-Hispanic, Other Population Average Household Size – Family Household Families at Poverty Level Population Unpaid Family Work; Age 16 and up Population Speaking only English at Home; Age 5 Population Employed Blue Collar Work; Age 16 andHispanic, White Population Families at Poverty Level with Children and Older up Population Speaking Spanish at Home; Age 5 and Population Employed White Collar Work; Age 16 andHispanic, Black Population Families above Poverty Level Older up Population Employed Service and Farm Work; Age 16Hispanic, American Indian Population Housing Units Owned by Occupant Families above Poverty Level with Children and up Population in Labor Force Employed by ArmedHispanic, Asian Population Housing Units Rented by Occupant Population Male Forces; Age 16 and up Average Length of Residence Population Female 17
  • 18. Our Findings: Profiling the User Types Unemploy % with Median tired of being surveyed! “I’m Summary of other socio-economic User Type -mentwasting our Bachelor’s Income The government is variables Rate time/money!” Degree Quitters $63,000 7.1% 13.4% Government and private, not-for (n=111) profit employment with large “I’m doing the best I can, but you’re asking a lot” household size conditions Returners $58,000 8.5% 10.7% More transient, socioeconomically (n=180) “I should do a good job at this, my disadvantageous conditions opinions are helping” Completers $61,000 8.1% 11% Less transient, socioeconomically (n=79) advantageous conditions18
  • 19. Our Conclusions & Recommendations19
  • 20. Our Conclusions • The tool gives us immediately visual, easy to interpret results that clearly bring out patterns – “Knew” the user types before we modeled it – Very easy to explain to clients or share across professionals • With visual mapping, we can: – Easily see the entire user experience – Determine if unique user types exist – See if usability problems exist with certain questions and people • Length interactions • Placement issues • Factual vs attitudinal questions20
  • 21. Our Conclusions • Provides great alternative for complex statistical investigation – May not give you anything useful or helpful – Usually empirically driven, so results can change frequently – Hard to determine what to ACTUALLY do! • A simple and effective way to communicate and examine a survey’s effectiveness to clients and other researchers – Can overlay respondent behavior with survey design – Natural extension of pilot-testing and cognitive testing21
  • 22. Our Conclusions • Most concerns are with total non-response. But this suggests specific item non-response patterns – Allows us to pinpoint the characteristics of those items – As well as the people behind that non-response • This suggests that item-level non-response adjustments may be necessary if variable is of key interest – Weigh option against client interests – Complexity can grow exponentially22
  • 23. Our Recommendations 1. Everyone knows the value of pre-testing a survey. This emphasizes it and the need for “true” test conditions. 2. Avoid areas where respondents are forced to engage for long periods 3. The design of a survey are critical and should not be left just to the statisticians! Example: paper surveys and interaction of questions and page location 4. Different users may required different persuasions techniques – Incentive levels – Customized invitations – Survey instructions – Different layout 5. Remember a reassessment of your key variables is always a good idea and can uncover significant issues (try this new tool!)23