Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

  • 917 views
Uploaded on

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
917
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Developing a User Interface for Large- scale, Multi-mode, Survey Data Collection Jennifer Beck & Elizabeth Sinclair U.S. Census Bureau User Focus 2012 October 19, 2012Saturday, January 5, 2013
  • 2. Economic Census • A measure of the health of U.S. businesses and the American economy every five years • Collects detailed information on revenue and business performance • Goes out to more than 4 million businesses across industries: 1. Multi-unit businesses = businesses with multiple locations 2. Single-unit businesses = businesses with a single location • Businesses receive a combination of forms tailored to their specific business activities – more than 800 different survey forms • The number of form combinations creates unique multi-mode data collection challengesSaturday, January 5, 2013
  • 3. Economic Census reporting options • The 2012 Economic Census has three different modes for data collection: 1. Paper forms 2. Surveyor – a software application 3. InternetSaturday, January 5, 2013
  • 4. Paper formSaturday, January 5, 2013
  • 5. SurveyorSaturday, January 5, 2013
  • 6. Internet optionSaturday, January 5, 2013
  • 7. 2012 Economic Census Internet option • Only available to single-unit businesses: – A pilot for future Economic Censuses – Alternative to Surveyor software • Respondents log into a Web interface through Census website • Respondents fill out and submit all information onlineSaturday, January 5, 2013
  • 8. Multi-mode data collection • Offering multiple modes of data collection has benefits: – Can save costs – Good customer service • Designing for multiple modes is challenging: – Mode consistency – Mode capabilities and constraintsSaturday, January 5, 2013
  • 9. Multi-mode challenges in the Economic Census • The paper forms, Surveyor (software program) and the Web interface “share” a creation database • The database contains “images” that will make up the form pages for all three modes • Because the systems “share” pages, the pages have to work for all three modes: – The “shared” design creates limitations and usability issues, especially for Surveyor and the Web – Changes made for the Web also make the same changes to SurveyorSaturday, January 5, 2013
  • 10. 2012 Economic Census Internet option: Usability testing goals 1. Evaluate the overall function and performance of the online interface 2. Assess respondents’ reactions to the interfaceSaturday, January 5, 2013
  • 11. Participants • 17 respondents from small businesses • Diverse job titles: accountants, office managers, business owners • From three different industries: – Retail/Service companies – Manufacturing companies – Construction companiesSaturday, January 5, 2013
  • 12. Methodology Where: • Met at respondents’ place of business What: • Respondents used their own computers (or iPad) How: • Respondents filled out the census forms • Completed tasks to test features of the interface • Used standard “think-aloud” protocolSaturday, January 5, 2013
  • 13. Usability findings • The Web interface performed well during the testing • We uncovered some key findings about the interface: 1. Questionnaire navigation 2. Scrolling 3. Response quality controlSaturday, January 5, 2013
  • 14. Questionnaire navigation • Survey navigation: – Forward and backward paging – Questionnaire navigation – response path through a survey • Skip patterns: – Questionnaire navigation path instructions for respondents – Help respondents locate and navigate to the relevant form sectionsSaturday, January 5, 2013
  • 15. Skip pattern: Paper questionnaireSaturday, January 5, 2013
  • 16. Skip pattern: Web interfaceSaturday, January 5, 2013
  • 17. Using skip patterns • Skip patterns can have a mode-specific advantage: – Paper: • Require reading, understanding, and following skip instruction navigation • Respondents do not always follow skip patterns on paper forms well • Can create response errors – On the Web: • Skip patterns can be automated • Directly navigate user down the correct questionnaire navigation path • Can reduce response errorSaturday, January 5, 2013
  • 18. Problems with skip patterns: Usability findings • The Web Interface (and Surveyor) had no automated skip patterns – Questions still contained skip instructions – Respondents had to page through irrelevant sections – Due to timing and programming complexity • Respondents were confused as to how to navigate to the appropriate section of the questionnaire – Expected forward navigation to take them to the appropriate screen – Expected to be able to click “Remarks” to navigate to that section • Lack of automation was especially problematic for the Web interface – Web interface had no navigation tree – Surveyor software has a navigation treeSaturday, January 5, 2013
  • 19. Surveyor navigation treeSaturday, January 5, 2013
  • 20. Fixing problems with skip patterns The recommendations: 1. Program automated skip patterns to navigate the respondent to the next questions (and remove skip instructions) OR 2. Include a navigation tree The resolutions: • Skip instruction stayed with the question – Shared image database made changing question layout and presentation not possible • Programming complexities and time constraints prevented adding a navigation treeSaturday, January 5, 2013
  • 21. Scrolling • Economic Census has a number of “long” questions – Tend to be detailed business activities and cumulative calculations – Do not fit on a single “page” • Long questions require mode-specific solutions: – Paper: • Pagination – Web interface (and Surveyor): • Paging • ScrollingSaturday, January 5, 2013
  • 22. PaperSaturday, January 5, 2013
  • 23. 23Saturday, January 5, 2013
  • 24. 24Saturday, January 5, 2013
  • 25. 25Saturday, January 5, 2013
  • 26. 26Saturday, January 5, 2013
  • 27. 27Saturday, January 5, 2013
  • 28. 28Saturday, January 5, 2013
  • 29. 29Saturday, January 5, 2013
  • 30. 30Saturday, January 5, 2013
  • 31. 31Saturday, January 5, 2013
  • 32. 32Saturday, January 5, 2013
  • 33. 33Saturday, January 5, 2013
  • 34. 34Saturday, January 5, 2013
  • 35. 35Saturday, January 5, 2013
  • 36. 36Saturday, January 5, 2013
  • 37. 37Saturday, January 5, 2013
  • 38. 38Saturday, January 5, 2013
  • 39. 39Saturday, January 5, 2013
  • 40. SurveyorSaturday, January 5, 2013
  • 41. 41Saturday, January 5, 2013
  • 42. 42Saturday, January 5, 2013
  • 43. 43Saturday, January 5, 2013
  • 44. 44Saturday, January 5, 2013
  • 45. 45Saturday, January 5, 2013
  • 46. 46Saturday, January 5, 2013
  • 47. 47Saturday, January 5, 2013
  • 48. 48Saturday, January 5, 2013
  • 49. 49Saturday, January 5, 2013
  • 50. 50Saturday, January 5, 2013
  • 51. 51Saturday, January 5, 2013
  • 52. InternetSaturday, January 5, 2013
  • 53. 53Saturday, January 5, 2013
  • 54. 54Saturday, January 5, 2013
  • 55. 55Saturday, January 5, 2013
  • 56. 56Saturday, January 5, 2013
  • 57. 57Saturday, January 5, 2013
  • 58. 58Saturday, January 5, 2013
  • 59. 59Saturday, January 5, 2013
  • 60. 60Saturday, January 5, 2013
  • 61. 61Saturday, January 5, 2013
  • 62. 62Saturday, January 5, 2013
  • 63. Choosing between paging and scrolling • Paging creates mode-consistency problems: – Paper: • Easy to “flip” back and forth between the question sections – Web interface (and Surveyor): • Requires forward and backward navigation to go back and forth between the question sections • Breaking up the item across pages can create response errors • Scrolling “preserves” mode consistency – The 2012 Economic Census Web interface employed scrollingSaturday, January 5, 2013
  • 64. Problems Scrolling: Usability findings • Scrolling created mode-specific problems: 1. Long items were “intimidating” to respondents 2. Length attenuated the benefits of scrolling over paging • Scrolling was especially problematic for the Web interface – Timing out: • Long items take longer to fill out • Data save linked to navigating to the next page • Respondents could lose dataSaturday, January 5, 2013
  • 65. Fixing problems with scrolling The recommendations: 1. Break up the items over multiple pages 2. (Make changes to the question to maintain mode consistency) 3. Include a navigation tree The resolutions: • Questions maintained scrolling design: – Shared image database made changing question layout and presentation not possible • Not enough time to re-program and add a navigation tree • Time out increased to 50 minutes • Data saved on time outSaturday, January 5, 2013
  • 66. Response Quality Control • Data quality is a big concern: – Follow-up is expensive – Post-collection editing is expensive and time consuming • Respondents find data edits helpful – Provides a “check” against errors – Communicates the importance of the information • Controlling for data quality is mode-specific advantage: – Paper forms offer no “built-in” data quality control – The Web interface allows for “real time data editing” – Can link responding to satisfying quality requirements • The web interface had edit triggers for individual items and for of theSaturday, January 5, 2013
  • 67. Item-specific data edits: Errors Errors = items respondents must fill out in order to submit their data to CensusSaturday, January 5, 2013
  • 68. Item-specific data edits: Warnings Warnings = items we would like respondents to fill out * * Respondents can still submit their data to Census with out filling them outSaturday, January 5, 2013
  • 69. Overall data edits: Form Review Form Review = indicated problem sections with color coding • One-click navigation to problem sectionSaturday, January 5, 2013
  • 70. Response quality control: Usability findings • Respondents liked the quality control features: – Item-specific data edits: • The edit messages were clear and helpful • Liked having the “extra check” – Form Review: • Really liked the layout of the form review • Liked the ease of navigating to problem sections • Capabilities were examples of successful design features • Provide in quality-control measure not possible in a paper modeSaturday, January 5, 2013
  • 71. Challenges to designing the 2012 Economic Census Web interface • The Economic Census is a large data-collection undertaking • Presents challenges for how to maintain consistency between modes and optimize mode advantages • More work needs to be done to refine web data collection for future Economic CensusesSaturday, January 5, 2013
  • 72. Contact Information Jen Beck: jennifer.l.beck@census.gov Elizabeth Sinclair: elizabeth.sinclair@census.gov 72Saturday, January 5, 2013