Developing a User Interface for Large-         scale, Multi-mode, Survey Data Collection                            Jennif...
Economic Census     •    A measure of the health of U.S. businesses and the American economy          every five years    ...
Economic Census reporting options     • The 2012 Economic Census has three different       modes for data collection:     ...
Paper formSaturday, January 5, 2013
SurveyorSaturday, January 5, 2013
Internet optionSaturday, January 5, 2013
2012 Economic Census                                Internet option     • Only available to single-unit businesses:       ...
Multi-mode data collection     • Offering multiple modes of data collection has       benefits:           – Can save costs...
Multi-mode challenges in the                       Economic Census     • The paper forms, Surveyor (software program) and ...
2012 Economic Census Internet             option: Usability testing goals     1. Evaluate the overall function and perform...
Participants     • 17 respondents from small businesses     • Diverse job titles: accountants, office managers,          b...
Methodology     Where:     • Met at respondents’ place of business     What:     • Respondents used their own computers (o...
Usability findings     • The Web interface performed well during the       testing     • We uncovered some key findings ab...
Questionnaire navigation     • Survey navigation:           – Forward and backward paging           – Questionnaire naviga...
Skip pattern: Paper questionnaireSaturday, January 5, 2013
Skip pattern: Web interfaceSaturday, January 5, 2013
Using skip patterns     • Skip patterns can have a mode-specific advantage:           – Paper:                  • Require ...
Problems with skip patterns:                       Usability findings     •    The Web Interface (and Surveyor) had no aut...
Surveyor navigation treeSaturday, January 5, 2013
Fixing problems with skip patterns     The recommendations:           1.     Program automated skip patterns to navigate t...
Scrolling     • Economic Census has a number of “long”       questions           – Tend to be detailed business activities...
PaperSaturday, January 5, 2013
23Saturday, January 5, 2013
24Saturday, January 5, 2013
25Saturday, January 5, 2013
26Saturday, January 5, 2013
27Saturday, January 5, 2013
28Saturday, January 5, 2013
29Saturday, January 5, 2013
30Saturday, January 5, 2013
31Saturday, January 5, 2013
32Saturday, January 5, 2013
33Saturday, January 5, 2013
34Saturday, January 5, 2013
35Saturday, January 5, 2013
36Saturday, January 5, 2013
37Saturday, January 5, 2013
38Saturday, January 5, 2013
39Saturday, January 5, 2013
SurveyorSaturday, January 5, 2013
41Saturday, January 5, 2013
42Saturday, January 5, 2013
43Saturday, January 5, 2013
44Saturday, January 5, 2013
45Saturday, January 5, 2013
46Saturday, January 5, 2013
47Saturday, January 5, 2013
48Saturday, January 5, 2013
49Saturday, January 5, 2013
50Saturday, January 5, 2013
51Saturday, January 5, 2013
InternetSaturday, January 5, 2013
53Saturday, January 5, 2013
54Saturday, January 5, 2013
55Saturday, January 5, 2013
56Saturday, January 5, 2013
57Saturday, January 5, 2013
58Saturday, January 5, 2013
59Saturday, January 5, 2013
60Saturday, January 5, 2013
61Saturday, January 5, 2013
62Saturday, January 5, 2013
Choosing between paging and scrolling     • Paging creates mode-consistency problems:           – Paper:                  ...
Problems Scrolling: Usability findings     • Scrolling created mode-specific problems:           1. Long items were “intim...
Fixing problems with scrolling     The recommendations:           1.     Break up the items over multiple pages           ...
Response Quality Control     •    Data quality is a big concern:           – Follow-up is expensive           – Post-colle...
Item-specific data edits: Errors                            Errors                            = items respondents         ...
Item-specific data edits: Warnings                            Warnings                            = items we would like   ...
Overall data edits: Form Review                            Form Review                            = indicated problem     ...
Response quality control:                           Usability findings     • Respondents liked the quality control feature...
Challenges to designing the 2012          Economic Census Web interface     • The Economic Census is a large data-collecti...
Contact Information     Jen Beck:     jennifer.l.beck@census.gov     Elizabeth Sinclair:     elizabeth.sinclair@census.gov...
Upcoming SlideShare
Loading in...5
×

Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

1,073

Published on

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,073
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

  1. 1. Developing a User Interface for Large- scale, Multi-mode, Survey Data Collection Jennifer Beck & Elizabeth Sinclair U.S. Census Bureau User Focus 2012 October 19, 2012Saturday, January 5, 2013
  2. 2. Economic Census • A measure of the health of U.S. businesses and the American economy every five years • Collects detailed information on revenue and business performance • Goes out to more than 4 million businesses across industries: 1. Multi-unit businesses = businesses with multiple locations 2. Single-unit businesses = businesses with a single location • Businesses receive a combination of forms tailored to their specific business activities – more than 800 different survey forms • The number of form combinations creates unique multi-mode data collection challengesSaturday, January 5, 2013
  3. 3. Economic Census reporting options • The 2012 Economic Census has three different modes for data collection: 1. Paper forms 2. Surveyor – a software application 3. InternetSaturday, January 5, 2013
  4. 4. Paper formSaturday, January 5, 2013
  5. 5. SurveyorSaturday, January 5, 2013
  6. 6. Internet optionSaturday, January 5, 2013
  7. 7. 2012 Economic Census Internet option • Only available to single-unit businesses: – A pilot for future Economic Censuses – Alternative to Surveyor software • Respondents log into a Web interface through Census website • Respondents fill out and submit all information onlineSaturday, January 5, 2013
  8. 8. Multi-mode data collection • Offering multiple modes of data collection has benefits: – Can save costs – Good customer service • Designing for multiple modes is challenging: – Mode consistency – Mode capabilities and constraintsSaturday, January 5, 2013
  9. 9. Multi-mode challenges in the Economic Census • The paper forms, Surveyor (software program) and the Web interface “share” a creation database • The database contains “images” that will make up the form pages for all three modes • Because the systems “share” pages, the pages have to work for all three modes: – The “shared” design creates limitations and usability issues, especially for Surveyor and the Web – Changes made for the Web also make the same changes to SurveyorSaturday, January 5, 2013
  10. 10. 2012 Economic Census Internet option: Usability testing goals 1. Evaluate the overall function and performance of the online interface 2. Assess respondents’ reactions to the interfaceSaturday, January 5, 2013
  11. 11. Participants • 17 respondents from small businesses • Diverse job titles: accountants, office managers, business owners • From three different industries: – Retail/Service companies – Manufacturing companies – Construction companiesSaturday, January 5, 2013
  12. 12. Methodology Where: • Met at respondents’ place of business What: • Respondents used their own computers (or iPad) How: • Respondents filled out the census forms • Completed tasks to test features of the interface • Used standard “think-aloud” protocolSaturday, January 5, 2013
  13. 13. Usability findings • The Web interface performed well during the testing • We uncovered some key findings about the interface: 1. Questionnaire navigation 2. Scrolling 3. Response quality controlSaturday, January 5, 2013
  14. 14. Questionnaire navigation • Survey navigation: – Forward and backward paging – Questionnaire navigation – response path through a survey • Skip patterns: – Questionnaire navigation path instructions for respondents – Help respondents locate and navigate to the relevant form sectionsSaturday, January 5, 2013
  15. 15. Skip pattern: Paper questionnaireSaturday, January 5, 2013
  16. 16. Skip pattern: Web interfaceSaturday, January 5, 2013
  17. 17. Using skip patterns • Skip patterns can have a mode-specific advantage: – Paper: • Require reading, understanding, and following skip instruction navigation • Respondents do not always follow skip patterns on paper forms well • Can create response errors – On the Web: • Skip patterns can be automated • Directly navigate user down the correct questionnaire navigation path • Can reduce response errorSaturday, January 5, 2013
  18. 18. Problems with skip patterns: Usability findings • The Web Interface (and Surveyor) had no automated skip patterns – Questions still contained skip instructions – Respondents had to page through irrelevant sections – Due to timing and programming complexity • Respondents were confused as to how to navigate to the appropriate section of the questionnaire – Expected forward navigation to take them to the appropriate screen – Expected to be able to click “Remarks” to navigate to that section • Lack of automation was especially problematic for the Web interface – Web interface had no navigation tree – Surveyor software has a navigation treeSaturday, January 5, 2013
  19. 19. Surveyor navigation treeSaturday, January 5, 2013
  20. 20. Fixing problems with skip patterns The recommendations: 1. Program automated skip patterns to navigate the respondent to the next questions (and remove skip instructions) OR 2. Include a navigation tree The resolutions: • Skip instruction stayed with the question – Shared image database made changing question layout and presentation not possible • Programming complexities and time constraints prevented adding a navigation treeSaturday, January 5, 2013
  21. 21. Scrolling • Economic Census has a number of “long” questions – Tend to be detailed business activities and cumulative calculations – Do not fit on a single “page” • Long questions require mode-specific solutions: – Paper: • Pagination – Web interface (and Surveyor): • Paging • ScrollingSaturday, January 5, 2013
  22. 22. PaperSaturday, January 5, 2013
  23. 23. 23Saturday, January 5, 2013
  24. 24. 24Saturday, January 5, 2013
  25. 25. 25Saturday, January 5, 2013
  26. 26. 26Saturday, January 5, 2013
  27. 27. 27Saturday, January 5, 2013
  28. 28. 28Saturday, January 5, 2013
  29. 29. 29Saturday, January 5, 2013
  30. 30. 30Saturday, January 5, 2013
  31. 31. 31Saturday, January 5, 2013
  32. 32. 32Saturday, January 5, 2013
  33. 33. 33Saturday, January 5, 2013
  34. 34. 34Saturday, January 5, 2013
  35. 35. 35Saturday, January 5, 2013
  36. 36. 36Saturday, January 5, 2013
  37. 37. 37Saturday, January 5, 2013
  38. 38. 38Saturday, January 5, 2013
  39. 39. 39Saturday, January 5, 2013
  40. 40. SurveyorSaturday, January 5, 2013
  41. 41. 41Saturday, January 5, 2013
  42. 42. 42Saturday, January 5, 2013
  43. 43. 43Saturday, January 5, 2013
  44. 44. 44Saturday, January 5, 2013
  45. 45. 45Saturday, January 5, 2013
  46. 46. 46Saturday, January 5, 2013
  47. 47. 47Saturday, January 5, 2013
  48. 48. 48Saturday, January 5, 2013
  49. 49. 49Saturday, January 5, 2013
  50. 50. 50Saturday, January 5, 2013
  51. 51. 51Saturday, January 5, 2013
  52. 52. InternetSaturday, January 5, 2013
  53. 53. 53Saturday, January 5, 2013
  54. 54. 54Saturday, January 5, 2013
  55. 55. 55Saturday, January 5, 2013
  56. 56. 56Saturday, January 5, 2013
  57. 57. 57Saturday, January 5, 2013
  58. 58. 58Saturday, January 5, 2013
  59. 59. 59Saturday, January 5, 2013
  60. 60. 60Saturday, January 5, 2013
  61. 61. 61Saturday, January 5, 2013
  62. 62. 62Saturday, January 5, 2013
  63. 63. Choosing between paging and scrolling • Paging creates mode-consistency problems: – Paper: • Easy to “flip” back and forth between the question sections – Web interface (and Surveyor): • Requires forward and backward navigation to go back and forth between the question sections • Breaking up the item across pages can create response errors • Scrolling “preserves” mode consistency – The 2012 Economic Census Web interface employed scrollingSaturday, January 5, 2013
  64. 64. Problems Scrolling: Usability findings • Scrolling created mode-specific problems: 1. Long items were “intimidating” to respondents 2. Length attenuated the benefits of scrolling over paging • Scrolling was especially problematic for the Web interface – Timing out: • Long items take longer to fill out • Data save linked to navigating to the next page • Respondents could lose dataSaturday, January 5, 2013
  65. 65. Fixing problems with scrolling The recommendations: 1. Break up the items over multiple pages 2. (Make changes to the question to maintain mode consistency) 3. Include a navigation tree The resolutions: • Questions maintained scrolling design: – Shared image database made changing question layout and presentation not possible • Not enough time to re-program and add a navigation tree • Time out increased to 50 minutes • Data saved on time outSaturday, January 5, 2013
  66. 66. Response Quality Control • Data quality is a big concern: – Follow-up is expensive – Post-collection editing is expensive and time consuming • Respondents find data edits helpful – Provides a “check” against errors – Communicates the importance of the information • Controlling for data quality is mode-specific advantage: – Paper forms offer no “built-in” data quality control – The Web interface allows for “real time data editing” – Can link responding to satisfying quality requirements • The web interface had edit triggers for individual items and for of theSaturday, January 5, 2013
  67. 67. Item-specific data edits: Errors Errors = items respondents must fill out in order to submit their data to CensusSaturday, January 5, 2013
  68. 68. Item-specific data edits: Warnings Warnings = items we would like respondents to fill out * * Respondents can still submit their data to Census with out filling them outSaturday, January 5, 2013
  69. 69. Overall data edits: Form Review Form Review = indicated problem sections with color coding • One-click navigation to problem sectionSaturday, January 5, 2013
  70. 70. Response quality control: Usability findings • Respondents liked the quality control features: – Item-specific data edits: • The edit messages were clear and helpful • Liked having the “extra check” – Form Review: • Really liked the layout of the form review • Liked the ease of navigating to problem sections • Capabilities were examples of successful design features • Provide in quality-control measure not possible in a paper modeSaturday, January 5, 2013
  71. 71. Challenges to designing the 2012 Economic Census Web interface • The Economic Census is a large data-collection undertaking • Presents challenges for how to maintain consistency between modes and optimize mode advantages • More work needs to be done to refine web data collection for future Economic CensusesSaturday, January 5, 2013
  72. 72. Contact Information Jen Beck: jennifer.l.beck@census.gov Elizabeth Sinclair: elizabeth.sinclair@census.gov 72Saturday, January 5, 2013
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×