Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

on

  • 1,317 views

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

Statistics

Views

Total Views
1,317
Views on SlideShare
1,317
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir) Presentation Transcript

  • 1. Developing a User Interface for Large- scale, Multi-mode, Survey Data Collection Jennifer Beck & Elizabeth Sinclair U.S. Census Bureau User Focus 2012 October 19, 2012Saturday, January 5, 2013
  • 2. Economic Census • A measure of the health of U.S. businesses and the American economy every five years • Collects detailed information on revenue and business performance • Goes out to more than 4 million businesses across industries: 1. Multi-unit businesses = businesses with multiple locations 2. Single-unit businesses = businesses with a single location • Businesses receive a combination of forms tailored to their specific business activities – more than 800 different survey forms • The number of form combinations creates unique multi-mode data collection challengesSaturday, January 5, 2013
  • 3. Economic Census reporting options • The 2012 Economic Census has three different modes for data collection: 1. Paper forms 2. Surveyor – a software application 3. InternetSaturday, January 5, 2013
  • 4. Paper formSaturday, January 5, 2013
  • 5. SurveyorSaturday, January 5, 2013
  • 6. Internet optionSaturday, January 5, 2013
  • 7. 2012 Economic Census Internet option • Only available to single-unit businesses: – A pilot for future Economic Censuses – Alternative to Surveyor software • Respondents log into a Web interface through Census website • Respondents fill out and submit all information onlineSaturday, January 5, 2013
  • 8. Multi-mode data collection • Offering multiple modes of data collection has benefits: – Can save costs – Good customer service • Designing for multiple modes is challenging: – Mode consistency – Mode capabilities and constraintsSaturday, January 5, 2013
  • 9. Multi-mode challenges in the Economic Census • The paper forms, Surveyor (software program) and the Web interface “share” a creation database • The database contains “images” that will make up the form pages for all three modes • Because the systems “share” pages, the pages have to work for all three modes: – The “shared” design creates limitations and usability issues, especially for Surveyor and the Web – Changes made for the Web also make the same changes to SurveyorSaturday, January 5, 2013
  • 10. 2012 Economic Census Internet option: Usability testing goals 1. Evaluate the overall function and performance of the online interface 2. Assess respondents’ reactions to the interfaceSaturday, January 5, 2013
  • 11. Participants • 17 respondents from small businesses • Diverse job titles: accountants, office managers, business owners • From three different industries: – Retail/Service companies – Manufacturing companies – Construction companiesSaturday, January 5, 2013
  • 12. Methodology Where: • Met at respondents’ place of business What: • Respondents used their own computers (or iPad) How: • Respondents filled out the census forms • Completed tasks to test features of the interface • Used standard “think-aloud” protocolSaturday, January 5, 2013
  • 13. Usability findings • The Web interface performed well during the testing • We uncovered some key findings about the interface: 1. Questionnaire navigation 2. Scrolling 3. Response quality controlSaturday, January 5, 2013
  • 14. Questionnaire navigation • Survey navigation: – Forward and backward paging – Questionnaire navigation – response path through a survey • Skip patterns: – Questionnaire navigation path instructions for respondents – Help respondents locate and navigate to the relevant form sectionsSaturday, January 5, 2013
  • 15. Skip pattern: Paper questionnaireSaturday, January 5, 2013
  • 16. Skip pattern: Web interfaceSaturday, January 5, 2013
  • 17. Using skip patterns • Skip patterns can have a mode-specific advantage: – Paper: • Require reading, understanding, and following skip instruction navigation • Respondents do not always follow skip patterns on paper forms well • Can create response errors – On the Web: • Skip patterns can be automated • Directly navigate user down the correct questionnaire navigation path • Can reduce response errorSaturday, January 5, 2013
  • 18. Problems with skip patterns: Usability findings • The Web Interface (and Surveyor) had no automated skip patterns – Questions still contained skip instructions – Respondents had to page through irrelevant sections – Due to timing and programming complexity • Respondents were confused as to how to navigate to the appropriate section of the questionnaire – Expected forward navigation to take them to the appropriate screen – Expected to be able to click “Remarks” to navigate to that section • Lack of automation was especially problematic for the Web interface – Web interface had no navigation tree – Surveyor software has a navigation treeSaturday, January 5, 2013
  • 19. Surveyor navigation treeSaturday, January 5, 2013
  • 20. Fixing problems with skip patterns The recommendations: 1. Program automated skip patterns to navigate the respondent to the next questions (and remove skip instructions) OR 2. Include a navigation tree The resolutions: • Skip instruction stayed with the question – Shared image database made changing question layout and presentation not possible • Programming complexities and time constraints prevented adding a navigation treeSaturday, January 5, 2013
  • 21. Scrolling • Economic Census has a number of “long” questions – Tend to be detailed business activities and cumulative calculations – Do not fit on a single “page” • Long questions require mode-specific solutions: – Paper: • Pagination – Web interface (and Surveyor): • Paging • ScrollingSaturday, January 5, 2013
  • 22. PaperSaturday, January 5, 2013
  • 23. 23Saturday, January 5, 2013
  • 24. 24Saturday, January 5, 2013
  • 25. 25Saturday, January 5, 2013
  • 26. 26Saturday, January 5, 2013
  • 27. 27Saturday, January 5, 2013
  • 28. 28Saturday, January 5, 2013
  • 29. 29Saturday, January 5, 2013
  • 30. 30Saturday, January 5, 2013
  • 31. 31Saturday, January 5, 2013
  • 32. 32Saturday, January 5, 2013
  • 33. 33Saturday, January 5, 2013
  • 34. 34Saturday, January 5, 2013
  • 35. 35Saturday, January 5, 2013
  • 36. 36Saturday, January 5, 2013
  • 37. 37Saturday, January 5, 2013
  • 38. 38Saturday, January 5, 2013
  • 39. 39Saturday, January 5, 2013
  • 40. SurveyorSaturday, January 5, 2013
  • 41. 41Saturday, January 5, 2013
  • 42. 42Saturday, January 5, 2013
  • 43. 43Saturday, January 5, 2013
  • 44. 44Saturday, January 5, 2013
  • 45. 45Saturday, January 5, 2013
  • 46. 46Saturday, January 5, 2013
  • 47. 47Saturday, January 5, 2013
  • 48. 48Saturday, January 5, 2013
  • 49. 49Saturday, January 5, 2013
  • 50. 50Saturday, January 5, 2013
  • 51. 51Saturday, January 5, 2013
  • 52. InternetSaturday, January 5, 2013
  • 53. 53Saturday, January 5, 2013
  • 54. 54Saturday, January 5, 2013
  • 55. 55Saturday, January 5, 2013
  • 56. 56Saturday, January 5, 2013
  • 57. 57Saturday, January 5, 2013
  • 58. 58Saturday, January 5, 2013
  • 59. 59Saturday, January 5, 2013
  • 60. 60Saturday, January 5, 2013
  • 61. 61Saturday, January 5, 2013
  • 62. 62Saturday, January 5, 2013
  • 63. Choosing between paging and scrolling • Paging creates mode-consistency problems: – Paper: • Easy to “flip” back and forth between the question sections – Web interface (and Surveyor): • Requires forward and backward navigation to go back and forth between the question sections • Breaking up the item across pages can create response errors • Scrolling “preserves” mode consistency – The 2012 Economic Census Web interface employed scrollingSaturday, January 5, 2013
  • 64. Problems Scrolling: Usability findings • Scrolling created mode-specific problems: 1. Long items were “intimidating” to respondents 2. Length attenuated the benefits of scrolling over paging • Scrolling was especially problematic for the Web interface – Timing out: • Long items take longer to fill out • Data save linked to navigating to the next page • Respondents could lose dataSaturday, January 5, 2013
  • 65. Fixing problems with scrolling The recommendations: 1. Break up the items over multiple pages 2. (Make changes to the question to maintain mode consistency) 3. Include a navigation tree The resolutions: • Questions maintained scrolling design: – Shared image database made changing question layout and presentation not possible • Not enough time to re-program and add a navigation tree • Time out increased to 50 minutes • Data saved on time outSaturday, January 5, 2013
  • 66. Response Quality Control • Data quality is a big concern: – Follow-up is expensive – Post-collection editing is expensive and time consuming • Respondents find data edits helpful – Provides a “check” against errors – Communicates the importance of the information • Controlling for data quality is mode-specific advantage: – Paper forms offer no “built-in” data quality control – The Web interface allows for “real time data editing” – Can link responding to satisfying quality requirements • The web interface had edit triggers for individual items and for of theSaturday, January 5, 2013
  • 67. Item-specific data edits: Errors Errors = items respondents must fill out in order to submit their data to CensusSaturday, January 5, 2013
  • 68. Item-specific data edits: Warnings Warnings = items we would like respondents to fill out * * Respondents can still submit their data to Census with out filling them outSaturday, January 5, 2013
  • 69. Overall data edits: Form Review Form Review = indicated problem sections with color coding • One-click navigation to problem sectionSaturday, January 5, 2013
  • 70. Response quality control: Usability findings • Respondents liked the quality control features: – Item-specific data edits: • The edit messages were clear and helpful • Liked having the “extra check” – Form Review: • Really liked the layout of the form review • Liked the ease of navigating to problem sections • Capabilities were examples of successful design features • Provide in quality-control measure not possible in a paper modeSaturday, January 5, 2013
  • 71. Challenges to designing the 2012 Economic Census Web interface • The Economic Census is a large data-collection undertaking • Presents challenges for how to maintain consistency between modes and optimize mode advantages • More work needs to be done to refine web data collection for future Economic CensusesSaturday, January 5, 2013
  • 72. Contact Information Jen Beck: jennifer.l.beck@census.gov Elizabeth Sinclair: elizabeth.sinclair@census.gov 72Saturday, January 5, 2013