• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)
 

Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir)

on

  • 1,187 views

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

Statistics

Views

Total Views
1,187
Views on SlideShare
1,187
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir) Developing a User Interface for Large-Scale Surveys (Jennifer Beck & Elizabeth Sincalir) Presentation Transcript

    • Developing a User Interface for Large- scale, Multi-mode, Survey Data Collection Jennifer Beck & Elizabeth Sinclair U.S. Census Bureau User Focus 2012 October 19, 2012Saturday, January 5, 2013
    • Economic Census • A measure of the health of U.S. businesses and the American economy every five years • Collects detailed information on revenue and business performance • Goes out to more than 4 million businesses across industries: 1. Multi-unit businesses = businesses with multiple locations 2. Single-unit businesses = businesses with a single location • Businesses receive a combination of forms tailored to their specific business activities – more than 800 different survey forms • The number of form combinations creates unique multi-mode data collection challengesSaturday, January 5, 2013
    • Economic Census reporting options • The 2012 Economic Census has three different modes for data collection: 1. Paper forms 2. Surveyor – a software application 3. InternetSaturday, January 5, 2013
    • Paper formSaturday, January 5, 2013
    • SurveyorSaturday, January 5, 2013
    • Internet optionSaturday, January 5, 2013
    • 2012 Economic Census Internet option • Only available to single-unit businesses: – A pilot for future Economic Censuses – Alternative to Surveyor software • Respondents log into a Web interface through Census website • Respondents fill out and submit all information onlineSaturday, January 5, 2013
    • Multi-mode data collection • Offering multiple modes of data collection has benefits: – Can save costs – Good customer service • Designing for multiple modes is challenging: – Mode consistency – Mode capabilities and constraintsSaturday, January 5, 2013
    • Multi-mode challenges in the Economic Census • The paper forms, Surveyor (software program) and the Web interface “share” a creation database • The database contains “images” that will make up the form pages for all three modes • Because the systems “share” pages, the pages have to work for all three modes: – The “shared” design creates limitations and usability issues, especially for Surveyor and the Web – Changes made for the Web also make the same changes to SurveyorSaturday, January 5, 2013
    • 2012 Economic Census Internet option: Usability testing goals 1. Evaluate the overall function and performance of the online interface 2. Assess respondents’ reactions to the interfaceSaturday, January 5, 2013
    • Participants • 17 respondents from small businesses • Diverse job titles: accountants, office managers, business owners • From three different industries: – Retail/Service companies – Manufacturing companies – Construction companiesSaturday, January 5, 2013
    • Methodology Where: • Met at respondents’ place of business What: • Respondents used their own computers (or iPad) How: • Respondents filled out the census forms • Completed tasks to test features of the interface • Used standard “think-aloud” protocolSaturday, January 5, 2013
    • Usability findings • The Web interface performed well during the testing • We uncovered some key findings about the interface: 1. Questionnaire navigation 2. Scrolling 3. Response quality controlSaturday, January 5, 2013
    • Questionnaire navigation • Survey navigation: – Forward and backward paging – Questionnaire navigation – response path through a survey • Skip patterns: – Questionnaire navigation path instructions for respondents – Help respondents locate and navigate to the relevant form sectionsSaturday, January 5, 2013
    • Skip pattern: Paper questionnaireSaturday, January 5, 2013
    • Skip pattern: Web interfaceSaturday, January 5, 2013
    • Using skip patterns • Skip patterns can have a mode-specific advantage: – Paper: • Require reading, understanding, and following skip instruction navigation • Respondents do not always follow skip patterns on paper forms well • Can create response errors – On the Web: • Skip patterns can be automated • Directly navigate user down the correct questionnaire navigation path • Can reduce response errorSaturday, January 5, 2013
    • Problems with skip patterns: Usability findings • The Web Interface (and Surveyor) had no automated skip patterns – Questions still contained skip instructions – Respondents had to page through irrelevant sections – Due to timing and programming complexity • Respondents were confused as to how to navigate to the appropriate section of the questionnaire – Expected forward navigation to take them to the appropriate screen – Expected to be able to click “Remarks” to navigate to that section • Lack of automation was especially problematic for the Web interface – Web interface had no navigation tree – Surveyor software has a navigation treeSaturday, January 5, 2013
    • Surveyor navigation treeSaturday, January 5, 2013
    • Fixing problems with skip patterns The recommendations: 1. Program automated skip patterns to navigate the respondent to the next questions (and remove skip instructions) OR 2. Include a navigation tree The resolutions: • Skip instruction stayed with the question – Shared image database made changing question layout and presentation not possible • Programming complexities and time constraints prevented adding a navigation treeSaturday, January 5, 2013
    • Scrolling • Economic Census has a number of “long” questions – Tend to be detailed business activities and cumulative calculations – Do not fit on a single “page” • Long questions require mode-specific solutions: – Paper: • Pagination – Web interface (and Surveyor): • Paging • ScrollingSaturday, January 5, 2013
    • PaperSaturday, January 5, 2013
    • 23Saturday, January 5, 2013
    • 24Saturday, January 5, 2013
    • 25Saturday, January 5, 2013
    • 26Saturday, January 5, 2013
    • 27Saturday, January 5, 2013
    • 28Saturday, January 5, 2013
    • 29Saturday, January 5, 2013
    • 30Saturday, January 5, 2013
    • 31Saturday, January 5, 2013
    • 32Saturday, January 5, 2013
    • 33Saturday, January 5, 2013
    • 34Saturday, January 5, 2013
    • 35Saturday, January 5, 2013
    • 36Saturday, January 5, 2013
    • 37Saturday, January 5, 2013
    • 38Saturday, January 5, 2013
    • 39Saturday, January 5, 2013
    • SurveyorSaturday, January 5, 2013
    • 41Saturday, January 5, 2013
    • 42Saturday, January 5, 2013
    • 43Saturday, January 5, 2013
    • 44Saturday, January 5, 2013
    • 45Saturday, January 5, 2013
    • 46Saturday, January 5, 2013
    • 47Saturday, January 5, 2013
    • 48Saturday, January 5, 2013
    • 49Saturday, January 5, 2013
    • 50Saturday, January 5, 2013
    • 51Saturday, January 5, 2013
    • InternetSaturday, January 5, 2013
    • 53Saturday, January 5, 2013
    • 54Saturday, January 5, 2013
    • 55Saturday, January 5, 2013
    • 56Saturday, January 5, 2013
    • 57Saturday, January 5, 2013
    • 58Saturday, January 5, 2013
    • 59Saturday, January 5, 2013
    • 60Saturday, January 5, 2013
    • 61Saturday, January 5, 2013
    • 62Saturday, January 5, 2013
    • Choosing between paging and scrolling • Paging creates mode-consistency problems: – Paper: • Easy to “flip” back and forth between the question sections – Web interface (and Surveyor): • Requires forward and backward navigation to go back and forth between the question sections • Breaking up the item across pages can create response errors • Scrolling “preserves” mode consistency – The 2012 Economic Census Web interface employed scrollingSaturday, January 5, 2013
    • Problems Scrolling: Usability findings • Scrolling created mode-specific problems: 1. Long items were “intimidating” to respondents 2. Length attenuated the benefits of scrolling over paging • Scrolling was especially problematic for the Web interface – Timing out: • Long items take longer to fill out • Data save linked to navigating to the next page • Respondents could lose dataSaturday, January 5, 2013
    • Fixing problems with scrolling The recommendations: 1. Break up the items over multiple pages 2. (Make changes to the question to maintain mode consistency) 3. Include a navigation tree The resolutions: • Questions maintained scrolling design: – Shared image database made changing question layout and presentation not possible • Not enough time to re-program and add a navigation tree • Time out increased to 50 minutes • Data saved on time outSaturday, January 5, 2013
    • Response Quality Control • Data quality is a big concern: – Follow-up is expensive – Post-collection editing is expensive and time consuming • Respondents find data edits helpful – Provides a “check” against errors – Communicates the importance of the information • Controlling for data quality is mode-specific advantage: – Paper forms offer no “built-in” data quality control – The Web interface allows for “real time data editing” – Can link responding to satisfying quality requirements • The web interface had edit triggers for individual items and for of theSaturday, January 5, 2013
    • Item-specific data edits: Errors Errors = items respondents must fill out in order to submit their data to CensusSaturday, January 5, 2013
    • Item-specific data edits: Warnings Warnings = items we would like respondents to fill out * * Respondents can still submit their data to Census with out filling them outSaturday, January 5, 2013
    • Overall data edits: Form Review Form Review = indicated problem sections with color coding • One-click navigation to problem sectionSaturday, January 5, 2013
    • Response quality control: Usability findings • Respondents liked the quality control features: – Item-specific data edits: • The edit messages were clear and helpful • Liked having the “extra check” – Form Review: • Really liked the layout of the form review • Liked the ease of navigating to problem sections • Capabilities were examples of successful design features • Provide in quality-control measure not possible in a paper modeSaturday, January 5, 2013
    • Challenges to designing the 2012 Economic Census Web interface • The Economic Census is a large data-collection undertaking • Presents challenges for how to maintain consistency between modes and optimize mode advantages • More work needs to be done to refine web data collection for future Economic CensusesSaturday, January 5, 2013
    • Contact Information Jen Beck: jennifer.l.beck@census.gov Elizabeth Sinclair: elizabeth.sinclair@census.gov 72Saturday, January 5, 2013