• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Display Matters: A Test of Visual Display Options in a Web-Based Survey
 

Display Matters: A Test of Visual Display Options in a Web-Based Survey

on

  • 373 views

Surveys are increasingly being conducted online, and it is pertinent to establish clear guidelines for presenting self-administered survey items on a computer screen. Goals should include reducing ...

Surveys are increasingly being conducted online, and it is pertinent to establish clear guidelines for presenting self-administered survey items on a computer screen. Goals should include reducing respondent burden and measurement error. Web survey designers often need to decide how to best present long lists of information and where to place navigation buttons (i.e., next and previous). We evaluated the usability of web-based National Survey of College Graduates (NSCG) questionnaire prototypes. In the first round of testing (n=8), respondents had difficulties proceeding through the survey because the ‘Next’ button was on the left side of the screen and the ‘Previous’ button was on the right. In the second round of testing (n=30), four versions of the survey were tested to assess usability of the ‘Next’ and ‘Previous’ buttons based on their placement on the screen (right or left side) and additionally, to assess the display format of a long list of occupation options (one-column versus two-columns). Dependent measures included participants’ comments from a think aloud protocol, self-reported ratings of satisfaction with the survey, responses to qualitative debriefing questions, time required to complete the job code item, and eye-tracking data focusing on which button (i.e., ‘Next’ or ‘Previous’) and which half of the job code list participants looked at first and more often. Qualitative and quantitative results revealed that participants preferred and performed quicker when the long list of occupation options was displayed in two columns rather than one and when ‘Next’ was displayed to the right of the ‘Previous’ button rather than vice versa. Our findings support usability best practice guidelines that recommend eliminating excessive scrolling on Web sites, and following reading conventions (i.e., looking to the right to move forward, as if turning a page in a book).

Statistics

Views

Total Views
373
Views on SlideShare
373
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Display Matters: A Test of Visual Display Options in a Web-Based Survey Display Matters: A Test of Visual Display Options in a Web-Based Survey Presentation Transcript

    • Display  Ma*ers:    A  Test  of  Visual  Display  Op6ons  in  a   Web-­‐Based  Survey   Jennifer  C.  Romano  Bergstrom1,  Jennifer  M.  Chen1,     Timothy  R.  Gilbert2  &  Ma*  Jans1   1  Center  for  Survey  Measurement   2  Demographic  Surveys  Division   U.S.  Census  Bureau   AAPOR  66th  Annual  Conference     May  13,  2011  
    • Current  Survey  Environment  •  Increasing  number  of  surveys  online  •  Design  considera6ons   –  Naviga6on  methods   –  Presenta6on  of  response  op6ons   2  
    • Current  Survey  Environment  •  Increasing  number  of  surveys  online  •  Design  considera6ons   –  Naviga6on  methods   –  Presenta6on  of  response  op6ons   3  
    • Background  on  Next  and  Previous  •  Next  should  be  on  the  leU   –  Reduces  the  amount  of  6me  to  move  cursor  to   primary  naviga6on  bu*on  (Couper,  2008)   –  Frequency  of  use  (Dillman  et  al.,  2009;  Faulkner,   1998;  Koyani  et  al.,  2004;  Wroblewski,  2008)   4  
    • Background  on  Next  and  Previous  •  Previous  should  be  on  the  leU   –  Web  applica6on  order   –  Everyday  devices   –  Logical  reading  order   5  
    • Background  on  Next  and  Previous  •  Previous  should  be  on  the  leU   –  Web  applica6on  order   –  Everyday  devices   –  Logical  reading  order   6  
    • Background  on  Next  and  Previous  •  Previous  should  be  below  Next   –  Bu*ons  can  be  closer  (Couper  et  al.,  2011;   Wroblewski,  2008)   7  
    • Background  on  Long  Lists  •  One  column   –  Visually  appear  to  belong  to  one  group   –  When  there  are  two  columns,  2nd  one  may  not  be   seen  (Smyth  et  al.,  1997)  •  Two  columns:  Double  banked   –  No  scrolling   –  See  all  op6ons  at  once   –  Appears  shorter   8  
    • Measuring  “Best”  Design  •  Typical:  In  the  Field   –  Drop-­‐off  rates   –  Keystrokes   –  Survey  comple6on  6mes  •  Our  Study:  In  the  Lab     –  User  sa6sfac6on   –  Eye-­‐tracking  data   –  Usability  metrics   9  
    • Usability  •  The  extent  to  which  a  product  can  be  used  by   specified  users  to  achieve  specified  goals  with   effec6veness,  efficiency,  and  sa6sfac6on.  ISO/ TR  16982:2002  •  For  web-­‐based  surveys,  the  design  must   –  Meet  respondents’  needs   –  Facilitate  easy  comple6on   –  Provide  a  sa6sfying  experience   –  Reduce  respondent  burden   –  Produce  high-­‐quality  data   10  
    • Na6onal  Survey  of  College  Graduates   (NSCG)  •  Collects  educa6on  and  job  informa6on  •  Respondents  have  Bachelor’s  degree  •  Was  available  in  PAPI  and  CATI  •  Usability  study  for  a  web-­‐based  self-­‐ administered  instrument   11  
    • Method    •  Lab-­‐based  usability  study  •  TA  read  introduc6on  and  leU  le*er  on  desk  •  Separate  rooms  •  R  read  le*er  and  logged  in  to  survey  •  Think  Aloud  (Olmsted-­‐Hawala  et  al.,  2010)  •  Eye  Tracking  •  Sa6sfac6on  Ques6onnaire  •  Debriefing   12  
    • Par6cipants  Gender   N   Age   N   Educa.on   N  Male   14   <  30   8   Bachelor’s   21  Female   16   31-­‐45   7   Master’s   6   46-­‐60   10   Ph.D.   3   >  60     5   Mean:  46   13  
    • Eye-­‐Tracking  Apparatus  14  
    • Ques6ons  Eye  Tracking  Can  Answer  •  Do  respondents  look  at  Next  and  Previous?  •  What  do  they  look  at  first?  •  Is  it  distrac6ng  when  Previous  is  located  in  a   par6cular  place  on  the  screen?  •  How  long  does  it  take  respondents  to  see  the   Next  bu*on?  •  Does  presenta6on  of  long  lists  affect  what   users  look  at  on  the  list?   15  
    • Previous  and  Next  Bu*ons  16  
    • One  Column  vs.  Two  Columns  17  
    • 4  Versions   N_P1   N_P2   Next  bu*on  on  leU,     Next  bu*on  on  leU,     1-­‐column  job  code   2-­‐column  job  code   PN1   PN2  Previous  bu*on  on  leU,     Previous  bu*on  on  leU,     1-­‐column  job  code   2-­‐column  job  code   18  
    • Results:  Sa6sfac6on  I   *  p  <  0.0001   19  
    • 8.5   Results:  Sa6sfac6on  II   8.5  Mean  Sa.sfac.on   Mean  Sa.sfac.on   8   8   Ra.ng   7.5   Ra.ng   7.5   7   7   6.5   6.5   6   6   Mean   N_P   PN   Mean   N_P   PN   Overall  reac6on  to  the  survey:     Informa6on  displayed  on  the  screens:     terrible  –  wonderful.  p  <  0.05.   inadequate  –  adequate.  p  =  0.07.     8.5   8.5  Mean  Sa.sfac.on   Mean  Sa.sfac.on   8   8   Ra.ng   7.5   Ra.ng   7.5   7   7   6.5   6.5   6   6   Mean   N_P   PN   Mean   N_P   PN   Arrangement  of  informa6on  on  the  screens:   Forward  naviga6on:     illogical  –  logical.  p  =  0.19.   impossible  –  easy.  p  =  0.13.     20  
    • Eye  Tracking:  Next  /  Previous  21  
    • Eye  Tracking:  Previous  /  Next  22  
    •  Eye  Tracking:  N_P  vs.  PN  •  Par6cipants  looked  at  Previous  and  Next  in  PN   condi6ons  •  Many  par6cipants  looked  at  Previous  in  the   N_P  condi6ons   –  Consistent  with  Couper  et  al.  (2011):  Previous  gets   used  more  when  it  is  on  the  right   23  
    •  Eye  Tracking:  Time  to  First  Fixa6on   8   7.5   7   6.5   Seconds   6      PN   5.5      N_P   5   4.5   4   Next   Previous   Mean  6me  to  first  look  at  the  naviga6on  bu*on   24  
    •  N_P  vs.  PN:  Respondent  Debriefing  •  N_P  version   –  Counterintui6ve   –  Don’t  like  the  “bu*ons  being  flipped.”   –  Next  on  the  leU  is  “really  irrita6ng.”   –  Order  is  “opposite  of  what  most  people  would   design.”  •  PN  version   –  “Pre*y  standard,  like  what  you  typically  see.”   –  The  loca6on  is  “logical.”   25  
    •  1  Column  vs.  2  Column   26  
    • Time  to  First  Fixa6on   25   20   15  Seconds   1  col   *  p  <  0.01   10   2  col   5   0   First  half  of  list   Second  half  of  list   27  
    • Total  Number  of  Fixa6ons   40   35   30  Number  of  Fixa.ons   25   20   1  col   15   2  col   10   5   0   First  half  of  list   Second  half  of  list   28  
    • Time  to  Complete  Item   120   100   80  Seconds   60   1  col   2  col   40   20   0   Mean   Min   Max   29  
    •  1  Col.  vs.  2  Col.:  Debriefing  •  25  had  a  preference   –  6  preferred  one  column   •  They  had  received  the  one-­‐column  version   –  19  preferred  2  columns   •  7  had  received  the  one-­‐column  version   •  Prefer  not  to  scroll   •  Want  to  see  and  compare  everything  at  once   •  It  is  easier  to  “look  through,”  to  scan,  to  read   •  Re  one  column,  “How  long  is  this  list  going  to  be?”   30  
    • Conclusions    •  Par6cipants  were  more  sa6sfied  when   Previous  was  on  the  leU.  •  Par6cipants  preferred  the  long  lists  in  two   columns.  •  Par6cipants  looked  at  the  first  half  of  the  list   sooner  than  the  second  half  when  in  one   column.  •  Par6cipants  looked  at  the  second  half  of  the   list  more  when  it  was  in  two  columns.   31  
    • Bigger  Picture:  Recap  on  Next  and  Previous   •  Next  should  be  on  the  leU   –  Reduces  the  amount  of  6me  to  move  cursor  to  primary   naviga6on  bu*on   –  Tab  order   –  Frequency  of  use   •  Previous  should  be  on  the  leU   –  Web  applica6on  order   –  Everyday  devices   –  Logical  reading  order   –  People  are  more  sa6sfied   –  It  takes  longer  to  first  look  at  Previous  when  on  the  right   32  
    • Bigger  Picture:  Recap  on  Long  Lists  •  One  column   –  Visually  appear  to  belong  to  one  group  •  Two  columns:  Double  banked   –  No  scrolling   –  See  all  op6ons  at  once   –  Appears  shorter   –  Second  column  may  not  be  seen   –  People  look  at  the  second  half  more   –  People  look  at  the  first  half  sooner  when  it  is  in  one   column   –  People  prefer  two  columns   33  
    • Future  Direc6ons    •  This  is  just  a  small  nugget.  •  N_P  vs.  P_N  study  in  progress   –  Same  layout   –  No  skip  pa*erns   –  Efficiency  measure  •  Long  list  of  items  condi6on   –  Which  items  do  people  pick?   –  Alphabe6zed  vs.  random  order   34  
    • Thank  you!  For  more  informa6on,  please  contact   Jennifer  Romano  Bergstrom   Jennifer.C.Romano@gmail.com   Jennifer.Romano@census.gov   Twi*er:  @romanocog   35