Display	  Ma*ers:	  	  A	  Test	  of	  Visual	  Display	  Op6ons	  in	  a	                 Web-­‐Based	  Survey	     Jenni...
Current	  Survey	  Environment	  •  Increasing	  number	  of	  surveys	  online	  •  Design	  considera6ons	     –  Naviga...
Current	  Survey	  Environment	  •  Increasing	  number	  of	  surveys	  online	  •  Design	  considera6ons	     –  Naviga...
Background	  on	  Next	  and	  Previous	  •  Next	  should	  be	  on	  the	  leU	      –  Reduces	  the	  amount	  of	  6m...
Background	  on	  Next	  and	  Previous	  •  Previous	  should	  be	  on	  the	  leU	      –  Web	  applica6on	  order	   ...
Background	  on	  Next	  and	  Previous	  •  Previous	  should	  be	  on	  the	  leU	      –  Web	  applica6on	  order	   ...
Background	  on	  Next	  and	  Previous	  •  Previous	  should	  be	  below	  Next	     –  Bu*ons	  can	  be	  closer	  (C...
Background	  on	  Long	  Lists	  •  One	  column	     –  Visually	  appear	  to	  belong	  to	  one	  group	     –  When	 ...
Measuring	  “Best”	  Design	  •  Typical:	  In	  the	  Field	       –  Drop-­‐off	  rates	       –  Keystrokes	       –  Su...
Usability	  •  The	  extent	  to	  which	  a	  product	  can	  be	  used	  by	     specified	  users	  to	  achieve	  speci...
Na6onal	  Survey	  of	  College	  Graduates	                       (NSCG)	  •    Collects	  educa6on	  and	  job	  informa...
Method 	  	  •    Lab-­‐based	  usability	  study	  •    TA	  read	  introduc6on	  and	  leU	  le*er	  on	  desk	  •    Se...
Par6cipants	  Gender	     N	      Age	             N	      Educa.on	       N	  Male	       14	     <	  30	         8	     ...
Eye-­‐Tracking	  Apparatus	  14	  
Ques6ons	  Eye	  Tracking	  Can	  Answer	  •  Do	  respondents	  look	  at	  Next	  and	  Previous?	  •  What	  do	  they	...
Previous	  and	  Next	  Bu*ons	  16	  
One	  Column	  vs.	  Two	  Columns	  17	  
4	  Versions	              N_P1	                                    N_P2	    Next	  bu*on	  on	  leU,	  	         Next	  b...
Results:	  Sa6sfac6on	  I	                             *	  p	  <	  0.0001	                                                ...
8.5	                                                 Results:	  Sa6sfac6on	  II	                                          ...
Eye	  Tracking:	  Next	  /	  Previous	  21	  
Eye	  Tracking:	  Previous	  /	  Next	  22	  
 Eye	  Tracking:	  N_P	  vs.	  PN	  •  Par6cipants	  looked	  at	  Previous	  and	  Next	  in	  PN	     condi6ons	  •  Man...
 Eye	  Tracking:	  Time	  to	  First	  Fixa6on	                           8	                        7.5	                  ...
 N_P	  vs.	  PN:	  Respondent	  Debriefing	  •  N_P	  version	      –  Counterintui6ve	      –  Don’t	  like	  the	  “bu*on...
 1	  Column	  vs.	  2	  Column	                                              26	  
Time	  to	  First	  Fixa6on	                25	                20	                15	  Seconds	                           ...
Total	  Number	  of	  Fixa6ons	                                 40	                                 35	                   ...
Time	  to	  Complete	  Item	                120	                100	                  80	  Seconds	                  60	  ...
 1	  Col.	  vs.	  2	  Col.:	  Debriefing	  •  25	  had	  a	  preference	      –  6	  preferred	  one	  column	          •  ...
Conclusions	  	  •  Par6cipants	  were	  more	  sa6sfied	  when	     Previous	  was	  on	  the	  leU.	  •  Par6cipants	  pr...
Bigger	  Picture:	  Recap	  on	  Next	  and	  Previous	    •  Next	  should	  be	  on	  the	  leU	         –  Reduces	  th...
Bigger	  Picture:	  Recap	  on	  Long	  Lists	  •  One	  column	     –  Visually	  appear	  to	  belong	  to	  one	  group...
Future	  Direc6ons	  	  •  This	  is	  just	  a	  small	  nugget.	  •  N_P	  vs.	  P_N	  study	  in	  progress	      –  Sa...
Thank	  you!	  For	  more	  informa6on,	  please	  contact	         Jennifer	  Romano	  Bergstrom	     Jennifer.C.Romano@g...
Upcoming SlideShare
Loading in...5
×

Display Matters: A Test of Visual Display Options in a Web-Based Survey

425
-1

Published on

Surveys are increasingly being conducted online, and it is pertinent to establish clear guidelines for presenting self-administered survey items on a computer screen. Goals should include reducing respondent burden and measurement error. Web survey designers often need to decide how to best present long lists of information and where to place navigation buttons (i.e., next and previous). We evaluated the usability of web-based National Survey of College Graduates (NSCG) questionnaire prototypes. In the first round of testing (n=8), respondents had difficulties proceeding through the survey because the ‘Next’ button was on the left side of the screen and the ‘Previous’ button was on the right. In the second round of testing (n=30), four versions of the survey were tested to assess usability of the ‘Next’ and ‘Previous’ buttons based on their placement on the screen (right or left side) and additionally, to assess the display format of a long list of occupation options (one-column versus two-columns). Dependent measures included participants’ comments from a think aloud protocol, self-reported ratings of satisfaction with the survey, responses to qualitative debriefing questions, time required to complete the job code item, and eye-tracking data focusing on which button (i.e., ‘Next’ or ‘Previous’) and which half of the job code list participants looked at first and more often. Qualitative and quantitative results revealed that participants preferred and performed quicker when the long list of occupation options was displayed in two columns rather than one and when ‘Next’ was displayed to the right of the ‘Previous’ button rather than vice versa. Our findings support usability best practice guidelines that recommend eliminating excessive scrolling on Web sites, and following reading conventions (i.e., looking to the right to move forward, as if turning a page in a book).

Published in: Design, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
425
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Display Matters: A Test of Visual Display Options in a Web-Based Survey

  1. 1. Display  Ma*ers:    A  Test  of  Visual  Display  Op6ons  in  a   Web-­‐Based  Survey   Jennifer  C.  Romano  Bergstrom1,  Jennifer  M.  Chen1,     Timothy  R.  Gilbert2  &  Ma*  Jans1   1  Center  for  Survey  Measurement   2  Demographic  Surveys  Division   U.S.  Census  Bureau   AAPOR  66th  Annual  Conference     May  13,  2011  
  2. 2. Current  Survey  Environment  •  Increasing  number  of  surveys  online  •  Design  considera6ons   –  Naviga6on  methods   –  Presenta6on  of  response  op6ons   2  
  3. 3. Current  Survey  Environment  •  Increasing  number  of  surveys  online  •  Design  considera6ons   –  Naviga6on  methods   –  Presenta6on  of  response  op6ons   3  
  4. 4. Background  on  Next  and  Previous  •  Next  should  be  on  the  leU   –  Reduces  the  amount  of  6me  to  move  cursor  to   primary  naviga6on  bu*on  (Couper,  2008)   –  Frequency  of  use  (Dillman  et  al.,  2009;  Faulkner,   1998;  Koyani  et  al.,  2004;  Wroblewski,  2008)   4  
  5. 5. Background  on  Next  and  Previous  •  Previous  should  be  on  the  leU   –  Web  applica6on  order   –  Everyday  devices   –  Logical  reading  order   5  
  6. 6. Background  on  Next  and  Previous  •  Previous  should  be  on  the  leU   –  Web  applica6on  order   –  Everyday  devices   –  Logical  reading  order   6  
  7. 7. Background  on  Next  and  Previous  •  Previous  should  be  below  Next   –  Bu*ons  can  be  closer  (Couper  et  al.,  2011;   Wroblewski,  2008)   7  
  8. 8. Background  on  Long  Lists  •  One  column   –  Visually  appear  to  belong  to  one  group   –  When  there  are  two  columns,  2nd  one  may  not  be   seen  (Smyth  et  al.,  1997)  •  Two  columns:  Double  banked   –  No  scrolling   –  See  all  op6ons  at  once   –  Appears  shorter   8  
  9. 9. Measuring  “Best”  Design  •  Typical:  In  the  Field   –  Drop-­‐off  rates   –  Keystrokes   –  Survey  comple6on  6mes  •  Our  Study:  In  the  Lab     –  User  sa6sfac6on   –  Eye-­‐tracking  data   –  Usability  metrics   9  
  10. 10. Usability  •  The  extent  to  which  a  product  can  be  used  by   specified  users  to  achieve  specified  goals  with   effec6veness,  efficiency,  and  sa6sfac6on.  ISO/ TR  16982:2002  •  For  web-­‐based  surveys,  the  design  must   –  Meet  respondents’  needs   –  Facilitate  easy  comple6on   –  Provide  a  sa6sfying  experience   –  Reduce  respondent  burden   –  Produce  high-­‐quality  data   10  
  11. 11. Na6onal  Survey  of  College  Graduates   (NSCG)  •  Collects  educa6on  and  job  informa6on  •  Respondents  have  Bachelor’s  degree  •  Was  available  in  PAPI  and  CATI  •  Usability  study  for  a  web-­‐based  self-­‐ administered  instrument   11  
  12. 12. Method    •  Lab-­‐based  usability  study  •  TA  read  introduc6on  and  leU  le*er  on  desk  •  Separate  rooms  •  R  read  le*er  and  logged  in  to  survey  •  Think  Aloud  (Olmsted-­‐Hawala  et  al.,  2010)  •  Eye  Tracking  •  Sa6sfac6on  Ques6onnaire  •  Debriefing   12  
  13. 13. Par6cipants  Gender   N   Age   N   Educa.on   N  Male   14   <  30   8   Bachelor’s   21  Female   16   31-­‐45   7   Master’s   6   46-­‐60   10   Ph.D.   3   >  60     5   Mean:  46   13  
  14. 14. Eye-­‐Tracking  Apparatus  14  
  15. 15. Ques6ons  Eye  Tracking  Can  Answer  •  Do  respondents  look  at  Next  and  Previous?  •  What  do  they  look  at  first?  •  Is  it  distrac6ng  when  Previous  is  located  in  a   par6cular  place  on  the  screen?  •  How  long  does  it  take  respondents  to  see  the   Next  bu*on?  •  Does  presenta6on  of  long  lists  affect  what   users  look  at  on  the  list?   15  
  16. 16. Previous  and  Next  Bu*ons  16  
  17. 17. One  Column  vs.  Two  Columns  17  
  18. 18. 4  Versions   N_P1   N_P2   Next  bu*on  on  leU,     Next  bu*on  on  leU,     1-­‐column  job  code   2-­‐column  job  code   PN1   PN2  Previous  bu*on  on  leU,     Previous  bu*on  on  leU,     1-­‐column  job  code   2-­‐column  job  code   18  
  19. 19. Results:  Sa6sfac6on  I   *  p  <  0.0001   19  
  20. 20. 8.5   Results:  Sa6sfac6on  II   8.5  Mean  Sa.sfac.on   Mean  Sa.sfac.on   8   8   Ra.ng   7.5   Ra.ng   7.5   7   7   6.5   6.5   6   6   Mean   N_P   PN   Mean   N_P   PN   Overall  reac6on  to  the  survey:     Informa6on  displayed  on  the  screens:     terrible  –  wonderful.  p  <  0.05.   inadequate  –  adequate.  p  =  0.07.     8.5   8.5  Mean  Sa.sfac.on   Mean  Sa.sfac.on   8   8   Ra.ng   7.5   Ra.ng   7.5   7   7   6.5   6.5   6   6   Mean   N_P   PN   Mean   N_P   PN   Arrangement  of  informa6on  on  the  screens:   Forward  naviga6on:     illogical  –  logical.  p  =  0.19.   impossible  –  easy.  p  =  0.13.     20  
  21. 21. Eye  Tracking:  Next  /  Previous  21  
  22. 22. Eye  Tracking:  Previous  /  Next  22  
  23. 23.  Eye  Tracking:  N_P  vs.  PN  •  Par6cipants  looked  at  Previous  and  Next  in  PN   condi6ons  •  Many  par6cipants  looked  at  Previous  in  the   N_P  condi6ons   –  Consistent  with  Couper  et  al.  (2011):  Previous  gets   used  more  when  it  is  on  the  right   23  
  24. 24.  Eye  Tracking:  Time  to  First  Fixa6on   8   7.5   7   6.5   Seconds   6      PN   5.5      N_P   5   4.5   4   Next   Previous   Mean  6me  to  first  look  at  the  naviga6on  bu*on   24  
  25. 25.  N_P  vs.  PN:  Respondent  Debriefing  •  N_P  version   –  Counterintui6ve   –  Don’t  like  the  “bu*ons  being  flipped.”   –  Next  on  the  leU  is  “really  irrita6ng.”   –  Order  is  “opposite  of  what  most  people  would   design.”  •  PN  version   –  “Pre*y  standard,  like  what  you  typically  see.”   –  The  loca6on  is  “logical.”   25  
  26. 26.  1  Column  vs.  2  Column   26  
  27. 27. Time  to  First  Fixa6on   25   20   15  Seconds   1  col   *  p  <  0.01   10   2  col   5   0   First  half  of  list   Second  half  of  list   27  
  28. 28. Total  Number  of  Fixa6ons   40   35   30  Number  of  Fixa.ons   25   20   1  col   15   2  col   10   5   0   First  half  of  list   Second  half  of  list   28  
  29. 29. Time  to  Complete  Item   120   100   80  Seconds   60   1  col   2  col   40   20   0   Mean   Min   Max   29  
  30. 30.  1  Col.  vs.  2  Col.:  Debriefing  •  25  had  a  preference   –  6  preferred  one  column   •  They  had  received  the  one-­‐column  version   –  19  preferred  2  columns   •  7  had  received  the  one-­‐column  version   •  Prefer  not  to  scroll   •  Want  to  see  and  compare  everything  at  once   •  It  is  easier  to  “look  through,”  to  scan,  to  read   •  Re  one  column,  “How  long  is  this  list  going  to  be?”   30  
  31. 31. Conclusions    •  Par6cipants  were  more  sa6sfied  when   Previous  was  on  the  leU.  •  Par6cipants  preferred  the  long  lists  in  two   columns.  •  Par6cipants  looked  at  the  first  half  of  the  list   sooner  than  the  second  half  when  in  one   column.  •  Par6cipants  looked  at  the  second  half  of  the   list  more  when  it  was  in  two  columns.   31  
  32. 32. Bigger  Picture:  Recap  on  Next  and  Previous   •  Next  should  be  on  the  leU   –  Reduces  the  amount  of  6me  to  move  cursor  to  primary   naviga6on  bu*on   –  Tab  order   –  Frequency  of  use   •  Previous  should  be  on  the  leU   –  Web  applica6on  order   –  Everyday  devices   –  Logical  reading  order   –  People  are  more  sa6sfied   –  It  takes  longer  to  first  look  at  Previous  when  on  the  right   32  
  33. 33. Bigger  Picture:  Recap  on  Long  Lists  •  One  column   –  Visually  appear  to  belong  to  one  group  •  Two  columns:  Double  banked   –  No  scrolling   –  See  all  op6ons  at  once   –  Appears  shorter   –  Second  column  may  not  be  seen   –  People  look  at  the  second  half  more   –  People  look  at  the  first  half  sooner  when  it  is  in  one   column   –  People  prefer  two  columns   33  
  34. 34. Future  Direc6ons    •  This  is  just  a  small  nugget.  •  N_P  vs.  P_N  study  in  progress   –  Same  layout   –  No  skip  pa*erns   –  Efficiency  measure  •  Long  list  of  items  condi6on   –  Which  items  do  people  pick?   –  Alphabe6zed  vs.  random  order   34  
  35. 35. Thank  you!  For  more  informa6on,  please  contact   Jennifer  Romano  Bergstrom   Jennifer.C.Romano@gmail.com   Jennifer.Romano@census.gov   Twi*er:  @romanocog   35  
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×