This document summarizes a presentation on web survey and form usability design and testing. It discusses how eye tracking research has provided insights into how users interact with forms and surveys. Key findings include that users focus on questions rather than instructions, have trouble with error messages and navigation, and may not read all pages in the same way. The document cautions that eye tracking is not always necessary and may not be suitable for tasks like recalling pre-existing answers. It encourages testing designs across devices to understand user experiences.

1. Web Survey and Forms Usability
Design & Testing
Jennifer Romano Bergstrom
August 13, 2014
AAPOR webinar
@romanocog @AAPOR

2. 2
@romanocog @AAPOR
What we design for…and reality
Krug, S. (2000) Don’t Make Me Think. Pearson Education *NEW 3rd Edition (2014)*

3. 3
Placement of Instructions

4. 4
http://www.pewinternet.org/three-technology-revolutions/
http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/
@romanocog @AAPOR
Technology Revolutions
%ofAmericanadults
90% of American adults have a cell phone
58% have a smartphone
42% have an tablet
32% have an e-reader

5. 5
@romanocog @AAPOR
The % of cell phone owners who use their cell phone to…
http://www.pewinternet.org/2013/09/19/cell-phone-activities-2013/

6. 6
@romanocog @AAPOR
Nielsen: The Cross-Platform Report, Quarter 2, 2012-US

7. 7
@romanocog @AAPOR
Users read what they need to read
He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan
Kaufmann.

8. Usability = “the extent
to which a product can
be used by specified
users to achieve
specified goals with
effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
@romanocog @AAPOR
+ emotions and
perceptions = UX

9. 9 User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
@romanocog @AAPOR
User Experience

10. 10
@romanocog @AAPOR
When to test

11. 11
https://www.youtube.com/watch?v=GrV2SZuRPv0
Low-Fi Paper Prototypes
@romanocog @AAPOR

12. 12
https://www.youtube.com/watch?v=GrV2SZuRPv0
Low-Fi Paper Prototypes
@romanocog @AAPOR

13. 13 https://uxmag.com/articles/eight-lessons-in-mobile-usability-testing
@romanocog @AAPOR
Mobile Low-Fi Paper Prototypes

14. 14
SUBJECTIVE
+ Satisfaction & difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial
@romanocog @AAPOR
UX Data

15. 15
OBSERVATIONAL
+ Time on page/task
+ Reaction time
+ Selection/click behavior
+ Success/fail rate
+ Conversion rate
SUBJECTIVE
+ Satisfaction & difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial
@romanocog @AAPOR
UX Data

16. 16
OBSERVATIONAL
+ Time on page/task
+ Reaction time
+ Selection/click behavior
+ Success/fail rate
+ Conversion rate
IMPLICIT
+ Eye tracking
+ Electrodermal activity (EDA)
+ Behavioral analysis
+ Verbalization analysis
+ Implicit associations
+ Pupil dilation
+ Facial expression analysis
SUBJECTIVE
+ Satisfaction & difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial
@romanocog @AAPOR
UX Data

17. @romanocog @AAPOR
Why should we measure implicit?

18. Traditional UX research is
good at explaining what
people say and do, not
what they think and feel.
@romanocog @AAPOR
Why should we measure implicit?

19. Explicit data
19
*Satisfaction Questionnaire: Please rate how difficult it was to log in on this device. 1= not difficult at all to 5= extremely difficult.
“Love the picture in the middle of it.”
“It looks very clean and very simple.”
“It looks pretty organized, it's a nice design.”
When asked how they would save information,
four of six participants said they would
bookmark the page or take a screenshot of
the information. Only two mentioned that
they would use the site functionality to save
for later use.
83%
9%
9%
Percentage of Difficulty Ratings*
1 & 2
3
>=4
@romanocog @AAPOR
Subjective Data
0
3
6
9
Participant Ratings
Likelihood to Recommend
Not likely at all or
Slightly likely
Moderately Likely
Very Likely
0
3
6
9
Participant Ratings
Likelihood to Use
Not likely at all or
Slightly likely
Moderately Likely
Very Likely
*Satisfaction Questionnaire: How likely would you be to use this site in the future?
How likely would you be to recommend this site to a friend?

20. Observational data
20
Details page about Military
Physician Assistant career, only
accessible through Military section.
Details page about Civilian
Physician Assistant career.
Six participants clicked on Salary &
Wages when looking for states with the
most jobs before looking under Job
Growth.
• Five participants thought they had
completed the task once they found the
Salary & Wages map on the wrong page.
0
5
10
15
20
Time on Page
Seconds
Average time on landing page
@romanocog @AAPOR
Observational Data
First click data

21. 21
0
20
40
60
80
100
120
Page 1 Page 2 Page 3
Time(seconds)
Time spent on each page of the instructions before working on form.
Aggregate fixation count heat map across all participants, Page 1.
Participants looked at ‘Purpose of Form’ section the most often.
Implicit Data @romanocog @AAPOR
Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of
methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,
Washington, DC.

22. 2222
Trouble with scrolling.
22
@romanocog @AAPOR
Implicit Data

23. 23
Modern eye tracking
Modern Eye Tracking

24. Copyright©2013TheNielsenCompany.Confidentialandproprietary.
24
Example Methodology
Participants:
• N = 74 | Average Age = 37
• Mix of gender, ethnicity, income
• Random assignment to diary condition
• New, Old, Prototype, Bilingual
Usability Testing session:
• Participants read a description of the
study.
• The moderator gave instructions and
calibrated the eye tracker.
• Participants completed Steps 1-5 in the
diary at their own pace.
• End-of-session satisfaction questionnaire
• Debriefing interview
Eye Tracker
Moderators worked from another room.
Control Room
Slide from: Walton, L., Romano Bergstrom, J., Hawkins, D. & Pierce, C. (2014). User
Experience and Eye-Tracking Study: Paper Diary Design Decisions. Paper presentation at
the American Association for Public Opinion Research (AAPOR) Conference, Anaheim, CA,
May 2014.

25. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
@romanocog #AAPOR
25
Poor forms lead to:
• Increased errors
• Lower response rate
• Higher nonresponse
• Loss of time
• Increased frustrations
• Poor user experiences.

26. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
Good forms = Good UX
@romanocog #AAPOR
26

27. @romanocog @AAPOR
What can we learn from
eye tracking forms and
surveys?

28. 28
@romanocog @AAPOR
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience
Design. Morgan Kaufmann.
People read pages
with questions on
them differently
than other pages.
The F-shaped eye-tracking pattern of the block of text at the top of the
page is completely different from the eye-tracking pattern on the question
and answer spaces at the bottom of the page.

29. 29
@romanocog @AAPOR
Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of
methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,
Washington, DC.
Fixation count gaze plot shows the
participant looks back and forth
between the two input options
People
don’t read
instructions.

30. 30
Users focus on the questions.

31. 31
@romanocog @AAPOR
Romano, J. C. & Chen, J. M. (2011). A usability and eye-tracking evaluation of four versions of the online National Survey for
College Graduates (NSCG): Iteration 2. Statistical Research Division (Study Series SSM2011-01). US Census Bureau.
Available online at <http://www.census.gov/srd/papers/pdf/ssm2011-01.pdf>.
Users focus on the
questions.
Participants did not read the text on the log-in
page, but they read the text on the very next page.

32. 32
@romanocog @AAPOR
Error messages must
be useful.

33. 33
@romanocog @AAPOR
Error messages must
be useful.
He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan
Kaufmann.

34. 34
@romanocog @AAPOR
Error messages must
be useful.
• “How do I advance to the next screen?”
• “It seems like it's stuck on the screen.”
Gaze Plot: After getting an error message, the
participant had to search all over the screen to
find the missing field.
He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan
Kaufmann.

35. 35
@romanocog @AAPOR
Users do not pay
attention to
multiple error
messages.
Romano, J. C. & Chen, J. M. (2011). A usability and eye-tracking evaluation of four versions of the online National Survey for
College Graduates (NSCG): Iteration 2. Statistical Research Division (Study Series SSM2011-01). US Census Bureau.
Available online at <http://www.census.gov/srd/papers/pdf/ssm2011-01.pdf>.

36. 36
@romanocog @AAPOR
Navigation must be intuitive.
Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data Collection
Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014
Intuitive ‘Next’ button location
• “Where and how you click is a bit counter-intuitive. [It’s] not super
obvious which button to click to get to next sections.”
• “I feel like the ‘Next’ should be at the bottom and not the top.”
Non-intuitive ‘Next’ button location

37. 37
@romanocog @AAPOR
It must be clear
how to get
started.
Users must click “Add a School” to start. The “Add a school”
button does not look like an actionable button. It is not clear that
this is how to start. The form did not look like it was clickable.
Users were unsure how to proceed once they had successfully
brought up two schools side-by-side.
Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of
methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,
Washington, DC.

38. @romanocog @AAPOR
Eye tracking is really cool!

39. Should I always include it
in my research?
@romanocog @AAPOR
Eye tracking is really cool!

40. Should I always include it
in my research?
@romanocog @AAPOR
Eye tracking is really cool!
Probably not

41. 41
@romanocog @AAPOR
When NOT to track
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

42. 42
@romanocog @AAPOR
Slot-In answers
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

43. 43
@romanocog @AAPOR
Gathered answers
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

44. 44
@romanocog @AAPOR
Created answers
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

45. 45
@romanocog @AAPOR
Third-party answers
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.

46. 46
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience
Design. Morgan Kaufmann.
@romanocog @AAPOR

47. See Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data
Collection Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014 for more about the UX across
devices.
• Not necessarily the same UX across all devices
• Different issues occur with different modes
• Match user expectations and survey objectives
• Conduct UX testing with real users
• Use eye tracking to understand the UX…cautiously.
@romanocog @AAPOR
47
Assess the UX across devicesAssess the UX across devices.

48. Thank you!
• Twitter: @forsmarshgroup
• LinkedIn: http://www.linkedin.com/company/fors-marsh-group
• Blog: http://www.forsmarshgroup.com/all/
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
AAPOR Webinar | August 13, 2014