talk at AAPOR 2014

1. Improving Federal Forms with User
Experience Testing and Eye Tracking
Jennifer Romano Bergstrom, Jon Strohl,
Andrew Hale, Sarah Keaton
May 17, 2014
AAPOR | Anaheim, CA
@romanocog #AAPOR

2. 2
@romanocog #AAPOR
+ emotions and
perceptions =
UX
Usability = “the extent
to which a product can
be used by specified
users to achieve
specified goals with
effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
2

3. 3
User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
User Experience
@romanocog #AAPOR
3

4. 4
OBSERVATION
AL
+ Ethnography
+ Time to complete task
+ Reaction time
+ Selection/click behavior
+ Ability to complete tasks
+ Accuracy
IMPLICIT
+ Facial expression analysis
+ Eye tracking
+ Electrodermal activity (EDA)
+ Behavioral analysis
+ Linguistic analysis of
verbalizations
+ Implicit associations
+ Pupil dilation
EXPLICI
T+ Post-task satisfaction
questionnaires
+ In-session difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial
UX Data
4
@romanocog #AAPOR

5. Traditional UX research
is good at explaining
what people say and do,
not what they think and
feel.
Why should we measure implicit?
@romanocog #AAPOR

6. 6
Modern eye tracking
Modern Eye Tracking

7. Copyright©2013TheNielsenCompany.Confidentialandproprietary.
7
Example Methodology
Participants:
• N = 74 | Average Age = 37
• Mix of gender, ethnicity, income
• Random assignment to diary condition
• New, Old, Prototype, Bilingual
Usability Testing session:
• Participants read a description of the
study.
• The moderator gave instructions and
calibrated the eye tracker.
• Participants completed Steps 1-5 in the
diary at their own pace.
• End-of-session satisfaction
questionnaire
• Debriefing interview
Eye Tracker
Moderators worked from another
room.
Control Room
Slide from Walton, Romano Bergstrom, Hawkins & Peirce – AAPOR 2014

8. 8
User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
@romanocog #AAPOR
Plain language
*Public Law 111-274. Retrieved from http://www.plainlanguage.gov/plLaw/
• President Obama
signed the Plain
Writing Act of 2010
requiring federal
agencies to use “clear
Government
communication that
the public can
understand and use.”
– Find what they need
– Understand what
they find
– Use what they find
to meet their needs
8

9. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
Government forms matter!
@romanocog #AAPOR
9
Poor forms lead to:
• Increased errors
• Lower response rate
• Higher nonresponse
• Loss of time
• Increased frustrations
• Poor user experiences.

10. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
Good forms = good UX
@romanocog #AAPOR
10

11. Issue: Instructions are not read, and the starting point is not clear
*Hochheiser, H. & Shneiderman, B. (2000). Performance benefits of simultaneous over sequential menus as task
complexity increases. International Journal of Human-Computer Interaction, 12(2), 173-192.
UX Best Practice: Use step-by-step menus for simple forward-moving tasks.*
“It’s not intuitive at all for where I’d go to
find information.”
Users select their desired job to plan a
career path.
• Keyword search
• Select from a series of dropdowns.
UX Findings:
• Instructions are not read
• Users enter their current job, not
desired job.
• It is not clear that users have the
option to either search or enter job
details when entering a starting job.
Fixation count gazeplot shows the participant looks back and forth
between the two input options
@romanocog #AAPOR
11

12. Issue: The layout does not encourage left-to-right processing
UX Best Practice: Allow users to enter information in ways that are logical to them.
@romanocog #AAPOR
12
Version 1 PrototypeVersion 2
Users enter their household TV viewing for one week.

13. Issue: The location of navigation buttons cause confusion
*Niemela, M. & Saarinen, J. (2000). Visual search for grouped versus ungrouped icons in a computer interface. Human
Factors, 42(4), 630-635.
UX Best Practice: Clearly differentiate navigation elements from one another.*
Users enter the job
title or keyword that
they want to search
for and click the
arrow pointed to the
right to run the
search.
UX Finding:
• The “Save this Search” button is closer to the box where the job title is entered than the
arrow icon that runs the search.
• Some participants incorrectly clicked on “Save and Search” to continue.
After clicking on the Save This Search button, one participant said: “Hmmm, create a new account and
I actually don’t want to create a new account…Maybe I can just scroll down and have a look here.”
@romanocog #AAPOR
13

14. Issue: It is not clear how to get started and enter data on the form
1: You want to compare the average cost of a public university to the average cost of a private university. Using this site,
approximately how much more is a 4-year private university likely to cost?
2: You’re deciding between the University of Virginia Main Campus and Georgetown, and you’re interested in graduation
rates. Which university graduates a higher percentage of students who enroll?
UX Best Practice: Ensure clickable parts of the form appear clickable.
Users must click “Add a School” to get started.
They select a college or university and enter
financial data.
UX Findings:
• The “Add a school” button does not look like an
actionable button. It is not clear that this is how
to start.
• The form did not look like it was clickable.
• Users were unsure how to proceed once they
had successfully brought up two schools side-
by-side.
10
17
2
15
10
4
Task
Accuracy
Public/Private Task1 Graduation Rate Task2
Success
Fail
n/a
@romanocog #AAPOR
14

15. Issue: The text on the home page is dense, and the form is not visible.
UX Best Practice: Ensure that the location of page elements do
not create the illusion that users have reached the bottom of a
page when they have not.1 Facilitate scanning by using clear
headings, short phrases and sentences, and short paragraphs. 2
Users enter information into the form and are shown their
cancer risk.
UX Findings:
• Users do not read the dense text.
• The form to enter information is below the fold, and the
layout creates the illusion that the bottom of the page
has been reached.
“It seems like a lot of information. I would skip past it and use the tool.”
“I see blah, blah, blah. A lot of information and I don’t really care for it.”
“I didn’t see it because I didn’t scroll down. If I saw this on the first
page, I would have been much happier.”
1: Ivory, M.Y., Sinha, R.R., & Hearst, M.A. (2000, June). Preliminary findings on quantitative measures for distinguishing
highly rated information-centric web pages. Proceedings of the 6th Conference on Human Factors and the Web. Retrieved
November 2005, from http://www.tri.sbc.com/hfweb/ivory/paper.html.
2: Morkes, J. & Nielsen, J. (1997). Concise, SCANNABLE, and objective: How to write for the Web. Retrieved November
2005, from http://www.useit.com/papers/webwriting/writing.html.3: Rosenholtz, R., Li, Y., Mansfield, J. & Jin, Z. (2005).
@romanocog #AAPOR
15

16. See Bristol, Romano Bergstrom & Link, AAPOR 2014 for more about the UX across devices.
• Not necessarily the same UX across all devices
• Different issues occur with different modes
• Match user expectations and survey objectives
• Conduct UX testing with real users
Assess the UX across devices.
@romanocog #AAPOR
16

17. Thank you!
• Twitter: @forsmarshgroup,
• LinkedIn: http://www.linkedin.com/company/fors-marsh-group
• Blog: www.forsmarshgroup.com/index.php/blog
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
AAPOR 2014| Anaheim, CA