7. 7
@romanocog @AAPOR
Users read what they need to read
He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan
Kaufmann.
8. Usability = “the extent
to which a product can
be used by specified
users to achieve
specified goals with
effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
@romanocog @AAPOR
+ emotions and
perceptions = UX
9. 9 User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
@romanocog @AAPOR
User Experience
18. Traditional UX research is
good at explaining what
people say and do, not
what they think and feel.
@romanocog @AAPOR
Why should we measure implicit?
19. Explicit data
19
*Satisfaction Questionnaire: Please rate how difficult it was to log in on this device. 1= not difficult at all to 5= extremely difficult.
“Love the picture in the middle of it.”
“It looks very clean and very simple.”
“It looks pretty organized, it's a nice design.”
When asked how they would save information,
four of six participants said they would
bookmark the page or take a screenshot of
the information. Only two mentioned that
they would use the site functionality to save
for later use.
83%
9%
9%
Percentage of Difficulty Ratings*
1 & 2
3
>=4
@romanocog @AAPOR
Subjective Data
0
3
6
9
Participant Ratings
Likelihood to Recommend
Not likely at all or
Slightly likely
Moderately Likely
Very Likely
0
3
6
9
Participant Ratings
Likelihood to Use
Not likely at all or
Slightly likely
Moderately Likely
Very Likely
*Satisfaction Questionnaire: How likely would you be to use this site in the future?
How likely would you be to recommend this site to a friend?
20. Observational data
20
Details page about Military
Physician Assistant career, only
accessible through Military section.
Details page about Civilian
Physician Assistant career.
Six participants clicked on Salary &
Wages when looking for states with the
most jobs before looking under Job
Growth.
• Five participants thought they had
completed the task once they found the
Salary & Wages map on the wrong page.
0
5
10
15
20
Time on Page
Seconds
Average time on landing page
@romanocog @AAPOR
Observational Data
First click data
21. 21
0
20
40
60
80
100
120
Page 1 Page 2 Page 3
Time(seconds)
Time spent on each page of the instructions before working on form.
Aggregate fixation count heat map across all participants, Page 1.
Participants looked at ‘Purpose of Form’ section the most often.
Implicit Data @romanocog @AAPOR
Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of
methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,
Washington, DC.
25. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
@romanocog #AAPOR
25
Poor forms lead to:
• Increased errors
• Lower response rate
• Higher nonresponse
• Loss of time
• Increased frustrations
• Poor user experiences.
26. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
Good forms = Good UX
@romanocog #AAPOR
26
28. 28
@romanocog @AAPOR
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience
Design. Morgan Kaufmann.
People read pages
with questions on
them differently
than other pages.
The F-shaped eye-tracking pattern of the block of text at the top of the
page is completely different from the eye-tracking pattern on the question
and answer spaces at the bottom of the page.
29. 29
@romanocog @AAPOR
Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of
methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,
Washington, DC.
Fixation count gaze plot shows the
participant looks back and forth
between the two input options
People
don’t read
instructions.
31. 31
@romanocog @AAPOR
Romano, J. C. & Chen, J. M. (2011). A usability and eye-tracking evaluation of four versions of the online National Survey for
College Graduates (NSCG): Iteration 2. Statistical Research Division (Study Series SSM2011-01). US Census Bureau.
Available online at <http://www.census.gov/srd/papers/pdf/ssm2011-01.pdf>.
Users focus on the
questions.
Participants did not read the text on the log-in
page, but they read the text on the very next page.
33. 33
@romanocog @AAPOR
Error messages must
be useful.
He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan
Kaufmann.
34. 34
@romanocog @AAPOR
Error messages must
be useful.
• “How do I advance to the next screen?”
• “It seems like it's stuck on the screen.”
Gaze Plot: After getting an error message, the
participant had to search all over the screen to
find the missing field.
He, Siu, Strohl, & Chaparro (2014). Mobile. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience Design. Morgan
Kaufmann.
35. 35
@romanocog @AAPOR
Users do not pay
attention to
multiple error
messages.
Romano, J. C. & Chen, J. M. (2011). A usability and eye-tracking evaluation of four versions of the online National Survey for
College Graduates (NSCG): Iteration 2. Statistical Research Division (Study Series SSM2011-01). US Census Bureau.
Available online at <http://www.census.gov/srd/papers/pdf/ssm2011-01.pdf>.
36. 36
@romanocog @AAPOR
Navigation must be intuitive.
Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data Collection
Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014
Intuitive ‘Next’ button location
• “Where and how you click is a bit counter-intuitive. [It’s] not super
obvious which button to click to get to next sections.”
• “I feel like the ‘Next’ should be at the bottom and not the top.”
Non-intuitive ‘Next’ button location
37. 37
@romanocog @AAPOR
It must be clear
how to get
started.
Users must click “Add a School” to start. The “Add a school”
button does not look like an actionable button. It is not clear that
this is how to start. The form did not look like it was clickable.
Users were unsure how to proceed once they had successfully
brought up two schools side-by-side.
Romano Bergstrom, J. C. & Strohl, J. (2014). Improving government websites and surveys with usability testing: A comparison of
methodologies. Proceedings from the Federal Committee on Statistical Methodology (FCSM) Conference, Nov 2013,
Washington, DC.
39. Should I always include it
in my research?
@romanocog @AAPOR
Eye tracking is really cool!
40. Should I always include it
in my research?
@romanocog @AAPOR
Eye tracking is really cool!
Probably not
41. 41
@romanocog @AAPOR
When NOT to track
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.
42. 42
@romanocog @AAPOR
Slot-In answers
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.
43. 43
@romanocog @AAPOR
Gathered answers
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.
44. 44
@romanocog @AAPOR
Created answers
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom &
Schall (Eds). Eye Tracking in User Experience Design. Morgan Kaufmann.
46. 46
Jarrett & Romano Bergstrom (2014). Forms and Surveys. In Romano Bergstrom & Schall (Eds). Eye Tracking in User Experience
Design. Morgan Kaufmann.
@romanocog @AAPOR
47. See Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data
Collection Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014 for more about the UX across
devices.
• Not necessarily the same UX across all devices
• Different issues occur with different modes
• Match user expectations and survey objectives
• Conduct UX testing with real users
• Use eye tracking to understand the UX…cautiously.
@romanocog @AAPOR
47
Assess the UX across devicesAssess the UX across devices.
48. Thank you!
• Twitter: @forsmarshgroup
• LinkedIn: http://www.linkedin.com/company/fors-marsh-group
• Blog: http://www.forsmarshgroup.com/all/
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
AAPOR Webinar | August 13, 2014
