Jennifer Romano Bergstrom gave a presentation on integrating usability testing methods in development. She discussed the importance of testing with end users, as things may seem straightforward to developers but not users. Usability testing can uncover issues developers may have overlooked. Bergstrom covered different testing methods like one-on-one sessions, focus groups, surveys, and prototypes of varying fidelity. Both qualitative and quantitative data should be collected through methods like observation, eye tracking, and questionnaires. Testing should occur throughout the development process from early concepts to final designs.

1. Launch w/ Confidence: Integrating Usability
Testing Methods in Development
Jennifer Romano Bergstrom
April 8, 2014
DCWW Code(Her) Workshop| Washington, DC
@romanocog

2. 2
About you
@forsmarshgroup @romanocog
1. What is your name?
2. What do you do?
3. Why are you here?
4. What does usability mean to you?

3. 3
@forsmarshgroup @romanocog
Measuring the UX
• How does it work for the
end user?
• What does the user
expect?
• How does it make the
user feel?
• What is the user’s
story and habits?
• What are the user’s
needs?
“the extent to which a
product can be used
by specified users to
achieve specified
goals with
effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
+ emotions

4. 4
Usability vs. User Experience (UX)?
The 5 Es to Understanding Users (W. Quesenbery): http://www.wqusability.com/articles/getting-started.html
User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
@forsmarshgroup @romanocog
Whitney‟s 5 Es of
Usability
Peter‟s User Experience
Honeycomb

5. 5
Krug, S. Don‟t Make Me Think
@forsmarshgroup @romanocog
What People do on the Web

6. 6
Why is testing important?
@forsmarshgroup @romanocog
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not
to your users.

7. 7
Why is testing important?
@forsmarshgroup @romanocog
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not
to your users.

8. 8
Why is testing important?
@forsmarshgroup @romanocog

9. 9
Why is testing important?
@forsmarshgroup @romanocog
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not
to your users.
• You might have overlooked something big!

10. When to test
10
@forsmarshgroup @romanocog

11. 11
@forsmarshgroup @romanocog
Where to test
• Controlled environment
• All participants have
the same experience
• Record and
communicate from
control room
• Observers watch from
control room and
provide additional
probes (via moderator)
in real time
• Incorporate
physiological measures
(e.g., eye tracking,
EDA)
• No travel costs
LABORATORY REMOTE IN THE FIELD
• Participants tend to be
more comfortable in
their natural
environments
• Recruit hard-to-reach
populations (e.g.,
children, doctors)
• Moderator travels to
various locations
• Bring equipment (e.g.,
eye tracker)
• Natural observations
• Participants in their
natural environments
(e.g., home, work)
• Use video chat
(moderated sessions)
or online programs
(unmoderated)
• Conduct many
sessions quickly
• Recruit participants in
many locations (e.g.,
states, countries)

12. 12
@forsmarshgroup @romanocog
How to test
• In-depth feedback from
each participant
• No group think
• Can allow participants
to take their own route
and explore freely
• No interference
• Remote in participant‟s
environment
• Flexible scheduling
• Qualitative and
Quantitative
ONE-ON-ONE
SESSIONS
FOCUS GROUPS SURVEYS• Representative
• Large sample sizes
• Collect a lot of data
quickly
• No interviewer bias
• No scheduling
sessions
• Quantitative analysis
• Participants may be
more comfortable with
others
• Interview many people
quickly
• Opinions collide
• Peer review
• Qualitative

13. 13
@forsmarshgroup @romanocog
What to measure
OBSERVATIO
NAL+ Ethnography
+ Time to complete task
+ Reaction time
+ Selection/click behavior
+ Ability to complete tasks
+ Accuracy
IMPLICIT
+ Facial expression analysis
+ Eye tracking
+ Electrodermal activity (EDA)
+ Behavioral analysis
+ Linguistic analysis of
verbalizations
+ Implicit associations
+ Pupil dilation
EXPLICI
T+ Post-task satisfaction
questionnaires
+ In-session difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial

14. Eye tracking
14
• Observing and recording eye
movements as a participant interacts
with a product
– Deeper insight into how users perform
tasks
– Objective behavioral data
@forsmarshgroup @romanocog

15. Qualitative heat maps
15
@forsmarshgroup @romanocog

16. Qualitative gaze plots
16
@forsmarshgroup @romanocog

17. Qualitative gaze plots
17
@forsmarshgroup @romanocog

18. Qualitative gaze plots
18
@forsmarshgroup @romanocog

19. Qualitative gaze plots and comments
19
@forsmarshgroup @romanocog
M
“Man, this is a long paragraph.”
“There's a lot of information, it'd be a lot better in list form. Ideally, you want to
get your information quick without reading through all this.”

20. 20
What can be tested?
@forsmarshgroup @romanocog
• Concept
• Existing websites/surveys
• Low-fidelity prototypes
• Paper mockups or mockups on computer
• Basic idea is there
• May lack functionality or graphical look
• High-fidelity prototypes
• As close as possible to final interface in look and feel

21. Focus group to test a concept
21
@forsmarshgroup @romanocog
• video

22. Usability test of a low-fidelity prototype
22
@forsmarshgroup @romanocog
• video

23. Usability test of a high-fidelity prototype
23
@forsmarshgroup @romanocog
• video

24. 24
@forsmarshgroup @romanocog
* Couper, M.P., Baker, R., & Mechling, J. (2011). Placement and Design of Navigation Buttons in Web Surveys. Survey
Practice, 4(1).
• “Where and how you click is a bit counter-intuitive. [It‟s] not
super obvious which button to click to get to next sections.”
• “I feel like the „Next‟ should be at the bottom and not the top.”Intuitive „Next‟ button location
Non-intuitive „Next‟ button location
Usability test across devices
• video

25. Combining qualitative and quantitative data
25
@forsmarshgroup @romanocog
• “I‟m not expecting them to email
or call me. I don‟t expect any
person to notify me.”
• “I‟m not sure when I‟d get an
answer. It‟s not like Yahoo!
Answers where it‟s immediate.”
Participants had different
expectations about what would
happen next. One expected an
immediate response, two said
they would call for
assistance, five expected to hear
back via email, and one
expected to see the answer
posted somewhere on the site.
• video

26. The older participants (ages 58, 64, and 65) tended to struggle more with the website.
Specifically, they seemed to have more difficulties with the tags and with learning how to
navigate to the answer pages (see videos below). We recommend testing a future iteration of
the site with more older participants to determine if the site is usable for this population.
Usability testing with diverse users
26
@forsmarshgroup @romanocog
Both participants click on the tags rather than the question text, and both were confused when the tags did not work as
expected.
• video• video

27. https://www.youtube.com/watch?v=aOyTG9MeKDw
Live usability test during the workshop!
27
@forsmarshgroup @romanocog

28. Thank you!
• Twitter: @forsmarshgroup
• LinkedIn: http://www.linkedin.com/company/fors-marsh-group
• Blog: www.forsmarshgroup.com/index.php/blog
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
DCWW Code(Her) Workshop| Washington, DC