This document provides an overview of usability testing and highlights from its history. It discusses why usability testing is important and how even simple, qualitative testing can identify major usability issues. Examples of usability metrics like effectiveness, efficiency and satisfaction are given. The document then describes how to plan and conduct DIY usability tests with only a few participants through defining goals, tasks, recruiting testing and debriefing. It also discusses testing accessibility, mobile usability, and using tools like prototyping and A/B testing.

1. Usability Testing
and QA
A practical approach
towards creating better user interfaces

2. Who am I?
Shilpa Thanawala
@skthana

3. And you?
Improving usability is about listening, shifting your
perspective, empathy.
Interview your neighbor, then introduce her/him.
Name
Role at your company or agency
Background (eg. designer? developer? etc.)
Experience with usability testing
How you think you’ll use what you learn today
Why this class?

4. Housekeeping
Initial and sign the roster
Evaluations at the end of the day
Cell phones on silent, calls outside
Breaks & lunch

5. Usability Testing
Some thoughts to start us off...

6. Designers know too much about their product to be objective judges;
the features they have come to love and prefer may not be understood
or preferred by future customers.
Don Norman, nngroup.com
Image credit: templatemonster.com

7. The test of the machine is the satisfaction it gives you. There isn't any
other test. If the machine produces tranquility it's right. If it disturbs
you it's wrong until either the machine or your mind is changed.
Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance: An Inquiry Into Values
Image credit: Optimum7.com

8. Supposing is good, but finding out is better.
Mark Twain
Image credit: Mediamatic

9. Usability Testing
Some historical highlights...

10. Early 20th Century
Improvements in industrial efficiency
In WW I, reduced work motions for soldiers to assemble &
take apart weapons in the dark.
http://www.measuringusability.com/blog/usability-history.php

11. 1947
John Karlin - how short phone cords should be

12. “It is not so much that Mr. Karlin trained
midcentury Americans how to use the telephone.
It is, rather, that by studying the psychological
capabilities and limitations of ordinary people, he
trained the telephone...”
Excerpt from New York Times article

13. 1980’s
New methodologies developed and published
Usability becomes a profession
Defined as a function of efficiency, effectiveness, and
satisfaction
first publishedPsychology of Everyday Things
1990’s
Usability matures
Diminishing returns from testing more than 3-5 participants
Books by Jakob Nielsen

14. 2000 - today
Steve Krug’s Don’t Make Me Think
Remote testing
Statistical analysis of usability data
Mobile devices, touch and gestures
http://www.measuringusability.com/blog/usability-history.php

15. Why?

16. Why spend valuable time and money doing
usability tests?
It's the best way to learn how users actually interact with
your website.
Watching > interviewing
You don't really know your users' needs. Users are not all the
same. And they're not your Mom.
All websites have problems, and the serious ones are easy to
find.

17. Why spend valuable time and money doing
usability tests?
You'll learn things you didn't know that you didn't know.
Watching users gets everyone on the same page.
Watching users will make you a better designer / developer /
product manager / content writer / etc.
Usability testing doesn't have to be expensive or time-
consuming.

18. What Testing Can't Tell You

19. Limitations of Usability Testing
Won't fix your site's problems (it's not a user training session)
Results are dependent on the questions you ask (tasks)
Can't model social interactions very well
Isn't always representative of real situations
For more on issues testing the social web see Dana Chisnell's talk

20. When?

21. When should usability testing be
done?
Before you design or build anything
Before you implement a change
Throughout the project
After implementing a fix
Throughout the life of the website
As soon as possible & repeat often!

22. Demo: A short usability test

23. Qualitative vs. Quantitative

24. Qualitative vs. Quantitative
Quantitative Usability Studies
Designed and conducted to produce data for statistical analysis
Each measurement represents a count or an amount
Distances, weights, quantities, etc.
Qualitative Usability Studies
Focused on verbal descriptions of users' experiences
Each measurement is a description or category
Words, sentences, feelings, yes/no
Categories are not quantitative, even if they are assigned
numbers
userfocus.co.uk

25. “...qualitative methods are much better suited for answering
questions about why or how to fix a problem, whereas quantitative
methods do a much better job answering ‘how many’ and ‘how much’
types of questions.”
More on Quantitative methods at
"When to Use Which User Experience Research Methods", nngroup.com
measuringusability.com

26. Usability Metrics

27. Effectiveness, Efficiency, and
Satisfaction
Effectiveness
Accuracy, correctness. Are users able to achieve their goals?
Efficiency
Ease of use, speed. How much effort (time) does it take to
complete a task?
Satisfaction
Users' perception. What does the user think about their
experience?
More at ,usability.gov wikipedia.org/wiki/Usability

28. Some Metrics for Common User Tasks
From Measuring the User Experience – Tullis & Albert

29. Performance-based Metrics
Task Success
Commonly used, pass / fail, or degrees of success, should
have clear end-state.
Task Time
Particularly important for repetitive tasks
Errors
Incorrect actions leading to significant time loss, additional
costs, or task failure
Efficiency
Effort and time needed to complete the task
Learnability
Effort and time needed for the user to learn how to use the
interface

30. Issue Metrics
Usability findings based on users' behavior
Consistent / repeatable, or idiosyncratic?
Remain open-minded to the unexpected

31. Other Metrics
Self-Reporting: Users' perceptions, feelings
Behavioral / Psychological: Eye-tracking, stress,
unprompted verbal expressions
Combinations / Comparisons: Interpretations based on
more than one metric
Card-Sorting: How to organize information in a way that
makes sense to users.
A/B Testing: Comparison of two alternate designs
Accessibility: Usability for those with disabilities (
, )
CCVA of
2010 WCAG

32. Bias
Every study contains some level of bias.
Participant background, knowledge, comfort level
Task definitions
Methodology (testing process, session length, how much
talking)
Artifacts of the prototype or product
Environment (lighting, noise, cameras, distractions)
The Facilitator (experience, skill, style)
Expectations
Measuring the User Experience – Tullis & Albert

33. Number of Test Participants

34. What's the right number of users
to recruit for a study?
3-5? 4-6? 10? 20? 100?
Those who favor a small number feel 80% of the most
important usability problems are detected by the first few
users.
Cases in which more users may be required:
You need to be sure you've captured as many problems as
possible
You have more than one user group (persona)
You have a large number of screens, complex design, wide
variety of tasks
You'd like to account for evaluator bias

35. Why we'll focus on qualitative testing today
Anyone can do it
No knowledge of statistics or data analysis needed; based on
verbal feedback, observation, and empathy
Cheap, fast, and easy
Great for small budgets, busy teams, minimal resources...
and frequent repeats
It gets the job done
More than sufficient to identify a website's most serious
usability problems

36. Break

37. DIY Usability Tests
Based largely on Rocket Surgery Made Easy – Steve Krug

38. Establish your Goals
What do you want to learn about your site?
Can users find products easily?
Is it clear to users what the site is about?
Is the information organized intuitively?
Is this new feature distracting?
Keep an open mind to allow for unexpected results.

39. Define your Tasks
Tasks → Activities
Determine which activities best target what you want to learn.
Create one or more scenarios based on
essential, real-world user goals.
Then, describe the activities in the context of your scenario.
Activities should be realistic
Activities should be actionable
Phrase them so as not to give away the answer
Print each activity on its own piece of paper.
(Avoid labels or numbers.)
"Turn User Goals into Task Scenarios for Usability Testing" — nngroup.com

40. Recruiting Test Participants
What kinds of participants?
Where to find them?
How many?
How to compensate them?

41. What kinds of participants?
How representative of actual users?
Most serious UI problems will be discovered by non-
representative users.
Testing with non-representative users is better than
obsessing over finding representative users.
Key requirements: not part of the team, fluent in the UI's
language

42. Where to find participants?
Where do your users hang out?
Advertise on the site, message boards, social networks
Craigslist

43. How many users?
For most of us doing DIY testing, 3-5 participants is fine.
No-shows or last-minute cancellations are common.
Have 1-2 backup participants lined up.
Phone-screen, scheduling, day-before reminders, directions,
parking, NDA's... use a checklist

44. Compensation
Shows appreciation for participant's time and effort
Encourages participants to be engaged and enthusiastic.
Gift cards, free product or service.

45. Equipment & Setup
Hint: you don't need a lab
Bare minimum (if you're the only one): a computer in a quiet
room, a pen and notepad, and 2 chairs
For a team, add a remote observation room, a good mic, and
screen-sharing (Skype or GHangout). And provide
irresistable snacks.
Can use screen-capture software (but most of the time it
won't be watched)
Can record the participant (but it's not really needed)

46. Make it a Team Effort
Try to get everyone to attend.
Assign a trustworthy team member to manage the
observation room

47. The Session Script
Start with a template & edit to suit your needs
Welcome / introduction
Logistics, release forms
Necessary background information
Test: general questions
Test: scenario activities
Wrapup, follow-up questions
Conclude and provide honorarium

48. Some Test Session Script Templates
Free templates at: , ,infodesign.com Steve Krug's site Book site
guide by Rubin, Chisnell & Spool

49. Debrief
Only for those who attended at least one session
Schedule it on the same day (next day latest), 60-90 min.
Provide food!
Get all observers to state (or stick) UX problems they
observed
Get general agreement on which are most serious & will be
tackled first (2-3 maximum)
Remind everyone when the next round of testing will take
place

50. Run your own usability test
1. Select a website to test
2. Determine goals: What's the question you're trying to answer
in this round of testing?
3. Create a scenario & 3 activities that will give you information
pertaining to your goal

51. Lunch
1 hour

52. Run your own usability test
1. Prepare your script (links to resources )
2. Rehearse and refine
3. Print scenario(s) and activities on 1 sheet (well-spaced so you
can cut them apart)
j.mp/ux-dec-11

53. Run your own usability test
1. Test two participants (~20 min each)
2. Take notes

54. Run your own usability test
1. Debrief within your original group
2. Identify 2-3 usability issues

55. Tools
Testing session checklists
, ,
Usability testing software
, ,
Online / Remote services
, , ,
Usability.gov Krug's checklist checklist on book site by
Rubin, Chisnell & Spool
Silverback (Mac) Morae (Win) Camtasia
Usertesting.com Loop11 usabilityhub.com many more

56. So how do we fix it?

57. Fixing usability issues
The bad news: most issues never get fixed
Make the smallest change possible to "stop the bleeding"
Large changes might introduce new issues
It's not fixed until you verify by ... testing again!

58. Demo: Prototyping a UI change using
Photoshop + InVision
One method (of many) to prototype a change:
Create some static mockups using screenshots and visual
editing software
Model interactions using the InVision app

59. A/B Testing

60. A/B Testing
A simple experiment testing two cases.
Before vs. after, one fix vs. another
Qualitative or quantitative

61. A/B Testing Possibilities
Layout and design
Relative sizes of UI elements
UI controls
Form elements (how many, field labels, helper-text)
Images
Copy length or content
Calls to action
...many more

62. A/B Testing Activity
Prototyping your proposed fixes

63. A Usability Test using Paper
Prototypes
Paper prototype of a kids’ website
Credit: BlueDuckLabs / YouTube.com
View online

64. A/B Testing Activity
1. Select one usability issue from your previous activity
2. Brainstorm at least two ways to fix it
3. Print the page and use either Balsamiq or paper prototyping
to mock up your two solutions.
4. Test each case on a separate participant in the other group
5. Debrief in your group. Which worked better, A or B?
More detail at: SM's Ultimate Guide to A/B Testing

65. Testing Accessibility

66. Why is accessibility important?
“The Web is an increasingly important resource in many aspects of
life: education, employment, government, commerce, health care,
recreation, and more. It is essential that the Web be accessible in order
to provide equal access and equal opportunity
to people with disabilities. ”
From The W3C's Web Accessibility Initiative

67. Is my site accessible?
Unplug your mouse / turn off your trackpad
Why - blind, low-vision, compromised motor-control, no
hands
How to test - start at url, use tab / shift-tab and arrows to see
whether you can access all controls, links, and fields on the
page, in the right order. Popups and dialogs should be
usable.
How to fix - proper HTML heirarchy, tabindex attribute,
javascript plugins

68. Is my site accessible?
Check color contrast
Why - low-vision; 1 out of 12 have color deficiency
How to test - online tools ( , ),
browser plugins ( )
How to fix - shift color palette to improve contrast, increase
font sizes for low contrast areas, avoid colorful background
images
Check my Colors Contrast Ratio
Grayscale Tool

69. Is my site accessible?
Turn off images
Why - images useless to low-vision or blind users, low
bandwidth
How to test - block images in browser, WebAIM tool
( )
How to fix - proper alt attributes, don't use images when text
can be used
WAVE

70. Accessibility Resources and Tools
WebAIM
W3C's Web Accessibility Initiative
Quick accessibility tests & fixes
SitePoint's Easy accessibility checks

71. Mobile / Touch

72. Mobile Usability
Context / environment (noise, lighting, glare)
Many more contexts beyond desktop
Bandwidth
Mobile devices force us to think about bandwidth again
Touch target size
Average fingertip size = 10mm
Adjustments in spacing, button size
Touch is also being used on large screens
Driving / attention
What usability issues contribute to driver distraction?

73. Mobile Usability Testing Resources
NNGroup recommendations
Article on UXMagazine

74. Break
10 minutes

75. Accessibility Testing Activity
1. Click the URL at the top of the browser window
2. Without using the mouse again, try using only the keyboard
to navigate
3. Can you access all the links, menus, and controls?
4. Can you determine where you are, or is it easy to lose your
place?

76. Wrap-up
Usability testing is an essential tool for improving your
website or product
Anyone can do a simple usability study
Test as early as possible, as frequently as possible
Tailor the type of testing you conduct (qualitative /
quantitative, metrics, number of participants) to give you the
information you're after
Use the available tools and resources to help you run your
sessions

77. Wrap-up
Get your whole team involved
Small adjustments are preferable (and faster) than a full
redesign
Remember to test your fixes, too
Stay aware of non-standard users (accessibility) and the
changing technological landscape (new platforms, new uses)

78. Resources
Websites / Blogs: , ,
,
Quantitative Testing: ,
,
nngroup.com usability.gov sensible.com
(Krug) UIE
measuringusability.com Measuring the
User Experience – Tullis & Albert usability.gov
Thank you!
Shilpa Thanawala | @skthana