Usability Testing Workshop

Usability Boot Camp Brian Sullivan DFW-UPA Workshop December 5, 2009

Introduction

What is Usability Testing?
Usability testing involves:
You Watch Customers
They Perform Tasks
You Note Their Problems
Making Recommendations
Rinse and repeat (i.e. iterate)
Jakob Nielsen’s Definition of Usability Usability is a quality attribute that assesses how easy user interfaces are to use. The word "usability" also refers to methods for improving ease-of-use during the design process.

Cast of Characters: Three Main Roles
Facilitator: This person oversees the entire test process—plan, test, and report.
Participant: Actual or potential customers. Representative users (marketing, designers) should not be used.
Observer: Records events as they occur. Limits interaction with the customer. Does contribute to the report.
In today’s workshop, you will play all three roles!

Benefits of Usability Testing: Feedback from Real Customers
If your website is difficult to use, customers leave .
If they get lost in your website, customers leave .
If a customer can’t FIND your product, they can’t BUY it. …. Then, they leave !
Real feedback directly from customers that use the product, which provides data—not opinions.

Benefits of Usability Testing: Informed Design Decisions Design A Design B
We encountered these results from a Usability Test:
Average Time to Book a Limo with Design A = 3:59
Average Time to Book a Limo with Design B = 8:00
12 of 12 customers successfully booked the correct limo with Design A.
7 of 12 customers successfully booked the correct limo with Design B.
All users were TRAINED and currently used Design B.

Benefits of Usability Testing: Mitigate Risk (Also, Rewarded) For years, companies have been performing usability testing. They just call it a product release. Teams that do usability testing before a release mitigate risk. Plus, they are rewarded . (ex: $300 Million Button)

Benefits of Usability Testing: Saves Time and Money
Randy Bias and Deborah Mayhew’s book provides hundreds of formulas to show the return on investment for usability testing.
Here is one formula:
Annual Savings in Productivity X Increase efficiency (in hours) X Average hourly wage of user X # of days per year X # of uses per day # of users Using the previous example, a switch to Design B saves over $2 million USD.

Benefits of Usability Testing: Makes Lots of Money Jared Spool The $300,000,000 Fix (per Jared Spool): In the above example, we have a form with two fields, two buttons, and one link. This form appeared on the bottom of a Shopping Cart. What are the problems? Hint: Current and New Customers both had problems.

Benefits of Usability Testing: Makes Lots of Money Jared Spool New Customers: New customers did not want to register. They resisted clicking the Register button. New customers wanted to buy what was in their shopping cart. They did not want a relationship. Most wanted to leave . As it turns out, customers are not required to register—it is optional.

Benefits of Usability Testing: Makes Lots of Money Jared Spool Existing Customers: Existing customers couldn’t remember their exact email and/or password. They clicked the “Forgot Password” link. Help Desk logs revealed this link was clicked about 167,000 each day! According to Spool, 75% of these people didn’t make a purchase. They did leave !

Benefits of Usability Testing: Makes Lots of Money Jared Spool Jared Spool’s $300,000,000 Fix: Make it obvious. The results: The number of customers purchasing went up by 45% . The extra purchases resulted in an extra $15,000,000 the first month. For the first year, the site saw an additional $300,000,000 . $ $

Usability Methods and Tools

Two Primary Methods of Data Collection Usability Methods Table

Quantitative and Qualitative Testing

Quantitative Testing Tool (User Zoom Demo)
Quantitative Usability Testing has grew in the last decade.
Many tools exist for quantitative testing tools are out there.
Keynote Systems has been around for a long time now.
You may currently use or know people that use Google Analytics.
Chalkmark is something we will see later today.
Lou Capone will give us a demo of UserZoom (30-minute demo).
Web IQ

Qualitative Testing Tools and Methods
Qualitative usability testing for the web has really matured in the last 30 years.
Quantitative usability testing tools has really matured in the last 10 years.
Morae and OvoStudios both offer mature solutions for usability professionals.
IntuitionHQ is very new to the market—I was just made aware of them this week.
We will use Morae today.

Three Types of Usability Tests (Qualitative Perspective) Today’s workshop will focus on summative usability testing.

Basic Usability Testing Process
Step 1: Planning Your Usability Test
Define what you are testing
Define which customers you are testing
Define what tasks you are testing
Recruit your customers
Write your usability scenarios and tasks
Step 2: Conducting Your Usability Test
Conduct a pilot test
Facilitate remaining tests
Step 3: Analyzing and Reporting Your Usability Test
Compile results
Make recommendations

A Typical Scene Behind The Glass in a Usability Test

Basic Usability Testing Process
Time = Money: Most usability tests are less than 90 minutes. Plus, you are taking your customer’s time. Put first things first—what needs testing!!! (Activities) (People) (Context)

A Simplified Version of Activities-People-Context Picture from Stephen Anderson Somebody (person) does something (activity) some where or some how (context). This model is Robert Bailey’s Human Performance Model. Let’s use this model to quickly plan out a test!

Here is the Royal Caribbean International Website

Describe Activities That Might Be Done…..
Create a group reservation
Plan an event (ex: wedding)
Update profile (ex: Passport)
Review Sailing Route
Picture from Stephen Anderson
Make a second payment
Move to a different ship
Upgrade to a new cabin
Buy an excursion deal

Describe People That Might Be Using the Site…..
Captain of the Ship
Group Traveler
Senior Citizen
College Student
Potential customer
Travel Agent
Special Needs Traveler
Worker (on the Ship)

Describe the Context of Use…..
Company website
IVR system
Kiosk (on the ship)
Printout (ex: Itinerary)
Social Media
Email/text messaging
Voice Mail
TV, Radio, DVR
Inside or outside
Public or private
Noisy or quiet
Legal regulations
Social attitudes
Company goals

It Is Not Too Complicated…..
People = Usability Participants
Context = Test Scenarios
Activities = Usability Tasks

Activities: What Concerns Does Your Product Team Have?
Ask the project team to list out their overall concerns with their product. Let’s go back to the Royal Caribbean International.
Overall Concerns:
Ease of learning
Ease of use
Ease of navigation
Usefulness of Help
Recovering for errors
Specific Concerns:
Booking Group Reservations
Splitting Group Payments
Availability of Group Cabins

People: Who are the Target Customers?
Ask the project team. They tend to focus on a single user.
Frame It This Way:
Who are primary customers? (Define their characteristics.)
Is there another group? (How are they like 1 st group?) (How are thy different?)
Is there another group? (How are they like both groups?) (How are they different?)
Travel Agent Event Planner

Context: Where and How are People Going to Use It?
Ask the team. Usually, the team focuses on the delivery device. They will need help with the other contextual factors.
Frame It This Way:
What device(s) is it on?
Is inside or outside?
Is it public or private?
Is it work or leisure-related?
Is it noisy or quiet?
Any legal regulations?

It Is Not Too Complicated….. Just Ask!!!

With People-Context-Activities Defined, Data Can Be Collected
Performance Data:
Average Task Completion
Average Time on Task
# of clicks to Help
# of omitted steps
# of errors made
# of hints
Subjective Data:
Prefer Design A or B
Design Suggestions
Task Ease Rating
Satisfaction Rating
User Comments
Ranking Features
Performance and subjective data are equally valuable. We will come back to data collection in a few minutes.

Basic Usability Testing Process—Recruting
(Activities) (People) (Context)

Number of Participants Debate

Number of Participants Debate (Sources) Jakob Nielsen (circa 1993) Five users will uncover about 85% of the usability issues of a product. Virzi backs up this claim around the same time. Jared Spool (circa 2000) Five users revealed only 35% of the usability issues of a product. This claim is backed up by Schroeder. Laura Faulkner (2003) - Five users uncovered 85% of usability issues on average. - But, the percent ranged from 55-100% (confidence interval). - Ten users uncovered 95% with a lower bound of 82%.

Number of Participants (Brian Sullivan, 2009)
General Rules:
Test with 6 customers (per group) —if time permits, use 8 customers.
More customers = more reliable results
More customers = fewer new results
For larger samples, do multiple rounds of smaller tests (rather than 1 large UT)
Base Your Number On:
Importance of project
Complexity of the product
Experience of testing team
Expected variety of user reactions

Recruiting Participants
Recruiting:
In-house sources
Recruiting agency
Incentives:
Gift checks ($100 per session)
Food or gift cards
Scheduling:
Slot times
Breaks between participants
Backups, if needed
Floaters, if needed

Selecting the Right Customers
User profiles:
Potential Customer
New Customer
Returning Customer
Primary or tertiary customers
Characteristics:
Job responsibilities (not their title)
Years experience
Expertise level (novice, intermediate, expert)
Specific skills (example: heart surgeons rather MDs or nurses)
Geographic location
Source: David Armano

Your Recruiting is Only As Good As the Screener
Steve Krug has said this:
I respectfully disagree. Because…
Your test is a waste of time , if you recruit the wrong people.
Your usability test is dangerous , if you act upon recommendations from a bad recruiting sample.
You must screen participants.
For recruiting, grade on a curve ! Grab a neighbor.

Basic Usability Testing Process—Write Out the Test
Step1: Planning Your Usability Test

Scenarios and Tasks
Some experts distinguish between tasks and scenarios:
Scenarios:
Provide the participant with motivation and context
Sets the stage or tells a story
Increases understanding
Makes the situation more realistic
Includes tasks
Tasks:
Quick and to the point
Do not need context to increase understanding

Example of Scenario with Tasks
Scenario #1:
You want to book a sailing on Royal Caribbean International for next June with your church group. The group is called “Saint Francis Summer 2010”. The group is selling out fast, so you want to book a cabin, which is close to an elevator because your leg hurts from a recent injury.
Open your browser
Click the link labeled “Royal Caribbean”
Tell me the available cabins in the “Saint Francis Summer 2010” group
Tell me a cabin number closest to an elevator
Book the cabin the best suits your needs

Practice Writing Scenarios and Tasks
We have two teams today. --Team Cool --Team Ninja Assassins
Pair up with someone.
Write a scenario with tasks.
We will pass out papers of some sample sites.
You will use your script later today to run some tests.

Step 2: Conducting Your Test

Basic Usability Process—Step 2

Don’t Rely on Luck!!!

You Do Not Need a Usability Lab to Conduct a Test

Test Side-by-Side in a Conference Room

Test Wherever Your Customers Might Be I have done usability testing at Starbuck’s and a User Conference.

Remote Usability Testing Easier Than Ever

Morae is Portable, Affordable, and ………………Usable

Shhhhh….you don’t need to record, either

Conduct a Pilot Test
Dry run of a complete usability test
Include all test material
Focus on the test scenarios, not findings
At least, one participant should be a pilot
Conduct before test begins (one day prior)
Make changes, if needed
May need to test, again

Again, Adopt a Workable Schedule

Tips for Being a Good Moderator
Know Your UCD Basics and Design Guidelines
Be a Quick Learner
Set an Instant Rapport with Participants
Have an Excellent Memory
Be a Good Listener (Use Active Listening)
Don’t Stress Out About Ambiguity
Be Very Flexible
Have a Long Attention Span
Show Empathy (ie Be a People Person)
Thing “Big Picture”
Be a Good Communicator
Be a Good Organizer and Coordinator

Tips for Getting the Most from Participants
Start with an Easy Task To Build Confidence
Choose the Right Format (remote, on-site UT)
Sit Beside the Person Not Behind the Glass
Use “Think-Out-Loud” Protocol
Give Them Time to Think It Through
Offer Appropriate Encouragement
Lead Participants, Don’t Answer Question (Being an Enabler)
Don’t Act Knowledgeable (Treat Them As the Experts)
Don’t Get Too Involved in Data Collection
Don’t Jump to Conclusions
Don’t Solve Their Problems Immediately (ie act as a Help Desk)

Team Cool Starts With Wizard of Oz Style
Wizard of Oz Style Testing:
Moderator
Tester
Wizard (Person moves screen shots)
@suredoc (Keith Anderson) is your guide today. NOTE: Keith is NOT the WWW – Wicked Witch of the West.

Rules for the Merry Old Land of Oz…..
Here are The Rules:
Get into Groups of three.
Each person picks a role.
Wizard does not talk.
Moderator analyzes Tester
Tester talks through the task.
Complete the Test.
Switch Roles.
Find two new people.
Repeat Steps 1-6.
Compile results via K-J Method (well do that later, though).

Team Ninja Assassins Will Be in Usability Lab
These are the Test Stations:
Two Waiting Participants
One Tester in Test Cell
Moderator
Test Cell Observer
Camera Operator
TV Operator
Remote Viewer Logger
Remote Viewer Manager
Results Compiler (2 people)
Two Observers
Rotate between each station
We’ll test four different sites today.

After 1.5 Hours, We Switch Locations
The Teams Switch:
Team Cool in Labs
Team Ninja Assassins (Wizard of Oz)

Step 3: Analyzing and Reporting

Basic Usability Process—Home Stretch Now!!!
Step 3: Analyzing and Reporting Your Usability Test
Compile results
Make recommendations

Dilbert’s Boss Solves Your Customer Problem

8 Steps to Analyze and Report Your Findings
Complete Your Notes
Summarize Your Findings
Compile and Tabulate Data
Analyze Your Findings
Prioritize Findings by Severity Level
Create Recommendations
Develop a Final Document or Presentation
Follow-up

Step 1—Complete Your Notes
After each session, fill in any gaps.
Basically, complete the short-hand used in the UT.
Mark any areas worth noting.
Use a whiteboard! It is really much easier, in the end.

Step 2—Summarize Findings
After each session, create a list of the major issues.
Organize by Issue and Participant.
Do Not Start Solution Finding Here!
Below is an Example of How to Summarize:

Step 3—Compile and Tabulate Data
Spreadsheet work best here
Do not work on solutions, yet
Try to find the real problem

Step 4—Analyze Findings
Which tasks were simple?
Which tasks were problematic?
Which participants had issues? (Look at it by group.)
Speculate on solutions.
Use Balsamiq to sketch out recommendations.
Sketched in Balsamiq in less than 5 minutes.

Step 5—Prioritize Findings
Severity 1 = Showstopper. Unless corrected all users will essentially have a problem.
Severity 2 = High Impact. Significant impact to usability or performance. High customer annoyance and frustration.
Severity 3 = Medium Impact. Some users experience degraded performance or frustration. Help Desk calls possible.
Severity 4 = Low Impact. Probably will not have a significant impact to user performance. Might be cosmetic in nature.

Step 6—Create Recommendations
Start with Positive Findings
Remind the team of people-activities-context
Be a problem solver
Present Issues by Severity Level (helps to prioritize)
It great to do a MOCKUP here
Tie recommendation back to USER PERSONAS, too

Nobody reads a “BIG, HONKING” report.

Step 7—Create Final Document or Presentation
Consider the audience
Some teams just want an Excel spreadsheet
Provide enough detail (executive summary v. full report)
Create a distribution list for key stakeholders
Purpose (identify issues, prioritize them, make recommendations)

Step 8—Follow-up
Always Provide Usability Next Steps
Usability Needs to Be Part of the Development/Design Cycle
Invite People to Other Studies
Create an Action Plan

Let’s Compile and Analyze Our Results

Conduct K-J Analysis of Wizard of Oz
List all observations on stickies
Post them on a wall
Cluster into groups
Name the group
Vote on each group
If possible, make recommendations

Conduct On Board Analysis of Lab Testing
Write issues on board after each participant (positive and negative)
Place checkmark when the issue is found by another user
Rank issues at the end of the test
If desired, we can think about solutions.

Wizard of Oz Testing on Chalkmark

Lab Testing of Four Sites (Sprouts)

Lab Testing of Four Sites (Consumer Search)

Lab Testing of Four Sites (Broccoli.com)

Lab Testing of Four Sites (Casa Ma ň ana)

Question & Answer

http://theusabilitycorner.blogspot.com	33
http://dfw-upa.org	29
http://bigdesignconference.com	17
http://www.slideshare.net	7
http://www.linkedin.com	3
http://bigdesignevents.com	1
http://www.dfw-upa.org	1
http://theusabilitycorner.blogspot.co.uk	1
http://theusabilitycorner.blogspot.com.ar	1

Usability Testing Workshop

by Brian Sullivan

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 93

Statistics