How to improve your organization’s website
through usability testing
Technology in the Arts Canada 2008 • Robert Barlow-Busch
Photo by srboisvert on Flickr

Photo by E v a on Flickr

Today
Setting the stage
Planning a usability test
Conducting a usability test
Determining what you’ve learned
Goal: Prepare you to test your own website.

Setting the stage

Usability testing in 5 words:
Observe people using your website

Usability testing in 21 words:
Observe people using your website
Identify issues…
By watching representative visitors…
Complete common tasks on your website.
Then fix the issues!

But why?
Make your website more useful to people.
Make your website more easy to use by people.
Make people want to visit — your website or your organization.

From informal, guerilla testing… …to the icing on the cake.
Photo by gorriti on Flickr Photo by chotda on Flickr

Welcome to the family!

Planning a usability test

Whom should you select as participants?
People representative of your actual (or desired) visitors.
Not yourself.
Not your designer(s).
Not people in your organization.

How many participants?
1? 8? 18? 80?
A gazillion?

Finding participants
Where to find them:
• A recruiting firm
• Your personal and professional network
• Current visitors (online or bricks ‘n mortar)
How to entice them:
• Don’t ask for help —offer them a fun opportunity!
• Provide incentives such as $$ or gift certificates

Where to conduct your usability test?

Roles on the testing team
Facilitator (mandatory)
• Interacts with the participant and runs the test.
Scribe (optional)
• Takes comprehensive notes.
Equipment jockey (optional)
• Sets up and monitors technology such as recording equipment.
Observer (optional)
• Quietly watches the test
• Usually a stakeholder from your organization.

Determining what to test
Option 1: Formulate tasks in advance
•
• Ensures you test specific parts of the website
• Appropriate when you know what people want to do on the site
Option 2: Interview participants and create ad-hoc tasks:
•
• Ensures you test the website for realistic activities
• Appropriate when you’d like to know what people want to do on
the site

Writing task instructions
Use the participant’s language.
Don’t inadvertently provide hints in your wording.
• “See if the museum has a guest policy for children’s strollers”
• “Find out if you’re allowed to bring a stroller to the museum”
Tasks can be very specific…
• “Can you visit the gallery this Saturday at 5:30 pm?”
…or rather general
• “Find out if this organization offers any services that interest you.”

What websites shall we test today?
Some ideas: Suggestions or volunteers?
• Technology in the Arts (.org) • Site should be of reasonable
• Royal Ontario Museum size/richness
• Waterloo Children’s Museum • Bonus: knowing business
• CBC Radio: Ideas goals so we can create tasks
• Canada Council for the Arts to test them

Exercise: Planning a usability test
1. Select a website to test.
2. Identify five tasks and write instructions for each.

Conducting a usability test

A common schedule
Orientation: 15 minutes
•
Tasks:
• 1 hour
Debriefing: 15 minutes
•
TOTAL:
• 90 minutes
Photo by Mrs. Maze on Flickr

Orienting your participant
Explain the purpose of your test
•
• “You’ll be using our website today to help us figure out where it needs
improvement.”
• “We’re not testing you. You’re helping us test the product.”
Interview them to:
•
• Confirm they meet your recruiting criteria (expect surprises sometimes)
• Gather information for planning ad-hoc tasks
• Establish rapport

Facilitating the test
Ask participants to “think aloud” while they perform tasks
•
• Can be very helpful if you demonstrate first
Probe or prompt for their thoughts if necessary
•
• Does their body language reveal or suggest anything?
• “What’s going through your mind now…?”
Avoid showing approval or disapproval.
•

Testing with younger audiences
Plan on less hands-on testing time
• More time needed to put them at ease
• Kids can find it hard to focus for long periods
Invite parents to accompany them
• But advise parents not to influence them; seat
them out of sight?
Expect more requests for help
• “What do you think you’d do here?... Any
guesses?... How about one last try…?”
Make it a playful experience!
Photo by Hey Paul on Flickr

Taking notes
DO record observations: what you see and hear
• E.g., Suzie never scrolled down the page
DON’T record inferences: why you think something happened
• E.g., Suzie doesn’t like to scroll
Take note of “sound bites”
• “It’s like trying to drive a car with the dashboard ripped out!”

Debriefing
Probe for qualitative feelings about the website
• What are the two things you most liked about the site? Disliked?
Collect quantitative data if desired. E.g., On a scale of 1–5:
• How successful were you in accomplishing today’s tasks?
• How would you rate the site’s visual design?
• How confident did you feel about where you were in the site at all
times?

Photo collage:
How I feel about your
organization…

Photo collage:
...and how I feel about
your website.

vs.

Exercise: Conducting a usability test
You have 15 minutes to conduct each test.
Tips for each role:
• Facilitator: Explain tasks, prompt to think aloud, deflect questions
• Scribe: Record positives, negatives, and good “sound bites”
• Participant: Do your best to role-play if you’re not the target audience
Rules:
• Do not be the participant if it’s your website
• When we switch, take on a role you haven’t tried yet

Determining what you’ve learned

Identify the highs and lows
Review your notes from each test
•
• Catalog both usability issues and what went well
Look for patterns across all your tests
•
• Try “affinity diagramming”
Prioritize the issues:
•
• Showstopper: Participant couldn’t proceed
• Serious: Caused major confusion or loss of confidence
• Improvement: A minor annoyance

Should you recommend fixes?
Photo by zerega on Flickr

Report options
Do nothing!
List highlights in an email.
Write a document outlining the methodology, observations,
and results.
Create a multimedia extravaganza.

Exercise: Determining what you’ve learned
Form groups of people who were Scribes for each site.
Each of you provide 1 positive finding, 1 issue, and 1 sound bite.
As a group, make one recommendation for a high-impact fix.

Q&A

Thank you!
Robert Barlow-Busch
Director of Product Design
Primal Fusion Inc.
bbb@primalfusion.com
www.primalfusion.com
Blog: www.chopsticker.com