TRENDS
in ti ng
> Tes
il ity
s ab
U
Kyle Soucy, Founding Principal
follow me: UsableInterface
PhillyCHI
June 24, 2009 www.usableinterface.com

Think Aloud Techniques

Think Aloud Techniques
Concurrent (CTA) Retrospective (RTA)
Vs.

Variation of RTA...
Show the participant their video
after they’re done testing.

Usability “Labs”

Vs.
Source: Kent State University
Image source: facit digital

Ideal “Lab-Less” Set Up

Remote vs. In-person
Testing
Vs.

Remote Testing
Pros & Cons
Pros: Cons:
Participants stay in their native environment. Can’t see the user’s facial expressions.
Accessible to a larger & more diverse pool of Can’t see if the user is using peripheral
participants. devices.
Recruiting is easier. Less no shows & easier
Dependent on an Internet connection.
last minute replacements.
Can only recruit participants with a high-
Opportunity to gather ethnographic data.
speed Internet connection.
Cost & time savings. No travel & lab rental
fees.
Easier for observers to “attend” a session.
Risk of compromising data because the
participant is using a foreign PC is removed.

Remote Testing Paper
Prototypes
(Yes, it is possible!)

Remote Testing Tools
• UserVue
• GoToMeeting
• LiveLook
• Adobe Connect
• Silverback
• Many more...

Remote Testing Tips
• Disable IM programs, Outlook meeting
reminders, or anything else that may be
distracting
• Use a phone recording controller.
• Use as few tools as possible
• Back up, back up, back up!

Automated Testing
(Unmoderated Remote Testing)

What You Can Learn
• Analytic data
• Task completion rates
• Number of clicks/clickstream
• Time on task
• Satisfaction ratings
• In-context feedback

When to Conduct
Automated Testing?
Not a replacement for moderated testing or research.
Best used to compliment your qualitative research.

Automated Testing
Tools
• Keynote WebEffective $$$
• UserZoom $$
• Treejack & Chalkmark $
• Loop11 (private beta)

Pre-defined Tasks
vs.
Interview-based Tasks

Improv Testing Process
1. Interview participant about how they use
the product.
2. Use the answers given in the interview to
create tasks on the fly.
When in doubt just ask, “what was the last thing you did
when you used this product? Can you show me?”

Observer Debriefing

Something different
to try...
1. Take notes by capturing specific quotes or observations
2. Create implications for each observation in the form
of:
- Wish for... (W4)
- How to... (H2)
- What if... (Wif)
3. Have each observer take turns sharing their
observations and implications.

Example
Observation: The user didn’t realize they were not
logged in and couldn’t access documents they wanted.
Implications:
“W4 a way the user can access documents
without logging in.”
“H2 make the log in link more visible.”
“Wif we didn’t force the user to log in?”

Benefits
• Helps to keep the observers on task and
actively involved in the testing.
• Refrains the observers from making
massive generalizations about “all users”.
• Informs the stakeholders that couldn’t
observe the tests and includes them in the
process.

Your turn...
• Capture an observation
• Make sure to write implications in the form
of:
- Wish for... (W4)
- How to... (H2)
- What if... (Wif)

Questions?
www.usableinterface.com