This document summarizes a research paper on using eye tracking data to improve usability testing moderation techniques. It introduces the triggered think-aloud protocol (TTA), which is a modified concurrent think-aloud where the moderator uses a participant's eye gaze data in real-time to determine when to prompt them to verbalize their thoughts. The benefits and limitations of different think-aloud techniques like concurrent, retrospective, and gaze-cued retrospective are discussed. Factors to consider when using the TTA approach are outlined. Next steps proposed include refining the TTA protocol and conducting experimental research to investigate the impact of various moderating factors.

1. Triggered Think-Aloud Protocol:
Using Eye Tracking to Improve Usability Test Moderation
Beverly Freeman | PayPal | CHI 2011

2. We Think-Aloud
“Thinking aloud may be
the single most valuable
usability engineering method.”

3. Concurrent vs. Retrospective Think-Aloud
+ –
CTA Thoughts are captured
while fresh
Talking may interfere
with doing
(concurrent think-aloud)
RTA Task performance is
uninterrupted
Memory decay,
fabrication
(retrospective think-aloud)
Task performance
Verbalization of task performance

4. Gaze-Cued RTA
+ –
Gaze-Cued RTA ET video provides an
additional memory aid
Participants may freak
out!
Memory decay,
fabrication
Task performance
Verbalization of task performance

5. Triggered Think-Aloud Protocol (TTA)
• Modified form of CTA
• Moderator uses real-time gaze data as an input to
determine when and how to prompt
Gaze-Cued RTA: TTA:
Gaze data  Participant Gaze data  Moderator  Participant

6. Triggered Think-Aloud Protocol (TTA)
Yes No
Instructions to “Share anything that “Continually think aloud”
would help me better
participants understand your
experience”
Moderator Triggered by observed Triggered by periods of
behaviors (e.g. “How silence
prompts would you describe
what you’re seeing?”)

7. What We Want to Avoid
“What are you “Did you
looking at now?” notice…?”
“You want me to keep “Oh right! I should’ve
talking? Um, I’m typing noticed this thing over
in my password now...” here…”
Task performance
Verbalization of task performance

8. TTA: A Preliminary Case Study
• Refrain from probing • Probe away!
– Avoid interfering with discovery – Capture initial interpretation

9. Using TTA: Factors to Consider
1. Research goals
– Formative vs. summative
2. Participant characteristics
– Talkative vs. reticent participants
3. Experience characteristics
– Static vs. dynamic content
– First-time encounter vs. habituated experience
– Diversity of user intents / strategies
4. ?
5. ?
6. ?

10. Next Steps for Us 
• Refinement of TTA protocol
– Initial participant instructions
– Moderator probes and prompts
• Experimental research
– Investigate impact of various factors (think-aloud variation, participant
characteristics, experience characteristics, etc.) on behavior and verbal
comments
Thank you!
Questions?
bfreeman@paypal.com