Exploring Eye Tracking for Games User Research: A case study of lessons learned

Exploring Eye Tracking for Games User Research! A case study of lessons learned Ania Rodriguez, M.S. CEO/Founder Key Lime Interactive @KeyLimeInteract Karl Steiner, Ph.D. Manager of Usability THQ, Inc @THQ © 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI

Overview
Introductions
Player Research Goals, Techniques and Challenges
Think Aloud vs. Debrief
Our Game Plan
Steps to Success
Q&A
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010

About Key Lime Interactive
Experienced market research and usability professionals
Founded January 2009
Expertise in both Eye-Tracking in-lab studies as well as remote usability testing techniques
Cross-Industry Expertise
Ability to scorecard against competition
Ability to suggest cross-industry recommendations to improve ease of use
Independent Third Party
Impartial with no-hidden agenda
Ability to work with business team and developers to “get everyone onboard”
© 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI Clients: 7/26/2010

About THQ
THQ Inc. (NASDAQ: THQI) is a leading worldwide developer and publisher of interactive entertainment software. The company develops its products for all popular game systems, personal computers and wireless devices. Headquartered in Los Angeles County, California, THQ sells product through its global network of offices located in North America, Europe and Asia Pacific.

What Makes Software Usable?
Speed?
Efficiency?
Simplicity?
Intuitiveness?

A “Usable” Game Click Here to WIN!!!

Usability ≠ Playability
For games, the user experience is the primary purpose of the product
Games must not only be usable…
But also enjoyable
Part of the appeal of (many) games is overcoming a level of challenge
Games should be difficult…
But not too difficult
Games employ a variety of devices, used by a variety of players, in a variety of physical settings
Despite common stereotypes, gamers represent an increasingly diverse group
Gameplay styles and techniques that are appropriate for one audience in one setting playing one type of game may not be acceptable in other contexts

Player Research Goals & Techniques
Player Research Goals
Describe and understand common players behaviors within a game
Determine where and why players encounters issues or difficulties when playing a game
Assess when and how players enjoy the experience of playing the game
Player Research Techniques
Qualitative Techniques
Lab-based “usability” style tests
Quantitative Techniques
Surveys and logs of player interactions

Using Observation and “Think Aloud” to diagnose player difficulties in a 3D Game
We can observe if player is having difficulties
Fails a task
Takes too long / too many attempts
Gets visibly frustrated
We require additional insights to determine why player is having difficulties
May observe player taking incorrect actions or using incorrect controls
Player may verbally describe incorrect assumptions about their goals or incorrect plans for achieving objectives (Think Aloud or Post-Session Debrief)

Issues With Think Aloud for Player Research in 3D Games
Many gameplay issues are related to player awareness and understanding of in-game cues and objects
However, player observation and think-aloud techniques do not always provide adequate information regarding player awareness and understanding of these elements
Players may notice details but fail to understand or act on the information (making observation difficult)
Players may not mention details they noticed while they play
Players may not remember details if they are queried after play

Our Game Plan
Take a 3D game that has already had been tested with Think Aloud Protocol and rerun the study with Eye-Tracking
Gather additional insights into player gaze and awareness, such as:
What objects in the 3D world are noticed? When are they noticed? Do players comment on or remember these objects?
What objects in the 3D world are missed?
What elements in the HUD are noticed? At what points are they noticed? Do players comment on or remember these HUD elements?
What elements in the HUD are missed?
Do overall gaze patterns change in different areas of the game?

Step#1: Computing Power
You need to network the eye-tracker to a separate PC or gaming console.
Pre-release games may not have been tuned for performance, so high-end hardware is typically required. If testing on a PC, consider:
CPU – A high-end recent CPU (e.g. Intel Core 2 Quad)
Video Card – Dedicated video card (laptops with integrated video or low-performance cards will likely not be sufficient)
Video output – make sure that your hardware supports the necessary game output (e.g. HDMI, VGA, etc.) and is compatible with the eye-tracker hardware
RAM – 4GB (8GB is always better)
Our gaming computer:
Gateway FX6831-01 PT.GAH02.004 Desktop PC - Intel Core i7-860 2.8GHz, 8GB DDR3, 1.5TB HDD, DVDRW, ATI Radeon HD5850, HDMI, Display Port, eSATA, FireWire, Windows 7 Home Premium

Step#2 : Double PC Setup Monitor #1 with SMI Eye Tracker Xbox Controller Monitor #2 with SMI Experiment Center Software Laptop running SMI iView X™ Software The Gaming Machine © 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI 7/26/2010

Step#3: Testing Protocol
In a game study, some players may not volunteer much information using just a Think Aloud.
The game itself may demand their attention at times, and describing the experience can introduce additional cognitive load or interruptions to game play
Our test scenarios included a natural stopping point midway through the study
Our suggestion: Use ‘debrief’ rather than Think Aloud because:
Ability to observe what the participants would naturally do without disrupting how they would play if they were on their own
Adding Think Aloud to this experience adds more cognitive load to the task than normally would be present

Step#4: Clarify Goals and Terminology
Clearly stated test goals and plans are important for any domain
Fleeting game events and unique game terminology makes prior planning and preparation even more important
Our suggestion: review test goals with project sponsors in front of the game
Game events may appear quickly – observers need to know what to look for and at what time
Game terminology can often be arcane and very specific. For example, an observer may need to know the visual differences between:
Space Marines and Chaos Marines
M-16s and AK-47s
Atomic Drops and a Facebreaker DDTs

Step #5: Run Study
8 gamers with third-person shooter experience
7 males, 1 female
1 day of testing; 60 minute sessions
30 minutes between subjects
Session overview:
Scene 1 (10-15 mins)
Debrief (5-10 mins)
Scene 2 (10-15 mins)
Debrief (5-10 mins)
One participant, 1 moderator
2 Observers/Note Takers

Step #5: Analysis
Fixations are generally an inadequate way of measuring subject attention when viewing dynamic media.
This is because a subject, when tracking a moving target (like a soccer ball), is actually performing smooth pursuit. This is similar to fixation, at least cognitively, but it is not detectable via normal fixation detection algorithms.
Instead, we recommend dwell time. This is the summary of time spent within an area whether in fixation, saccade, or smooth pursuit, and is a better way to quantify attention on a moving object.
Glance time is a related and equally useful measure.

Definitions
Dwell Time: Dwell time starts at the moment the AOI is fixated and ends at the moment the last fixation on the AOI ends (equal to the sum of durations from all fixations and saccades that hit the AOI)
Glance Duration = Saccade duration for entering the object + sum of all fixation durations and saccade durations before the eyes begin to leave the AOI (equal to the dwell time + duration of saccade entering AOI)
7/26/2010 © 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI

Step #6: Gridded AOIs [Dwell Time] © 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI After 30 seconds End 60 seconds Example: Gridded Areas of Interests (AOIs) for E3 Promotional Video 7/26/2010

Step #7: AOIs and KPIs © 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI Create AOIs
AOIs can be
Any shape or form
Made invisible (turn on/off)
Enabled/Disabled
7/26/2010

Step #9: Create Report
Create typical usability report with findings
Executive Summary
Methodology
Findings and Recommendations
Detailed Findings and Metrics
Appendix
Present
Include everyone you can (e.g., usability team, senior leadership, developers, project managers, etc.)

Ania Rodriguez, M.S. Key Lime Interactive Founder [email_address] @keylimeinteract Karl Steiner, Ph.D. Manger of Usability Research [email_address] © 2010 Key Lime Interactive + THQ, IUE2010 Ann Arbor, MI

http://keylimeinteractive.com	526
url_unknown	4
http://a0.twimg.com	3
http://www.slideshare.net	1
http://webcache.googleusercontent.com	1
http://www.techgig.com	1

Exploring Eye Tracking for Games User Research: A case study of lessons learned

by keylimeinteractive on Jul 26, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

6 Embeds 536

Statistics

Exploring Eye Tracking for Games User Research: A case study of lessons learned — Presentation Transcript