„ Keeping it real” while thinking aloud Ensuring credibility of eye-tracking usability studies employing think aloud method. Tobii User meeting, 2007-04-18

Agenda About Symetria Think aloud method Misconceptions about eyetracking Problems with eyetracking in think aloud studies Context of situations as an important analysis factor Concept of checkpoints

About Symetria

Think aloud method

Misconceptions about eyetracking

Problems with eyetracking in think aloud studies

Context of situations as an important analysis factor

Concept of checkpoints

About Symetria Usability E-marketing Media buying Usability evaluation User-Centered Design Eyetracking analysis (layouts, ads) Usability consulting Marketing communication on the Web Web campaigns E-mail marketing Market analysis Media planning Ad production Media buying Monitoring Design & planning Production & promotion Broadcasting

Think aloud as conducted by Symetria Trunk test : What is this site? What is it for? What is your first impression? Tasks : The more users talk, the better User can stop and explain No heavy time pressure Task fulfillment levels as usability measures

Trunk test :

What is this site?

What is it for?

What is your first impression?

Tasks :

The more users talk, the better

User can stop and explain

No heavy time pressure

Task fulfillment levels as usability measures

Eyetracking misconceptions Eyetracking results can be obtained by simply summarizing data for each page Every piece of data is important The more data gathered, the more accurate the results

Eyetracking results can be obtained by simply summarizing data for each page

Every piece of data is important

The more data gathered, the more accurate the results

Eyetracking & think aloud convergence – problems Result analysis errors User specific problems Speech and gaze correlation

Eyetracking & thinking aloud convergence – problems Speech and gaze correlation Users tend to look at the elements they are describing, influencing the results Questions asked influence reactions – this is not always intended

Speech and gaze correlation

Users tend to look at the elements they are describing, influencing the results

Questions asked influence reactions – this is not always intended

Eyetracking & thinking aloud convergence – problems User specific problems Overeager users – concentrate too much on talking Distractible users – forget about the task, Tendency to describe the page element by element

User specific problems

Overeager users – concentrate too much on talking

Distractible users – forget about the task,

Tendency to describe the page element by element

Eyetracking & thinking aloud convergence – problems Result analysis errors Summarizing data without inspecting the context Mixing gaze data for different types of activities User describes User operates page screen actions reactions task 1 task 2 task n

Result analysis errors

Summarizing data without inspecting the context

Mixing gaze data for different types of activities

Eyetracking misconceptions corrected Eyetracking is only a tool enhancing the study Gaze data cannot be summarized without analyzing the context It is not important to track the whole study, but specific moments Users that concentrate on talking make results unreliable

Eyetracking is only a tool enhancing the study

Gaze data cannot be summarized without analyzing the context

It is not important to track the whole study, but specific moments

Users that concentrate on talking make results unreliable

„ Keeping it real” during the study Don’t push users to talk Ask users to describe their reactions , not their actions The recording is enough to understand what happened It is not important for user to know why he/she did something In case of overeager users, suggest concentrating on the tasks Plan ahead what questions to ask on pages which will be analyzed

Don’t push users to talk

Ask users to describe their reactions , not their actions

The recording is enough to understand what happened

It is not important for user to know why he/she did something

In case of overeager users, suggest concentrating on the tasks

Plan ahead what questions to ask on pages which will be analyzed

Planning eyetracking analysis for thinking aloud studies Typical situations can be distinguished for each task Online store – user looks for a 2GB mp3 player : Home page Recommended buyings Product description Product list Another Product description Shipment info Product description Shopping cart Checkout

Typical situations can be distinguished for each task

Online store – user looks for a 2GB mp3 player :

Home page

Recommended buyings

Product description

Product list

Another Product description

Shipment info

Product description

Shopping cart

Checkout

Concept of checkpoints An instance of user interaction with a page/type of pages that happens in a specific context defined before the study Example: „ Product list page” checkpoint Type of checkpoint: task fulfillment Purpose of interaction: user is looking for a mp3 player Condition: User is seeing a product list for the first time Hypothesis: user looks at text labels rather than icons

An instance of user interaction with a page/type of pages that happens in a specific context defined before the study

Example:

„ Product list page” checkpoint

Type of checkpoint: task fulfillment

Purpose of interaction: user is looking for a mp3 player

Condition: User is seeing a product list for the first time

Hypothesis: user looks at text labels rather than icons

Types of checkpoints Free interaction checkpoints: Home page Products page Task fulfillment checkpoints: Registration form Product description Question-response checkpoints: What page is currently displayed? How would you go around contacting the site’s owner?

Free interaction checkpoints:

Home page

Products page

Task fulfillment checkpoints:

Registration form

Product description

Question-response checkpoints:

What page is currently displayed?

How would you go around contacting the site’s owner?

Conclusion Plan situations in which eyetracking data will be comparable between the users Plan questions to ask During the study, look for situations identified before Don’t summarize results automatically

Plan situations in which eyetracking data will be comparable between the users

Plan questions to ask

During the study, look for situations identified before

Don’t summarize results automatically

Thank you for your attention Wojtek Chojnacki [email_address]