• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Effects of Age and Think-Aloud Protocol on Eye-Tracking Data and Usability Measures
 

Effects of Age and Think-Aloud Protocol on Eye-Tracking Data and Usability Measures

on

  • 581 views

 

Statistics

Views

Total Views
581
Views on SlideShare
580
Embed Views
1

Actions

Likes
0
Downloads
2
Comments
0

1 Embed 1

http://www.twylah.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Effects of Age and Think-Aloud Protocol on Eye-Tracking Data and Usability Measures Effects of Age and Think-Aloud Protocol on Eye-Tracking Data and Usability Measures Presentation Transcript

    • Effects of Age andThink-Aloud Protocol onEye-Tracking Data and Usability Measures Jennifer Romano Bergstrom @romanocog Fors Marsh Group @forsmarshgroup & Erica Olmsted-Hawala U.S. Census Bureau EyeTrackUX | Las Vegas, NV | June 2012
    • Usability Testing• Technique in user-centered design to evaluate a product by testing it with actual users of the product• Value is getting direct feedback on how real users work with the product
    • How Usability is Measured• International Organization for Standardization (ISO) 9242:11 The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
    • Usability Testing• Metrics – Task accuracy – Task efficiency – User satisfaction• Usability professional (test administrator) identifies problem areas of product based on user performance & verbalized feedback
    • Think-Aloud Protocol• Primary tool used by usability professionals when conducting usability tests• Test administrator — Listens to what participant says — Watches what participant does — Cannot directly observe what a participant thinks 5
    • Concurrent Think Aloud (CTA) Protocol• Participant encouraged to “think aloud” while working on task – Running commentary of thought-action process – Participant articulates as they work on task 6
    • Retrospective Think-Aloud (RTA) Protocol• Participant completes session in silence – Watches a video replay of session • Sometime with eye tracking – Articulates while watching video replay
    • Research on CTA and RTA• Usability studies have compared effects of CTA and RTA on usability performance – Accuracy – Efficiency – Satisfaction – Verbal data quality• Results vary by study
    • Pros and Cons of Think Aloud CTA RTA• Metrics might not be accurate • Metrics “more pure”• Emotions • Reflection• Users describing the what • Users describe their own• Real sense of users’ interpretation or experience with site understanding of why• Shorter sessions • Users’ insight into what the• Real time responses design issues are • Longer sessions • Memory is fallible
    • Cognitive Aging
    • Aging: The Facts• The population is increasing.
    • The Facts 390,000 370,000Population of US in thousands 350,000 330,000 310,000 290,000 270,000 250,000 2000 2005 2009 2010 2015 2020 2025 2030
    • The Facts• The population is increasing.• Older adults are the fastest growing part of the population.
    • The Facts 120,000 100,000Population of US in thousands 80,000 18 to 24 years 60,000 25 to 44 years 45 to 64 years 40,000 65 years and over 20,000 0 2000 2005 2009 2010 2015 2020 2025 2030
    • The FactsPercent Change in US Population and in Older Adults: 2000-2029.
    • The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.
    • The Facts 90 80Percent of households with Internet 70 60 50 40 30 20 10 0 1997 2000 2001 2003 2007 2009
    • The Facts
    • The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.• Cognitive decline is inherent with age.
    • The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.• Cognitive decline is inherent with age.
    • Cognition and Internet Examples• Spatial skills – Where things are located• Short-term memory – Remembering the purpose• Working memory – Remembering information entered earlier to respond to later questions• Sustained attention – Complex Web sites
    • The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.• Cognitive decline is inherent with age.• Web site designers often do not take cognitive decline into account when designing Web sites and surveys.
    • Cognitive Aging, Think-Aloud Protocoland Usability Performance
    • Challenges of Dual tasks• Performance for older adults decreases when they attempt to complete dual tasks
    • Research Questions• Does the dual task of thinking aloud while working on finding information impact usability performance and eye fixation patterns?• What think-aloud protocol (CTA / RTA) should practitioners use when conducting usability studies with older adults?
    • Our Study• 2 (CTA / RTA) x 3 (Age) between-subjects study.• Outcome Variables – Accuracy – Efficiency – Satisfaction – Eye-tracking data – Verbal reports
    • Participants• 95 participants• Metro DC area• Three age groups – Young adults ages 18-28 – Middle-age adults ages 40-50 – Older adults ages 64-76• One year of prior experience using Internet• Reimbursed $40.00
    • Participant Characteristics Young adults Middle-age adults Older adults CTA RTA CTA RTA CTA RTAGender 3M / 8F 6M / 15F 5M / 7F 9M / 10F 5M / 6F 9M / 12FMean Age 22 22 45 46 67 68Mean Years ofeducation 15 15 15 14 16 16Ease of learninga new Web site* 1.36 1.47 1.83 1.42 2.36 2.38Ease ofnavigating the 1.09 1.05 1.33 1.16 1.72 1.57Internet* *Scale: 1 (Not difficult at all) – 5 (Extremely difficult).
    • Methods• 5 information-gathering tasks – Typical tasks users come to site to accomplish – Tasks organized by easy to hard – Collected accuracy and efficiency measures – Eye tracking on all tasks• Satisfaction questionnaire and debriefing at end• CTA group – CTA on all tasks• RTA group – RTA on last task only – Gaze video playback• Test administrator and participant were in separate rooms
    • Website Tested• US Census Bureau’s American FactFinder – Primary website about population, housing, and economy of the US – Legacy version of site
    • AFF Website: http://factfinder.census.gov
    • Example Tasks• First easy task: You want to learn more about Maryland, and specifically about how many people live there. How many people live in Maryland?• First hard task: You are doing a report on schooling in the U.S. What percent of the population in Florida, California and Texas completed college in 2008?
    • Results• Task accuracy• Task efficiency• User satisfaction• Eye Tracking• Verbal reports
    • Young and middle age adults completed more tasks successfully than older adults ** 80% * 70% 60% 50% 40% CTA 30% RTA 20% 10% 0% Young adults Middle-age adults Older adults*p < 0.05; **p < 0.01
    • Middle-age adults in CTA had higher accuracy in hard task than in RTA 80% * 70% 60% 50% 40% CTA RTA 30% 20% 10% 0%*p = 0.07 Middle-age adult--1st hard task
    • Young adults completed tasks faster than older adults ** 4:19 3:50 3:21 2:52 2:24 CTA 1:55 RTA 1:26 0:57 0:28 0:00 Young adults Middle-age adults Older adults**p = 0.05, *p = 0.10
    • Middle-age adults in RTA completed hard task faster than in CTA 10:48 * 9:36 8:24 7:12 6:00 CTA 4:48 RTA 3:36 2:24 1:12 0:00*p = 0.10 Middle age adults--1st hard task
    • No significant age-related differences in user satisfaction98765 CTA4 RTA3210 Young adults Middle-age adults Older adults
    • Speed / Accuracy Trade Off• Cognitive demands influence the speed and accuracy with which adults process information• Our findings for the middle-age adults are consistent with earlier cognitive work. – Higher accuracy for hard task with CTA – Slower for hard task with CTA
    • Interaction of Age and Think Aloud• Older adults’ performance decreases when they perform dual tasks.• Our findings for the middle age adults are consistent with earlier cognitive work. – Older adults: floor level – Young adults: ceiling level
    • Eye Tracking• Total number of fixations per AOIs – Task 1• Total number of fixations on main page – Task 1
    • AOIsTop banner Top navigation Population clockLeft navigation Whole screen
    • Left navigation: No difference between CTA and RTA2520151050 CTA RTA
    • Left navigation: Older adults had fewer fixations than young adults for CTA only 10 * 9 8 7 6 5 CTA 4 RTA 3 2 1 0*p < 0.05 Young adults Middle-age adults Older adults
    • Top banner: More fixations during CTA 4 3 2 CTA RTA 1 0 CTA RTAp = 0.06
    • Top banner: No significantdifferences, but pattern is the same543 CTA2 RTA10 Young adults Middle-age adults Older adults
    • Top navigation: No difference between CTA and RTA21 CTA RTA0 CTA RTA
    • Top navigation: Older adults had fewer fixations than young adults for CTA only 4 * 3 2 CTA RTA 1 0 Young adults Middle-age adults Older adults*p < 0.05
    • Population clock: No difference between CTA and RTA6543 CTA2 RTA10 CTA RTA
    • Population clock: No significantdifferences, but CTA pattern is similar 8 7 6 5 4 CTA 3 RTA 2 1 0 Young adults Middle-age adults Older adults
    • Whole screen: No difference between CTA and RTA60504030 CTA RTA20100 CTA RTA
    • Whole screen: Middle-age adults had fewer fixations than young 80 adults for CTA only 70 * 60 50 40 CTA 30 RTA 20 10 0 Young adults Middle-age adults Older adults*p = 0.05
    • Seconds to complete Task 1: No significant differences16014012010080 CTA60 RTA4020 0 Young adults Middle-age adults Older adults
    • Accuracy for Task 1: No significant differences100% 90% 80% 70% 60% 50% CTA 40% RTA 30% 20% 10% 0% Young adults Middle-age adults Older adults
    • Language Analysis• Present tense and past tense verbalizations• Also examined – Positive emotion words (e.g., good, helpful) – Words of assent (e.g., absolutely, cool) – Negative emotion words (e.g., annoy, awful) – Negations (e.g., aren’t, can’t) – Affective processing (e.g., caring, insult)
    • Participants in CTA spoke more present tense; Participants in RTA spoke more past tense14%12%10% 8% CTA 6% RTA 4% 2% 0% Past tense Present tense
    • Young adults in CTA spoke 5% fewer utterances in present tense16%14%12%10% Young CTA8% Middle-age CTA6% Older CTA4%2%0% Present tense
    • Young adults in RTA follow similar pattern: fewer utterances of present tense12%10% 8% Young RTA 6% Middle-age RTA Older RTA 4% 2% 0% Present tense
    • Words of Insight and Cognition• Test administrator observations during sessions – CTA more immediate what they were doing – RTA reflections on actions• Insight words – Complex – Meaning – Prove• Cognitive processes – Ambiguous – Hesitate – Solution
    • Participants in RTA spoke more words of insight and cognition25%20%15% CTA10% RTA5%0% Insight words Cognitive processes
    • Summary of Usability Results I• Accuracy – Young and middle-age adults completed more tasks successfully (higher accuracy) than older adults.• Efficiency – Young adults completed tasks faster than older adults.• Satisfaction – Age did not affect satisfaction.
    • Summary of Usability Results II• Condition effect – Middle-age adults who completed tasks in silence (RTA) completed the most difficult task faster and with higher accuracy than middle-age adults who thought aloud (CTA) while completing tasks.
    • Summary of Eye-Tracking Results• When age was not a factor, there appeared to be no differences in fixations between CTA and RTA• When age was taken into account – Older had fewer fixations than young in CTA only • Left navigation • Top navigation • Similar pattern for other AOIs – Middle-age had fewer fixations than young in CTA only • Whole screen – Not related to accuracy or efficiency
    • Implications• Recruit participants of varying age groups for usability testing; conduct usability tests with older adults.• Be aware of cognitive demands and effects that thinking aloud has on aging adults.• Consider pros and cons when creating a protocol and selecting a think-aloud method. – Use CTA for a richer verbal commentary, particularly with respect to participants’ emotional reaction to the site. – Use RTA when interested in verbal feedback related to participants’ insight on the issues they had with the application. – Use RTA when pure eye-tracking data is necessary.
    • Future Analysis• Fixation duration• Sum across all tasks – Easy vs. hard tasks• Performance in first 10 seconds• Correlations – Education – Experience – Cognitive abilities• Verbalizations – Phrases – Number of words, phrases• Click data• Identified usability issues
    • Future Research• Include wider ranges of age to test the assertion that the age-related effect is incremental and occurs slowly through a lifetime• Alter RTA protocol• Replicate with different web sites• Are there specific gaze patterns by age and think-aloud protocol?
    • Thank You! Jennifer Romano Bergstrom Fors Marsh Group 571.858.3795 jbergstrom@forsmarshgroup.com @romanocogProceedings paper from STC re verbalizations is available. Eye-tracking paper is forthcoming.