Effects of Age andThink-Aloud Protocol onEye-Tracking Data and  Usability Measures   Jennifer Romano Bergstrom @romanocog ...
Usability Testing• Technique in user-centered design to  evaluate a product by testing it with actual  users of the produc...
How Usability is Measured• International Organization for  Standardization (ISO) 9242:11  The extent to which a product ca...
Usability Testing• Metrics  – Task accuracy  – Task efficiency  – User satisfaction• Usability professional (test administ...
Think-Aloud Protocol• Primary tool used by usability  professionals when conducting usability  tests• Test administrator  ...
Concurrent Think Aloud (CTA)            Protocol• Participant encouraged to “think aloud”  while working on task  – Runnin...
Retrospective Think-Aloud (RTA)             Protocol• Participant completes session in silence  – Watches a video replay o...
Research on CTA and RTA• Usability studies have compared effects of  CTA and RTA on usability performance  – Accuracy  – E...
Pros and Cons of Think Aloud                CTA                               RTA•   Metrics might not be accurate   •   M...
Cognitive Aging
Aging: The Facts• The population is increasing.
The Facts                                390,000                                370,000Population of US in thousands      ...
The Facts• The population is increasing.• Older adults are the fastest growing part of  the population.
The Facts                                120,000                                100,000Population of US in thousands      ...
The FactsPercent Change in US Population and in Older Adults: 2000-2029.
The Facts• The population is increasing.• Older adults are the fastest growing part of  the population.• More people are u...
The Facts                                      90                                      80Percent of households with Intern...
The Facts
The Facts• The population is increasing.• Older adults are the fastest growing part of  the population.• More people are u...
The Facts• The population is increasing.• Older adults are the fastest growing part of  the population.• More people are u...
Cognition and Internet Examples• Spatial skills  – Where things are located• Short-term memory  – Remembering the purpose•...
The Facts• The population is increasing.• Older adults are the fastest growing part of  the population.• More people are u...
Cognitive Aging,  Think-Aloud Protocoland Usability Performance
Challenges of Dual tasks• Performance for older adults decreases  when they attempt to complete dual tasks
Research Questions• Does the dual task of thinking aloud while  working on finding information impact  usability performan...
Our Study• 2 (CTA / RTA) x 3 (Age) between-subjects  study.• Outcome Variables  – Accuracy  – Efficiency  – Satisfaction  ...
Participants• 95 participants• Metro DC area• Three age groups  – Young adults ages 18-28  – Middle-age adults ages 40-50 ...
Participant Characteristics                         Young adults            Middle-age adults            Older adults     ...
Methods• 5 information-gathering tasks   –   Typical tasks users come to site to accomplish   –   Tasks organized by easy ...
Website Tested• US Census Bureau’s American FactFinder  – Primary website about population, housing,    and economy of the...
AFF Website: http://factfinder.census.gov
Example Tasks• First easy task: You want to learn more  about Maryland, and specifically about  how many people live there...
Results•   Task accuracy•   Task efficiency•   User satisfaction•   Eye Tracking•   Verbal reports
Young and middle age adults     completed more tasks successfully             than older adults                     **    ...
Middle-age adults in CTA had higher      accuracy in hard task than in RTA    80%                      *    70%    60%    ...
Young adults completed tasks               faster than older adults                                          **         4:...
Middle-age adults in RTA completed        hard task faster than in CTA    10:48                    *     9:36     8:24    ...
No significant age-related    differences in user satisfaction98765                                                       ...
Speed / Accuracy Trade Off• Cognitive demands influence the speed  and accuracy with which adults process  information• Ou...
Interaction of Age and Think Aloud• Older adults’ performance decreases  when they perform dual tasks.• Our findings for t...
Eye Tracking• Total number of fixations per AOIs  – Task 1• Total number of fixations on main page  – Task 1
AOIsTop banner                          Top navigation                         Population clockLeft navigation            ...
Left navigation: No difference        between CTA and RTA2520151050        CTA          RTA
Left navigation: Older adults had       fewer fixations than young adults                  for CTA only       10          ...
Top banner: More fixations during                    CTA 4 3 2                            CTA                             ...
Top banner: No significantdifferences, but pattern is the same543                                                      CTA...
Top navigation: No difference       between CTA and RTA21                       CTA                        RTA0     CTA   ...
Top navigation: Older adults had          fewer fixations than young adults                     for CTA only      4       ...
Population clock: No difference       between CTA and RTA6543                        CTA2                        RTA10    ...
Population clock: No significantdifferences, but CTA pattern is similar 8 7 6 5 4                                         ...
Whole screen: No difference       between CTA and RTA60504030                      CTA                        RTA20100    ...
Whole screen: Middle-age adults         had fewer fixations than young      80               adults for CTA only      70  ...
Seconds to complete Task 1: No    significant differences16014012010080                                                   ...
Accuracy for Task 1: No significant           differences100% 90% 80% 70% 60% 50%                                         ...
Language Analysis• Present tense and past tense  verbalizations• Also examined  – Positive emotion words (e.g., good, help...
Participants in CTA spoke more present tense; Participants in RTA spoke more past tense14%12%10% 8%                       ...
Young adults in CTA spoke 5%  fewer utterances in present tense16%14%12%10%                             Young CTA8%       ...
Young adults in RTA follow similar pattern:   fewer utterances of present tense12%10% 8%                                  ...
Words of Insight and Cognition• Test administrator observations during sessions  – CTA more immediate what they were doing...
Participants in RTA spoke more   words of insight and cognition25%20%15%                                             CTA10...
Summary of Usability Results I• Accuracy – Young and middle-age adults  completed more tasks successfully (higher  accurac...
Summary of Usability Results II• Condition effect – Middle-age adults who  completed tasks in silence (RTA)  completed the...
Summary of Eye-Tracking Results• When age was not a factor, there appeared  to be no differences in fixations between CTA ...
Implications• Recruit participants of varying age groups for usability  testing; conduct usability tests with older adults...
Future Analysis• Fixation duration• Sum across all tasks   – Easy vs. hard tasks• Performance in first 10 seconds• Correla...
Future Research• Include wider ranges of age to test the  assertion that the age-related effect is  incremental and occurs...
Thank You!            Jennifer Romano Bergstrom                Fors Marsh Group                   571.858.3795            ...
Upcoming SlideShare
Loading in …5
×

Effects of Age and Think-Aloud Protocol on Eye-Tracking Data and Usability Measures

735
-1

Published on

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
735
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Effects of Age and Think-Aloud Protocol on Eye-Tracking Data and Usability Measures

  1. 1. Effects of Age andThink-Aloud Protocol onEye-Tracking Data and Usability Measures Jennifer Romano Bergstrom @romanocog Fors Marsh Group @forsmarshgroup & Erica Olmsted-Hawala U.S. Census Bureau EyeTrackUX | Las Vegas, NV | June 2012
  2. 2. Usability Testing• Technique in user-centered design to evaluate a product by testing it with actual users of the product• Value is getting direct feedback on how real users work with the product
  3. 3. How Usability is Measured• International Organization for Standardization (ISO) 9242:11 The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
  4. 4. Usability Testing• Metrics – Task accuracy – Task efficiency – User satisfaction• Usability professional (test administrator) identifies problem areas of product based on user performance & verbalized feedback
  5. 5. Think-Aloud Protocol• Primary tool used by usability professionals when conducting usability tests• Test administrator — Listens to what participant says — Watches what participant does — Cannot directly observe what a participant thinks 5
  6. 6. Concurrent Think Aloud (CTA) Protocol• Participant encouraged to “think aloud” while working on task – Running commentary of thought-action process – Participant articulates as they work on task 6
  7. 7. Retrospective Think-Aloud (RTA) Protocol• Participant completes session in silence – Watches a video replay of session • Sometime with eye tracking – Articulates while watching video replay
  8. 8. Research on CTA and RTA• Usability studies have compared effects of CTA and RTA on usability performance – Accuracy – Efficiency – Satisfaction – Verbal data quality• Results vary by study
  9. 9. Pros and Cons of Think Aloud CTA RTA• Metrics might not be accurate • Metrics “more pure”• Emotions • Reflection• Users describing the what • Users describe their own• Real sense of users’ interpretation or experience with site understanding of why• Shorter sessions • Users’ insight into what the• Real time responses design issues are • Longer sessions • Memory is fallible
  10. 10. Cognitive Aging
  11. 11. Aging: The Facts• The population is increasing.
  12. 12. The Facts 390,000 370,000Population of US in thousands 350,000 330,000 310,000 290,000 270,000 250,000 2000 2005 2009 2010 2015 2020 2025 2030
  13. 13. The Facts• The population is increasing.• Older adults are the fastest growing part of the population.
  14. 14. The Facts 120,000 100,000Population of US in thousands 80,000 18 to 24 years 60,000 25 to 44 years 45 to 64 years 40,000 65 years and over 20,000 0 2000 2005 2009 2010 2015 2020 2025 2030
  15. 15. The FactsPercent Change in US Population and in Older Adults: 2000-2029.
  16. 16. The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.
  17. 17. The Facts 90 80Percent of households with Internet 70 60 50 40 30 20 10 0 1997 2000 2001 2003 2007 2009
  18. 18. The Facts
  19. 19. The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.• Cognitive decline is inherent with age.
  20. 20. The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.• Cognitive decline is inherent with age.
  21. 21. Cognition and Internet Examples• Spatial skills – Where things are located• Short-term memory – Remembering the purpose• Working memory – Remembering information entered earlier to respond to later questions• Sustained attention – Complex Web sites
  22. 22. The Facts• The population is increasing.• Older adults are the fastest growing part of the population.• More people are using the Internet.• Cognitive decline is inherent with age.• Web site designers often do not take cognitive decline into account when designing Web sites and surveys.
  23. 23. Cognitive Aging, Think-Aloud Protocoland Usability Performance
  24. 24. Challenges of Dual tasks• Performance for older adults decreases when they attempt to complete dual tasks
  25. 25. Research Questions• Does the dual task of thinking aloud while working on finding information impact usability performance and eye fixation patterns?• What think-aloud protocol (CTA / RTA) should practitioners use when conducting usability studies with older adults?
  26. 26. Our Study• 2 (CTA / RTA) x 3 (Age) between-subjects study.• Outcome Variables – Accuracy – Efficiency – Satisfaction – Eye-tracking data – Verbal reports
  27. 27. Participants• 95 participants• Metro DC area• Three age groups – Young adults ages 18-28 – Middle-age adults ages 40-50 – Older adults ages 64-76• One year of prior experience using Internet• Reimbursed $40.00
  28. 28. Participant Characteristics Young adults Middle-age adults Older adults CTA RTA CTA RTA CTA RTAGender 3M / 8F 6M / 15F 5M / 7F 9M / 10F 5M / 6F 9M / 12FMean Age 22 22 45 46 67 68Mean Years ofeducation 15 15 15 14 16 16Ease of learninga new Web site* 1.36 1.47 1.83 1.42 2.36 2.38Ease ofnavigating the 1.09 1.05 1.33 1.16 1.72 1.57Internet* *Scale: 1 (Not difficult at all) – 5 (Extremely difficult).
  29. 29. Methods• 5 information-gathering tasks – Typical tasks users come to site to accomplish – Tasks organized by easy to hard – Collected accuracy and efficiency measures – Eye tracking on all tasks• Satisfaction questionnaire and debriefing at end• CTA group – CTA on all tasks• RTA group – RTA on last task only – Gaze video playback• Test administrator and participant were in separate rooms
  30. 30. Website Tested• US Census Bureau’s American FactFinder – Primary website about population, housing, and economy of the US – Legacy version of site
  31. 31. AFF Website: http://factfinder.census.gov
  32. 32. Example Tasks• First easy task: You want to learn more about Maryland, and specifically about how many people live there. How many people live in Maryland?• First hard task: You are doing a report on schooling in the U.S. What percent of the population in Florida, California and Texas completed college in 2008?
  33. 33. Results• Task accuracy• Task efficiency• User satisfaction• Eye Tracking• Verbal reports
  34. 34. Young and middle age adults completed more tasks successfully than older adults ** 80% * 70% 60% 50% 40% CTA 30% RTA 20% 10% 0% Young adults Middle-age adults Older adults*p < 0.05; **p < 0.01
  35. 35. Middle-age adults in CTA had higher accuracy in hard task than in RTA 80% * 70% 60% 50% 40% CTA RTA 30% 20% 10% 0%*p = 0.07 Middle-age adult--1st hard task
  36. 36. Young adults completed tasks faster than older adults ** 4:19 3:50 3:21 2:52 2:24 CTA 1:55 RTA 1:26 0:57 0:28 0:00 Young adults Middle-age adults Older adults**p = 0.05, *p = 0.10
  37. 37. Middle-age adults in RTA completed hard task faster than in CTA 10:48 * 9:36 8:24 7:12 6:00 CTA 4:48 RTA 3:36 2:24 1:12 0:00*p = 0.10 Middle age adults--1st hard task
  38. 38. No significant age-related differences in user satisfaction98765 CTA4 RTA3210 Young adults Middle-age adults Older adults
  39. 39. Speed / Accuracy Trade Off• Cognitive demands influence the speed and accuracy with which adults process information• Our findings for the middle-age adults are consistent with earlier cognitive work. – Higher accuracy for hard task with CTA – Slower for hard task with CTA
  40. 40. Interaction of Age and Think Aloud• Older adults’ performance decreases when they perform dual tasks.• Our findings for the middle age adults are consistent with earlier cognitive work. – Older adults: floor level – Young adults: ceiling level
  41. 41. Eye Tracking• Total number of fixations per AOIs – Task 1• Total number of fixations on main page – Task 1
  42. 42. AOIsTop banner Top navigation Population clockLeft navigation Whole screen
  43. 43. Left navigation: No difference between CTA and RTA2520151050 CTA RTA
  44. 44. Left navigation: Older adults had fewer fixations than young adults for CTA only 10 * 9 8 7 6 5 CTA 4 RTA 3 2 1 0*p < 0.05 Young adults Middle-age adults Older adults
  45. 45. Top banner: More fixations during CTA 4 3 2 CTA RTA 1 0 CTA RTAp = 0.06
  46. 46. Top banner: No significantdifferences, but pattern is the same543 CTA2 RTA10 Young adults Middle-age adults Older adults
  47. 47. Top navigation: No difference between CTA and RTA21 CTA RTA0 CTA RTA
  48. 48. Top navigation: Older adults had fewer fixations than young adults for CTA only 4 * 3 2 CTA RTA 1 0 Young adults Middle-age adults Older adults*p < 0.05
  49. 49. Population clock: No difference between CTA and RTA6543 CTA2 RTA10 CTA RTA
  50. 50. Population clock: No significantdifferences, but CTA pattern is similar 8 7 6 5 4 CTA 3 RTA 2 1 0 Young adults Middle-age adults Older adults
  51. 51. Whole screen: No difference between CTA and RTA60504030 CTA RTA20100 CTA RTA
  52. 52. Whole screen: Middle-age adults had fewer fixations than young 80 adults for CTA only 70 * 60 50 40 CTA 30 RTA 20 10 0 Young adults Middle-age adults Older adults*p = 0.05
  53. 53. Seconds to complete Task 1: No significant differences16014012010080 CTA60 RTA4020 0 Young adults Middle-age adults Older adults
  54. 54. Accuracy for Task 1: No significant differences100% 90% 80% 70% 60% 50% CTA 40% RTA 30% 20% 10% 0% Young adults Middle-age adults Older adults
  55. 55. Language Analysis• Present tense and past tense verbalizations• Also examined – Positive emotion words (e.g., good, helpful) – Words of assent (e.g., absolutely, cool) – Negative emotion words (e.g., annoy, awful) – Negations (e.g., aren’t, can’t) – Affective processing (e.g., caring, insult)
  56. 56. Participants in CTA spoke more present tense; Participants in RTA spoke more past tense14%12%10% 8% CTA 6% RTA 4% 2% 0% Past tense Present tense
  57. 57. Young adults in CTA spoke 5% fewer utterances in present tense16%14%12%10% Young CTA8% Middle-age CTA6% Older CTA4%2%0% Present tense
  58. 58. Young adults in RTA follow similar pattern: fewer utterances of present tense12%10% 8% Young RTA 6% Middle-age RTA Older RTA 4% 2% 0% Present tense
  59. 59. Words of Insight and Cognition• Test administrator observations during sessions – CTA more immediate what they were doing – RTA reflections on actions• Insight words – Complex – Meaning – Prove• Cognitive processes – Ambiguous – Hesitate – Solution
  60. 60. Participants in RTA spoke more words of insight and cognition25%20%15% CTA10% RTA5%0% Insight words Cognitive processes
  61. 61. Summary of Usability Results I• Accuracy – Young and middle-age adults completed more tasks successfully (higher accuracy) than older adults.• Efficiency – Young adults completed tasks faster than older adults.• Satisfaction – Age did not affect satisfaction.
  62. 62. Summary of Usability Results II• Condition effect – Middle-age adults who completed tasks in silence (RTA) completed the most difficult task faster and with higher accuracy than middle-age adults who thought aloud (CTA) while completing tasks.
  63. 63. Summary of Eye-Tracking Results• When age was not a factor, there appeared to be no differences in fixations between CTA and RTA• When age was taken into account – Older had fewer fixations than young in CTA only • Left navigation • Top navigation • Similar pattern for other AOIs – Middle-age had fewer fixations than young in CTA only • Whole screen – Not related to accuracy or efficiency
  64. 64. Implications• Recruit participants of varying age groups for usability testing; conduct usability tests with older adults.• Be aware of cognitive demands and effects that thinking aloud has on aging adults.• Consider pros and cons when creating a protocol and selecting a think-aloud method. – Use CTA for a richer verbal commentary, particularly with respect to participants’ emotional reaction to the site. – Use RTA when interested in verbal feedback related to participants’ insight on the issues they had with the application. – Use RTA when pure eye-tracking data is necessary.
  65. 65. Future Analysis• Fixation duration• Sum across all tasks – Easy vs. hard tasks• Performance in first 10 seconds• Correlations – Education – Experience – Cognitive abilities• Verbalizations – Phrases – Number of words, phrases• Click data• Identified usability issues
  66. 66. Future Research• Include wider ranges of age to test the assertion that the age-related effect is incremental and occurs slowly through a lifetime• Alter RTA protocol• Replicate with different web sites• Are there specific gaze patterns by age and think-aloud protocol?
  67. 67. Thank You! Jennifer Romano Bergstrom Fors Marsh Group 571.858.3795 jbergstrom@forsmarshgroup.com @romanocogProceedings paper from STC re verbalizations is available. Eye-tracking paper is forthcoming.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×