Your SlideShare is downloading. ×
Measuring Attention with Mouse Movements
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Measuring Attention with Mouse Movements


Published on

Recently, market research institute MetrixLab has developed a computer-based tool for …

Recently, market research institute MetrixLab has developed a computer-based tool for
tracking visual attention. This tool, the FocusTracker™, is based on the assumption that
mouse movements provide a reliable indication of when and where attention is allocated
on a computer screen. Because it is internet based, the tool can be used with hundreds of
participants from any location in the world, allowing maximal freedom in targeting
specific groups.

Published in: Technology, Health & Medicine

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Action as a Window to Perception:Measuring Attention with Mouse Movements A Validation Study of the MetrixLab FocusTracker Prof. dr. A. Johnson and dr. ir. L.J.M. Mulder
  • 2. Action as a Window to Perception: Measuring Attention with Mouse MovementsIntroductionAttention has been described as the interface between memory and events in the world.We have to attend to information if we are to encode it, and retrieval of past experiencesdepends on attention to appropriate cues in the environment (Logan & Compton, 1998).In visual processing, such as when shopping for a product on the internet or scanningsupermarket shelves, attention is needed to locate relevant information and to guideaction. Although it is possible to move the focus of attention at least a few degrees ofvisual angle away from the focus of the eyes (von Helmholtz, 1894), we almost alwaysattend where we look (Johnson & Proctor, 2004). Therefore, if we want to know whethersomeone has attended to information, we will want to know if they have looked at it. Eyemovement tracking is one means of measuring attention to scenes (Duchowski, 2002). Inpractice, however, the inconvenience and cost of collecting and analyzing eye-movementdata limit the effectiveness of the technique for evaluating visual information displays.Recently, market research institute MetrixLab has developed a computer-based tool fortracking visual attention. This tool, the FocusTracker™, is based on the assumption thatmouse movements provide a reliable indication of when and where attention is allocatedon a computer screen. Because it is internet based, the tool can be used with hundreds ofparticipants from any location in the world, allowing maximal freedom in targetingspecific groups. It has the advantage that participants stay in a natural setting and that nolaboratory or specialized equipment is necessary. Participants are first trained to point themouse at high speeds, moving over images or texts on the computer screen. After thisshort training period, the displays of interest are presented with the instruction to theparticipant to “point to whatever catches your eye.” “Scan path” data from theFocusTracker can then be replayed using MetrixLab’s online FT Replay™ program todetermine how attention is allocated to objects in the scene.The FocusTracker is based on the assumption that there is a one-to-one relationshipbetween where we fix our gaze, where we point via a mouse, and what we are attendingto. The question addressed in this research is thus whether the hand can be trained tofollow visual spatial attention and whether attentional processing can be measured bytracking pointing movements with a handheld computer mouse. A related question iswhether viewers can be adequately instructed to perform the task from their own homes,without the direct intervention of a researcher. In brief, the results of the research are verypromising, showing high correlations between the scan paths for the eye and the mouseas well as high correlations between the percentage of time spent in designated regions ofinterest for the mouse and the eye. An additional comparison of the data with that of agroup who did not use a mouse while viewing the experimental stimuli showed thatviewing patterns were not disrupted by mouse use. © 1 2006
  • 3. The Experiment: Tracking the Focus of AttentionIn order to determine whether the hand can effectively follow the eye while viewingvisual information, an experiment was conducted in which 21 advertisements wereviewed for 5 seconds each. Participants were instructed to view the advertisements and toattempt to move the mouse in the same way as the eyes. Eye movements were registeredwith an eye tracker and hand movements were registered by logging the position of ahandheld computer mouse. Two conditions were compared: one in which the participantsreceived a short, verbal instruction and demonstration of how the mouse should bemoved, and one in which participants followed the FocusTracker instruction program.MethodParticipants. Each group included 15 participants. The FocusTracker instruction groupincluded 5 men and 10 women, 11 of which were university students. The mean age inthis group was 25 years old (sd = 7.15). The verbal instruction group included 3 men and12 women; 11 of which were university students. The mean age in this group was 25years old (sd = 9.58). All but one participant had completed either a VWO or HBOprogram.Stimuli. The stimuli were relatively unknown advertisements1 taken from relativelyexpensive magazines. Advertisements were 25.5 cm high and 16-20.5 cm wide and werepresented on a 17-inch computer screen. Each advertisement was made up of four regionsof interest (ROIs): A headline, an illustration, text, and a trademark (see Figure 1). Theposition of each ROI varied across the advertisements and in some cases they overlapped.Apparatus. Eye movements were recorded with an Applied Science Laboratories model504 eye tracker equipped with a pan/tilt camera.2 Eye position was determined 50 timesper s. A Logitech infrared mouse was used to record hand position. Mouse position wasalso sampled 50 times per s. Participants were tested individually in a dimly lit room.Procedure. Participants were randomly assigned to either the verbal instruction orFocusTracker training groups. Participants in both groups were told that they shouldmove the mouse to follow their eye movements. Participants in the verbal instructiongroup were also given a demonstration of how the mouse should be moved along with theeyes. Participants in the FocusTracker training group performed the tasks in theFocusTracker training: following a moving butterfly with the mouse, moving the mouseto each of a series of sequentially presented objects of the same type, moving the mouseto each of a series of sequentially presented objects of different types and moving the1 The familiarity of the advertisements was tested in a pilot study in which 15 people (aged 17-40) wereasked whether they had ever seen each of the advertisements. Only advertisements that were recognized byno more than 3 of the 15 participants in the pilot study were used as stimuli in the experiment.2 Eye position is determined by comparing the pupil and the corneal reflection of infrared light emittedfrom the camera. © 2 2006
  • 4. (b) (a) (c) (d) Figure 1. A sample advertisement showing the four regions of interest (ROIs): the headline (a), illustration (b), text (c) and trademark (d). ROIs were defined for analysis by enclosing them in rectangles. In some cases (see ROI d), two rectangles were used to define the region. In case of overlap, the smaller ROI was subtracted from the larger ROI.mouse along with the eyes while viewing each of two advertisements. The training wasfollowed independently by each participant, without the intervention of the researcher,and lasted approximately 1.5 – 2 min.The session began with the calibration of the eye tracker. The training was then given andwas immediately followed by the experimental trials. Trials were separated with thepresentation of the mouse cursor centered in a light gray screen. This screen was shownfor 6 s before the first trial and for 2 s between subsequent trials. Participants wereinstructed to look at the mouse cursor until the advertisement appeared and were told thatit would be impossible to move the mouse during these 2 s. The 21 advertisements werepresented for 5 s each in the same order for all participants.After participants had viewed all of the advertisements, a surprise memory test wasgiven. In Part 1 of the memory test, participants were shown either the headline,illustration or trademark from one of the advertisements and were asked to recall theother two attributes of the advertisement (e.g., if the trademark was shown, participantsshould report the illustration and the headline; recall of the text was not tested). Each cuewas used seven times. Part 2 of the memory test was a recognition test in which 42 © 3 2006
  • 5. Eye scan path Mouse scan pathFigure 2. Eye and mouse scan paths from a single participant.trademarks were presented, 21 from the advertisements used in the experiment and 21from related products. The participant’s task was to classify each trademark as havingbeen presented or not. Finally, participants were asked how familiar they were with eachof the advertisements.At the end of the experimental session personal data (e.g., age, level of education,familiarity with the computer) was collected and participants were debriefed. The entireexperiment lasted approximately 1 hour.Data analysis. Not all trials could be included in the analysis because of missing orunreliable eye movement data. If more than 0.5 s data was not usable the trial was notanalyzed. Missing data resulted from the camera losing the eye position, extreme eyeposition readings as a result of the camera misreading the reflection point and as a resultof correcting for eye blinks. Approximately 33 trials per group (9.6% of the data) wereexcluded from analysis. Additionally, occasional extreme values (outliers) were removedand replaced by the average of the two values before and after the outliers. The sameprocedure was applied to brief eye blinks.The relation between eye and hand movements was tested by (1) comparing the scanpaths for the eye and hand using bidimensional regression techniques and (2) bycomparing the percentages of time spent by the eye and hand, respectively in each of thefour ROIs. Because mouse movements lagged behind eye movements, it was necessary tocompensate for the lag on each trial. This was done by determining the best fit(bidimensional r) between the mouse and eye data. On average, 0.63 s of the mouse dataat the beginning of the trial was discarded, as was a corresponding amount of the eyemovement data at the end of the trial. The FocusTracker training group took on average0.58 s to move the mouse whereas the verbal instruction group needed 0.69 s (F(1, 25) = © 4 2006
  • 6. Table 1. Average Values from the Bimendimensional Regression as a Function of Group (standard error in parentheses) Group FocusTracker training Verbal Instruction Value Correlation (r) 0.87 (0.04) 0.89 (0.04) Rotation (θ) 0.14 (0.06) 0.09 (0.05) Expansion (φ) 0.80 (0.02) 0.79 (0.02) Left-right translation (α1) 90.2 (13.3) 87.0 (12.0) Up-down translation (α2) 79.8 (9.2) 70.9 (6.1)4.37, p = .047). Both groups were slower to move the mouse on the first advertisement(m = 1.17 s) as compared to all other advertisements (m = 0.61 s; (F(1, 25) = 22.63, p <.001).ResultsThe relation of mouse and eye scan paths. Sample eye and mouse scan paths are shownin Figure 2. The degree to which the mouse followed the position of the eyes wasassessed with bidimensional regression. The overall correlation between the scan paths(r), and the rotation (θ), expansion (φ), and translation (to the right or left or up or down;α) of the mouse scan path relative to that of the eye were determined individually foreach participant and each advertisement (see Table 1). Correlations were Fishertransformed for analysis.Large correlations (ranging from .83 - .92 across advertisements) were found between themouse and eye scan paths. These correlations did not significantly differ as a function ofgroup. Differences between the advertisements were also minimal.3 Analysis of therotation parameter, θ, revealed that the mouse scan path showed a slight (m = 11.5°)rotation to the right relative to the eye scan path. The expansion parameter, φ, with anaverage value of 0.8, reflected that the mouse scan path covered a somewhat smaller areathan that covered by the eye. Finally, the mouse scan path was shifted, on average, about4 cm above and to the right of the eye scan path.3 One advertisement significantly differed from two others; no other differences between advertisementswere found. © 5 2006
  • 7. Table 2 Percent Time Spent in Each Region of Interest by Eye and Mouse as a Function of Group (standard error in parentheses) Group FocusTracker Training Verbal Instruction Region of Interest % time eye % time mouse % time eye % time mouse Illustration 41.2 (3.12) 47.1 (3.51) 46.3 (2.32) 50.1 (2.16) Headline 30.1 (1.83) 28.0 (2.15) 27.7 (1.61) 26.9 (1.75) Text 14.8 (1.89) 14.8 (2.10) 11.6 (1.76) 10.9 (1.69) Trademark 11.3 (1.27) 9.5 (1.27) 11.0 (1.22) 10.9 (1.44)Percentage time in each region of interest. The overall percent time spent in each of theROIs by the eye and mouse, respectively, is shown in Table 2. Two analyses were carriedout: A correlational analysis in which the correlation between the time spent in eachregion by the eye and the mouse was computed across all advertisements for eachparticipant, and an ANOVA with ROI (illustration, headline, text or trademark), effector(eye or mouse) and group as factors. Correlations were Fisher transformed for analysis.Overall, the correlation between time spent in each of the ROIs for the eye and mousewas high (r = .88). No significant difference in the correlation as a function of group wasfound (r = .88 and r = .87 for the FocusTracker and verbal instruction groups,respectively). Additional analysis carried out per advertisement showed averagecorrelation coefficients ranging from .68 to .97. Figure 3 shows the time spent in eachROI by the mouse as a function of the time spent in each ROI by the eye.An ANOVA with ROI (illustration, headline, text or trademark) and effector (eye ormouse) as within subject factors and group (FocusTracker training or verbal instruction)as a between subject factor showed a main effect of ROI (F(3, 84) = 102.81, p < .001).Follow-up tests showed that significantly more time was spent on the illustration than theheadline, and on these two ROIs than on the text or trademark. Furthermore, there was asignificant Effector x ROI interaction (F(3, 84) = 14.14, p < .001). As can be seen inTable 2, the tendency to spend the most time on the illustration was more pronounced forthe mouse than for the eye. No differences between groups were found.To investigate in more detail the differences in percentage time spent in the ROIs, thenumber of cases in which the eye visited a ROI not visited by the mouse, and vice versa,was computed. These percentages are shown in Table 3. The eye was more likely to visitan area not visited by the mouse than vice versa (F(1, 28) = 31.98, p < .001). This wasespecially the case for the trademark. © 6 2006
  • 8. Region of Interest 100 100,00 aandachtsgebieden headline Headline percentage tijd muis per aandachtsgebied Percent time mouse in ROI tekst Text afbeelding Illustration 80 80,00 merk Trademark 60 60,00 40 40,00 20 20,00 0 0,00 0,00 20,00 40,00 60,00 80,00 100,00 0 20 40 60 80 100 percentage tijd oog per aandachtsgebied Percent time eye in ROIFigure 3. Percent time spent in each region of interest (ROI) by the mouse as a function of percenttime spent in each region of interest by the eyeTable 3Average Percentage of Trials in Which Either the Eye Visited a Region of Interest Not Visited by theMouse or the Mouse Visited a Region of Interest Not Visited by the Eye Group FocusTracker Training Verbal InstructionRegion of Interest Eye only Mouse only Eye only Mouse onlyIllustration 1.47 (0.65) 0.0 (0.0) 0.73 (0.50) 0.33 (0.33)Headline 6.27 (1.78) 0.0 (0.0) 7.73 (1.73) 1.07 (0.57)Text 9.47 (2.58) 5.53 (1.66) 10.40 (2.30) 8.27 (1.30)Trademark 13.00 (2.01) 1.73 (1.03) 7.53 (1.62) 0.33 (0.33) Effects of Mouse Use on Viewing Behavior The results of the experiment comparing mouse and eye scan paths suggest that “mouse tracking” can be an excellent substitute for eye tracking. High correlations were found between the forms of the scan paths for mouse and eye and for the amount of time spent in each ROI by the mouse and eye. Before using mouse tracking to evaluate observer behavior, however, it is important to know whether the use of the mouse leads to a different way of looking. That is, it is important to know that mouse use does not result in viewing behavior that is different than looking under normal conditions. In order to © 7 2006
  • 9. Table 4Number and Duration of Fixations as a Function of Group and Region of Interest (standard error inparentheses) Group FocusTracker Training Verbal Instruction Eye Tracker OnlyRegion of Number of Duration of Number of Duration of Number of Duration ofInterest Fixations Fixations Fixations Fixations Fixations FixationsIllustration 8.5 (0.57) 204 (9.50) 9.7 (0.65) 203 (11.15) 8.3 (0.43) 196 (11.33)Headline 6.6 (0.38) 191 (9.25) 6.0 (0.40) 189 (3.41) 6.3 (0.43) 171 (8.27)Text 3.7 (0.49) 168 (5.92) 2.9 (0.43) 181 (7.68) 3.7 (0.41) 163 (8.39)Trademark 2.6 (0.13) 168 (6.14) 2.3 (0.25) 184 (6.94) 2.8 (0.19) 157 (5.59)Total 21.3 (0.41) 188 (8.15) 20.8 (0.57) 190 (8.17) 21.1 (0.41) 176 (7.90) examine this issue, a group of observers comparable to the experimental groups was tested under the experimental conditions but without use of the mouse. The number and length of fixations in each ROI was then computed. Table 4 shows the average number and duration of fixations within each ROI as a function of whether or not observers used the mouse while viewing the advertisements. Separate ANOVAs with ROI (illustration, headline, text or trademark) as a within- subjects factor and group (FocusTracker training, verbal instruction or eye tracking only) as a between-subjects factor were conducted on the percentage of time spent in each ROI, the number of fixations in each ROI, and the average duration of the fixations. No differences between groups were found, nor did group interact with ROI. In short, effects of using the mouse on looking behavior are minimal. Effects of Mouse Use on Recognition and Recall If the mouse is used to track attention, and attention is subsequently tested with recognition and recall questions, it is important to know whether mouse use has an effect on memory for the viewed material. In order to test this we measured recall of the illustration, headline and trademark, and recognition of trademarks, for each of the three groups. For the recall test, one element of the advertisement was shown (e.g., the headline), and observers were asked to recall the other ROIs, excluding the text. Table 5 shows recall performance as a function of group, ROI and cue (e.g., either the headline or the trademark could be shown as a cue for the illustration). An ANOVA with ROI (illustration, headline or trademark) as a within-subject factor and group (FocusTracker training, verbal instruction or eye tracking only) as a between-subject factor showed significant effects of ROI (F(2, 88) = 87.44, p < .001) and group (F(2, 44) = 6.75, p = © 8 2006
  • 10. Table 5 Percent Correct Recall as a Function of Group, Region of Interest and Cue (standard error in parentheses) Group Region of Interest FocusTracker training Verbal Instruction Eye Tracker Only Illustration Headline cue 43% (4.83) 23% (5.88) 47% (3.83) Trademark cue 41% (3.91) 38% (5.34) 61% (4.98) Headline Illustration cue 14% (3.94) 9% (3.36) 42% (6.87) Trademark cue 14% (4.18) 14% (4.62) 29% (5.05) Trademark Illustration cue 21% (3.91) 24% (5.34) 36% (5.63) Headline cue 18% (4.06) 17% (4.67) 18% (3.19).003), as well as a significant ROI x Group interaction (F(4, 88) = 6.17, p < .001). Bothgroups who used the mouse remembered fewer elements of the advertisements than thegroup who performed the task without the mouse. This effect was significant for theROIs “illustration” and “headline”, but not for the ROI “trademark”.For the recognition test, participants were presented with a list of the 21 trademarks seenin the advertisement, combined with a list of 21 similar trademarks. On average 33% ofthe presented trademarks were recognized. Of the similar trademarks, 13% wereincorrectly classified as having been seen. Recognition performance did not differbetween groups.SummaryHigh correlations between eye and mouse scan paths and between percentage of timespent in each ROI by the eye and the mouse indicate that the mouse is a viable alternativeto the eye tracker for measuring attention under natural viewing conditions. More than75% of the variability in eye movements is captured by the mouse. Moreover, the lack ofdifferences between the FocusTracker and verbal instruction groups suggest thatinstructions can be given remotely without any decrement to the accuracy of thetechnique.Use of the mouse had little influence on the way in which observers viewed theadvertisements. Thus, the results would seem to be generalizable to other viewing © 9 2006
  • 11. situations. Use of the mouse did influence how much could be remembered of what wasseen. This suggests that using the mouse does make demands on mental resources.RecommendationsSeveral aspects of the data should be taken into account when using the mouse to trackattention. First, both the FocusTracker training and verbal instruction groups showed alower correlation between eye and mouse scan paths on the first advertisement than onsubsequent advertisements. This suggests that at least one “practice trial” should be usedbefore the stimuli of interest are shown. Second, consideration should be taken of the factthat the scan path of the mouse covers a smaller area than that of the eye, and is shiftedsomewhat to the right and to the top of the display. The smaller area is due to the fact thatthe eye sometimes made movements that were not followed with the mouse. The shift tothe right and to the top of the display may be a result of using the right hand to move themouse. Another factor that may have played a role in this shift is that in many of theadvertisements, the trademark was at the bottom of the advertisement. In at least somecases, the eye moved to the trademark while the mouse lagged behind, perhaps becausethe cursor would get in the way of reading the text or because the trial ended before themouse movement was completed.Specifically, the following recommendations are made: • Present a “practice” stimulus after training and before beginning the evaluation of the stimuli of interest • Define regions of interest to take into account the tendency of the mouse to be shifted to the right and top of the display • Do not analyze the first 0.63 s of the trial data, or measure the time taken to move the mouse • Do not rely on memory for viewed advertisements as an indication of where observers allocated their attention.Mouse tracking is a viable alternative to eye tracking for determining which elements ofadvertisements receive attention during a short viewing period. Moreover, theFocusTracker training is a viable, on-line method of instructing observers to perform thetask. © 10 2006
  • 12. ReferencesDuchowski, A. T. (2002). A breadth-first survey of eye-tracking applications. Behavior Research Methods Instruments & Computers, 34, 455-470.Johnson, A., & Proctor, R. W. (2004). Attention: Theory and practice. Thousand Oaks, CA: Sage Publications.Logan, G. D., & Compton, B. J. (1998). Attention and automaticity. In R. D. Wright (Ed.), Visual attention. Vancouver studies in cognitive science (Vol. 8, pp. 108-131). New York: Oxford University Press.von Helmholtz, H. (1894). Über den Ursprung der richtigen Deutung unserer Sinneseindrücke (The origin of the correct interpretation of our sensory impressions). Translated in R. M. Warren & R. P. Warren (1968). Helmholtz on perception, its physiology, and development (pp. 249-260). New York: Wiley. © 11 2006