Goldberg Scanpath Clustering And Aggregation


Published on

Eye tracking specialists often need to understand and represent aggregate scanning strategies, but methods to identify similar scanpaths and aggregate multiple scanpaths have been elusive. A
new method is proposed here to identify scanning strategies by aggregating groups of matching scanpaths automatically. A dataset of scanpaths is first converted to sequences of viewed area names, which are then represented in a dotplot. Matching sequences in the dotplot are found with linear regressions, and then used to cluster the scanpaths hierarchically. Aggregate scanning strategies are generated for each cluster and presented in an interactive dendrogram. While the clustering and aggregation method works in a bottom-up fashion, based on pair-wise matches, a top-down extension is also described, in which a scanning strategy is first input by cursor gesture, then matched against the dataset. The ability to discover both bottom-up and top-down strategy matches provides a powerful tool for scanpath analysis, and for understanding group scanning strategies.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Goldberg Scanpath Clustering And Aggregation

  1. 1. Scanpath Clustering and Aggregation Joseph H. Goldberg and Jonathan I. Helfman Applications User Experience Oracle USA; Abstract data, to identify aggregate scanning strategies within and among observers, groups, and conditions. Eye tracking specialists often need to understand and represent aggregate scanning strategies, but methods to identify similar 1.1 Scanpaths scanpaths and aggregate multiple scanpaths have been elusive. A new method is proposed here to identify scanning strategies by Most eye trackers sample gaze locations 50-120 times a second, aggregating groups of matching scanpaths automatically. A then reduce the samples to sequences of fixations and saccades dataset of scanpaths is first converted to sequences of viewed called ‘scanpaths.’ Several algorithms are available for this re- area names, which are then represented in a dotplot. Matching duction [Salvucci and Goldberg 2000]. Scanpaths are essentially sequences in the dotplot are found with linear regressions, and records of visual attention while performing a task. Scanpaths then used to cluster the scanpaths hierarchically. Aggregate are typically represented as a single visual image with a se- scanning strategies are generated for each cluster and presented quence of connected nodes and edges, where node diameter is in an interactive dendrogram. While the clustering and aggrega- often displayed as proportional to fixation duration (within de- tion method works in a bottom-up fashion, based on pair-wise fined minimum and maximum bounds) and edges connect suc- matches, a top-down extension is also described, in which a cessive fixations. Figure 1 displays a hypothetical scanpath with scanning strategy is first input by cursor gesture, then matched 5 fixations and 4 saccades. Scanpath tracings from real data can against the dataset. The ability to discover both bottom-up and be quite complex, with frequent revisiting of AOIs and overlap- top-down strategy matches provides a powerful tool for scanpath ping saccades. analysis, and for understanding group scanning strategies. CR Categories: H.5.2. User Interfaces: Evalua- tion/methodology, /Theory and methods, I.5.3. Pattern Recogni- tion: Clustering/ algorithms, I.5.5. Pattern Recognition: Imple- mentation/Interactive systems Keywords: Eye Tracking, Usability Evaluation, Pattern Analysis, Dotplot, String Analysis, Sequential Clustering, Se- quence Analysis Figure 1. Scanpath showing a sequence of fixations and sac- cades. Longer fixation durations are denoted by larger diameter circles. 1 Introduction 1.2 Areas of Interest Eye tracking studies collect large amounts of fixation and sac- cade data while observers complete tasks or view scenes. To Areas of interest (AOIs) are regions that associate scanpath data improve the design of software and visual media, investigators with features of the scanned scene, including task target areas. make usability inferences by comparing task or interface condi- Recent studies have defined AOIs, for example, as separate list- tions between groups of users. The challenge for the researcher ings on a search results page [Cutrell and Guan 2007] and as is to quickly comprehend both individual and group scanning lines of text [Beymer and Russell 2005]. Scanpath data is typi- strategies in each tested condition. cally combined with a set of AOIs by determining the contain- ment of fixations within AOIs. AOI statistics can include fixa- The number of studies using eye tracking methods have in- tion time within AOIs, time to first view an AOI, order of AOI creased dramatically in the past few years, due to improvements viewing, transitions among AOIs, time to reach specific AOIs, in hardware, ease of calibration, and rapidly reported results. and other data. While new eye-tracking technologies have solved many niggling issues, there is still a significant gap between the results and a Figure 2 illustrates the relationship between scanpaths and AOIs. real understanding of users’ visual strategies while solving tasks. It shows a design wireframe with 5 AOIs, A-E, and a back- The present paper proposes a method to cluster users’ scanning ground area, F. Two scanpaths are shown. One scanpath, shown Copyright © 2010 by the Association for Computing Machinery, Inc. as solid red, contains 9 fixations. When each fixation is repre- Permission to make digital or hard copies of part or all of this work for personal or sented by its AOI identifier, the solid red scanpath’s fixations classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the form the sequence: CCABDFEFF. The other scanpath, in dashed first page. Copyrights for components of this work owned by others than ACM must be blue, contains 8 fixations, which form the sequence: DCDAB- honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on FEE. Note that some analysis methods eliminate repeated, con- servers, or to redistribute to lists, requires prior specific permission and/or a fee. secutive AOI visits, and may also eliminate background areas, Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail which would reduce these strings to CABDE and DCDABE, ETRA 2010, Austin, TX, March 22 – 24, 2010. © 2010 ACM 978-1-60558-994-7/10/0003 $10.00 227
  2. 2. respectively. Although not immediately obvious, the two scan- Sequence alignment techniques first attempt to align sequences paths share the sequence of AOIs: CABFE. Techniques for as closely as possible, before computing their dissimilarity [Jo- identifying shared sequences (also called ‘sequential matches’ or sephson and Holmes 2002]. Alignment techniques, however, ‘alignments’) are described in the following section. depend on initial pair-wise alignments and have difficulties aligning sequences of vastly different lengths or with very long distinct subsequences [Higgins, et al. 1996]. Scanpath sequences typically consist of AOI names as tokens, but other tokens are also possible, such as: • Saccade Angles. By coding each successive saccade as an angular direction of travel, path direction sequences can be compared. The direction of travel could be measured in either absolute angles (with respect to a common coordi- nate system) or relative to the current direction of motion. • Fixation Durations. Longer fixation durations may indi- cate greater stimulus complexity or observer confusion [Goldberg and Kotval 1999; 1998]. Scanpaths with longer fixation durations may indicate problematic aspects of an interface. Figure 2. Two hypothetical scanpaths browsing a page with six • Saccade Distances. Coding the distances between fixa- defined AOIs. Each scanpath started at the ‘+’. tions as tokens could potentially locate denser areas of in- terfaces that result in very short saccades. AOIs may overlap or may be nested within each other. When AOIs overlap it is still possible to establish a process for deter- Finding scanpath differences among groups of users and condi- mining which AOI corresponds to a particular fixation. One tions is somewhat more difficult than comparing two scanpaths. such process, for example, is to sort AOIs by area, smallest first, One algorithm first computes the pairwise sequence alignment and then iterate through the sorted list, stopping at the first (i.e. difference between scanpaths within each compared group, then the smallest) AOI containing the fixation. computes the difference between groups. A reference distribu- tion is generated by Monte Carlo simulation, allowing the defi- Spatial clustering techniques can also be used to empirically nition of statistically significant differences among scanpaths define AOIs [Salvucci and Goldberg 2000], in one case from [Feusner and Lukoff 2008]. Matching alignments from multiple over 5000 scanpaths [Wooding 2002]. A mean shift approach sequences may also represent the ‘averaged’ scanpath from a set can generate AOIs by iteratively moving sampled gaze locations of users [Hembrooke, et al., 2006]. Unsupervised learning algo- to locations of higher gaze density on a page [Santella and De- rithms can classify natural groups of scanpath sequences, form- Carlo 2004]. Similarity coefficients within and between partici- ing a hierarchical clustering of sequences in a hierarchy tree pants and images can then be used to generate a parsing tree to [West, et al. 2006]. assign AOIs automatically [Heminghous and Duchowski 2006]. Hidden Markov modeling has been used to model scanpaths, by 1.3 Comparing and Aggregating Scanpaths developing probability distributions for sequences of AOI transi- tions [Salvucci and Goldberg 2000]. While these models can Although individual scanpaths can appear to be extremely ran- determine the overall transition probabilities among AOIs, the dom and noisy, methods are available to compare them, and to composite probabilities don’t necessarily represent the aggregate aggregate them to find group trends or to uncover cognitive sequence across observers. This is due, in part, because Hidden strategies. Markov models are usually only first or second order, including only the prior one or two fixations in successive probability String comparison methods are often used to compute the simi- estimates [Josephson and Holmes 2002]. larity between two scanpaths. Scanpaths are first coded as a string of AOI names, numbers or letters. The Levenshtein dis- Aggregate representations of group sequential scanning strate- tance between the strings is then computed as the minimum gies are not easily developed from current methods. While number of substitutions, insertions, and/or deletions required to heatmaps provide a view of aggregated visual attention over a transform one string into the other [Levenshtein 1966; Smith and specific time period, they cannot adequately convey user and Waterman 1981]. Dynamic programming methods have been group scanning strategies. A single heatmap cannot show used to help determine minimum Levenshtein distances between changes over time and, therefore, cannot show sequential infor- scanpath sequences [Josephson and Holmes 2002]. A cost is mation about scanpath fixations. Sequential analysis of scan- assigned to each of the operations to result in a dissimilarity paths is required to understand the flow of visual attention on a value that ranges from 0 (identical scanpaths) to 1 (completely task. different scanpaths). For example, Figure 2’s blue scanpath (DCDABFEE) can be transformed into the red scanpath Current scanpath comparison and aggregation methods suffer (CCABDFEFF) with 5 substitutions and 1 addition, a total of 6 from several drawbacks: operations. Costs may be differentially assigned to each opera- tion type, but can be difficult to define objectively, especially • Scanpath Length. Scanpaths of different lengths have when scanpaths are of vastly different lengths. very different similarity scores, and comparison of scan- paths of differering lengths can throw off alignment calcu- lations. 228
  3. 3. • Intervening Tokens. In some cases, different length that smaller AOIs will result in greater precision for representing scanpaths are due to non-matching tokens that interrupt the aggregate strategy. what would otherwise be long matching sub-sequences. The pair-wise comparison of scanpaths is scaled to an entire • Transformation Costs. The relative cost of string opera- study by: (1) concatenating the scanpath AOI sequences into one tions is hard to define objectively, and can greatly alter the sequence, (2) plotting the sequence with a dotplot, and (3) using similarity metric. linear regression to identify pair-wise matches in the plot. Large • Scaling. String analysis methods work well for comparing datasets result in a dense dotplot, such as that shown in Figure 4. a limited number of sequences, but they don’t scale effec- Green lines indicate boundaries between scanpaths. Red lines tively to comparisons of all scanpaths in all conditions of indicate a sequential match between two scanpath segments. The a study. dotplot is symmetric, so only matches in the upper half are shown. The plot density has also been relieved somewhat using • Sequential Aggregation of Multiple Scanpaths. While inverse frequency weighting, in which extremely frequent heatmaps aggregate positional fixation information, they matches (e.g., revisits to the background AOI) have been down- do not represent sequential scanning sequences effec- weighted [Church and Helfman 1993]. tively. 1.4 Dotplots A dotplot is a graphical technique for visualizing similarities within a sequence or between two or more concatenated se- quences. Dotplots have been used to find insertions, deletions, matches, and reverse matches in genetic sequences [Huang and Zhang 2004], and have been applied to finding repetition in literature, detecting plagiarism, aligning translated documents, A. B. and identifying copied source code [Church and Helfman 1993; Helfman 1994]. The dotplot can also be considered as an inter- Figure 3. Finding matching sequences using a dotplot. A. Dot- mediate representation that is used for finding patterns algo- plot of sequences from Figure 2, with collinear data forming a rithmically; it does not need to be exposed to an eye tracking matching sequence. B. Aggregate strategy sequence ‘CABFE’ researcher directly. plotted on task background. Advantages of dotplots over string editing and alignment tech- Additional dotplots may contain only selected scanpaths. An niques include: example is shown in Figure 5. Regression lines are again shown in red, with stronger matching (i.e., a greater number of match- • Dotplots do not require a cost matrix to judge the similar- ing AOI names) noted by darker cell backgrounds. Regressions ity between two strings. with negative slopes indicate forward matching patterns (e.g., CABFE from Figure 3). Positive slopes indicate reverse match- • Dotplots support the calculation of millions of matches at ing patterns (e.g., EFBAC). near interactive rates by pre-computing positions and fre- quencies of distinct tokens [Church and Helfman 1993]. • Dotplots provide a visual representation that is useful for interactive exploration and validation. • Doplots can robustly handle non-matching tokens that in- terrupt what would otherwise be long matching sub- sequences. 2 Pattern Analysis Method 2.1 Sequence Matching using Dotplots Dotplots can be applied to pattern finding among scanpaths by listing one sequence of scanpath AOI names on the horizontal axis, and one sequence on the vertical axis of a matrix. A dot is placed in the intersecting cells of any matching AOIs. Figure 3A shows an example of a dotplot for the scanpaths from Figure 2. The red scanpath sequence (CCABDFEFF) is plotted on the horizontal axis, and the blue sequence (DCDABFEE) is on the vertical axis of the matrix. A linear regression identifies a se- quence of five dots, representing the sequence of matching AOIs, CABFE. An aggregate scanning strategy can now be represented by the Figure 4. Dotplot resulting from concatenated sequence of sequence of matching AOIs, with each aggregate fixation lo- scanpaths across study participants. Green lines separate each cated near the center of its associated AOI (Figure 3B). Note scanpath, and red marks indicate matching sequences. 229
  4. 4. The matching sequences that are discovered by the dotplot re- gression now become a reference for hierarchical clustering and aggregation. The dotplot and its matches need only be computed once for a dataset, a computational advantage. 2.3 Hierarchical Strategy Clustering Matching scanpaths can be hierarchically clustered to find strat- egies – sets of scanned regions that match increasingly greater numbers of scanpaths and individuals. The process starts by considering every individual scanpath as a leaf node cluster. At each iteration of the process, the two ‘closest’ clusters are merged into a single cluster. Clustering ends when only one cluster remains. The concept of ‘closeness’ is determined from the dotplot matches between the sequences associated with each cluster according to Algorithm 1. Because closeness between clusters must be determined repeatedly, the algorithm includes a basic optimization: closeness results between a pair of clusters are cached using a key formed from the two cluster IDs, and closeness is only calculated if it is not already in the cache. Algorithm 1. Computation of cluster distance. if(no matches between cluster sequences) Figure 5. Subset of previous dotplot, showing matching se- quences as red lines. Darker cells represent those patterns that distance = MAXIMUM; //not close matched a greater number of AOI names. else distance = 2.2 Regressions on Dotplots matchLength/maxMatchLength; Like heatmaps and other research-oriented visualizations, dot- plots show patterns that are recognized by the human visual where: system quickly, but are much less discernable to a machine. The present approach uses linear regression to pull out statistically • ‘matchLength’ is the number of sequentially matching significant sequence matches from a dense dotplot. It uses ad- AOI names between the pair of sequences, justable threshold R2 and residual data distances to find signifi- cant matches, as follows: • ‘maxMatchLength’ is the largest number of sequen- tially matching AOI names in the entire dataset (i.e. 1. Start with an inverse-frequency filtered dotplot comparing between each pair of sequences). multiple concatenated scanpath sequences. At each step of the clustering process, the two closest clusters 2. Iterate over the dotplot cells for each pair of scanpaths, are merged into a new cluster. The merged cluster stores a identifying the ‘darkest’ dots in the cell, which correspond unique name, references to its two child clusters, and the ‘dis- to sub-sequences with significant matches. High-pass fre- tance’ between the child clusters. For clustering to continue, quency thresholds of 0.1-0.5 work well, using a 1µ+1σ cri- each new cluster must also be assigned a sequence of AOI names. One possibility would be to form a sequence from the terion. matching AOI names. We have found, however, that because the matches are typically shorter than either child sequence, this 3. Fit a linear regression to the darkest points, moving to the approach guarantees that cluster hierarchies will be small and next cell if R2 value is too low (e.g., <0.5). shallow. In contrast, we have found that deeper clusters are obtained when the merged cluster is assigned one of the child 4. Compute a 1µ+1σ threshold from the Euclidean distances sequences. Because each child sequence contains the matching between the regression line and the data, and identify those sequence of AOI names, either one will match at least as many other scanpaths as the matching sequence. In fact, we have data within the threshold. found that the deepest clusters are obtained when the merged cluster is assigned the child sequence that matches the most 5. Recompute the regression only for data within the thresh- other sequences in the dataset. old. Dots falling on (or close to) the regression line are con- sidered elements of the matching sequence. 2.4 Aggregation of Strategies 6. (Optionally) return to (2) after removing matching dots, and An aggregate representation is assigned to each cluster of scan- re-compute the regression to find further matches between ning strategies. The representation uses the sequence of match- ing AOI names between the sequences associated with the clus- the pair of scanpaths. ter’s two children. If the sequences associated with the cluster’s children have no matches, the aggregate strategy uses the se- 230
  5. 5. quence of the child cluster with the greatest distance value (i.e. relative angle radial histograms can also provide informa- the child cluster with the best matching children of its own). tion about the shape of a scanpath or aggregate strategy. Choosing an aggregate strategy in this way ensures that the ag- gregates will correspond to the closest actual scanpath matches in their clusters whenever possible. The choice of aggregate strategy representation depends entirely on the sequential matches found between scanpath pairs: if the scanpaths have few matches, the results of the clustering and aggregation methods will be dubious. Dotplot regression is parameterized, however, Figure 6. Compact visual scanpath representations: A. Scaled and it is usually possible to vary the parameters to find addi- trace, B. Vertical time expansion, C. Horizontal time expansion, tional sequential matches (See Section 3.2). The present aggre- D. Radial histogram, based on absolute angles, E. Radial histo- gation method is a proposed solution, and further evaluation is gram, based on relative angles. required to fully judge the validity of this approach. 3 Implementation of Eye Tracking Pattern 2.5 Visual Representation of Aggregate Analysis Tool Strategies The dotplot-based, pattern analysis method was implemented in Aggregate sequences can be represented in multiple ways to a tool that clusters, aggregates, and represents scanning strate- highlight different properties. Five example visual representa- gies across datasets of scanpaths collected from multiple studies. tions are shown in Figure 6, and described below. Further back- The tool was written in HTML/Javascript, and runs within the ground on these representations are available in a companion Firefox and Chrome web browsers. Sub-queries may be con- paper [Goldberg and Helfman, in press]. ducted within each dataset, such as including only correct trials or only certain targets. The tool is intended for usability profes- • Scanpath Trace. Like those shown by Yarbus [1967], sionals who are using eye tracking methods to discover scanning these scaled down scanpath traces show the general distri- strategies of groups of individuals, and to compare design alter- bution and extent of scanning. While fixations are not ex- natives. plicitly shown, a general impression of fixation density, scanpath complexity, and individual and aggregated scan- A dataset is developed from data that are exported from a Tobii ning strategy can be obtained (Figure 6A). T60 eye tracker, running Tobii Studio 1.15 software. The fol- lowing elements are imported into the pattern analysis tool: • Vertical Time Expansion. A record of horizontal scanning is provided by substituting time for the vertical coordinates • Background image from each task. of each fixation, allowing for analysis of horizontal scan- ning trends and backtracking frequency (Figure 6B). The • AOI names and coordinates for background task images. analyst can determine, for example, whether scanning was • List of tasks by participants, including (if available) in- primarily in one direction or whether it moved right and left tended target AOI, AOI of selected response, and whether frequently. the response was an error. • Horizontal Time Expansion. Replacing the horizontal • List of fixations by participant by task, including time- coordinates of each fixation with time provides a record of stamp, fixation duration, and x/y location. vertical scanning within the overall scanpath or aggregate strategy (Figure 6C). Similar representations have been 3.1 Analysis Parameters used, in larger scale, for scanning lines of computer code [Uwano, et al. 2006] and listings of search results [Raiha, et A dialog panel (Figure 7) is available in the tool to modify pa- al. 2005, Aula, et al. 2005]. rameters of the regressions, in order to increase or decrease the • Radial Histogram—Absolute Angles. A radial histogram number of matches found in the dotplot. Clusters that are too shallow to be meaningful can be modified by resetting the re- counts the number of saccade angles associated with each gression values and re-conducting the regression. More liberal of several predefined angle ranges (Figure 6D). Angles are regression parameters result in a greater number of matching measured with respect to an absolute scale (e.g. counter- sequences in the dotplot, which in turn can result in greater clockwise, with 0o on the positive X-axis). A red dot indi- matching and deeper clusters. More liberal regressions, how- cates the mean shift, computed as the spatial average of the ever, also run the risk of matches between insignificant or overly histogram. This representation provides angular informa- frequent sequences, resulting in non-representative aggregate tion about the scanpath, including number of different an- strategy clusters. The proper settings of regression parameters gles, and whether scanning was direct or convoluted. result in conservative clusters that are sufficiently deep. Regres- • Radial Histogram—Relative Angles. Similar to Absolute sion parameters that can be modified include: Angle radial histogram, except each angle is determined relative to its prior saccade direction (Figure 6E). Moving • Xµ+Yσ, where X and Y adjust the threshold Euclidean to the left from the prior direction defines a 90o relative an- distance of data points, to be included in the regression. gle, but could define a very different absolute angle. Radial Choices include 0, 0.5, 1, 2. Larger values result in greater histograms made from relative angles are more sensitive to number of significant regressions. small directional shifts in scanpaths, than are those made • Minimum number of collinear points required for a suc- from absolute angles. Comparison between absolute and cessful regression. Choices include 3, 4, 5, 10. Larger val- ues result in fewer, but longer, sequential matches. 231
  6. 6. • Minimum R2 value required for a significant regression. ters at a specified depth level. Scanpath and aggregate strategy Choices include: 0, .001, .01, .1, .5. Larger values result in traces (in red) are also shown within the dendrogram. fewer sequential matches. Selecting a cluster in the dendrogram updates a detail pane (Fig- • Qµ+Rσ, where Q and R adjust the threshold distance of ure 9) with both the aggregated strategy and the individual scan- data points from the initial regression line that are used to paths that match the aggregate strategy. Figure 9 displays a se- compute the final regression line. Choices include: 0, .5, 1, lected cluster (-31), its aggregate strategy cluster, and the 7 2. Larger values result in greater number of significant scanpaths contained in the cluster. The aggregate strategy is also matches. shown on the task background image in Figure 10, both with (Figure 10A) and without (Figure 10B) its constituent scanpaths. The aggregate matching strategy necessarily provides the lowest common denominator among the matching scanpaths, so ap- pears as a simpler, more compact scanpath. In this case, it re- veals a matching strategy that starts at the left-most column header, scans to the middle of the table, then scans leftward. In general, larger clusters result in less complex aggregate scanning strategies that match more scanpaths than those derived from smaller clusters. Figure 7. Parameter setting panel for dotplot analysis in the pattern analysis tool. Default values are shown. The content of the sequences evaluated by the dotplot can also be selected. Sequence content choices include region names, saccade lengths, fixation durations, absolute saccade angles, and relative saccade angles. Although not the focus of the present paper, sequences composed of these alternate metrics provide additional strategic value for eye tracking analysis. While se- quences of AOI names are dependent on the specific placement of AOIs on a background image, sequences of saccade angles can be matched and aggregated independently from background images. For example, the sequence 90o, 0o, 270o (absolute an- gles) defines a scanpath that moves up, right, then down, regard- less of AOIs or background image. Matching scanning se- Figure 8. Interactive dendrogram shows results of the clustering quences could be identified across images and tasks using sac- process. Clicking selects a cluster and its aggregate strategy, cade angles. Similarly, sequences of fixation durations could which is viewed in other panes. Darker circles indicate stronger potentially identify cognitive states or task complexity [Marshall matches, and a cut-plane of strategy clusters is selected by 2007, Goldberg and Kotval 1999]. dragging the vertical dotted line. When based on quantitative metrics, sequences for dotplot analysis and scanpath representation are tallied into user- definable bins. For example, saccade angles have default bin sizes of 20o, but larger or smaller sizes can be selected. Larger bin widths provide coarser analyses, but generally make it easier to form clusters of similar sequences. Smaller bin widths pro- vide a higher resolution of analysis, but may make it more diffi- cult to form clusters. 3.2 Interactive Dendrogram The results of hierarchical clustering are shown in an interactive dendrogram, a type of binary tree diagram (Figure 8). The hier- archical pattern of sequential matches is represented graphically by the linear branches of the tree diagram. Matching distance (‘closeness’) within each cluster is represented by a number (0- 1) and a dot, where darker dots are associated with stronger matching distance. Strategy clusters can be viewed by vertical scrolling, and any cluster can be selected by clicking on it. A Figure 9. Detail pane showing selected aggregate strategy clus- dotted blue line can also be dragged horizontally to slice across ter (-31) and the 7 individual scanpaths that matched the aggre- the cluster hierarchy, allowing the selection and viewing of clus- gate strategy. Additional visual representations are also pro- vided to the right of the scanpaths. 232
  7. 7. 3.3 Positing Strategies via Gesture matching would find other similar saccade angle sequences, saccade distances, fixation durations, or other metrics. Presentation of the pattern analysis tool to this point has focused on its ability to find and cluster matching strategy sequences 4 Discussion within an eye tracking dataset. This ‘bottom-up’ analysis essen- tially finds ad hoc matches in a dataset. The pattern analysis tool also includes a method to input (or posit) a ‘top-down’ strategy, 4.1 Pattern Analysis Method which generates a new dotplot and regressions. The new strategy may be input by dragging the mouse cursor over a scaled-down The present work was motivated by the need for a method to version of the background task image. The input strategy is con- extract group scanning strategies from large eye tracking data- sets. While heatmaps and cumulative fixation times provide verted to a sequence of AOI names, much like a regular scan- path, and a dotplot is used to match the posited strategy against valuable insight into which AOIs are viewed, they do not pro- the dataset. Scanpaths that match the strategy are treated as a vide sufficient detail about the sequential scanning strategies on a page. Sequential information is needed to understand common new query result, clustered, and presented in a new interactive dendrogram. scanning behavior between groups of observers, between differ- ent interface designs, or between interface conditions. Although other researchers have also proposed solutions to the problem of defining the aggregate scanpath for a group [e.g., Hembrooke, et al. 2006, West, et al. 2006, Feusner and Lukoff, 2008], the present approach provides certain advantages. (1) It is scalable to large numbers of participant scanpaths. (2) The un- derlying dotplot representations have already been validated for sequence matching in other domains, such as biology and pro- gram code comparison. (3) Unlike other string analysis and se- quential alignment methods, no cost matrix is required for dot- Figure 10. Background task image with superimposed aggre- plots. (4) Matching sequences can be discovered even with the gate strategy with (A), and without (B) individual scanpath data. presence of intervening non-matching tokens. (5) The method is efficient, requiring that a dotplot be computed only once for a As an example, Figure 11 displays a cursor-input path that traces dataset. (6) The matching process is highly tunable by modify- a scanpath traveling down the left column, then rightward across ing regression properties. Although default values usually result the bottom of the presented table. Upon mouse-up, the query is in a wide range of discovered strategies, adjusting these values input, and a new result set is clustered and presented in a den- can create deeper clusters with scanning strategies that match drogram. Selection of a cluster in the dendrogram, shown at the greater numbers of scanpath segments across a dataset. (7) The left in Figure 12, displays the aggregate strategy, along with the dotplot method can be extended to other sequences besides AOI four matching scanpaths. The aggregate strategy is shown super- names, such as saccade angles, or fixation durations. Each of imposed on the task background image at the right of the figure. these can provide additional information about strategies. In this case, the aggregate strategy matched a portion of the downward scanning, and much of the rightward scanning. 4.2 Pattern Analysis Tool The methods described in this paper were implemented within an eye tracking pattern analysis tool to provide researchers a way to rapidly discover sequential group scanning strategies in studies. While details about the dotplot matching process are easily hidden from an end-user, a selectable dendrogram allows interactive exploration of the hierarchically clustered scanning strategies. A range of scanning strategies and cluster depths are visible by scrolling the dendrogram. Slicing the dendrogram at a Figure 11. Cursor-input query to filter the dataset. particular depth provides a rapid way to control which clusters are shown in a detail panel. Scanpaths and aggregate strategies can be selected for display on the background task image. The pair-wise sequential matches that underlie the aggregate strate- gies can also be easily viewed. Graphic representations of scanpaths and aggregate strategies in our tool extend others’ ideas and introduce new functionality. They allow the researcher to quickly visualize similarities and differences between scanpaths. The representations provide a Figure 12. Resulting portion of dendrogram, scanpaths, and diagnostic aid to help understand why scanpaths were included aggregate strategy for selected cluster, based upon the input within selected clusters. Although not explored in the present posited strategy of Figure 11. paper, features of these representations (e.g., bin lengths in ra- dial histograms, mean shift distances from origins) can poten- Although not yet implemented in the present tool, it is also pos- tially aid the algorithmic discovery of larger scanning strategies, sible to convert the gesture input to other sequences besides AOI such as clockwise, downward, rightward, even more complex names. Examples include saccade angles, saccade distances, scanning tendencies [Goldberg and Helfman, in press]. fixations durations, and other metrics. In these cases, dotplot 233
  8. 8. Gesture-based strategy input supports a valuable top-down GOLDBERG, J. AND HELFMAN, J. In Press, in press. Visual Scan- analysis by imposing a hypothesized strategy, then discovering path Representation. In Proc. ETRA 2010, ACM Press. matching sequential strategies in the dataset. Top-down analysis enables the researcher to find out whether observers collectively GOLDBERG, J., AND KOTVAL, X. 1998. Eye Movement-based scanned a stimulus image in a specified way, such as left column Evaluation of the Computer Interface. In Kumar, J. (Ed.), downward, or bottom row rightward. Ultimately, a library of Advances in Occupational Ergonomics and Safety, IOS hypothesized strategies can be developed and saved, then com- Press, 529-532. pared against scanpath collections. A higher-level description of GOLDBERG, J., AND KOTVAL, X. 1999. Computer Interface individual and aggregate scanning strategies could ultimately Evaluation using Eye Movements: Methods and Constructs. result from a pattern of these matches. Int. J. Industrial Ergonomics, 14, 631-645. 4.3 Areas for Future Enhancements HELFMAN, J. 1994. Similarity Patterns in Language. In Proc. 1994 IEEE Symp. Vis. Lang., IEEE Press, 173-175. Although our pattern analysis tool makes a large contribution to HEMBROOKE, H., FEUSNER, M., AND GAY, G. 2006. Averaging the empirical analysis of collections of scanpath data, there are Scan Patterns and What They Can Tell Us. In Proc. ETRA several areas for improvement, such as: 2006, ACM Press, 41. • The current hierarchical clustering process forces each HIGGINS, D., THOMPSON, J., GIBSON, T. 1996. Using CLUSTAL scanpath to be in only one cluster. Scanpaths, however, for Multiple Sequence Alignments. Methods Enzymol., 266, may match strategies from multiple clusters. While our 383-402. technique of associating clusters with sequences that HUANG, Y., AND ZHANG, L. 2004. Rapid and Sensitive Dot- match the most other sequences in the dataset is an at- Matrix Methods for Genome Analysis. Bioinformatics, 20, tempt to mitigate this problem, future work will evaluate 4, 460-466. alternative modifications of the clustering algorithm to al- low non-matching sub-sequences of scanpaths to contrib- JOSEPHSON, S., AND HOLMES, M. 2002. Visual Attention to Re- ute to other clusters. peated Internet Images: Testing the Scanpath Theory on the World Wide Web. In Proc. ETRA 2002, ACM Press, 43-49. • The present strategy matching process is efficient and can form relatively deep clusters. As one explores a strategy LEVENSHTEIN, V.I. 1966. Binary Codes Capable of Correcting cluster tree, parent strategies typically look like common Deletions, Insertions and Reversals. Doklady Physics, 10, subsets of child strategy representations. However, with 707-710. the present algorithm, it is possible to find parent strate- gies that resemble deeper ancestors, rather than direct MARSHALL, S.P. 2007. Identifying Cognitive State from Eye child strategies. Metrics. Aviation, Space, and Environmental Medicine, 78(5), B165-B185. • While the pattern analysis tool presently supports match- ing and clustering of any sequences, it can only define ag- RAIHA, K., AULA, A., MAJARANTA, P., RANTALA, H., AND KOI- gregate strategies for clustered sequences of AOI names. VUNEN, K. 2005. Static Visualization of Temporal Eye- Aggregate strategies that are based on sequences of fixa- Tracking Data. In Proc. INTERACT 2005, Int. Fed. Info tion durations, absolute/relative saccade angles, saccade Proc, 946-949. lengths, and other properties can potentially match scan- SALVUCCI, D., AND GOLDBERG, J. 2000. Identifying Fixations and paths that are not tied to particular tasks and background Saccades in Eye-tracking Protocols. In Proc. ETRA 2000, images. Future work will investigate clustering and aggre- ACM Press, 71-78. gation methods based upon these and other criteria. SANTELLA, A. AND DECARLO, D. 2004. Robust Clustering of Eye Movement Recordings for Quantification of Visual Interest. In Proc. ETRA 2004, ACM Press, 1-8. References AULA, A., MAJARANTA, P., AND RAIHA, K. 2005. Eye-tracking SMITH, T. AND WATERMAN, S. 1981. Identification of Common Reveals the Personal Styles for Search Result Evaluation, Molecular Subsequences. J. Molecular Biol., 147, 1, 195- In Proc. INTERACT 2005, Int. Fed. Info Proc, 1058-1061. 197. BEYMER, D., AND RUSSELL, D. 2005. WebGazeAnalyzer: A Sys- UWANO, H., NAKUMURA, M., MONDEN, A., AND MATSUMOTO, K. tem for Capturing and Analyzing Web Reading Behavior 2006. Analyzing Individual Performance of Source Code using Eye Gaze. In Proc. CHI 2005, ACM Press, 1913- Review Using Reviewers’ Eye Movement. In Proc. ETRA 1916. 2006, ACM Press, 133-140. CHURCH, K., AND HELFMAN, J. 1993. Dotplot: A Program for WEST, J., HAAKE, A., ROZANSKI, E., AND KARN, K. 2006. EyePat- Exploring Self-similarity in Millions of Lines of Text and terns: Software for Identifying Patterns and Similarities Code. J. Comp. and Graphical Statistics, 2, 153-174. across Fixation Sequences. In Proc. ETRA 2006, ACM Press, 149-154. CUTRELL, E., AND GUAN, Z. 2007. What are You Looking For? An Eye-tracking Study of Information Usage in Web WOODING, D. 2002. Eye Movements of Large Populations: II. Search. In Proc. CHI 2007, ACM Press, 407-416. Deriving Regions of Interest, Coverage, and Similarity us- ing Fixation Maps. Behavior Research Methods, Instru- FEUSNER, M., AND LUKOFF, B. 2008. Testing for Statistically ments & Computers, 34, 4, 518-528. Significant Differences Between Groups of Scan Patterns. In Proc. ETRA 2008, ACM Press, 43-46. YARBUS, A.L. 1967. Eye Movements and Vision. New York, Plenum Press. 234