Presented at IEEE 21st International Conference on Information Reuse and Integration for Data Science (IEEE IRI 2020).

1. Automated Filtering of Eye Gaze
Metrics from Dynamic Areas of
Interest
Gavindya Jayawardena (@Gavindya2)
Sampath Jayarathna (@Openmaze)
Department of Computer Science
Old Dominion University, Norfolk VA
IEEE 21st International Conference on Information Reuse and Integration for Data Science (IEEE IRI
2020)
IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

2. Introduction
● Eye-tracking
○ Enables investigation of the cognitive behavior when,
■ Visually exploring a stimulus
■ Carrying out cognitively demanding tasks
○ Size of the pupil diameter correlates to the task complexity
● Areas of Interests (AOIs)
○ Static AOIs
■ Explore eye gaze patterns and statistically compared gaze transitions between static AOIs
○ Dynamic AOIs
■ Use Spatio-temporal clustering of data to find potential AOIs
2IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

3. AOI Creation
● Manual Annotation
○ Softwares - Tobii Pro Studio, [1], [2]
○ Cons - Frame-by-frame annotation takes time, effort, and demands costly labor
● Physical Markers
○ Place the physical markers in the field of view of the subject
○ Cons - Irregular surfaces or motion of the surface
[1] S. Lessing and L. Linge, “Iicap? a new environment for eye tracking data analysis,” 2002.
[2] S. Stellmach, L. Nacke, and R. Dachselt, “Advanced gaze visualizations for three-dimensional virtual environments,” in Proceedings of the 2010 symposium on
eye-tracking research & Applications, 2010, pp. 109–112
3IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

4. We Propose
● A Dynamic AOI-Mapped Gaze Extraction Workflow
○ Using deep neural networks for object detection in video streams
○ Filter gaze locations that fall into the identified dynamic AOIs for the analysis of eye gaze metrics
● We develop the proposed Dynamic AOI-Mapped Gaze Extraction Workflow as a Component of
RAEMAP
4IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

5. RAEMAP: Real-time Advanced Eye
Movements Analysis Pipeline
5IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Figure 1. The Proposed Architecture of the RAEMAP

6. Object Detection
● Detect objects as dynamic AOIs
● Transfer learning to remodel existing image
classifiers for dynamic AOI detection
● 4 CNN-based object detectors that were
pre-trained on MS COCO images dataset
as the baseline models
● Detectron2* is used to load object detection
algorithms
6IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
* https://github.com/facebookresearch/detectron2

7. The Workflow
7IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Figure 2. The Workflow of the RAEMAP which Processes Eye-tracking Data and Dynamic Eye-tracking Stimuli to Detect Dynamic AOIs.

8. Dataset
● Eye-tracking database for a set of standard video sequences [3]
○ 15 Participants (2 F and 13 M)
○ 12 Video sequences
● Head-mounted eye tracker which had two 30 fps cameras
■ The eye camera
■ The scene camera
● Pre-processed gaze data of each participant separately for each video
○ Filtered out the incorrect gaze locations
[3] H. Hadizadeh, M. J. Enriquez, and I. V. Bajic, “Eye-tracking database for a set of standard video sequences,” IEEE Transactions on Image
Processing, vol. 21, no. 2, pp. 898–903, 2011.
8IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

9. Evaluation - 1. Dynamic AOI Detection
● Four video sequences with dominant objects
○ Foreman
○ Bus
○ Mother and Daughter
○ Hall Monitor
● Apply an object detector using the RAEMAP
and dynamically detect the AOIs
● Ground truth dataset
○ Manually annotating each video sequence
BeaverDam video annotation tool
9IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Figure 3. Frames with AOIs, captured from
manually annotated AOIs (Top Row) & dynamic
AOIs detected from YOLOv3 (Bottom Row)

10. Evaluation Metrics
● Intersection over Union (IoU) threshold - 0.5
○ The predicted bounding box is True Positive if IoU
≥ 0.5
○ The predicted bounding box is False Positive if IoU
< 0.5
○ Precision and Recall
10
● Mean Average Precision (mAP)
○ COCO mAP - 101 point interpolated AP
○ Pascal VOC2008 mAP - 11 point
interpolated AP
IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

11. Results - 1. Dynamic AOI Detection
11IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

12. Evaluation - 2. Dynamic AOI-mapped Eye Movements
12IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

13. Results - 2. Dynamic AOI-mapped Eye Movements
13
Figure 4. Visualization of manually annotated AOIs (in blue), AOIs detected by an object detector (in red), the gaze positions of a participant in
five consecutive frames (in green).
IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

14. Results - 2. Dynamic AOI-mapped Eye Movements
14IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

15. Conclusions
● Limitation - We only used videos with dominant objects
● Computer vision methods could be incorporated for offline detection of dynamic AOIs in video
streams
● Goal of our implementation and evaluation of object detectors was to integrate in the RAEMAP to
filter eye movement data within dynamic AOIs
● Faster R-CNN object detector with ResNet-101-FPN backbone will be used as the object
detector of proposed Dynamic AOI-Mapped Gaze Extraction Workflow of RAEMAP
15

16. Future Work
● Evaluate our methodology using one-stage object detectors such as SSD [4], DSSD [5], and Retinanet
[6].
● Use segmentation instead of rectangular boundaries when defining dynamic AOIs for the extraction of
eye movements.
[4] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C.Berg, “Ssd: Single shot multibox detector,” in European conference on computer vision Springer,
2016, pp. 21–37.
[5] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, “Dssd: Deconvolutional single shot detector,” arXiv preprint arXiv:1701.06659, 2017
[6] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dolĺar, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer
vision, 2017, pp. 2980–2988.
16IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

17. Thank You! Questions?
Gavindya Jayawardena
gavindya@cs.odu.edu
@Gavindya2
17IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2