This document summarizes an approach for generating synthetic eye-tracking data using recurrent neural networks. The approach treats eye-tracking records as textual sequences and applies natural language processing and deep learning techniques. Specifically, a long short-term memory network is used to model eye movements as sequences. The model is trained on a dataset of over 2 million eye-tracking records from 59 participants. Experimental results show the model can accurately predict future eye movements, achieving around 75% accuracy. The generative model provides a way to synthetically create additional eye-tracking data when real datasets are limited or unavailable.

1. NCTA2020
GenerativeModelingof SyntheticEye-TrackingData:
NLP-BasedApproachwithRecurrentNeuralNetworks
Mahmoud Elbattah, Jean-Luc Guérin, Romuald Carette, Federica Cilia, Gilles Dequen
Laboratoire MIS
Université de Picardie Jules Verne (UPJV), France
mahmoud.elbattah@u-picardie.fr
https://www.scitepress.org/PublicationsDetail.aspx?ID=Gjb/MLVTCAw=&t=1

2. NCTA2020
Background:Eye-TrackingTechnology
2
Image Source: https://imotions.com/blog/eye-tracking/
J.H. Goldberg, and J.I. Helfman, “Visual scanpath representation”, In Proceedings of the 2010 Symposium on Eye-Tracking Research &
Applications, ACM, 2010, pp. 203-210.
Scan-path

3. NCTA2020
Motivation
Challenge:
• The lack or difficulty of acquiring eye-tracking datasets presents a key challenge for
eye-tracking studies.
Possible Solution:
• Generating synthetic eye-tracking data.
3

4. NCTA2020
RELATEDWORK
• Hand-crafted algorithmic methods, such as:
• (Ma and Deng, 2009).
• (Le, Ma, and Deng, 2012).
• (Duchowski et al. 2016)
• Machine Learning approaches, such as:
• (Assens et al. 2018), using convolutional-recurrent architecture
• (Zemblys, Niehorster, and Holmqvist, 2019), using sequence-to-sequenceLSTM-based
architecture
• (Fuhl, 2020), using variational autoencoder
4

5. NCTA2020
Our Approach:Key Ideas
• Considering Eye-tracking records as textual sequences.
• Applying methods from the Natural Language Processing (NLP) domain to
transform and model eye-tracking sequences.
• Using a long short-term memory (LSTM) model for the generative modeling
task.
5

6. NCTA2020
Data Description
• The eye-tracking dataset was constructed over 25 sessions.
• More than 2M records stored in structured CSV files.
6
Number of Participants(ASD, TD) 59 (29, 30)
Gender Distribution (M, F) 38 (≈ 64%), 21 (≈ 36%)
Age (Mean, Median) years 7.88, 8.1
Carette, R., Elbattah, M., Dequen, G., Guérin, J. L., & Cilia, F. (2018, September). Visualization of eye-
tracking patterns in autism spectrum disorder: method and dataset. In 2018 Thirteenth International
Conference on Digital Information Management (ICDIM) (pp. 248-253). IEEE.

7. NCTA2020
Data Transformation
7
Segmentation of sequences was partly inspired by the K-mer representation, widely used in Genomics.

8. NCTA2020
GenerativeModelArchitecture
8

9. NCTA2020
ExperimentalResults
9
≈ 75% accuracy of
prediction

10. NCTA2020
Summary
• We presented an NLP-based approach for generating synthetic eye-tracking data.
• Using a sequence-based representation of the saccadic eye movement, eye-tracking
records could be modelled as textual strings with an LSTM model.
• The lack of open-access eye-tracking datasets could make our approach attractive
for further studies.
• For instance, the generative model can serve as an alternative method for data
augmentation in a wide range of eye-tracking applications.
10

11. NCTA2020
Thank You!