SlideShare a Scribd company logo
Automated Filtering of Eye Gaze
Metrics from Dynamic Areas of
Interest
Gavindya Jayawardena (@Gavindya2)
Sampath Jayarathna (@Openmaze)
Department of Computer Science
Old Dominion University, Norfolk VA
IEEE 21st International Conference on Information Reuse and Integration for Data Science (IEEE IRI
2020)
IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Introduction
● Eye-tracking
○ Enables investigation of the cognitive behavior when,
■ Visually exploring a stimulus
■ Carrying out cognitively demanding tasks
○ Size of the pupil diameter correlates to the task complexity
● Areas of Interests (AOIs)
○ Static AOIs
■ Explore eye gaze patterns and statistically compared gaze transitions between static AOIs
○ Dynamic AOIs
■ Use Spatio-temporal clustering of data to find potential AOIs
2IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
AOI Creation
● Manual Annotation
○ Softwares - Tobii Pro Studio, [1], [2]
○ Cons - Frame-by-frame annotation takes time, effort, and demands costly labor
● Physical Markers
○ Place the physical markers in the field of view of the subject
○ Cons - Irregular surfaces or motion of the surface
[1] S. Lessing and L. Linge, “Iicap? a new environment for eye tracking data analysis,” 2002.
[2] S. Stellmach, L. Nacke, and R. Dachselt, “Advanced gaze visualizations for three-dimensional virtual environments,” in Proceedings of the 2010 symposium on
eye-tracking research & Applications, 2010, pp. 109–112
3IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
We Propose
● A Dynamic AOI-Mapped Gaze Extraction Workflow
○ Using deep neural networks for object detection in video streams
○ Filter gaze locations that fall into the identified dynamic AOIs for the analysis of eye gaze metrics
● We develop the proposed Dynamic AOI-Mapped Gaze Extraction Workflow as a Component of
RAEMAP
4IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
RAEMAP: Real-time Advanced Eye
Movements Analysis Pipeline
5IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Figure 1. The Proposed Architecture of the RAEMAP
Object Detection
● Detect objects as dynamic AOIs
● Transfer learning to remodel existing image
classifiers for dynamic AOI detection
● 4 CNN-based object detectors that were
pre-trained on MS COCO images dataset
as the baseline models
● Detectron2* is used to load object detection
algorithms
6IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
* https://github.com/facebookresearch/detectron2
The Workflow
7IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Figure 2. The Workflow of the RAEMAP which Processes Eye-tracking Data and Dynamic Eye-tracking Stimuli to Detect Dynamic AOIs.
Dataset
● Eye-tracking database for a set of standard video sequences [3]
○ 15 Participants (2 F and 13 M)
○ 12 Video sequences
● Head-mounted eye tracker which had two 30 fps cameras
■ The eye camera
■ The scene camera
● Pre-processed gaze data of each participant separately for each video
○ Filtered out the incorrect gaze locations
[3] H. Hadizadeh, M. J. Enriquez, and I. V. Bajic, “Eye-tracking database for a set of standard video sequences,” IEEE Transactions on Image
Processing, vol. 21, no. 2, pp. 898–903, 2011.
8IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Evaluation - 1. Dynamic AOI Detection
● Four video sequences with dominant objects
○ Foreman
○ Bus
○ Mother and Daughter
○ Hall Monitor
● Apply an object detector using the RAEMAP
and dynamically detect the AOIs
● Ground truth dataset
○ Manually annotating each video sequence
BeaverDam video annotation tool
9IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Figure 3. Frames with AOIs, captured from
manually annotated AOIs (Top Row) & dynamic
AOIs detected from YOLOv3 (Bottom Row)
Evaluation Metrics
● Intersection over Union (IoU) threshold - 0.5
○ The predicted bounding box is True Positive if IoU
≥ 0.5
○ The predicted bounding box is False Positive if IoU
< 0.5
○ Precision and Recall
10
● Mean Average Precision (mAP)
○ COCO mAP - 101 point interpolated AP
○ Pascal VOC2008 mAP - 11 point
interpolated AP
IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Results - 1. Dynamic AOI Detection
11IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Evaluation - 2. Dynamic AOI-mapped Eye Movements
12IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Results - 2. Dynamic AOI-mapped Eye Movements
13
Figure 4. Visualization of manually annotated AOIs (in blue), AOIs detected by an object detector (in red), the gaze positions of a participant in
five consecutive frames (in green).
IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Results - 2. Dynamic AOI-mapped Eye Movements
14IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Conclusions
● Limitation - We only used videos with dominant objects
● Computer vision methods could be incorporated for offline detection of dynamic AOIs in video
streams
● Goal of our implementation and evaluation of object detectors was to integrate in the RAEMAP to
filter eye movement data within dynamic AOIs
● Faster R-CNN object detector with ResNet-101-FPN backbone will be used as the object
detector of proposed Dynamic AOI-Mapped Gaze Extraction Workflow of RAEMAP
15
Future Work
● Evaluate our methodology using one-stage object detectors such as SSD [4], DSSD [5], and Retinanet
[6].
● Use segmentation instead of rectangular boundaries when defining dynamic AOIs for the extraction of
eye movements.
[4] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C.Berg, “Ssd: Single shot multibox detector,” in European conference on computer vision Springer,
2016, pp. 21–37.
[5] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, “Dssd: Deconvolutional single shot detector,” arXiv preprint arXiv:1701.06659, 2017
[6] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dolĺar, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer
vision, 2017, pp. 2980–2988.
16IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
Thank You! Questions?
Gavindya Jayawardena
gavindya@cs.odu.edu
@Gavindya2
17IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2

More Related Content

Similar to Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest

DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...
DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...
DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...
IRJET Journal
 
Vision-Based Motorcycle Crash Detection and Reporting Using Deep Learning
Vision-Based Motorcycle Crash Detection and Reporting Using Deep LearningVision-Based Motorcycle Crash Detection and Reporting Using Deep Learning
Vision-Based Motorcycle Crash Detection and Reporting Using Deep Learning
IRJET Journal
 
Real Time Moving Object Detection for Day-Night Surveillance using AI
Real Time Moving Object Detection for Day-Night Surveillance using AIReal Time Moving Object Detection for Day-Night Surveillance using AI
Real Time Moving Object Detection for Day-Night Surveillance using AI
IRJET Journal
 
Implementation of face and eye detection on DM6437 board using simulink model
Implementation of face and eye detection on DM6437 board using simulink modelImplementation of face and eye detection on DM6437 board using simulink model
Implementation of face and eye detection on DM6437 board using simulink model
journalBEEI
 
Object Detection for Autonomous Cars using AI/ML
Object Detection for Autonomous Cars using AI/MLObject Detection for Autonomous Cars using AI/ML
Object Detection for Autonomous Cars using AI/ML
IRJET Journal
 
IRJET- Object Detection in Real Time using AI and Deep Learning
IRJET- Object Detection in Real Time using AI and Deep LearningIRJET- Object Detection in Real Time using AI and Deep Learning
IRJET- Object Detection in Real Time using AI and Deep Learning
IRJET Journal
 
IRJET- Driver Drowsiness Detection System using Computer Vision
IRJET- Driver Drowsiness Detection System using Computer VisionIRJET- Driver Drowsiness Detection System using Computer Vision
IRJET- Driver Drowsiness Detection System using Computer Vision
IRJET Journal
 
IRJET - Computer Vision-based Image Processing System for Redundant Objec...
IRJET -  	  Computer Vision-based Image Processing System for Redundant Objec...IRJET -  	  Computer Vision-based Image Processing System for Redundant Objec...
IRJET - Computer Vision-based Image Processing System for Redundant Objec...
IRJET Journal
 
Object Detection and Tracking AI Robot
Object Detection and Tracking AI RobotObject Detection and Tracking AI Robot
Object Detection and Tracking AI Robot
IRJET Journal
 
AIFrienz_Webinar_Tomomi_Research_Inc).pdf
AIFrienz_Webinar_Tomomi_Research_Inc).pdfAIFrienz_Webinar_Tomomi_Research_Inc).pdf
AIFrienz_Webinar_Tomomi_Research_Inc).pdf
Seong-Hun Choe
 
GP_Slides_V3 .pptx
GP_Slides_V3 .pptxGP_Slides_V3 .pptx
GP_Slides_V3 .pptx
AhmedEldairy
 
Motion capture for Animation
Motion capture for AnimationMotion capture for Animation
Motion capture for Animation
IRJET Journal
 
IRJET - Floor Cleaning Robot with Vision
IRJET - Floor Cleaning Robot with VisionIRJET - Floor Cleaning Robot with Vision
IRJET - Floor Cleaning Robot with Vision
IRJET Journal
 
DYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTION
DYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTIONDYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTION
DYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTION
IRJET Journal
 
IRJET- Real-time Eye Tracking for Password Authentication
IRJET- Real-time Eye Tracking for Password AuthenticationIRJET- Real-time Eye Tracking for Password Authentication
IRJET- Real-time Eye Tracking for Password Authentication
IRJET Journal
 
(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization
(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization
(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization
e8xu
 
Restoration of the video by removing rain streaks
Restoration of the video by removing rain streaksRestoration of the video by removing rain streaks
Restoration of the video by removing rain streaks
IRJET Journal
 
IRJET- Iris Controlled Access Mechanism for the Physically Challenged
IRJET- Iris Controlled Access Mechanism for the Physically ChallengedIRJET- Iris Controlled Access Mechanism for the Physically Challenged
IRJET- Iris Controlled Access Mechanism for the Physically Challenged
IRJET Journal
 
IRJET- Driver’s Sleep Detection
IRJET-  	  Driver’s Sleep DetectionIRJET-  	  Driver’s Sleep Detection
IRJET- Driver’s Sleep Detection
IRJET Journal
 
IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...
IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...
IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...
IRJET Journal
 

Similar to Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest (20)

DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...
DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...
DSNet Joint Semantic Learning for Object Detection in Inclement Weather Condi...
 
Vision-Based Motorcycle Crash Detection and Reporting Using Deep Learning
Vision-Based Motorcycle Crash Detection and Reporting Using Deep LearningVision-Based Motorcycle Crash Detection and Reporting Using Deep Learning
Vision-Based Motorcycle Crash Detection and Reporting Using Deep Learning
 
Real Time Moving Object Detection for Day-Night Surveillance using AI
Real Time Moving Object Detection for Day-Night Surveillance using AIReal Time Moving Object Detection for Day-Night Surveillance using AI
Real Time Moving Object Detection for Day-Night Surveillance using AI
 
Implementation of face and eye detection on DM6437 board using simulink model
Implementation of face and eye detection on DM6437 board using simulink modelImplementation of face and eye detection on DM6437 board using simulink model
Implementation of face and eye detection on DM6437 board using simulink model
 
Object Detection for Autonomous Cars using AI/ML
Object Detection for Autonomous Cars using AI/MLObject Detection for Autonomous Cars using AI/ML
Object Detection for Autonomous Cars using AI/ML
 
IRJET- Object Detection in Real Time using AI and Deep Learning
IRJET- Object Detection in Real Time using AI and Deep LearningIRJET- Object Detection in Real Time using AI and Deep Learning
IRJET- Object Detection in Real Time using AI and Deep Learning
 
IRJET- Driver Drowsiness Detection System using Computer Vision
IRJET- Driver Drowsiness Detection System using Computer VisionIRJET- Driver Drowsiness Detection System using Computer Vision
IRJET- Driver Drowsiness Detection System using Computer Vision
 
IRJET - Computer Vision-based Image Processing System for Redundant Objec...
IRJET -  	  Computer Vision-based Image Processing System for Redundant Objec...IRJET -  	  Computer Vision-based Image Processing System for Redundant Objec...
IRJET - Computer Vision-based Image Processing System for Redundant Objec...
 
Object Detection and Tracking AI Robot
Object Detection and Tracking AI RobotObject Detection and Tracking AI Robot
Object Detection and Tracking AI Robot
 
AIFrienz_Webinar_Tomomi_Research_Inc).pdf
AIFrienz_Webinar_Tomomi_Research_Inc).pdfAIFrienz_Webinar_Tomomi_Research_Inc).pdf
AIFrienz_Webinar_Tomomi_Research_Inc).pdf
 
GP_Slides_V3 .pptx
GP_Slides_V3 .pptxGP_Slides_V3 .pptx
GP_Slides_V3 .pptx
 
Motion capture for Animation
Motion capture for AnimationMotion capture for Animation
Motion capture for Animation
 
IRJET - Floor Cleaning Robot with Vision
IRJET - Floor Cleaning Robot with VisionIRJET - Floor Cleaning Robot with Vision
IRJET - Floor Cleaning Robot with Vision
 
DYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTION
DYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTIONDYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTION
DYNAMIC ENERGY MANAGEMENT USING REAL TIME OBJECT DETECTION
 
IRJET- Real-time Eye Tracking for Password Authentication
IRJET- Real-time Eye Tracking for Password AuthenticationIRJET- Real-time Eye Tracking for Password Authentication
IRJET- Real-time Eye Tracking for Password Authentication
 
(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization
(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization
(Research note) Visual Inertial Odometry using Coupled Nonlinear Optimization
 
Restoration of the video by removing rain streaks
Restoration of the video by removing rain streaksRestoration of the video by removing rain streaks
Restoration of the video by removing rain streaks
 
IRJET- Iris Controlled Access Mechanism for the Physically Challenged
IRJET- Iris Controlled Access Mechanism for the Physically ChallengedIRJET- Iris Controlled Access Mechanism for the Physically Challenged
IRJET- Iris Controlled Access Mechanism for the Physically Challenged
 
IRJET- Driver’s Sleep Detection
IRJET-  	  Driver’s Sleep DetectionIRJET-  	  Driver’s Sleep Detection
IRJET- Driver’s Sleep Detection
 
IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...
IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...
IRJET- A Review Paper on Object Detection using Zynq-7000 FPGA for an Embedde...
 

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Zilliz
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Zilliz
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 

Recently uploaded (20)

UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 

Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest

  • 1. Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest Gavindya Jayawardena (@Gavindya2) Sampath Jayarathna (@Openmaze) Department of Computer Science Old Dominion University, Norfolk VA IEEE 21st International Conference on Information Reuse and Integration for Data Science (IEEE IRI 2020) IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 2. Introduction ● Eye-tracking ○ Enables investigation of the cognitive behavior when, ■ Visually exploring a stimulus ■ Carrying out cognitively demanding tasks ○ Size of the pupil diameter correlates to the task complexity ● Areas of Interests (AOIs) ○ Static AOIs ■ Explore eye gaze patterns and statistically compared gaze transitions between static AOIs ○ Dynamic AOIs ■ Use Spatio-temporal clustering of data to find potential AOIs 2IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 3. AOI Creation ● Manual Annotation ○ Softwares - Tobii Pro Studio, [1], [2] ○ Cons - Frame-by-frame annotation takes time, effort, and demands costly labor ● Physical Markers ○ Place the physical markers in the field of view of the subject ○ Cons - Irregular surfaces or motion of the surface [1] S. Lessing and L. Linge, “Iicap? a new environment for eye tracking data analysis,” 2002. [2] S. Stellmach, L. Nacke, and R. Dachselt, “Advanced gaze visualizations for three-dimensional virtual environments,” in Proceedings of the 2010 symposium on eye-tracking research & Applications, 2010, pp. 109–112 3IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 4. We Propose ● A Dynamic AOI-Mapped Gaze Extraction Workflow ○ Using deep neural networks for object detection in video streams ○ Filter gaze locations that fall into the identified dynamic AOIs for the analysis of eye gaze metrics ● We develop the proposed Dynamic AOI-Mapped Gaze Extraction Workflow as a Component of RAEMAP 4IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 5. RAEMAP: Real-time Advanced Eye Movements Analysis Pipeline 5IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2 Figure 1. The Proposed Architecture of the RAEMAP
  • 6. Object Detection ● Detect objects as dynamic AOIs ● Transfer learning to remodel existing image classifiers for dynamic AOI detection ● 4 CNN-based object detectors that were pre-trained on MS COCO images dataset as the baseline models ● Detectron2* is used to load object detection algorithms 6IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2 * https://github.com/facebookresearch/detectron2
  • 7. The Workflow 7IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2 Figure 2. The Workflow of the RAEMAP which Processes Eye-tracking Data and Dynamic Eye-tracking Stimuli to Detect Dynamic AOIs.
  • 8. Dataset ● Eye-tracking database for a set of standard video sequences [3] ○ 15 Participants (2 F and 13 M) ○ 12 Video sequences ● Head-mounted eye tracker which had two 30 fps cameras ■ The eye camera ■ The scene camera ● Pre-processed gaze data of each participant separately for each video ○ Filtered out the incorrect gaze locations [3] H. Hadizadeh, M. J. Enriquez, and I. V. Bajic, “Eye-tracking database for a set of standard video sequences,” IEEE Transactions on Image Processing, vol. 21, no. 2, pp. 898–903, 2011. 8IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 9. Evaluation - 1. Dynamic AOI Detection ● Four video sequences with dominant objects ○ Foreman ○ Bus ○ Mother and Daughter ○ Hall Monitor ● Apply an object detector using the RAEMAP and dynamically detect the AOIs ● Ground truth dataset ○ Manually annotating each video sequence BeaverDam video annotation tool 9IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2 Figure 3. Frames with AOIs, captured from manually annotated AOIs (Top Row) & dynamic AOIs detected from YOLOv3 (Bottom Row)
  • 10. Evaluation Metrics ● Intersection over Union (IoU) threshold - 0.5 ○ The predicted bounding box is True Positive if IoU ≥ 0.5 ○ The predicted bounding box is False Positive if IoU < 0.5 ○ Precision and Recall 10 ● Mean Average Precision (mAP) ○ COCO mAP - 101 point interpolated AP ○ Pascal VOC2008 mAP - 11 point interpolated AP IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 11. Results - 1. Dynamic AOI Detection 11IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 12. Evaluation - 2. Dynamic AOI-mapped Eye Movements 12IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 13. Results - 2. Dynamic AOI-mapped Eye Movements 13 Figure 4. Visualization of manually annotated AOIs (in blue), AOIs detected by an object detector (in red), the gaze positions of a participant in five consecutive frames (in green). IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 14. Results - 2. Dynamic AOI-mapped Eye Movements 14IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 15. Conclusions ● Limitation - We only used videos with dominant objects ● Computer vision methods could be incorporated for offline detection of dynamic AOIs in video streams ● Goal of our implementation and evaluation of object detectors was to integrate in the RAEMAP to filter eye movement data within dynamic AOIs ● Faster R-CNN object detector with ResNet-101-FPN backbone will be used as the object detector of proposed Dynamic AOI-Mapped Gaze Extraction Workflow of RAEMAP 15
  • 16. Future Work ● Evaluate our methodology using one-stage object detectors such as SSD [4], DSSD [5], and Retinanet [6]. ● Use segmentation instead of rectangular boundaries when defining dynamic AOIs for the extraction of eye movements. [4] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C.Berg, “Ssd: Single shot multibox detector,” in European conference on computer vision Springer, 2016, pp. 21–37. [5] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, “Dssd: Deconvolutional single shot detector,” arXiv preprint arXiv:1701.06659, 2017 [6] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dolĺar, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988. 16IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2
  • 17. Thank You! Questions? Gavindya Jayawardena gavindya@cs.odu.edu @Gavindya2 17IRI 2020 Automated Filtering of Eye Gaze Metrics from Dynamic Areas of Interest @Gavindya2