SlideShare a Scribd company logo
1© Fraunhofer IOSB
Eye Tracking, Gaze Analysis and Gaze-Based
Interaction
Jan Hendrik Hammer
Karlsruhe Institute of Technology, Germany
Evaluating Use and Impact
Workshop of the
Scottish Network on Digital Cultural Resources Evaluation,
Glasgow, United Kingdom,
March 31, 2016
Real-Time 3D Gaze Analysis in
Mobile Applications 2
Outline
Applications for eye tracking
Mobile and stationary eye tracking
From eye tracking to gaze analysis
What can we get from eye tracking? An example.
Further options for eye tracking
Real-Time 3D Gaze Analysis in
Mobile Applications 3
Applications for Eye Tracking
Assistive technology as interaction modality for people with
disabilities
User experience and interaction (interface design)
Marketing and consumer research
 Interaction with products
 Perception of advertisement
Compare viewing behaviour of experts and novices
 During sports
 During the perception of artworks
...
Real-Time 3D Gaze Analysis in
Mobile Applications 4
Mobile and Stationary
Devices
Real-Time 3D Gaze Analysis in
Mobile Applications 5
Stationary Devices
Small headbox
Only for gaze on displays
(mostly only one)
Prices:
 Device: 80 .. 40,000+ £
 Software:
 SDKs for free  programming
 Analysis software: 800 .. 8,000+ £
Tobii Pro TX300
(www.tobiipro.com)
SMI RED250
(www.smivision.com)
Smart eye pro
(http://smarteye.se)
SMI RED250Mobile
(www.smivision.com)
Tobii Pro X3-120
(www.tobiipro.com)
Aurora (http://smarteye.se)
Gazepoint GP3 (www.gazept.com)
Real-Time 3D Gaze Analysis in
Mobile Applications 6
Mobile Devices
Binocular eye tracking
Below 100 g
Hardware and live viewing:
1,800-10,000 £
Gaze analysis
 SDKs for free or software
(~8,000 £)
 Gaze registration  further costs
 Semi-automatic: gaze annotations
 Fully-automatic: pose estimation needed
Dikablis Professional
(www.ergoneers.com)
Tobii Pro Glasses 2
(www.tobiipro.com)
SMI ETG 2 Professional
(www.eyetracking-glasses.com)
Pupil labs eye tracker (https://pupil-labs.com)
Real-Time 3D Gaze Analysis in
Mobile Applications 7
Estimation of 3D Eye Ball Position
Pose estimation of scene camera
 Inside-out tracking
 Scene camera
 Marker tracking
 Markers distributed over the scene
 Distraction of visual attention
 Outside-in tracking
 External cameras + infrared light sources
 Infrared marker attached to device
 High accuracy
 Costs: 4,000 .. 150,000+ £
Eye tracker e.g. from Pupil Labs
(https://pupil-labs.com)
Marker attached in
scene
Infrared markers
attached to e.g. SMI ETG 2
(http://www.eyetracking-glasses.com)
Camera tracking system e.g. from ART
(http://www.ar-tracking.com)
Dikablis Professional
(www.ergoneers.com)
Real-Time 3D Gaze Analysis in
Mobile Applications 8
From Eye Tracking to Gaze
Analysis
Real-Time 3D Gaze Analysis in
Mobile Applications 9
Gaze Analysis Overview
Eye Tracking
• Line-of-sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations, saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas-of-interest (AOIs)
• visually most relevant objects
Real-Time 3D Gaze Analysis in
Mobile Applications 10
Gaze Analysis Overview
Eye Tracking
• Line-of-sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations, saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas-of-interest (AOIs)
• visually most relevant objects
Real-Time 3D Gaze Analysis in
Mobile Applications 11
 Nodal point of eye relative to scene camera
 Viewing direction
view cone
Eye Tracking Data for Line of Sight Reconstruction
Guestrin, E. D. D. , Eizenman, M. (2006) General
theory of remote gaze estimation using the pupil
center and corneal reflections. IEEE transactions on
bio-medical engineering, 53, 1124-1133
Real-Time 3D Gaze Analysis in
Mobile Applications 12
Estimation of 3D Eye Ball Position
Pose estimation of scene camera
 Inside-out tracking
 Scene camera
 Marker tracking
 Markers distributed over the scene
 Distraction of visual attention
 Outside-in tracking
 External cameras + infrared light sources
 Infrared marker attached to device
 High accuracy
 Costs: 4,000 .. 150,000+ £
Eye tracker e.g. from Pupil Labs
(https://pupil-labs.com)
Marker attached in
scene
Infrared markers
attached to e.g. SMI ETG 2
(http://www.eyetracking-glasses.com)
Camera tracking system e.g. from ART
(http://www.ar-tracking.com)
Dikablis Professional
(www.ergoneers.com)
Real-Time 3D Gaze Analysis in
Mobile Applications 13
3D Modell of Environment
Manual creation
 Very time consuming
 Only static scenes
Valencian Kitchen, National Museum of Decorative Arts
(Madrid, Ministry of Culture, Spain)
The Laboratory of Lavoisier, Musée des
arts et mètiers, (Paris, France)
SmartControlRoom, Fraunhofer IOSB
(Karlsruhe, Germany)
Real-Time 3D Gaze Analysis in
Mobile Applications 14
Line-of-Sight Reconstruction
 3D gaze point computation =
intersection line-of-sight with 3D
world
Real-Time 3D Gaze Analysis in
Mobile Applications 15
Gaze analysis overview
Eye Tracking
• Line of sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations , saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas of interest (AOIs)
• visually most relevant objects
Real-Time 3D Gaze Analysis in
Mobile Applications 16
3D Scanpath Visualization
 Gazepoints (violet):
7221 gaze points, 4 min 48 s 8820 gaze points, 5 min 52 s
Real-Time 3D Gaze Analysis in
Mobile Applications 17
3D Scanpath Visualization
 Fixations (≥100 ms, red):
7221 gaze points, 4 min 48 s 8820 gaze points, 5 min 52 s
Real-Time 3D Gaze Analysis in
Mobile Applications 18
3D Scanpath Visualization
 Fixations and saccades:
Real-Time 3D Gaze Analysis in
Mobile Applications 19
Gaze analysis overview
Eye Tracking
• Line of sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations, saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas of interest (AOIs)
• visually most relevant objects
Real-Time 3D Gaze Analysis in
Mobile Applications 20
Areas of Interest (AOIs)
Definition of AOIs Visualization of a hit AOI
Real-Time 3D Gaze Analysis in
Mobile Applications 21
Areas of Interest (AOIs)
Definition of AOIs Visualization of a hit AOI
Real-Time 3D Gaze Analysis in
Mobile Applications 22
Entry time + 1st fixation number
Duration of 1st fixation
Number of fixations on AOI
Cumulative fixation time
Dwell time
 Cumulative fixation time + cumulative duration of saccades on AOI
Revisits
 Number of saccades entering the AOI minus 1
AOI-based Metrics
Real-Time 3D Gaze Analysis in
Mobile Applications 23
What can we get from eye
tracking? An example.
Real-Time 3D Gaze Analysis in
Mobile Applications 24
Two types of experiments
 T1: Freely viewing artworks
 How do subjects look at the scene?
 What are the most attractive areas?
 T2: Freely viewing artworks while
listening to audio guide
 Do people follow the told stories?
Experiments in the EU-Project ARtSENSE
Real-Time 3D Gaze Analysis in
Mobile Applications 25
Different Scanning Behavior
 Normalized heat map (green to red)
Freely viewing (5 minutes) Listening to audio guide (6.5 minutes)
Real-Time 3D Gaze Analysis in
Mobile Applications 26
Different Scanning Behavior
Food and kitchen tools
Important persons and animals (part of audio guide stories)
Real-Time 3D Gaze Analysis in
Mobile Applications 27
Cumulative Fixation Time (CFT)
Food and kitchen tools
Important persons and animals (part of audio guide stories)
CFT = 45 seconds (41 %)
CFT = 64 seconds (59 %)
CFT = 30 seconds (14%)
CFT = 181 seconds (86 %)
Real-Time 3D Gaze Analysis in
Mobile Applications 28
Scanpath Comparison
 Scanpath 1: Freely viewing
 Scanpath 2: Listening to audio guide
Real-Time 3D Gaze Analysis in
Mobile Applications 30
Live Computation of Visually Most Relevant Objects
in Small Time Windows
0
625
532
751
328
0 200 400 600 800 1000
tablet3
tablet2
servant2
tablet1
servant1
Time/ms
CFT in 4 s Time Window at Time t
766
469
469
548
250
0 200 400 600 800 1000
tablet3
tablet2
servant2
tablet1
servant1
Time/ms
CFT in 4 s Time Window at Time t + 1 s
 Time window [t - 4 s, t]
 Time window [t - 3 s, t + 1 s]
 Basis for recommendations in AR Glasses
tablet1
tablet2
tablet3
servant1 servant2
Real-Time 3D Gaze Analysis in
Mobile Applications 31
Gaze-Based Interaction
Real-Time 3D Gaze Analysis in
Mobile Applications 32
Stationary Gaze Key Press
 Video stream selection using gaze key press
Real-Time 3D Gaze Analysis in
Mobile Applications 33
Mobile Gaze Key Press – First tests
Selection by pointing
Real-Time 3D Gaze Analysis in
Mobile Applications 34
Mobile Gaze Key Press – First tests
Pointing (26.5 sec) Mobile gaze key press (16 sec, ≈3/5)
Real-Time 3D Gaze Analysis in
Mobile Applications 35
Further Interesting Stuff
Stationary eye tracking via webcam (www.eyezag.de)
 No eye tracker needed
 Subjects can participate from all over the world
Real-Time 3D Gaze Analysis in
Mobile Applications 36
Further Interesting Stuff
Eye tracking in virtual reality
 You can walk around and the immersion
is great!
 Cheap pose estimation for 3D eye
tracking
 Modeling of 3D world needed
 Eye tracking hardware and analysis
software not available for all VR devices
– but in probably in the future
HTC Vive
(www.htcvive.com)
Real-Time 3D Gaze Analysis in
Mobile Applications 37
Summary
What do you want the people to look at?
 Images on display  stationary eye tracking
 Large volumes  mobile eye tracking
Workflow: Eye tracking  gaze movement computation  gaze
analysis
Gaze-based interaction (gaze key press)
Alternatives to usual eye tracking and eye tracking in VR
Real-Time 3D Gaze Analysis in
Mobile Applications 38
Thank you for your
attention!
Questions?
Real-Time 3D Gaze Analysis in
Mobile Applications 39
Contribution
1st fully automated real-time capable 3D gaze analysis for
mobile applications
 Publication: Jan Hendrik Hammer, Michael Maurus, Jürgen Beyerer.
Real-time 3D gaze analysis in mobile applications. In: Proceedings
of the 2013 Conference on Eye Tracking South Africa, ETSA '13, S.
75-78, ACM, August 2013.
Real-Time 3D Gaze Analysis in
Mobile Applications 40
Realistic Heatmap Generation in 3D Environments
Visual acuity decreases with
deviation from visual axis
 Projected Gaussian
Projected Gaussian Occlusion test
Real-Time 3D Gaze Analysis in
Mobile Applications 41
Contribution
1st method for realistic heatmap visualization of gaze data
in 3D environments
 Publication: Michael Maurus, Jan Hendrik Hammer, Jürgen Beyerer.
Realistic Heatmap Visualization for Interactive Analysis of 3D Gaze
Data. In: Proceedings of the Symposium on Eye Tracking Research
and Applications, S. 295--298, ACM, März 2014.
Real-Time 3D Gaze Analysis in
Mobile Applications 42
Real-Time 3D Gaze Analysis in
Mobile Applications 43
Real-Time 3D Gaze Analysis in
Mobile Applications 44
Real-Time 3D Gaze Analysis in
Mobile Applications 45
Motivation: Adaptive AR Museum Guide
1. Freely viewing artwork
3. Recommendation
4. Gesture interaction
2. Detection of visually
most relevant objects

More Related Content

What's hot

Augmented Human: AR 4.0 Beyond Time and Space
Augmented Human: AR 4.0 Beyond Time and SpaceAugmented Human: AR 4.0 Beyond Time and Space
Augmented Human: AR 4.0 Beyond Time and Space
Woontack Woo
 
A Short Introduction to Computer Vision-based Marker Tracking
A Short Introduction to  Computer Vision-based  Marker TrackingA Short Introduction to  Computer Vision-based  Marker Tracking
A Short Introduction to Computer Vision-based Marker Tracking
Jens Grubert
 
Human activity recognition
Human activity recognition Human activity recognition
Human activity recognition srikanthgadam
 
„Emotional Flowers“ User Centered Game Design
„Emotional Flowers“ User Centered Game Design„Emotional Flowers“ User Centered Game Design
„Emotional Flowers“ User Centered Game DesignMartin Ortner
 
Human Activity Recognition using Smartphone's sensor
Human Activity Recognition using Smartphone's sensor Human Activity Recognition using Smartphone's sensor
Human Activity Recognition using Smartphone's sensor
Pankaj Mishra
 
Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...
Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...
Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...
Bennett King
 
Godeye An Efficient System for Blinds
Godeye An Efficient System for BlindsGodeye An Efficient System for Blinds
Godeye An Efficient System for Blinds
ijtsrd
 
Keynote at VR in Science and Industry
Keynote at VR in Science and Industry Keynote at VR in Science and Industry
Keynote at VR in Science and Industry
Christian Sandor
 
Tridelity
TridelityTridelity
Tridelity
Tradelity
 
Sixth sense technology
Sixth sense technology Sixth sense technology
Sixth sense technology
Pulkit Singhal
 
Eye Tracking & Design
Eye Tracking & DesignEye Tracking & Design
Eye Tracking & Design
Optimal Usability
 
IRJET - Smart E – Cane for the Visually Challenged and Blind using ML Con...
IRJET -  	  Smart E – Cane for the Visually Challenged and Blind using ML Con...IRJET -  	  Smart E – Cane for the Visually Challenged and Blind using ML Con...
IRJET - Smart E – Cane for the Visually Challenged and Blind using ML Con...
IRJET Journal
 
Magic Leap Augmented Reality Strategy Insights from Patents
Magic Leap Augmented Reality Strategy Insights from PatentsMagic Leap Augmented Reality Strategy Insights from Patents
Magic Leap Augmented Reality Strategy Insights from Patents
Alex G. Lee, Ph.D. Esq. CLP
 
3D printing weekly update - 8.17.15
3D printing weekly update - 8.17.153D printing weekly update - 8.17.15
3D printing weekly update - 8.17.15
Sandra Kahn
 
Andrew Myers portfolio
Andrew Myers portfolioAndrew Myers portfolio
Andrew Myers portfolio
AndrewMyers76
 
The Sensor Web - New Opportunities for MediaMixing
The Sensor Web - New Opportunities for MediaMixingThe Sensor Web - New Opportunities for MediaMixing
The Sensor Web - New Opportunities for MediaMixingMediaMixerCommunity
 
Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...
Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...
Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...
Kalle
 
IRJET-Unconstraint Eye Tracking on Mobile Smartphone
IRJET-Unconstraint Eye Tracking on Mobile SmartphoneIRJET-Unconstraint Eye Tracking on Mobile Smartphone
IRJET-Unconstraint Eye Tracking on Mobile Smartphone
IRJET Journal
 
Sixth sense technology documentation
Sixth sense technology documentationSixth sense technology documentation
Sixth sense technology documentation
PrashanthBeemanathi
 

What's hot (19)

Augmented Human: AR 4.0 Beyond Time and Space
Augmented Human: AR 4.0 Beyond Time and SpaceAugmented Human: AR 4.0 Beyond Time and Space
Augmented Human: AR 4.0 Beyond Time and Space
 
A Short Introduction to Computer Vision-based Marker Tracking
A Short Introduction to  Computer Vision-based  Marker TrackingA Short Introduction to  Computer Vision-based  Marker Tracking
A Short Introduction to Computer Vision-based Marker Tracking
 
Human activity recognition
Human activity recognition Human activity recognition
Human activity recognition
 
„Emotional Flowers“ User Centered Game Design
„Emotional Flowers“ User Centered Game Design„Emotional Flowers“ User Centered Game Design
„Emotional Flowers“ User Centered Game Design
 
Human Activity Recognition using Smartphone's sensor
Human Activity Recognition using Smartphone's sensor Human Activity Recognition using Smartphone's sensor
Human Activity Recognition using Smartphone's sensor
 
Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...
Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...
Cyborg Design: Multimodal Interactions, Information, and Environments for Wea...
 
Godeye An Efficient System for Blinds
Godeye An Efficient System for BlindsGodeye An Efficient System for Blinds
Godeye An Efficient System for Blinds
 
Keynote at VR in Science and Industry
Keynote at VR in Science and Industry Keynote at VR in Science and Industry
Keynote at VR in Science and Industry
 
Tridelity
TridelityTridelity
Tridelity
 
Sixth sense technology
Sixth sense technology Sixth sense technology
Sixth sense technology
 
Eye Tracking & Design
Eye Tracking & DesignEye Tracking & Design
Eye Tracking & Design
 
IRJET - Smart E – Cane for the Visually Challenged and Blind using ML Con...
IRJET -  	  Smart E – Cane for the Visually Challenged and Blind using ML Con...IRJET -  	  Smart E – Cane for the Visually Challenged and Blind using ML Con...
IRJET - Smart E – Cane for the Visually Challenged and Blind using ML Con...
 
Magic Leap Augmented Reality Strategy Insights from Patents
Magic Leap Augmented Reality Strategy Insights from PatentsMagic Leap Augmented Reality Strategy Insights from Patents
Magic Leap Augmented Reality Strategy Insights from Patents
 
3D printing weekly update - 8.17.15
3D printing weekly update - 8.17.153D printing weekly update - 8.17.15
3D printing weekly update - 8.17.15
 
Andrew Myers portfolio
Andrew Myers portfolioAndrew Myers portfolio
Andrew Myers portfolio
 
The Sensor Web - New Opportunities for MediaMixing
The Sensor Web - New Opportunities for MediaMixingThe Sensor Web - New Opportunities for MediaMixing
The Sensor Web - New Opportunities for MediaMixing
 
Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...
Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...
Kohlbecker Low Latency Combined Eye And Head Tracking System For Teleoperatin...
 
IRJET-Unconstraint Eye Tracking on Mobile Smartphone
IRJET-Unconstraint Eye Tracking on Mobile SmartphoneIRJET-Unconstraint Eye Tracking on Mobile Smartphone
IRJET-Unconstraint Eye Tracking on Mobile Smartphone
 
Sixth sense technology documentation
Sixth sense technology documentationSixth sense technology documentation
Sixth sense technology documentation
 

Viewers also liked

Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...
Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...
Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...
scotdigich
 
eyegaze communication system
eyegaze communication system eyegaze communication system
eyegaze communication system
Manasa Reddy
 
Eye tracking and its economic feasibility
Eye tracking and its economic feasibilityEye tracking and its economic feasibility
Eye tracking and its economic feasibility
Jeffrey Funk
 
how to use the virtual keyboard
how to use the virtual keyboardhow to use the virtual keyboard
how to use the virtual keyboard
Sri Rakesh
 
Blue brain
Blue brainBlue brain
Blue brain
RJREJESH
 
Eye Mouse
Eye MouseEye Mouse
Eye Mouse
Amir Al-Ansary
 
The eye-gaze-communication-system-1.doc(updated)
The eye-gaze-communication-system-1.doc(updated)The eye-gaze-communication-system-1.doc(updated)
The eye-gaze-communication-system-1.doc(updated)
NIRAJ KUMAR
 
Virtual keyboard
Virtual keyboardVirtual keyboard
Virtual keyboard
siddhantranjan
 
Virtual keyboard
Virtual keyboardVirtual keyboard
Virtual keyboard
Stephen Raj
 
Blue brain document
Blue brain documentBlue brain document
Blue brain documentkoustuba
 
Blue brain seminar by bhaumik patel
Blue brain seminar by bhaumik patelBlue brain seminar by bhaumik patel
Blue brain seminar by bhaumik patelbhaumikpatel004
 
Blue brain by MAYANK SAHU
Blue brain by MAYANK SAHUBlue brain by MAYANK SAHU
Blue brain by MAYANK SAHU
mayank843
 
Blue brain
Blue brain Blue brain
Blue brain
Aurobindo Nayak
 
Eye gaze communication
Eye gaze communicationEye gaze communication
Eye gaze communication
PRADEEP Cheekatla
 
Best Ever PPT Of Bluebrain
Best Ever PPT Of BluebrainBest Ever PPT Of Bluebrain
Best Ever PPT Of Bluebrain
Prakash Thulaseedharan
 
Blue brain project ppt
Blue brain project pptBlue brain project ppt
Blue brain project pptLishita Shah
 
EYE GAZE COMMUNICATION by njn
EYE GAZE COMMUNICATION by njnEYE GAZE COMMUNICATION by njn
EYE GAZE COMMUNICATION by njn
Nijin Nijin
 
Eye Movement based Human Computer Interaction Technique
Eye Movement based Human Computer Interaction TechniqueEye Movement based Human Computer Interaction Technique
Eye Movement based Human Computer Interaction Technique
Jobin George
 

Viewers also liked (20)

Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...
Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...
Areti Galani Newcastle 'Experience driven evaluation' Scottish Network of Dig...
 
eyegaze communication system
eyegaze communication system eyegaze communication system
eyegaze communication system
 
Eye tracking and its economic feasibility
Eye tracking and its economic feasibilityEye tracking and its economic feasibility
Eye tracking and its economic feasibility
 
how to use the virtual keyboard
how to use the virtual keyboardhow to use the virtual keyboard
how to use the virtual keyboard
 
Blue brain
Blue brainBlue brain
Blue brain
 
Eye Mouse
Eye MouseEye Mouse
Eye Mouse
 
The eye-gaze-communication-system-1.doc(updated)
The eye-gaze-communication-system-1.doc(updated)The eye-gaze-communication-system-1.doc(updated)
The eye-gaze-communication-system-1.doc(updated)
 
Virtual keyboard
Virtual keyboardVirtual keyboard
Virtual keyboard
 
Business model canvas
Business model canvasBusiness model canvas
Business model canvas
 
Virtual keyboard
Virtual keyboardVirtual keyboard
Virtual keyboard
 
Blue brain document
Blue brain documentBlue brain document
Blue brain document
 
Blue brain seminar by bhaumik patel
Blue brain seminar by bhaumik patelBlue brain seminar by bhaumik patel
Blue brain seminar by bhaumik patel
 
Blue brain by MAYANK SAHU
Blue brain by MAYANK SAHUBlue brain by MAYANK SAHU
Blue brain by MAYANK SAHU
 
Blue brain
Blue brain Blue brain
Blue brain
 
Eye gaze communication
Eye gaze communicationEye gaze communication
Eye gaze communication
 
Best Ever PPT Of Bluebrain
Best Ever PPT Of BluebrainBest Ever PPT Of Bluebrain
Best Ever PPT Of Bluebrain
 
Blue brain project ppt
Blue brain project pptBlue brain project ppt
Blue brain project ppt
 
EYE GAZE COMMUNICATION by njn
EYE GAZE COMMUNICATION by njnEYE GAZE COMMUNICATION by njn
EYE GAZE COMMUNICATION by njn
 
Blue Brain
Blue Brain Blue Brain
Blue Brain
 
Eye Movement based Human Computer Interaction Technique
Eye Movement based Human Computer Interaction TechniqueEye Movement based Human Computer Interaction Technique
Eye Movement based Human Computer Interaction Technique
 

Similar to Jan Hendrik Hammer, Fraunhofer, KIT, Eyetracking and Gaze Analysis

Augmented reality(my ppt)
Augmented reality(my ppt)Augmented reality(my ppt)
Augmented reality(my ppt)Srilakshmi Alla
 
eng.pptx
eng.pptxeng.pptx
eng.pptx
Zuine
 
Mobile eyetracking voor_uxd_testing
Mobile eyetracking voor_uxd_testingMobile eyetracking voor_uxd_testing
Mobile eyetracking voor_uxd_testingMonkeyshot
 
Seminar presentation
Seminar presentationSeminar presentation
Seminar presentation
Brij Kishore
 
Virtual Reality
Virtual RealityVirtual Reality
Virtual Reality
SubhradeepMaji
 
Artec english- 2011
Artec  english- 2011Artec  english- 2011
Artec english- 2011
pmorillo
 
Eye Tracking Technologies: VDC-Whitepaper
Eye Tracking Technologies: VDC-WhitepaperEye Tracking Technologies: VDC-Whitepaper
Eye Tracking Technologies: VDC-Whitepaper
Virtual Dimension Center (VDC) Fellbach
 
Mar general pan hui ppt
Mar general pan hui ppt Mar general pan hui ppt
Mar general pan hui ppt
luruiyang
 
Sensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle Process
Sensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle ProcessSensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle Process
Sensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle Process
Shivam Gupta
 
3 d body scanning
3 d body scanning3 d body scanning
3 d body scanningArka Das
 
Augmented Reality - the next big thing in mobile
Augmented Reality - the next big thing in mobileAugmented Reality - the next big thing in mobile
Augmented Reality - the next big thing in mobile
Hari Gottipati
 
ACVT Capabilities Show
ACVT Capabilities ShowACVT Capabilities Show
Mobile Augmented Reality
Mobile Augmented RealityMobile Augmented Reality
Mobile Augmented Reality
Marios Bikos
 
Immersive technologies.pptx
Immersive technologies.pptxImmersive technologies.pptx
Immersive technologies.pptx
AnandSri5
 
Augmented reality
Augmented realityAugmented reality
Augmented reality
Nitin Meena
 
COSC 426 Lecture 1: Introduction to Augmented Reality
COSC 426 Lecture 1: Introduction to Augmented RealityCOSC 426 Lecture 1: Introduction to Augmented Reality
COSC 426 Lecture 1: Introduction to Augmented Reality
Mark Billinghurst
 
Augmented Reality
Augmented RealityAugmented Reality
Augmented Reality
Md. Al-Faran Hasan Sadat
 
IRJET- Smart Mirror using Eye Gaze Tracking
IRJET- Smart Mirror using Eye Gaze TrackingIRJET- Smart Mirror using Eye Gaze Tracking
IRJET- Smart Mirror using Eye Gaze Tracking
IRJET Journal
 
virtual reality Barkha manral seminar on augmented reality.ppt
virtual reality Barkha manral seminar on augmented reality.pptvirtual reality Barkha manral seminar on augmented reality.ppt
virtual reality Barkha manral seminar on augmented reality.ppt
Barkha Manral
 
Augmented reality
Augmented realityAugmented reality
Augmented reality
Narendra kumar Jha
 

Similar to Jan Hendrik Hammer, Fraunhofer, KIT, Eyetracking and Gaze Analysis (20)

Augmented reality(my ppt)
Augmented reality(my ppt)Augmented reality(my ppt)
Augmented reality(my ppt)
 
eng.pptx
eng.pptxeng.pptx
eng.pptx
 
Mobile eyetracking voor_uxd_testing
Mobile eyetracking voor_uxd_testingMobile eyetracking voor_uxd_testing
Mobile eyetracking voor_uxd_testing
 
Seminar presentation
Seminar presentationSeminar presentation
Seminar presentation
 
Virtual Reality
Virtual RealityVirtual Reality
Virtual Reality
 
Artec english- 2011
Artec  english- 2011Artec  english- 2011
Artec english- 2011
 
Eye Tracking Technologies: VDC-Whitepaper
Eye Tracking Technologies: VDC-WhitepaperEye Tracking Technologies: VDC-Whitepaper
Eye Tracking Technologies: VDC-Whitepaper
 
Mar general pan hui ppt
Mar general pan hui ppt Mar general pan hui ppt
Mar general pan hui ppt
 
Sensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle Process
Sensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle ProcessSensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle Process
Sensor | Augment Reality | Virtual Reality | Google Tesser-OCR | Recycle Process
 
3 d body scanning
3 d body scanning3 d body scanning
3 d body scanning
 
Augmented Reality - the next big thing in mobile
Augmented Reality - the next big thing in mobileAugmented Reality - the next big thing in mobile
Augmented Reality - the next big thing in mobile
 
ACVT Capabilities Show
ACVT Capabilities ShowACVT Capabilities Show
ACVT Capabilities Show
 
Mobile Augmented Reality
Mobile Augmented RealityMobile Augmented Reality
Mobile Augmented Reality
 
Immersive technologies.pptx
Immersive technologies.pptxImmersive technologies.pptx
Immersive technologies.pptx
 
Augmented reality
Augmented realityAugmented reality
Augmented reality
 
COSC 426 Lecture 1: Introduction to Augmented Reality
COSC 426 Lecture 1: Introduction to Augmented RealityCOSC 426 Lecture 1: Introduction to Augmented Reality
COSC 426 Lecture 1: Introduction to Augmented Reality
 
Augmented Reality
Augmented RealityAugmented Reality
Augmented Reality
 
IRJET- Smart Mirror using Eye Gaze Tracking
IRJET- Smart Mirror using Eye Gaze TrackingIRJET- Smart Mirror using Eye Gaze Tracking
IRJET- Smart Mirror using Eye Gaze Tracking
 
virtual reality Barkha manral seminar on augmented reality.ppt
virtual reality Barkha manral seminar on augmented reality.pptvirtual reality Barkha manral seminar on augmented reality.ppt
virtual reality Barkha manral seminar on augmented reality.ppt
 
Augmented reality
Augmented realityAugmented reality
Augmented reality
 

Recently uploaded

Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
CatarinaPereira64715
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 

Recently uploaded (20)

Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 

Jan Hendrik Hammer, Fraunhofer, KIT, Eyetracking and Gaze Analysis

  • 1. 1© Fraunhofer IOSB Eye Tracking, Gaze Analysis and Gaze-Based Interaction Jan Hendrik Hammer Karlsruhe Institute of Technology, Germany Evaluating Use and Impact Workshop of the Scottish Network on Digital Cultural Resources Evaluation, Glasgow, United Kingdom, March 31, 2016
  • 2. Real-Time 3D Gaze Analysis in Mobile Applications 2 Outline Applications for eye tracking Mobile and stationary eye tracking From eye tracking to gaze analysis What can we get from eye tracking? An example. Further options for eye tracking
  • 3. Real-Time 3D Gaze Analysis in Mobile Applications 3 Applications for Eye Tracking Assistive technology as interaction modality for people with disabilities User experience and interaction (interface design) Marketing and consumer research  Interaction with products  Perception of advertisement Compare viewing behaviour of experts and novices  During sports  During the perception of artworks ...
  • 4. Real-Time 3D Gaze Analysis in Mobile Applications 4 Mobile and Stationary Devices
  • 5. Real-Time 3D Gaze Analysis in Mobile Applications 5 Stationary Devices Small headbox Only for gaze on displays (mostly only one) Prices:  Device: 80 .. 40,000+ £  Software:  SDKs for free  programming  Analysis software: 800 .. 8,000+ £ Tobii Pro TX300 (www.tobiipro.com) SMI RED250 (www.smivision.com) Smart eye pro (http://smarteye.se) SMI RED250Mobile (www.smivision.com) Tobii Pro X3-120 (www.tobiipro.com) Aurora (http://smarteye.se) Gazepoint GP3 (www.gazept.com)
  • 6. Real-Time 3D Gaze Analysis in Mobile Applications 6 Mobile Devices Binocular eye tracking Below 100 g Hardware and live viewing: 1,800-10,000 £ Gaze analysis  SDKs for free or software (~8,000 £)  Gaze registration  further costs  Semi-automatic: gaze annotations  Fully-automatic: pose estimation needed Dikablis Professional (www.ergoneers.com) Tobii Pro Glasses 2 (www.tobiipro.com) SMI ETG 2 Professional (www.eyetracking-glasses.com) Pupil labs eye tracker (https://pupil-labs.com)
  • 7. Real-Time 3D Gaze Analysis in Mobile Applications 7 Estimation of 3D Eye Ball Position Pose estimation of scene camera  Inside-out tracking  Scene camera  Marker tracking  Markers distributed over the scene  Distraction of visual attention  Outside-in tracking  External cameras + infrared light sources  Infrared marker attached to device  High accuracy  Costs: 4,000 .. 150,000+ £ Eye tracker e.g. from Pupil Labs (https://pupil-labs.com) Marker attached in scene Infrared markers attached to e.g. SMI ETG 2 (http://www.eyetracking-glasses.com) Camera tracking system e.g. from ART (http://www.ar-tracking.com) Dikablis Professional (www.ergoneers.com)
  • 8. Real-Time 3D Gaze Analysis in Mobile Applications 8 From Eye Tracking to Gaze Analysis
  • 9. Real-Time 3D Gaze Analysis in Mobile Applications 9 Gaze Analysis Overview Eye Tracking • Line-of-sight reconstruction • Gaze point computation Gaze movement computation • Fixations, saccades, smooth pursuits, ... Gaze analysis • Gaze metrics • Areas-of-interest (AOIs) • visually most relevant objects
  • 10. Real-Time 3D Gaze Analysis in Mobile Applications 10 Gaze Analysis Overview Eye Tracking • Line-of-sight reconstruction • Gaze point computation Gaze movement computation • Fixations, saccades, smooth pursuits, ... Gaze analysis • Gaze metrics • Areas-of-interest (AOIs) • visually most relevant objects
  • 11. Real-Time 3D Gaze Analysis in Mobile Applications 11  Nodal point of eye relative to scene camera  Viewing direction view cone Eye Tracking Data for Line of Sight Reconstruction Guestrin, E. D. D. , Eizenman, M. (2006) General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE transactions on bio-medical engineering, 53, 1124-1133
  • 12. Real-Time 3D Gaze Analysis in Mobile Applications 12 Estimation of 3D Eye Ball Position Pose estimation of scene camera  Inside-out tracking  Scene camera  Marker tracking  Markers distributed over the scene  Distraction of visual attention  Outside-in tracking  External cameras + infrared light sources  Infrared marker attached to device  High accuracy  Costs: 4,000 .. 150,000+ £ Eye tracker e.g. from Pupil Labs (https://pupil-labs.com) Marker attached in scene Infrared markers attached to e.g. SMI ETG 2 (http://www.eyetracking-glasses.com) Camera tracking system e.g. from ART (http://www.ar-tracking.com) Dikablis Professional (www.ergoneers.com)
  • 13. Real-Time 3D Gaze Analysis in Mobile Applications 13 3D Modell of Environment Manual creation  Very time consuming  Only static scenes Valencian Kitchen, National Museum of Decorative Arts (Madrid, Ministry of Culture, Spain) The Laboratory of Lavoisier, Musée des arts et mètiers, (Paris, France) SmartControlRoom, Fraunhofer IOSB (Karlsruhe, Germany)
  • 14. Real-Time 3D Gaze Analysis in Mobile Applications 14 Line-of-Sight Reconstruction  3D gaze point computation = intersection line-of-sight with 3D world
  • 15. Real-Time 3D Gaze Analysis in Mobile Applications 15 Gaze analysis overview Eye Tracking • Line of sight reconstruction • Gaze point computation Gaze movement computation • Fixations , saccades, smooth pursuits, ... Gaze analysis • Gaze metrics • Areas of interest (AOIs) • visually most relevant objects
  • 16. Real-Time 3D Gaze Analysis in Mobile Applications 16 3D Scanpath Visualization  Gazepoints (violet): 7221 gaze points, 4 min 48 s 8820 gaze points, 5 min 52 s
  • 17. Real-Time 3D Gaze Analysis in Mobile Applications 17 3D Scanpath Visualization  Fixations (≥100 ms, red): 7221 gaze points, 4 min 48 s 8820 gaze points, 5 min 52 s
  • 18. Real-Time 3D Gaze Analysis in Mobile Applications 18 3D Scanpath Visualization  Fixations and saccades:
  • 19. Real-Time 3D Gaze Analysis in Mobile Applications 19 Gaze analysis overview Eye Tracking • Line of sight reconstruction • Gaze point computation Gaze movement computation • Fixations, saccades, smooth pursuits, ... Gaze analysis • Gaze metrics • Areas of interest (AOIs) • visually most relevant objects
  • 20. Real-Time 3D Gaze Analysis in Mobile Applications 20 Areas of Interest (AOIs) Definition of AOIs Visualization of a hit AOI
  • 21. Real-Time 3D Gaze Analysis in Mobile Applications 21 Areas of Interest (AOIs) Definition of AOIs Visualization of a hit AOI
  • 22. Real-Time 3D Gaze Analysis in Mobile Applications 22 Entry time + 1st fixation number Duration of 1st fixation Number of fixations on AOI Cumulative fixation time Dwell time  Cumulative fixation time + cumulative duration of saccades on AOI Revisits  Number of saccades entering the AOI minus 1 AOI-based Metrics
  • 23. Real-Time 3D Gaze Analysis in Mobile Applications 23 What can we get from eye tracking? An example.
  • 24. Real-Time 3D Gaze Analysis in Mobile Applications 24 Two types of experiments  T1: Freely viewing artworks  How do subjects look at the scene?  What are the most attractive areas?  T2: Freely viewing artworks while listening to audio guide  Do people follow the told stories? Experiments in the EU-Project ARtSENSE
  • 25. Real-Time 3D Gaze Analysis in Mobile Applications 25 Different Scanning Behavior  Normalized heat map (green to red) Freely viewing (5 minutes) Listening to audio guide (6.5 minutes)
  • 26. Real-Time 3D Gaze Analysis in Mobile Applications 26 Different Scanning Behavior Food and kitchen tools Important persons and animals (part of audio guide stories)
  • 27. Real-Time 3D Gaze Analysis in Mobile Applications 27 Cumulative Fixation Time (CFT) Food and kitchen tools Important persons and animals (part of audio guide stories) CFT = 45 seconds (41 %) CFT = 64 seconds (59 %) CFT = 30 seconds (14%) CFT = 181 seconds (86 %)
  • 28. Real-Time 3D Gaze Analysis in Mobile Applications 28 Scanpath Comparison  Scanpath 1: Freely viewing  Scanpath 2: Listening to audio guide
  • 29. Real-Time 3D Gaze Analysis in Mobile Applications 30 Live Computation of Visually Most Relevant Objects in Small Time Windows 0 625 532 751 328 0 200 400 600 800 1000 tablet3 tablet2 servant2 tablet1 servant1 Time/ms CFT in 4 s Time Window at Time t 766 469 469 548 250 0 200 400 600 800 1000 tablet3 tablet2 servant2 tablet1 servant1 Time/ms CFT in 4 s Time Window at Time t + 1 s  Time window [t - 4 s, t]  Time window [t - 3 s, t + 1 s]  Basis for recommendations in AR Glasses tablet1 tablet2 tablet3 servant1 servant2
  • 30. Real-Time 3D Gaze Analysis in Mobile Applications 31 Gaze-Based Interaction
  • 31. Real-Time 3D Gaze Analysis in Mobile Applications 32 Stationary Gaze Key Press  Video stream selection using gaze key press
  • 32. Real-Time 3D Gaze Analysis in Mobile Applications 33 Mobile Gaze Key Press – First tests Selection by pointing
  • 33. Real-Time 3D Gaze Analysis in Mobile Applications 34 Mobile Gaze Key Press – First tests Pointing (26.5 sec) Mobile gaze key press (16 sec, ≈3/5)
  • 34. Real-Time 3D Gaze Analysis in Mobile Applications 35 Further Interesting Stuff Stationary eye tracking via webcam (www.eyezag.de)  No eye tracker needed  Subjects can participate from all over the world
  • 35. Real-Time 3D Gaze Analysis in Mobile Applications 36 Further Interesting Stuff Eye tracking in virtual reality  You can walk around and the immersion is great!  Cheap pose estimation for 3D eye tracking  Modeling of 3D world needed  Eye tracking hardware and analysis software not available for all VR devices – but in probably in the future HTC Vive (www.htcvive.com)
  • 36. Real-Time 3D Gaze Analysis in Mobile Applications 37 Summary What do you want the people to look at?  Images on display  stationary eye tracking  Large volumes  mobile eye tracking Workflow: Eye tracking  gaze movement computation  gaze analysis Gaze-based interaction (gaze key press) Alternatives to usual eye tracking and eye tracking in VR
  • 37. Real-Time 3D Gaze Analysis in Mobile Applications 38 Thank you for your attention! Questions?
  • 38. Real-Time 3D Gaze Analysis in Mobile Applications 39 Contribution 1st fully automated real-time capable 3D gaze analysis for mobile applications  Publication: Jan Hendrik Hammer, Michael Maurus, Jürgen Beyerer. Real-time 3D gaze analysis in mobile applications. In: Proceedings of the 2013 Conference on Eye Tracking South Africa, ETSA '13, S. 75-78, ACM, August 2013.
  • 39. Real-Time 3D Gaze Analysis in Mobile Applications 40 Realistic Heatmap Generation in 3D Environments Visual acuity decreases with deviation from visual axis  Projected Gaussian Projected Gaussian Occlusion test
  • 40. Real-Time 3D Gaze Analysis in Mobile Applications 41 Contribution 1st method for realistic heatmap visualization of gaze data in 3D environments  Publication: Michael Maurus, Jan Hendrik Hammer, Jürgen Beyerer. Realistic Heatmap Visualization for Interactive Analysis of 3D Gaze Data. In: Proceedings of the Symposium on Eye Tracking Research and Applications, S. 295--298, ACM, März 2014.
  • 41. Real-Time 3D Gaze Analysis in Mobile Applications 42
  • 42. Real-Time 3D Gaze Analysis in Mobile Applications 43
  • 43. Real-Time 3D Gaze Analysis in Mobile Applications 44
  • 44. Real-Time 3D Gaze Analysis in Mobile Applications 45 Motivation: Adaptive AR Museum Guide 1. Freely viewing artwork 3. Recommendation 4. Gesture interaction 2. Detection of visually most relevant objects

Editor's Notes

  1. Eyetrackers.net/en/buy SMI: iView X READ 18900 € ETG 2 9.900 € RED m 14.900 € Experiment Suit 360: 4.900 € Gazepoint (http://www.gazept.com/shop/): GP3 Eye Tracker: $ 495 Gazepoint Analysis Professional Edition Software $995.00 Tobii: Tobii Pro Analystics SDK: free and for all their screen based eye trackers Tobii Pro Studio: Basic: 4700 $, Profession: 8900$, Enterprise: 12900$ Tobii X60 29,900 $ Tobii Pro X3-120: ca. 100 € Smart eye pro: Smart eye Aurora:
  2. Pubil: 2300 € Tobii pro glasses 2 live vieweing: 15000 $ Tobii pro glasses 2 incl. Analysis software 25000 $ ETG Analysis Pro (60 Hz Glasses, Data Recorder, Analysis Software): ~ 30,000 $
  3. Outside tracking: Costs: 5,000 .. 200,000 $+
  4. Eye tracking solves the question of where your eyes are directed to in the given scene. This is NOT what you are looking at because “looking” involves some kind processing in our brains. Eye tracking is the processing of the raw data from the used device to reconstruct the line of sight for gaze point computation. What you are looking at is more part of the gaze movement computation. Our gaze path consists of different gaze movements. The two most important ones, I want you to remember are fixations and saccades. Fixations are periods during which our gaze remains still at some point and we process what we see. Fixations range between 60 ms and somewhere below 2 seconds. Between fixations, the points we look at, our gaze jumps and these ballistic movements are called saccades. Using these gaze path we can then analyse the gaze using different gaze metrics and AOIs, so called areas of interest, to e.g. infer the visually most relevant objects of a scene.
  5. Geometrical model for computation of the viewing direction
  6. Outside tracking: Costs: 5,000 .. 200,000 €+
  7. Definition of AOIs in Valencian Kitchen Hit AOIs are highlighted in realtime
  8. Metrics for attractiveness of an AOI
  9. Duke of Wellington
  10. audio guides – tablets -> further information HMD -> figure out where people look Scanpath -> tartlets System determines tartlets as visually most relevant objects -> recommendation system Question -> Explicit interaction by hand gestures Today - implicit gaze interaction