Jan Hendrik Hammer, Fraunhofer, Karlsruhe Institute of Technology, Germany, "Eyetracking, Gaze Analysis and Gaze-based interaction. Paper presented at the 3d workshop 'Evaluating Use and Impact' of the Scottish Network of Digital Cultural Heritage Resources Evaluation, Glasgow, 31 March 2016

1. 1© Fraunhofer IOSB
Eye Tracking, Gaze Analysis and Gaze-Based
Interaction
Jan Hendrik Hammer
Karlsruhe Institute of Technology, Germany
Evaluating Use and Impact
Workshop of the
Scottish Network on Digital Cultural Resources Evaluation,
Glasgow, United Kingdom,
March 31, 2016

2. Real-Time 3D Gaze Analysis in
Mobile Applications 2
Outline
Applications for eye tracking
Mobile and stationary eye tracking
From eye tracking to gaze analysis
What can we get from eye tracking? An example.
Further options for eye tracking

3. Real-Time 3D Gaze Analysis in
Mobile Applications 3
Applications for Eye Tracking
Assistive technology as interaction modality for people with
disabilities
User experience and interaction (interface design)
Marketing and consumer research
 Interaction with products
 Perception of advertisement
Compare viewing behaviour of experts and novices
 During sports
 During the perception of artworks
...

4. Real-Time 3D Gaze Analysis in
Mobile Applications 4
Mobile and Stationary
Devices

5. Real-Time 3D Gaze Analysis in
Mobile Applications 5
Stationary Devices
Small headbox
Only for gaze on displays
(mostly only one)
Prices:
 Device: 80 .. 40,000+ £
 Software:
 SDKs for free  programming
 Analysis software: 800 .. 8,000+ £
Tobii Pro TX300
(www.tobiipro.com)
SMI RED250
(www.smivision.com)
Smart eye pro
(http://smarteye.se)
SMI RED250Mobile
(www.smivision.com)
Tobii Pro X3-120
(www.tobiipro.com)
Aurora (http://smarteye.se)
Gazepoint GP3 (www.gazept.com)

6. Real-Time 3D Gaze Analysis in
Mobile Applications 6
Mobile Devices
Binocular eye tracking
Below 100 g
Hardware and live viewing:
1,800-10,000 £
Gaze analysis
 SDKs for free or software
(~8,000 £)
 Gaze registration  further costs
 Semi-automatic: gaze annotations
 Fully-automatic: pose estimation needed
Dikablis Professional
(www.ergoneers.com)
Tobii Pro Glasses 2
(www.tobiipro.com)
SMI ETG 2 Professional
(www.eyetracking-glasses.com)
Pupil labs eye tracker (https://pupil-labs.com)

7. Real-Time 3D Gaze Analysis in
Mobile Applications 7
Estimation of 3D Eye Ball Position
Pose estimation of scene camera
 Inside-out tracking
 Scene camera
 Marker tracking
 Markers distributed over the scene
 Distraction of visual attention
 Outside-in tracking
 External cameras + infrared light sources
 Infrared marker attached to device
 High accuracy
 Costs: 4,000 .. 150,000+ £
Eye tracker e.g. from Pupil Labs
(https://pupil-labs.com)
Marker attached in
scene
Infrared markers
attached to e.g. SMI ETG 2
(http://www.eyetracking-glasses.com)
Camera tracking system e.g. from ART
(http://www.ar-tracking.com)
Dikablis Professional
(www.ergoneers.com)

8. Real-Time 3D Gaze Analysis in
Mobile Applications 8
From Eye Tracking to Gaze
Analysis

9. Real-Time 3D Gaze Analysis in
Mobile Applications 9
Gaze Analysis Overview
Eye Tracking
• Line-of-sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations, saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas-of-interest (AOIs)
• visually most relevant objects

10. Real-Time 3D Gaze Analysis in
Mobile Applications 10
Gaze Analysis Overview
Eye Tracking
• Line-of-sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations, saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas-of-interest (AOIs)
• visually most relevant objects

11. Real-Time 3D Gaze Analysis in
Mobile Applications 11
 Nodal point of eye relative to scene camera
 Viewing direction
view cone
Eye Tracking Data for Line of Sight Reconstruction
Guestrin, E. D. D. , Eizenman, M. (2006) General
theory of remote gaze estimation using the pupil
center and corneal reflections. IEEE transactions on
bio-medical engineering, 53, 1124-1133

12. Real-Time 3D Gaze Analysis in
Mobile Applications 12
Estimation of 3D Eye Ball Position
Pose estimation of scene camera
 Inside-out tracking
 Scene camera
 Marker tracking
 Markers distributed over the scene
 Distraction of visual attention
 Outside-in tracking
 External cameras + infrared light sources
 Infrared marker attached to device
 High accuracy
 Costs: 4,000 .. 150,000+ £
Eye tracker e.g. from Pupil Labs
(https://pupil-labs.com)
Marker attached in
scene
Infrared markers
attached to e.g. SMI ETG 2
(http://www.eyetracking-glasses.com)
Camera tracking system e.g. from ART
(http://www.ar-tracking.com)
Dikablis Professional
(www.ergoneers.com)

13. Real-Time 3D Gaze Analysis in
Mobile Applications 13
3D Modell of Environment
Manual creation
 Very time consuming
 Only static scenes
Valencian Kitchen, National Museum of Decorative Arts
(Madrid, Ministry of Culture, Spain)
The Laboratory of Lavoisier, Musée des
arts et mètiers, (Paris, France)
SmartControlRoom, Fraunhofer IOSB
(Karlsruhe, Germany)

14. Real-Time 3D Gaze Analysis in
Mobile Applications 14
Line-of-Sight Reconstruction
 3D gaze point computation =
intersection line-of-sight with 3D
world

15. Real-Time 3D Gaze Analysis in
Mobile Applications 15
Gaze analysis overview
Eye Tracking
• Line of sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations , saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas of interest (AOIs)
• visually most relevant objects

16. Real-Time 3D Gaze Analysis in
Mobile Applications 16
3D Scanpath Visualization
 Gazepoints (violet):
7221 gaze points, 4 min 48 s 8820 gaze points, 5 min 52 s

17. Real-Time 3D Gaze Analysis in
Mobile Applications 17
3D Scanpath Visualization
 Fixations (≥100 ms, red):
7221 gaze points, 4 min 48 s 8820 gaze points, 5 min 52 s

18. Real-Time 3D Gaze Analysis in
Mobile Applications 18
3D Scanpath Visualization
 Fixations and saccades:

19. Real-Time 3D Gaze Analysis in
Mobile Applications 19
Gaze analysis overview
Eye Tracking
• Line of sight reconstruction
• Gaze point computation
Gaze movement
computation
• Fixations, saccades, smooth pursuits, ...
Gaze analysis
• Gaze metrics
• Areas of interest (AOIs)
• visually most relevant objects

20. Real-Time 3D Gaze Analysis in
Mobile Applications 20
Areas of Interest (AOIs)
Definition of AOIs Visualization of a hit AOI

21. Real-Time 3D Gaze Analysis in
Mobile Applications 21
Areas of Interest (AOIs)
Definition of AOIs Visualization of a hit AOI

22. Real-Time 3D Gaze Analysis in
Mobile Applications 22
Entry time + 1st fixation number
Duration of 1st fixation
Number of fixations on AOI
Cumulative fixation time
Dwell time
 Cumulative fixation time + cumulative duration of saccades on AOI
Revisits
 Number of saccades entering the AOI minus 1
AOI-based Metrics

23. Real-Time 3D Gaze Analysis in
Mobile Applications 23
What can we get from eye
tracking? An example.

24. Real-Time 3D Gaze Analysis in
Mobile Applications 24
Two types of experiments
 T1: Freely viewing artworks
 How do subjects look at the scene?
 What are the most attractive areas?
 T2: Freely viewing artworks while
listening to audio guide
 Do people follow the told stories?
Experiments in the EU-Project ARtSENSE

25. Real-Time 3D Gaze Analysis in
Mobile Applications 25
Different Scanning Behavior
 Normalized heat map (green to red)
Freely viewing (5 minutes) Listening to audio guide (6.5 minutes)

26. Real-Time 3D Gaze Analysis in
Mobile Applications 26
Different Scanning Behavior
Food and kitchen tools
Important persons and animals (part of audio guide stories)

27. Real-Time 3D Gaze Analysis in
Mobile Applications 27
Cumulative Fixation Time (CFT)
Food and kitchen tools
Important persons and animals (part of audio guide stories)
CFT = 45 seconds (41 %)
CFT = 64 seconds (59 %)
CFT = 30 seconds (14%)
CFT = 181 seconds (86 %)

28. Real-Time 3D Gaze Analysis in
Mobile Applications 28
Scanpath Comparison
 Scanpath 1: Freely viewing
 Scanpath 2: Listening to audio guide

29. Real-Time 3D Gaze Analysis in
Mobile Applications 30
Live Computation of Visually Most Relevant Objects
in Small Time Windows
0
625
532
751
328
0 200 400 600 800 1000
tablet3
tablet2
servant2
tablet1
servant1
Time/ms
CFT in 4 s Time Window at Time t
766
469
469
548
250
0 200 400 600 800 1000
tablet3
tablet2
servant2
tablet1
servant1
Time/ms
CFT in 4 s Time Window at Time t + 1 s
 Time window [t - 4 s, t]
 Time window [t - 3 s, t + 1 s]
 Basis for recommendations in AR Glasses
tablet1
tablet2
tablet3
servant1 servant2

30. Real-Time 3D Gaze Analysis in
Mobile Applications 31
Gaze-Based Interaction

31. Real-Time 3D Gaze Analysis in
Mobile Applications 32
Stationary Gaze Key Press
 Video stream selection using gaze key press

32. Real-Time 3D Gaze Analysis in
Mobile Applications 33
Mobile Gaze Key Press – First tests
Selection by pointing

33. Real-Time 3D Gaze Analysis in
Mobile Applications 34
Mobile Gaze Key Press – First tests
Pointing (26.5 sec) Mobile gaze key press (16 sec, ≈3/5)

34. Real-Time 3D Gaze Analysis in
Mobile Applications 35
Further Interesting Stuff
Stationary eye tracking via webcam (www.eyezag.de)
 No eye tracker needed
 Subjects can participate from all over the world

35. Real-Time 3D Gaze Analysis in
Mobile Applications 36
Further Interesting Stuff
Eye tracking in virtual reality
 You can walk around and the immersion
is great!
 Cheap pose estimation for 3D eye
tracking
 Modeling of 3D world needed
 Eye tracking hardware and analysis
software not available for all VR devices
– but in probably in the future
HTC Vive
(www.htcvive.com)

36. Real-Time 3D Gaze Analysis in
Mobile Applications 37
Summary
What do you want the people to look at?
 Images on display  stationary eye tracking
 Large volumes  mobile eye tracking
Workflow: Eye tracking  gaze movement computation  gaze
analysis
Gaze-based interaction (gaze key press)
Alternatives to usual eye tracking and eye tracking in VR

37. Real-Time 3D Gaze Analysis in
Mobile Applications 38
Thank you for your
attention!
Questions?

38. Real-Time 3D Gaze Analysis in
Mobile Applications 39
Contribution
1st fully automated real-time capable 3D gaze analysis for
mobile applications
 Publication: Jan Hendrik Hammer, Michael Maurus, Jürgen Beyerer.
Real-time 3D gaze analysis in mobile applications. In: Proceedings
of the 2013 Conference on Eye Tracking South Africa, ETSA '13, S.
75-78, ACM, August 2013.

39. Real-Time 3D Gaze Analysis in
Mobile Applications 40
Realistic Heatmap Generation in 3D Environments
Visual acuity decreases with
deviation from visual axis
 Projected Gaussian
Projected Gaussian Occlusion test

40. Real-Time 3D Gaze Analysis in
Mobile Applications 41
Contribution
1st method for realistic heatmap visualization of gaze data
in 3D environments
 Publication: Michael Maurus, Jan Hendrik Hammer, Jürgen Beyerer.
Realistic Heatmap Visualization for Interactive Analysis of 3D Gaze
Data. In: Proceedings of the Symposium on Eye Tracking Research
and Applications, S. 295--298, ACM, März 2014.

41. Real-Time 3D Gaze Analysis in
Mobile Applications 42

42. Real-Time 3D Gaze Analysis in
Mobile Applications 43

43. Real-Time 3D Gaze Analysis in
Mobile Applications 44

44. Real-Time 3D Gaze Analysis in
Mobile Applications 45
Motivation: Adaptive AR Museum Guide
1. Freely viewing artwork
3. Recommendation
4. Gesture interaction
2. Detection of visually
most relevant objects