Eyeing the Web

Steffen Staab: Eyeing the Web Keynote at WebIST-2019 1Institute for Web Science and Technologies · University of Koblenz-Landau, Germany
Web and Internet Science Group · ECS · University of Southampton, UK &
Eyeing the Web
Steffen Staab
Joint work with
Chandan Kumar, Raphael Menges, Korok Sengupta
Christoph Schäfer, Tina Walber @ EyeVido

Steffen Staab: Eyeing the Web Keynote at WebIST-2019 2
Here‘s looking at you, kid

Remote Eye Tracking

Anatomy of the Human Eye
Unavoidable issues wrt eye tracking
• accuracy → bias → calibration
• precision → variance → focal area
• intention → Midas touch

Web Interaction
Experience the Web Manipulate the Web
Eyes
Ears
Hands
Mouth
Feet
Other
Voice
Haptic
smell
taste
balance
Understand Augment

Eye & Web: Why should one care?
Accessibility
Standardization
Sound architecture
Interaction bandwidth
Web community
HCI community

Eye & Web: Why should one care?
If you make it there,
You make it everywhere
On the
Web
Accessibility
Standardization
Sound architecture
Interaction bandwidth

Web Interaction
Part I Part II
Eyes
Ears
Hands
Mouth
Feet
Other
Voice
Haptic
smell
taste
balance
Understand Augment

Understanding
Users‘ Understanding of Web Sites

• Spin-out of WeST in 2015
• Major companies as clients:
„If a user is
having a
problem, it‘s
your problem.“
- Steve Jobs -

Cloud-Software EYEVIDO Lab
. . .. . .
. . . . . . . . .
. . .
Creation and evaluation of usability
studies in the web browser
Storage and analysis of
test data in the cloud
Data collection with eye
trackers anywhere and
anytime
Various possibilities for
data visualization,
analysis, and data export
0100101
0101000
Secure server in Germany,
security approved (TÜV label)
Eye tracker
https://youtu.be/-wbdHx0Idbk

Multiple users interact with an interface
Usability experts want to
analyze user behavior!

User Semantics&Pragmatics vs Web Site Syntax
Usability Objective
Understanding the
Semantics and
Pragmatics
of
Many Users‘
Interactions
Issues with the Web site
Dynamics implemented
using syntax of
• URL change
• z-index, or visibility, or
opacity, or transform …
– In the page,
in the style sheet,
– in the code

Data recording during interaction
• Video recording
• Mouse data
• Eye-tracking data
• Voice data…
One could watch every video with data, but…
How to make the analysis effective and efficient?
Research Question

Frames from a video are stitched to a single image
First Approach: Stitching (Menges et al. 2018)

What if the pages look different?
Dynamics in the Web Interface
Frame n Frame n+1

Paradigms of Usability Analysis
Traditional
• Passive stimuli
– Static images
Required
• Active stimuli
– Carousels
– Video
– Ajax
– ...
Both
• Correlating stimuli and (lack of) user activities
– Clicks
– Gaze
– Voice
• Coming up with suggestions for improved usability

Discovery of Active Visual Stimuli

Novel Framework for Discovering Visual Stimuli

Ground Truth of Visual Change
Visual Change?No Yes
Frame n Frame n+1

Ground Truth of Visual Change: Refined

Training of the Classifier

… compute for each pair of sequential frames.
Features for Classifiers of Visual Change

• Data recording on 12 Web sites, each 3 pages
• 4 participants, total of 155 minutes of videos
• In total 23,571 frame pairs have been labeled
• 4,446 have been labeled as visually different /
changing
Ground Truth Data Set
Interface for Labeling

For each frame pair: Features and Label. Try to predict!
Visual Change Classifier: Within Site
Same-Site Classifications with 4-fold cross validation
*F1-scores everywhere

Same-Category Classifications
Across-Category Classifications
Feature Importances
Visual Change Classifier: Across Sites

Framework for Visual Stimuli Discovery
Remember the proposed framework…
This we have covered
Splitting Merging

Splitting
Input
Video with
frames
Output
Stimulus shots
with stitched
frames

Merging
Stimulus shots
with stitched
frames of each
video
Output
Stitched
frames across
videos
Input
Output of Splitting

Input to the framework

Output of the framework

More Output of the framework

• Efficiency for the UI expert?
• Better features for the machine learning?
• How to handle borderline cases?
• How to consider UI expert feedback and
personalization?
– Not every analysis must have the same purpose!
Open questions

Web Interaction using your Eyes

The Assumption
Photo by User Gflores on en.wikipedia - http://www.epa.gov/win/winnews/images05/0510keyboard.gif, Public Domain,
https://commons.wikimedia.org/w/index.php?curid=36965777

Web Accessibility

EU Project MAMEM 2015-2018
https://youtu.be/42yGmr3NE0k

GazeTheWeb

User Experience:
Issues with Emulation
Precision, Accuracy, Midas Touch
• Filtering, calibration, element size and
feedback
• Better Pointing and Selection Method
(e.g., replace mouse and keyboard)
EyeTracker
(1)
(2)
(4)
GazeData
Cursor /
Button Press/
Scroll Wheel
Key Press
No semantic
feedback!
Monitor
Reflection of user
Emulation Software
ScreenEmulation
Filtering
DesktopWeb Browser
a)
b)
KeyboardMouse
Pixel Data
? ?
(3)
(5)
(6)
Gaze Emulation

User Experience: Issues with Emulation
User experience depends on application control
– Emulation of mouse and keyboard
incurs interaction and visual overhead
Example: Type a search query

Context-adapted Gaze Interaction:
GazeTheWeb
• Optimize gaze interaction experience
– Minimize the required input actions
– Minimize inspection overhead
• Extract the semantics of the interface
– Text input
– Hyperlink
– Video
– …
• Adapt the means of interaction
– Appropriate input translation
– Visual indicators
– Contextual Feedback
Pixel Data
Commands
Webpage
Elements/
Information
Interface
Web Engine
<ContextualData/>
<html>
<head> ... </head>
<body>
... <input/>
... <a/> ...
</body>
</html>
WebPage
Screen
EyeTracker
(1)
(3)
(4)
(5)
(6)
(7)
(8)
GazeData
Monitor
Reflection of user
(2)
Filtering
Open
URL or
Bookmark
Back
Mark
Bookmark
Remove
Tab
(b) Tab 1 Tab 2 New Tab
Tabs
Duckduckgo.com
(a)
T
Back Forward TabsHyperlink
Germany |
(c)
Text Input
Scrolling

GazeTheWeb: Adaptations

GazeTheWeb Video
https://www.youtube.com/watch?v=x1ESgaoQR9Y (48s)

Web Interface Introspection
• Web page semantics
– Identification and Tracking of Interaction Elements
• Web browser semantics
– Page Meta-Information and Control Handles
• Dynamics in modern Web
– Structure and layout often changes

Examples of ideosyncratic engineering
Google Search.
• Stacks two input fields over each other using CSS z-index
– one displays suggestions
– the other the actual input
– figure out by specific programming which is which
Facebook Chat prohibits insertion of text using JavaScript.
• We simulate keystrokes through the Web engine
YouTube Video
• Videos in general: how to operate if no button is visible?
– We introduce an invisible button!
• YouTube prevents making the video full screen.
– Make the 7th parent full screeen.
Rule of thumb:
Generalizing methods donot work on most sophisticated Web sites

Task completion
Usability
Workload
Gaze based emulation GazeTheWeb
UX Evaluation:
Summative lab study
EyeTracker
(1)
(2)
(4)
GazeData
Cursor /
Button Press/
Scroll Wheel
Key Press
No semantic
feedback!
Monitor
Reflection of user
Emulation Software
ScreenEmulation
Filtering
DesktopWeb Browser
a)
b)
KeyboardMouse
Pixel Data
? ?
(3)
(5)
(6)
Pixel Data
Commands
Webpage
Elements/
Information
Interface
Web Engine
<ContextualData/>
<html>
<head> ... </head>
<body>
... <input/>
... <a/> ...
</body>
</html>
WebPage
Screen
EyeTracker
(1)
(3)
(4)
(5)
(6)
(7)
(8)
GazeData
Monitor
Reflection of user
(2)
Filtering
Open
URL or
Bookmark
Back
Mark
Bookmark
Remove
Tab
Tabs
Duckduckgo.com
(a)
T
Germany |
(c)
Text Input
Scrolling

UX Evaluation: Results
Task completion time Workload –NASA TLX
Usability
Design Heuristics

Task completion
Usability
Workload
Gaze based emulation GazeTheWeb
UX Evaluation:
Summative lab study
EyeTracker
(1)
(2)
(4)
GazeData
Cursor /
Button Press/
Scroll Wheel
Key Press
No semantic
feedback!
Monitor
Reflection of user
Emulation Software
ScreenEmulation
Filtering
DesktopWeb Browser
a)
b)
KeyboardMouse
Pixel Data
? ?
(3)
(5)
(6)
Pixel Data
Commands
Webpage
Elements/
Information
Interface
Web Engine
<ContextualData/>
<html>
<head> ... </head>
<body>
... <input/>
... <a/> ...
</body>
</html>
WebPage
Screen
EyeTracker
(1)
(3)
(4)
(5)
(6)
(7)
(8)
GazeData
Monitor
Reflection of user
(2)
Filtering
Open
URL or
Bookmark
Back
Mark
Bookmark
Remove
Tab
Tabs
Duckduckgo.com
(a)
T
Germany |
(c)
Text Input
Scrolling
👍
👍
👍
👍
👍
👍

Trial with Patients • Neuro muscular disease
• Spinal cord injury
• Parkinson disease

MAMEM Trials: Phase I
SUS Scores
Target Group: 77.36
Control Group: 72.17
P=.2
SUS Scores Phase II: 70 (NMD), 75.5 (PD), 73.3 (SCI)

Conclusion

It‘s the semantics, of course!
Experience the Web Manipulate the Web
Eyes
Ears
Hands
Mouth
Feet
Other
Voice
Haptic
smell
taste
balance
Semantic questions
asked by UI expert
Semantic
comprehension
required by UI

Discussion
Challenges
• Re-engineering the
Semantics
• Semantics-oriented
syntax
(HTML5 <nav>)
not much used
Outlook
• How to get user
interaction semantics
annotated?
• How to re-engineer the
user interaction
semantics?
How to move away from single-use
solutions towards Web way of working?

What‘s next?

Don‘t touch!

Talk-and-Gaze for Text Editing
For
everyone
!

Usability Data Eco System
Stakeholders
• UI Experts
• Test subjects
• Users
Data
• UI Experts
– Observations
– Analyses
• Test subjects
– Gaze
– Mouse
– Voice
– ....
• Users
– Mouse
– ....
feed
answers
Usability
Data
Mining

Thank you!

Control
R. Menges, C. Kumar, S, Staab. Improving user experience of eye tracking-based interaction:
Introspecting and adapting interfaces. In: ACM Transactions on Computer-Human Interaction.
Accepted May 2019.
C. Kumar, R. Menges, S. Staab. Eye-Controlled Interfaces for Multimedia Interaction. IEEE MultiMedia
23(4): 6-13, 2016.
C. Kumar, D. Akbari, R. Menges, S. MacKenzie, S. Staab. TouchGazePath: Multimodal Interaction with
Touch and Gaze Path for Secure Yet Efficient PIN Entry. In: 21st ACM International Conference on
Multimodal Interaction (ICMI 2019), Suzhou, Jiangsu, China. October 14-18, 2019.
K. Sengupta, S. Bhattarai, S. Sarcar, S. MacKenzie, S. Staab. Leveraging Error Correction in Voice-
based Text Entry by Talk-and-Gaze. Tech Report.
R. Menges, C. Kumar, S. Staab. Eye tracking for Interaction: Adapting Multimedia Interfaces. Book
chapter. IET. To appear 2020.
Analysis
R. Menges, H. Tamimi, C. Kumar, T. Walber, C. Schäfer, S. Staab. Enhanced Representation of Web
Pages for Gaze-based Attention Analysis. In: Proc. of ETRA 2018 ACM Symposium on Eye
Tracking Research & Applications. Warsaw, Poland, June 14-17, 2018.
R. Menges, C. Kumar, S. Staab. What did my Users Experience? Discovering Visual Stimuli of Dynamic
User Interfaces. Tech Report 2019.
T. Walber, A. Scherp, S. Staab. Smart Photo Selection: Interpret Gaze as Personal Interest. In: Proc. of
2014 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI ‘14, Toronto,
Canada, April 26-May 1, 2014.
References

Eyeing the Web

Recommended

Recommended

More Related Content

Similar to Eyeing the Web

Similar to Eyeing the Web (20)

More from Steffen Staab

More from Steffen Staab (20)

Recently uploaded

Recently uploaded (20)

Eyeing the Web

Editor's Notes