SlideShare a Scribd company logo
Heads-Up Multitasker: Simulating Attention
Switching On Optical Head-Mounted Displays
Bai Yunpeng, Aleksi Ikkala, Antti Oulasvirta*, Shengdong Zhao*, Lucia J. Wang, Pengzhi Yang, Peisen Xu
Background: Heads-Up Computing Paradigm
Convenient and
ubiquitous reading
Daily task
planning
Cooking
Exercising
Move and read on smart glasses
How to design optimal interfaces?
Background: Modeling Objectives
● Attention switches
In the heart of an ancient forest, where
the sunlight gently filtered through a high
canopy, a small, forgotten pond mirrored
the sky so precisely that it
Reading on smart glasses
while walking
Walk control
● Walk speed
● Walk direction
We predict:
Vision attention control:
● Attention to the environment
● Text reading on smart glasses
Application: An Evaluation Tool
Which text spacing
is the best?
5 different
text spacings
Run our simulator to see!
Best Text
Spacing:
Layout 4
1
2
3
4
5
Application: An Evaluation Tool
Simple road read and walk City read and walk Read, walk, see signs
Flexible to differnt design factors, users, scenarios, and tasks!
Handcrafted policies
Flexibly learn policies by
reinforcement learning
o, r
a
rSC
rSC
rR
rS
External state
(MuJoCo Simulation)
In the heart of an
ancient forest,
where the
sunlight gently
Our Model
FPV TPV
Read (R)
Scan (S)
Supervisory
Controller
(SC)
b
Locomotion
Control
(LC)
e
w
Oculomotor
Control
(OC)
Internal
state
a
o
Memory
Hierarchical
Reinforcement
Learning
External state
Oculomotor
Control
(OC)
Locomotion
Control
(LC)
o, r
a
Internal
state
Memory
a
o
Our Model
Supervisory
Controller
(SC)
Read (R)
Scan (S)
rSC
rSC
b
w
rR
e
rS
Assumptions:
● Decompose complex cognitive task to simpler subtasks in terms of
solvability and trainability.
● High-fidelity simulation of primitive actions – eye movement and walking.
Model – Supervisory Level – Supervisory Control (SC)
● State
○ Reading progress.
○ Walking status.
○ Current task.
○ Environmental situation.
● Action
○ Attention deployment.
● Observation
○ Time awareness.
○ Current task.
○ Reading progress.
○ Walking speed.
● Reward Function
○ rt
= wR
* rR
+ wR
* rR
- wA
* cA
- cT
● Transition
○ Deterministic
Task Description:
When to switch attention?
Learning Objectives:
Minimize the environmental information loss
while maximizing the reading progress.
Github code
Model – Task Level – Read (R)
● State
○ Target word should be read.
○ Current fixation.
● Action
○ The word index.
● Observation
○ Time awareness.
○ Current task.
○ Belief: Probability representation on words.
● Reward Function
○ rt
= rtime cost
(-0.1) + Bonus (+-10)
● Transition
○ Deterministic
Task Description:
Resume reading.
Learning Objectives:
Relocate the correct word quickly.
Github code
Model – Motor Level – Oculomotor Control (OC)
● State
○ Target word should be fixated.
○ Current fixation.
● Action
○ The eyeball motor control: x and y rotations.
● Observation
○ Time awareness.
○ Vision perception (image captured in the simulator)
○ Proprioception: Current fixation.
● Reward Function
○ rt
= 0.1 * (e-10*d
- 1) + Bonus (+10)
● Transition
○ Stochastic:
at+1
~ N (target fixation position, 0.08 * saccade amplitude)
Task Description:
Fixate on the target word.
Learning Objective:
Fixate on the target object ASAP under
the noisy perception and control
conditions.
Github code
Study Overview
Study 1: Attention switches.
Simulation data
Tasks
1
Study 2: Reading while walking.
Simulation data vs. Human data
2
Study 3: Resume reading after attention switches.
Simulation data vs. Human data
3
Study 4: Read, walk, and see env signs.
Simulation data vs. Human data
4
Study 1: Attention Switches Adapt to Agents and Walking Speed
Reward = wread
⨉ r1
+ wwalk
⨉ r2
+ wattention switch
⨉ r3
● Higher wread
, the agent priorities read more.
● Higher wwalk
, the agent priorities walk safety more.
We could train different agents by designing different reward functions.
Olaf
Env
events
Texts
Shakespeare Norman
Read walk attention swithces
Study 1: Attention Switches Adapt to Agents and Walking Speed
Other metrics:
● Reading interruption positions
● Reading speed
● Walking error rate
Number of attention switch
Study 2: Reading Speed Adapts to Walking
Simulation
Reading deterioration due
to head perturbations
Reading speed ratio = RSwalk
/ RSstand
Results
(N=12)
Human Sim
Real world
Study 3: Reading Resumptions Adapt to Text Layouts
Simulation
Real world
In the quiet town of
Willow Creek,
whispers of a
mysterious figure
Simulate the reading resumption adaptation to different text layouts.
Study 3: Reading Resumptions Adapt To Text Layouts
Layout 1
Layout 2
Layout 3
Completion Time (s)
N=12
Error Rate (%)
N=12
Study 3: Reading Resumptions Adapt To Text Layouts
Internal
state
Memory
Normal model
Internal
state
Ablation model:
No Memory Module
VS.
Layout 1
Layout 2
Layout 3
Completion Time (s)
Ablation: huge discrepency
Study 4: Read, Walk, and See Env Signs
Task: Read, walk, see signs
Experiment setup: real world vs. simulation.
Route: a rectangle path
Study 4: Read, Walk, and See Env Signs
Simulation Human
N=12
Conclusion and Takeaways
A flexible simulator Hierarchical
POMDPs
Evaluate Extend Oculomotor Locomotion
● What is the task to optimize?
● How to describe user behaviors into
sequential decision-making processes?
● What cognitive processes to include?
Thank You!
Github codes and data Know more about me;
Contact me if you are looking for jobs
In the heart of an
ancient forest,
where the sunlight
gently filtered
through a

More Related Content

Similar to Heads-Up Multitasker: CHI 2024 Presentation.pdf

An introduction to reinforcement learning
An introduction to reinforcement learningAn introduction to reinforcement learning
An introduction to reinforcement learning
Subrat Panda, PhD
 
anintroductiontoreinforcementlearning-180912151720.pdf
anintroductiontoreinforcementlearning-180912151720.pdfanintroductiontoreinforcementlearning-180912151720.pdf
anintroductiontoreinforcementlearning-180912151720.pdf
ssuseradaf5f
 
Oop concept
Oop conceptOop concept
Oop concept
Waseem Mehmood
 
Wattpad - Spark Stories
Wattpad - Spark StoriesWattpad - Spark Stories
Wattpad - Spark Stories
Rylan Halteman
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
MLconf
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
Xavier Amatriain
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
Xavier Amatriain
 
Python week 7 8 2019-2020 for g10 by eng.osama ghandour
Python week 7 8 2019-2020 for g10 by eng.osama ghandourPython week 7 8 2019-2020 for g10 by eng.osama ghandour
Python week 7 8 2019-2020 for g10 by eng.osama ghandour
Osama Ghandour Geris
 
130704798265658191
130704798265658191130704798265658191
130704798265658191
Tanzeel Ahmad
 
Wearable Computing - Part III: The Activity Recognition Chain (ARC)
Wearable Computing - Part III: The Activity Recognition Chain (ARC)Wearable Computing - Part III: The Activity Recognition Chain (ARC)
Wearable Computing - Part III: The Activity Recognition Chain (ARC)
Daniel Roggen
 
Python week 6 2019 2020 for grade 10
Python week 6 2019 2020 for grade 10 Python week 6 2019 2020 for grade 10
Python week 6 2019 2020 for grade 10
Osama Ghandour Geris
 
Inventing the future Business Programming Language
Inventing the future  Business Programming LanguageInventing the future  Business Programming Language
Inventing the future Business Programming Language
ESUG
 
Notes About Linden Scripting 2009 Congress
Notes About Linden Scripting 2009 CongressNotes About Linden Scripting 2009 Congress
Notes About Linden Scripting 2009 Congress
draceina
 
Probabilistic content models,
Probabilistic content models,Probabilistic content models,
Probabilistic content models,
Bryan Gummibearehausen
 
Hierarchical Reinforcement Learning
Hierarchical Reinforcement LearningHierarchical Reinforcement Learning
Hierarchical Reinforcement Learning
ahmad bassiouny
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Xavier Amatriain
 
Designing Object Oriented Software - lecture slides 2013
Designing Object Oriented Software - lecture slides 2013Designing Object Oriented Software - lecture slides 2013
Designing Object Oriented Software - lecture slides 2013
Jouni Smed
 
Building data "Py-pelines"
Building data "Py-pelines"Building data "Py-pelines"
Building data "Py-pelines"
Rob Winters
 
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Seattle Apache Flink Meetup
 
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
Bowen Li
 

Similar to Heads-Up Multitasker: CHI 2024 Presentation.pdf (20)

An introduction to reinforcement learning
An introduction to reinforcement learningAn introduction to reinforcement learning
An introduction to reinforcement learning
 
anintroductiontoreinforcementlearning-180912151720.pdf
anintroductiontoreinforcementlearning-180912151720.pdfanintroductiontoreinforcementlearning-180912151720.pdf
anintroductiontoreinforcementlearning-180912151720.pdf
 
Oop concept
Oop conceptOop concept
Oop concept
 
Wattpad - Spark Stories
Wattpad - Spark StoriesWattpad - Spark Stories
Wattpad - Spark Stories
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
 
Python week 7 8 2019-2020 for g10 by eng.osama ghandour
Python week 7 8 2019-2020 for g10 by eng.osama ghandourPython week 7 8 2019-2020 for g10 by eng.osama ghandour
Python week 7 8 2019-2020 for g10 by eng.osama ghandour
 
130704798265658191
130704798265658191130704798265658191
130704798265658191
 
Wearable Computing - Part III: The Activity Recognition Chain (ARC)
Wearable Computing - Part III: The Activity Recognition Chain (ARC)Wearable Computing - Part III: The Activity Recognition Chain (ARC)
Wearable Computing - Part III: The Activity Recognition Chain (ARC)
 
Python week 6 2019 2020 for grade 10
Python week 6 2019 2020 for grade 10 Python week 6 2019 2020 for grade 10
Python week 6 2019 2020 for grade 10
 
Inventing the future Business Programming Language
Inventing the future  Business Programming LanguageInventing the future  Business Programming Language
Inventing the future Business Programming Language
 
Notes About Linden Scripting 2009 Congress
Notes About Linden Scripting 2009 CongressNotes About Linden Scripting 2009 Congress
Notes About Linden Scripting 2009 Congress
 
Probabilistic content models,
Probabilistic content models,Probabilistic content models,
Probabilistic content models,
 
Hierarchical Reinforcement Learning
Hierarchical Reinforcement LearningHierarchical Reinforcement Learning
Hierarchical Reinforcement Learning
 
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
 
Designing Object Oriented Software - lecture slides 2013
Designing Object Oriented Software - lecture slides 2013Designing Object Oriented Software - lecture slides 2013
Designing Object Oriented Software - lecture slides 2013
 
Building data "Py-pelines"
Building data "Py-pelines"Building data "Py-pelines"
Building data "Py-pelines"
 
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
 
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...Approximate queries and graph streams on Flink, theodore vasiloudis,  seattle...
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
 

Recently uploaded

Physiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptxPhysiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptx
fatima132662
 
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
Sérgio Sacani
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Selcen Ozturkcan
 
Clinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdfClinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdf
RAYMUNDONAVARROCORON
 
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
ABHISHEK SONI NIMT INSTITUTE OF MEDICAL AND PARAMEDCIAL SCIENCES , GOVT PG COLLEGE NOIDA
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
University of Maribor
 
Anti-Universe And Emergent Gravity and the Dark Universe
Anti-Universe And Emergent Gravity and the Dark UniverseAnti-Universe And Emergent Gravity and the Dark Universe
Anti-Universe And Emergent Gravity and the Dark Universe
Sérgio Sacani
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
Sérgio Sacani
 
Sustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart AgricultureSustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart Agriculture
International Food Policy Research Institute- South Asia Office
 
Compexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titrationCompexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titration
Vandana Devesh Sharma
 
GBSN - Biochemistry (Unit 6) Chemistry of Proteins
GBSN - Biochemistry (Unit 6) Chemistry of ProteinsGBSN - Biochemistry (Unit 6) Chemistry of Proteins
GBSN - Biochemistry (Unit 6) Chemistry of Proteins
Areesha Ahmad
 
The cost of acquiring information by natural selection
The cost of acquiring information by natural selectionThe cost of acquiring information by natural selection
The cost of acquiring information by natural selection
Carl Bergstrom
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
Scintica Instrumentation
 
Male reproduction physiology by Suyash Garg .pptx
Male reproduction physiology by Suyash Garg .pptxMale reproduction physiology by Suyash Garg .pptx
Male reproduction physiology by Suyash Garg .pptx
suyashempire
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
hozt8xgk
 
IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE  AND ITS BENIFITS.pptxIMPORTANCE OF ALGAE  AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
OmAle5
 
gastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptxgastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptx
Shekar Boddu
 
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
eitps1506
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
PirithiRaju
 
fermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptxfermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptx
ananya23nair
 

Recently uploaded (20)

Physiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptxPhysiology of Nervous System presentation.pptx
Physiology of Nervous System presentation.pptx
 
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
SDSS1335+0728: The awakening of a ∼ 106M⊙ black hole⋆
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
 
Clinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdfClinical periodontology and implant dentistry 2003.pdf
Clinical periodontology and implant dentistry 2003.pdf
 
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
MICROBIAL INTERACTION PPT/ MICROBIAL INTERACTION AND THEIR TYPES // PLANT MIC...
 
Randomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNERandomised Optimisation Algorithms in DAPHNE
Randomised Optimisation Algorithms in DAPHNE
 
Anti-Universe And Emergent Gravity and the Dark Universe
Anti-Universe And Emergent Gravity and the Dark UniverseAnti-Universe And Emergent Gravity and the Dark Universe
Anti-Universe And Emergent Gravity and the Dark Universe
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
 
Sustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart AgricultureSustainable Land Management - Climate Smart Agriculture
Sustainable Land Management - Climate Smart Agriculture
 
Compexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titrationCompexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titration
 
GBSN - Biochemistry (Unit 6) Chemistry of Proteins
GBSN - Biochemistry (Unit 6) Chemistry of ProteinsGBSN - Biochemistry (Unit 6) Chemistry of Proteins
GBSN - Biochemistry (Unit 6) Chemistry of Proteins
 
The cost of acquiring information by natural selection
The cost of acquiring information by natural selectionThe cost of acquiring information by natural selection
The cost of acquiring information by natural selection
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
 
Male reproduction physiology by Suyash Garg .pptx
Male reproduction physiology by Suyash Garg .pptxMale reproduction physiology by Suyash Garg .pptx
Male reproduction physiology by Suyash Garg .pptx
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
 
IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE  AND ITS BENIFITS.pptxIMPORTANCE OF ALGAE  AND ITS BENIFITS.pptx
IMPORTANCE OF ALGAE AND ITS BENIFITS.pptx
 
gastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptxgastroretentive drug delivery system-PPT.pptx
gastroretentive drug delivery system-PPT.pptx
 
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
CLASS 12th CHEMISTRY SOLID STATE ppt (Animated)
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
 
fermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptxfermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptx
 

Heads-Up Multitasker: CHI 2024 Presentation.pdf

  • 1. Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays Bai Yunpeng, Aleksi Ikkala, Antti Oulasvirta*, Shengdong Zhao*, Lucia J. Wang, Pengzhi Yang, Peisen Xu
  • 2. Background: Heads-Up Computing Paradigm Convenient and ubiquitous reading Daily task planning Cooking Exercising Move and read on smart glasses How to design optimal interfaces?
  • 3. Background: Modeling Objectives ● Attention switches In the heart of an ancient forest, where the sunlight gently filtered through a high canopy, a small, forgotten pond mirrored the sky so precisely that it Reading on smart glasses while walking Walk control ● Walk speed ● Walk direction We predict: Vision attention control: ● Attention to the environment ● Text reading on smart glasses
  • 4. Application: An Evaluation Tool Which text spacing is the best? 5 different text spacings Run our simulator to see! Best Text Spacing: Layout 4 1 2 3 4 5
  • 5. Application: An Evaluation Tool Simple road read and walk City read and walk Read, walk, see signs Flexible to differnt design factors, users, scenarios, and tasks! Handcrafted policies Flexibly learn policies by reinforcement learning
  • 6. o, r a rSC rSC rR rS External state (MuJoCo Simulation) In the heart of an ancient forest, where the sunlight gently Our Model FPV TPV Read (R) Scan (S) Supervisory Controller (SC) b Locomotion Control (LC) e w Oculomotor Control (OC) Internal state a o Memory Hierarchical Reinforcement Learning
  • 7. External state Oculomotor Control (OC) Locomotion Control (LC) o, r a Internal state Memory a o Our Model Supervisory Controller (SC) Read (R) Scan (S) rSC rSC b w rR e rS Assumptions: ● Decompose complex cognitive task to simpler subtasks in terms of solvability and trainability. ● High-fidelity simulation of primitive actions – eye movement and walking.
  • 8. Model – Supervisory Level – Supervisory Control (SC) ● State ○ Reading progress. ○ Walking status. ○ Current task. ○ Environmental situation. ● Action ○ Attention deployment. ● Observation ○ Time awareness. ○ Current task. ○ Reading progress. ○ Walking speed. ● Reward Function ○ rt = wR * rR + wR * rR - wA * cA - cT ● Transition ○ Deterministic Task Description: When to switch attention? Learning Objectives: Minimize the environmental information loss while maximizing the reading progress. Github code
  • 9. Model – Task Level – Read (R) ● State ○ Target word should be read. ○ Current fixation. ● Action ○ The word index. ● Observation ○ Time awareness. ○ Current task. ○ Belief: Probability representation on words. ● Reward Function ○ rt = rtime cost (-0.1) + Bonus (+-10) ● Transition ○ Deterministic Task Description: Resume reading. Learning Objectives: Relocate the correct word quickly. Github code
  • 10. Model – Motor Level – Oculomotor Control (OC) ● State ○ Target word should be fixated. ○ Current fixation. ● Action ○ The eyeball motor control: x and y rotations. ● Observation ○ Time awareness. ○ Vision perception (image captured in the simulator) ○ Proprioception: Current fixation. ● Reward Function ○ rt = 0.1 * (e-10*d - 1) + Bonus (+10) ● Transition ○ Stochastic: at+1 ~ N (target fixation position, 0.08 * saccade amplitude) Task Description: Fixate on the target word. Learning Objective: Fixate on the target object ASAP under the noisy perception and control conditions. Github code
  • 11. Study Overview Study 1: Attention switches. Simulation data Tasks 1 Study 2: Reading while walking. Simulation data vs. Human data 2 Study 3: Resume reading after attention switches. Simulation data vs. Human data 3 Study 4: Read, walk, and see env signs. Simulation data vs. Human data 4
  • 12. Study 1: Attention Switches Adapt to Agents and Walking Speed Reward = wread ⨉ r1 + wwalk ⨉ r2 + wattention switch ⨉ r3 ● Higher wread , the agent priorities read more. ● Higher wwalk , the agent priorities walk safety more. We could train different agents by designing different reward functions. Olaf Env events Texts Shakespeare Norman Read walk attention swithces
  • 13. Study 1: Attention Switches Adapt to Agents and Walking Speed Other metrics: ● Reading interruption positions ● Reading speed ● Walking error rate Number of attention switch
  • 14. Study 2: Reading Speed Adapts to Walking Simulation Reading deterioration due to head perturbations Reading speed ratio = RSwalk / RSstand Results (N=12) Human Sim Real world
  • 15. Study 3: Reading Resumptions Adapt to Text Layouts Simulation Real world In the quiet town of Willow Creek, whispers of a mysterious figure Simulate the reading resumption adaptation to different text layouts.
  • 16. Study 3: Reading Resumptions Adapt To Text Layouts Layout 1 Layout 2 Layout 3 Completion Time (s) N=12 Error Rate (%) N=12
  • 17. Study 3: Reading Resumptions Adapt To Text Layouts Internal state Memory Normal model Internal state Ablation model: No Memory Module VS. Layout 1 Layout 2 Layout 3 Completion Time (s) Ablation: huge discrepency
  • 18. Study 4: Read, Walk, and See Env Signs Task: Read, walk, see signs Experiment setup: real world vs. simulation. Route: a rectangle path
  • 19. Study 4: Read, Walk, and See Env Signs Simulation Human N=12
  • 20. Conclusion and Takeaways A flexible simulator Hierarchical POMDPs Evaluate Extend Oculomotor Locomotion ● What is the task to optimize? ● How to describe user behaviors into sequential decision-making processes? ● What cognitive processes to include?
  • 21. Thank You! Github codes and data Know more about me; Contact me if you are looking for jobs In the heart of an ancient forest, where the sunlight gently filtered through a