Takumi Yamamoto, Katsutoshi Masai, Anusha Withana, and Yuta Sugiura. 2023. Masktrap: Designing and Identifying Gestures to Transform Mask Strap into an Input Interface. In Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI '23). Association for Computing Machinery, New York, NY, USA, 762–775. https://doi.org/10.1145/3581641.3584062

1. Masktrap: Designing and Identifying Gestures
to Transform Mask Strap into an Input Interface
T a k u m i Y a m a m o t o 1 ) , K a t s u t o s h i M a s a i 1 ) , 2 ) , A n u s h a W i t h a n a 3 ) , Y u t a S u g i u r a 1 )
1 ) K e i o U n i v e r s i t y , 2 ) N T T C o m m u n i c a t i o n S c i e n c e L a b o r a t o r i e s ,
3 ) T h e U n i v e r s i t y o f S y d n e y

2. Face Mask
• Necessity during the pandemic
2
Motivation: Face Mask
Many types of mask
In public space

3. 5
Our Approach
Input Information Using Mask Straps

4. 6
Related Work
[Kunimi, 2021] [Hiraki, 2021]
Silent speech interface
[Kusabuka, 2020]
Breath gesture
Mask-based input interface
• Many type of gestures: pulling, touching…
• Good for Hygienic conditions
Not focus on mask straps
Touching Ear
[Chen, 2020]
Touching Face
[Xu, 2020]
Touching Hat
[Dierk, 2019]
[Olwal, 2018]
[Olwal, 2020]
Face to Hand Gesture or String Gesture
Face to hand gesture String gesture
Not explored face’s string gestures

5. 7
3 Contributions
Gesture Design Implementation Technical Evaluation
User Elicitation Study
(N = 20)
Ear-Hook Device User Study
(N = 8)

6. 8
Gesture Design
• Gestures using mask straps are not explored.
• Orientation, position
• Designing method for gesture sets
• User elicitation study approach [Wobbrock, 2009]
Implementation
Gesture Design Evaluation
String based interface
[Olwal, 2020]
Show participants the operations Design the gesture for the operation
Turning volume up!
• Intuitive, preferable [Morris, 2010]
• Standard method and applied in various domains

7. 9
Elicitation Study: Task and Procedure
• 20 participants
• Select 21 daily tasks from previous work [Masai, 2020][Fukahori,
2015]
• Rules
• Only use mask straps
• Avoid conflicting gestures within the same group
• Not thinking about the technical limitations
Number of
participants
(Male/Female)
20(12/8)
Age 26.5±10.2
Dominant hand All right-
handed
The list of tasks
Participant information
Implementation
Gesture Design Evaluation

8. 10
Analysis of Proposed Gestures: Type of Gestures
• Most Proposed: Pull gestures
Type of Gestures
Implementation
Gesture Design Evaluation

9. 11
Analysis of Proposed Gestures: Interaction Area
• Right vs Left: using right straps occupy 72.9%
• Upper vs Lower: Trace and Touch use upper straps.
• Others use lower straps.
Interaction Area
Implementation
Gesture Design Evaluation

10. • Agreement Rates (AR) [Vatavu, 2015]
• 0 ≤ AR ≤ 1
• Result:Average:0.062
• Low agreement (<0.1) [Vatavu, 2015]
12
Elicitation Study: Agreement Rates (AR)
Agreement Rate
• 𝑟: referent (operation)
• 𝑃 : the set of all proposals for referent 𝑟
• |𝑃|: the number of the set
• |𝑃𝑖|: the subsets of the same proposals from 𝑃
Implementation
Gesture Design Evaluation

11. 13
Elicitation Study: Final Gesture Set
• Final gesture sets: gesture with the highest number of proposals
• Gestures in the same group cannot be duplicated.
• No gesture can be duplicated for operation 0.
Final Gesture Sets
Implementation
Gesture Design Evaluation

12. 14
Implementation: Hardware
• Re-usable ear-hook device
• Not attached sensors to masks (Considering the disposable nature of masks)
• 8 photo-reflective sensors and 2 microphones
(a)3D CAD model and Sensor positions
(b)Prototype device
User wear the device
Implementation
Gesture Design Evaluation

13. 15
Implementation: Software
• Record about 1.2 sec for each gesture
• Feature Extractions
• Photo-reflective sensors: Extract the mean and variance value
• Microphone sensors: FFT and Mel spectrogram features
• Random Forest
Feature extractions
Implementation
Gesture Design Evaluation

14. 16
Gesture Set for the Evaluation
• 10 Gestures are selected from Elicitation Study’s Results
• Same gesture on both sides was only used on the right.
• Gestures with multiple times were reduced to once.
• Add removing and wearing masks
Gesture Sets for Recognition
Implementation
Gesture Design Evaluation

15. • 8 Participants wear device and masks in their right ear.
• 8 participants× 12 gestures × 20 times = 1920 data
• Use same masks
17
Technical Evaluation: Procedure
Participants
(Male/Female)
8/0
Age 22.88± 2.42
Dominant hands Right-handed
Type of Mask
Color: White
Thickness : 5mm,
Type : Non-woven type
Information of Experiment
Implementation
Gesture Design Evaluation

16. 18
Technical Evaluation: Result
Using photo-reflective sensors
92.61 %
Using microphone sensors
42.40 %
Using both sensors
93.07 %
Implementation
Gesture Design Evaluation

17. • Another Use case:
• Situation where people wear masks for other purposes
• Experiment was conducted in the controlled settings
• In the wild experiment
• Changing the mask type
• Investigate the gesture’s usability
19
Discussion and Future Work
Healthcare
Cleaning
Other purposes

18. 20
Summary
Background
Masks are used
in many situations and for many purposes
Related work
Not focus on mask straps
Hand to face’s straps gestures are not explored
Proposal Input information using mask straps
Elicitation
Study
User defined gesture sets for daily task
Implementation
Using photo-reflective sensors and microphones sensors
Classify 12 gestures using random forest
Evaluation We got 93.07% Using both type of sensors
Future work
In the wild experiment
Considering about typical usage scenarios