ISWC2023

1. Riku Kitamura1) , Takumi Yamamoto1) , Yuta Sugiura1)
TouchLog
Finger Micro Gesture Recognition
Using Photo-Reflective Sensors
1)Keio University

2. • Human fingertips are dexterous and capable of precise movements
• Intuitive and diverse input capabilities
• Many wearable devices are equipped with fingertip gesture input
• Finger micro gestures (FMG) are gaining attention
2
Background
TipText: Capacitance sensing
[Xu, 2019]
Efring: Electric field sensing
[Chen, 2023]
Pyro: PIR sensors
[Gong, 2017]

3. • FMG using the thumb and index finger
• Social acceptability[1]
• Haptic Feedback
• Reduced fatigue
• Confidential input
• No environmental sound restrictions
• Input method that is one-handed and does not cover the fingertip
3
Motivation
[1] Radu-Daniel Vatavu, IFAD Gestures: Understanding Users’ Gesture Input Performance with Index-Finger Augmentation Devices, CHI’23, 2023

4. 4
Our Approach
• Use fingernail-type device
• Fingertip skin deformation information is acquired
using 7 photo-reflective sensors
Gesture Input Method Utilizing Fingertip Skin Deformation

5. • Fingernail-type device
• 7 photo reflective sensors are placed on fingernail-shaped output by a 3D printer
• Sensors face the finger side
5
Implementation: Hardware
Sensor direction

6. • Record for 2.4 sec on average (160 frames) per gesture
• Feature extractions
• Time-series data was divided into 5 parts(with 32-frame intervals)
• Statistical features were extracted for each divided part
• Feature value of 175 dimensions(for 7 sensor values × 5 divisions × 5 statistical features)
• Random forest
6
Implementation: Software
1
2
3
4
5
6
7
Time-series data (160 frames)
Feature Extraction
・・・
Divide into 5 parts
s
Mean
Variance
Max
Min
Median

7. • Selected from FMG related work[2,3]
• Ensure diversity of gestures
7
Gesture Set
Gesture sets
[2] Taizhou Chen, EFRing: Enabling Thumb-to-Index-Finger Microgesture Interaction through Electric Field Sensing Using Single Smart Ring, IMWUT, 2023
[3] Jung Gong, Pyro: Thumb-Tip Gesture Recognition Using Pyroelectric Infrared Sensing, UIST’17, 2017

8. • General Model
• Individual Model
• LOOCV (Leave One Out Cross Validation) Model
8
Three Models for Identification Accuracy
9 user’s data
1 user’s data
× 10 participants
One user’s data
Training data
Test data
90%
10%
× 10 participants
Training data
Test data
All user’s data
Training data
Test data
90%
10%

9. • 10 right-handed participants wore the device on their right index fingers
• 10 participants × 11 gestures × 20 times = 2200 data
• Before the experiment, each gesture was practiced
• No device removal during experiment
9
Experiment
Count of participants (Male/Female) 10 (5/5)
Average age 22.4(SD = 1.57)
Dominant hand Right
Information of Participants

10. • General Model : 91.1%
• Individual Model: 91.5% (SD = 3.1%)
• LOOCV Model : 67.6% (SD = 14.6%)
10
Results

11. • General Model : 91.1%
• Individual Model: 91.5% (SD = 3.1%)
• LOOCV Model : 67.6% (SD = 14.6%)
11
Results

12. • Continuous input
12
Discussion: Other Input Possibilities

13. • Contact pressure input
13
Discussion: Other Input Possibilities

14. • User dependence
• Finger size, dexterity, fingertip skin condition
• Investigate multiple device sizes, fingertip humidity and softness
• Use in a real environment
• Effects of sunlight
• Use the device while walking or running
• Conduct verification under conditions similar to real-world environments
14
Limitations and Future Work

15. 15
Summary
Background
Human fingertips are dexterous and capable
of fine movements
Motivation
Input method that is one-handed and does not cover the
belly of the fingers
Related work
Loss of finger belly perception
Photo reflective sensors are used
for skin deformation acquisition
Our approach
Gesture input method utilizing
fingertip skin deformation
Implementation
Device using photo-reflective sensors
Classify 11 gestures using Random forest
Evaluation
General Model: 91.1%
Individual Model: 91.5%
LOOCV Model: 67.6%
Future work Device that can be worn comfortably in daily life