EarTouch is a system that uses photo-reflective sensors mounted on wireless earphones to detect skin deformation on the ear caused by the user pulling or deforming their ear in different directions. This allows for directional gestures to be recognized with 90% accuracy as well as symbolic gestures to be recognized with 77% accuracy. The system was evaluated with 8 participants performing predefined gestures while sitting, walking, and re-inserting the earphones. The results demonstrate the feasibility of using the ear as an input interface for mobile devices.

1. EarTouch:
Turning the Ear into an Input Surface
Takashi Kikuchi, Yuta Sugiura, Katsutoshi Masai,
Maki Sugimoto, Bruce H. Thomas
Keio University, University of South Australia

2. • Earphones are becoming smaller and wireless
as the Apple Airpods
Motivation
Sep 6, 2017 MobileHCI2017 EarTouch
Apple AirPods Air by crazybabyJabra Elite Sport Skybuds
2

3. Existing Interaction Styles for Wireless Earphone
Sep 6, 2017 MobileHCI2017 EarTouch
Tapping Voice Control
difficult to apply in noisy environments or
environments where the user cannot speak
the sensor is small that this input
method is limited
3
New type of input method is required

4. EarTouch: Your Ear as Input Interface
Sep 6, 2017 MobileHCI2017 EarTouch 4

5. • Distance from the earphone to the ear changes
when the skin is deformed.
• Measure ear deformation with photo-reflective
sensors
Principle
Sep 6, 2017 MobileHCI2017 EarTouch 5
Photo transistor
Infrared LED
Photo
transistor
Infrared
LED
Photo-reflective sensor
Earphone
Photo
reflective
Sensors
Lobule
Helix
Antihelix

6. • We created a device with four photo-reflective
sensors mounted on an earphone
Hardware
Sep 6, 2017 MobileHCI2017 EarTouch 6
Photo reflective sensors
Earphone

7. Related work
Sep 6, 2017 MobileHCI2017 EarTouch 7
EarPut [Lissermann 2014]Cord Input [Schwarz 2010]EarFieldSensing [Matthies 2017]
Skin deformation detection using photo-reflective sensors
SenSkin [Ogata 2013]
Intelligent Earphone
AffectiveWear [Masai 2015]

8. • Using the sensor data, it enables
to recognize the direction which
the user pulled the ear
• SVM is applied to the sensor
data for recognition
Recognition of Directional-gestures
Sep 6, 2017 MobileHCI2017 EarTouch
Recognition of directional-gesturesTraining the SVM with direction dataset
8
Neutral Up
Forward
Down
Backward

9. • From the directional input, a stroke input is
created by plotting 2D points
• $1 Unistroke Recognizer is used for gesture
recognition
Recognition of Symbolic Gestures
Sep 6, 2017 MobileHCI2017 EarTouch 9
Line v caret
square stairs

10. Hat or Gloves...
Sep 6, 2017 MobileHCI2017 EarTouch 10

11. Evaluation - Directional Gestures -
Sep 6, 2017 MobileHCI2017 EarTouch
• Information of the evaluation
- 8 participants
- Frame rate 30 fps
- 5 directions * 100 frame * 5 trial
→ Cross Validation
• Precision of 5 basic directions with 3 conditions
11
We conducted a user study to investigate the
recognition accuracy of directional gestures.
Sitting Walking
Re-inserting
the device

12. Accuracy Concerning Directional Gestures
Sep 6, 2017 MobileHCI2017 EarTouch
• The average recognition accuracy was 90%
• Walking condition is lower than other conditions
Confusion matrix of
recognition accuracy
when sitting
Recognition accuracy
12

13. • The average recognition accuracy was 77%
• The accuracy was higher without visual aids.
• With visual aids : 75%
• Eyes free: 79%
• Many participants tried to improve their gesture
input by going back and forth with visual aids.
Accuracy of Symbolic Gestures
Sep 6, 2017 MobileHCI2017 EarTouch
The five symbols used for the user study Recognition accuracy of symbolic gestures
13

14. Applications
Sep 6, 2017 MobileHCI2017 EarTouch 14

15. • Don’t pull ear strongly. Otherwise,
your ear will get damage
• Affected by ambient light such as
sunlight
• Power Consumption → Use one
sensor as a switch to turn the other
LEDs on
• Users moves (e.g. running and
jumping) increases false positives →
Combing other sensors to recognize
the user’s state
Limitations & Future work
Sep 6, 2017 MobileHCI2017 EarTouch 15

16. Thank you
Sep 6, 2017 MobileHCI2017 EarTouch 16
Keio University, University of South Australia
Takashi Kikuchi, Yuta Sugiura, Katsutoshi Masai, Maki Sugimoto, Bruce H. Thomas
Motivation
&
Basic Idea
Implementation
Evaluation
Earphones are becoming smaller and wireless Ear as input interface
Photo-reflective sensors measure skin deformation
Directional and symbolic
gestures can be recognized
90% accuracy for directional gestures 77% accuracy for symbolic gestures