Mobile Interaction Based on Human Gesture Analysis Ricardo Gamboa Tiago Guerreiro, Joaquim Jorge {rjssg,tjvg,jaj}@immi.inesc-id.pt Hugo Gamboa [email_address]

Actual Interaction Successful & Appropriate?

Screen and Keypad size are Limited

Multi-Task Graphical-based devices

Slow and Visually Demanding Interaction Mobility Issues!

Possible solution Key Shortcuts

Memory Issues – Where is each shortcut?

Other Solution Voice shortcuts

Voice Recognition Issues

Low Social Acceptance

Proposed Solution Gestural Interface!

Gestures ease Communication

Task Analysis - User observation Actual panorama on mobile usage and shortcutting habits. Results proved: Mobile Interaction is “keystroke consuming” Shortcuts are ineffective.

Propposed Approach – Mnemonical Body Shortcuts “ Phone in Silence Mode”

Propposed Approach – Mnemonical Body Shortcuts “ Phone your Girlfriend”

Related Work RFID Accelerometers EMG Cameras Touch Screens

RFID Prototype Pocket LOOX 720 ACG RF RFID reader RFID Tags

Evaluation

Mnemonical Body Shortcuts Evaluation Mouth Hand Chest Head Wrist Eye Finger Ear SMS 10 1 6 Call 3 1 12 Contacts 3 5 2 1 Clock 10 1 Photos 2 8 Calculator 3 Mp3 2 Agenda 1 3 1 Alarm-clock 2 2 2 3

20 Users 5 chosen Applications 20 Shortcuts Key Shortcuts Vs Mnemonics 94% Recognition Rate

Accelerometer ADXL 330 MEMS Bioplux4 Device

Why Accelerometers?

Accelerometer Applicable in Contextual & Explicit Human Motion

Contextual Interaction

Stopped Holding Picked Walking Running Stopped Movement Analysis - Amplitude Holding Time (s) Acc (g) Amp = √( x^2+y^2+z^2 )

Fall Detection

Explicit Interaction Explicit Interaction

Mnemonical Body Shortcuts X Y

Mnemonical Body Shortcuts Position – Y axis Acceleration Calibrated Acceleration Thresholded

Mnemonical Body Shortcuts Position – Y axis ∫ dt ∫ dt Velocity Position

Mnemonical Body Shortcuts Position + Final Rotation

Mnemonical Body Shortcuts Evaluation – Default Gestures TOTAL RECOGNITION 82% Mouth - 85% Chest – 85% Navel – 90% Shoulder - 75% Neck – 100% Ear - 60% Head – 85% Leg – 80% Wrist – 85% Eye – 75% 10 users – 5 Gestures – No training – 20 Recognitions each

Mnemonical Body Shortcuts Evaluation – Trainable Gestures TOTAL RECOGNITION 71% 10 users – 5 Gestures – 5x Train for each gesture – 20 Recognitions Leg - 85% Mouth – 94% Navel – 64% Neck – 95% Ear - 55% 5 most common gestures

Mnemonical Body Shortcuts Discussion Default Gestures Good Recognition Rate. Limited to 10 pre-defined gestures. Users have to learn the gestures. Treinable gestures Lower Recognition Rate – Similar gestures are choosen. Position isn’t very effective to desambiguate gestures outside x,y plan. One training error spoils the recognition – outlier detection is needed.

Default Gestures

Good Recognition Rate.

Limited to 10 pre-defined gestures.

Users have to learn the gestures.

Treinable gestures

Lower Recognition Rate – Similar gestures are choosen.

Position isn’t very effective to desambiguate gestures outside x,y plan.

One training error spoils the recognition – outlier detection is needed.

Tilt – Angle Calculation Time (s/1000) Angle (degrees) X Y Z

Tilt - Centralization and Joining Y & Z X Y + Z Angle (degrees) - start position variation Time (s/1000)

Tilt - Thresholding X Y + Z Angle (degrees) - start position variation Time (s/1000)

Tilt - Decision LEFT RIGHT UP DOWN X Y + Z Time (s/1000) Angle (degrees)

Tilt Evaluation

Tilt Results - Recognition LEFT TILT 85% RIGHT TILT 94% UP TILT 75% DOWN TILT 93% TOTAL RECOGNITION 86%

Future Work Mobility tests on Mnemonical Body Shortcuts Try a Feature Based Algorithm Design a full prototype ( + feedback and shortcuts)

Mobility tests on Mnemonical Body Shortcuts

Try a Feature Based Algorithm

Design a full prototype

( + feedback and shortcuts)