Details in: http://m-accessibility.blogspot.com This presentation is focused on mnemonical body shortcuts (check http://immi.inesc-id.pt/~tjvg/ for Publications on the subject). In the presentation we detail information we can obtain from an accelerometer and how it can be used to improve mobile device interaction. The presentation was performed at ISHF 2007.

1. Mobile Interaction Based on Human Gesture Analysis Ricardo Gamboa Tiago Guerreiro, Joaquim Jorge {rjssg,tjvg,jaj}@immi.inesc-id.pt Hugo Gamboa [email_address]

2. Actual Interaction Successful & Appropriate?

3. Screen and Keypad size are Limited

4. Multi-Task Graphical-based devices

5. Slow and Visually Demanding Interaction Mobility Issues!

6. Possible solution Key Shortcuts

7. Memory Issues – Where is each shortcut?

8. Other Solution Voice shortcuts

9. Voice Recognition Issues

10. Low Social Acceptance

11. Proposed Solution Gestural Interface!

12. Gestures ease Communication

13. Task Analysis - User observation Actual panorama on mobile usage and shortcutting habits. Results proved: Mobile Interaction is “keystroke consuming” Shortcuts are ineffective.

14. Propposed Approach – Mnemonical Body Shortcuts “ Phone in Silence Mode”

15. Propposed Approach – Mnemonical Body Shortcuts “ Phone your Girlfriend”

16. Related Work RFID Accelerometers EMG Cameras Touch Screens

17. RFID Prototype Pocket LOOX 720 ACG RF RFID reader RFID Tags

18. Evaluation

19. Mnemonical Body Shortcuts Evaluation Mouth Hand Chest Head Wrist Eye Finger Ear SMS 10 1 6 Call 3 1 12 Contacts 3 5 2 1 Clock 10 1 Photos 2 8 Calculator 3 Mp3 2 Agenda 1 3 1 Alarm-clock 2 2 2 3

20. 20 Users 5 chosen Applications 20 Shortcuts Key Shortcuts Vs Mnemonics 94% Recognition Rate

21. Accelerometer ADXL 330 MEMS Bioplux4 Device

22. Why Accelerometers?

23. Accelerometer Applicable in Contextual & Explicit Human Motion

24. Contextual Interaction

25. Stopped Holding Picked Walking Running Stopped Movement Analysis - Amplitude Holding Time (s) Acc (g) Amp = √( x^2+y^2+z^2 )

26. Fall Detection

27. Explicit Interaction Explicit Interaction

28. Mnemonical Body Shortcuts X Y

29. Mnemonical Body Shortcuts Position – Y axis Acceleration Calibrated Acceleration Thresholded

30. Mnemonical Body Shortcuts Position – Y axis ∫ dt ∫ dt Velocity Position

31. Mnemonical Body Shortcuts Position + Final Rotation

32. Mnemonical Body Shortcuts Evaluation – Default Gestures TOTAL RECOGNITION 82% Mouth - 85% Chest – 85% Navel – 90% Shoulder - 75% Neck – 100% Ear - 60% Head – 85% Leg – 80% Wrist – 85% Eye – 75% 10 users – 5 Gestures – No training – 20 Recognitions each

33. Mnemonical Body Shortcuts Evaluation – Trainable Gestures TOTAL RECOGNITION 71% 10 users – 5 Gestures – 5x Train for each gesture – 20 Recognitions Leg - 85% Mouth – 94% Navel – 64% Neck – 95% Ear - 55% 5 most common gestures

35. Tilt – Angle Calculation Time (s/1000) Angle (degrees) X Y Z

36. Tilt - Centralization and Joining Y & Z X Y + Z Angle (degrees) - start position variation Time (s/1000)

37. Tilt - Thresholding X Y + Z Angle (degrees) - start position variation Time (s/1000)

38. Tilt - Decision LEFT RIGHT UP DOWN X Y + Z Time (s/1000) Angle (degrees)

39. Tilt Evaluation

40. Tilt Results - Recognition LEFT TILT 85% RIGHT TILT 94% UP TILT 75% DOWN TILT 93% TOTAL RECOGNITION 86%