Gesture-aware remote controls: guidelines and interaction techniques

566 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
566
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Gesture-aware remote controls: guidelines and interaction techniques

  1. 1. GESTURE-AWARE REMOTE CONTROLS: GUIDELINES & INTERACTION TECHNIQUES Gilles Bailly, Dong-Bach Vo, Eric Lecolinet and Yves Guiard Telecom ParisTech, LTCI - CNRSmardi 15 novembre 2011
  2. 2. Interactive television More and more services and contents: • Electronic program guide • Video on demand • Web based widgets • Web applications • Personal content 2 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  3. 3. Interactive television More and more services and contents: • Electronic program guide • Video on demand • Web based widgets • Web applications • Personal content 3 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  4. 4. Interactive television More and more services and contents: • Electronic program guide • Video on demand • Web based widgets • Web applications • Personal content 4 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  5. 5. Interactive television More and more services and contents: • Electronic program guide • Video on demand • Web based widgets • Web applications • Personal content 5 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  6. 6. Interactive television More and more services and contents: • Electronic program guide • Video on demand • Web based widgets • Web applications • Personal content 6 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  7. 7. Interacting with interactive television Inspired from computer interaction: • Deep hierarchical menus • Long lists 7 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  8. 8. Interacting with interactive television Inspired from computer interaction: • Deep hierarchical menus • Long lists 8 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  9. 9. Interacting with interactive television Inspired from computer interaction: • Deep hierarchical menus • Long lists 9 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  10. 10. Interacting with interactive television Inspired from computer interaction: • Deep hierarchical menus • Long lists 10 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  11. 11. Remote control Problems: • Browsing • Commands 11 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  12. 12. Remote control Problems: • Browsing • Commands 12 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  13. 13. Remote control Problems: • Browsing • Commands 13 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  14. 14. Remote control Problems: • Browsing • Commands Limited expressiveness 14 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  15. 15. Improving expressiveness Solution? • Add a new button for each new command 15 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  16. 16. Improving expressiveness Solution? • Add a new button for each new command Pultius TV remote control [Artlebedev 07] 16 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  17. 17. Improving expressiveness Solution? • Add a new button for each new command Can we benefit from moving the remote control in 3D space to improve expressiveness? 17 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  18. 18. Approach Advantages of 3D gestures: • Allow large vocabulary of gestures • Can be easy to remember • Free visual focus from the remote control • Comfortable to perform gestures at home [Rico et al. 10] • Remote controls can remain small and easy to hold 18 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  19. 19. Approach Advantages of 3D gestures: • Allow large vocabulary of gestures • Can be easy to remember • Free visual focus from the remote control • Comfortable to perform gestures at home [Rico et al. 10] • Remote controls can remain small and easy to hold 18 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  20. 20. 1. Interactive television2. Related work3. Observing spontaneous gestures4. Exploring new gesture dimension5. Designing new interaction techniques6. Conclusion 19 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  21. 21. Related work 3D gestures offer 6 degrees of freedom: • 3 translations Y • 3 rotations Z X 20 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  22. 22. Related work 3D gestures offer 6 degrees of freedom: • 3 translations Y • 3 rotations Z Yaw Roll Pitch X 21 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  23. 23. Related work Rotations: • 1 dimension • 2 dimensions • Discretization of tilt angle with visual feedback [Rahman et al. 09] Motion marking menu system [Oakley 03] Pitch Dong-Bach Vo ‒ ICMI 2011 X 22mardi 15 novembre 2011
  24. 24. Related work Rotations: • 1 dimension • 2 dimensions • Discretization of tilt angle with visual feedback [Rahman et al. 09] GesText [Jones et al. 10] Roll Z Pitch Dong-Bach Vo ‒ ICMI 2011 X 23mardi 15 novembre 2011
  25. 25. Related work Rotations: • 1 dimension Y • 2 dimensions Z • Discretization of tilt angle Yaw with visual feedback 12 levels [Rahman et al. 09] Roll 16 levels 24 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  26. 26. 1. Interactive television2. Related work3. Observing spontaneous gestures4. Exploring new gesture dimension5. Designing new interaction techniques6. Conclusion 25 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  27. 27. Experiment 1: spontaneous gestures What gestures users perform spontaneously for moving targets? Y • Which degree(s) of freedom? Z Yaw • Is it context dependant (smartphone vs. remote controls)? Roll Pitch X 26 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  28. 28. Experiment 1: spontaneous gestures Task: • Perform the gesture that would move the target to this position Procedure: • Smartphone vs. Remote control • Observation from video recordings 27 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  29. 29. Experiment 1: spontaneous gestures Task: • Perform a gesture which has produced this effect Procedure: • Smartphone vs. Remote control • Observation from video recordings 28 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  30. 30. Experiment 1: spontaneous gestures 9 Number of participants 8 7 6 5 Results: 4 3 • Either rotations or translations 2 1 • 14 of 18 users perform rotations 0 (S) Condition (RC) Condition • Pitch and Roll only in smartphone Pitch & Roll rotations condition Pitch & Yaw rotations • Pitch and Yaw in remote control X & Y translations condition • Roll can be used for something else 29 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  31. 31. Experiment 1: spontaneous gestures 9 Number of participants 8 7 6 5 Results: 4 3 • Either rotations or translations 2 1 • 14 of 18 users perform rotations 0 (S) Condition (RC) Condition • Pitch and Roll only in smartphone Pitch & Roll rotations condition Pitch & Yaw rotations • Pitch and Yaw in remote control X & Y translations condition • Roll can be used for something else Z Roll Pitch X 30 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  32. 32. Experiment 1: spontaneous gestures 9 Number of participants 8 7 6 5 Results: 4 3 • Either rotations or translations 2 1 • 14 of 18 users perform rotations 0 (S) Condition (RC) Condition • Pitch and Roll only in smartphone Pitch & Roll rotations condition Pitch & Yaw rotations • Pitch and Yaw in remote control X & Y translations condition Y • Roll can be used for something else Yaw ntrol condition Pitch X 31 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  33. 33. Experiment 1: spontaneous gestures 9 Number of participants 8 7 6 5 Results: 4 3 • Either rotations or translations 2 1 • 14 of 18 users perform rotations 0 (S) Condition (RC) Condition • Pitch and Roll only in smartphone Pitch & Roll rotations condition Pitch & Yaw rotations • Pitch and Yaw in remote control X & Y translations condition Y • Roll can be used for something else Z Yaw Roll Pitch X 32 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  34. 34. 1. Interactive television2. Related work3. Observing spontaneous gestures4. Exploring roll gestures5. Designing new interaction techniques6. Conclusion 33 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  35. 35. Experiment 2: exploring roll gestures What we know: • 16 items selection with visual feedback [Rahman 09] • 3 items selection in eyes free selection [Oakley 07] 34 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  36. 36. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: • 9 participants • Items set size (5, 7, 9, 11 items) 35 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  37. 37. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: • 9 participants • Items set size (5, 7, 9, 11 items) 36 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  38. 38. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: Step 1: Roll to prepare the gesture • 9 participants • Items set size (5, 7, 9, 11 items) 37 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  39. 39. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: Step 1: Roll to prepare the gesture • 9 participants • Items set size (5, 7, 9, 11 items) 37 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  40. 40. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: Step 2: Press the trigger • 9 participants • Items set size (5, 7, 9, 11 items) 38 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  41. 41. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: Step 3: Roll gesture • 9 participants • Items set size (5, 7, 9, 11 items) 39 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  42. 42. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: Step 4: Release the trigger • 9 participants • Items set size (5, 7, 9, 11 items) 40 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  43. 43. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: • 9 participants • Items set size (5, 7, 9, 11 items) 41 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  44. 44. Experiment 2: exploring roll gestures Question: How many levels users can control without visual feedback? Task: Use any roll amplitude you need to reach the target • Roll - Press - Roll - Release strategy • No feedback provided Procedure: • 9 participants • Items set size (5, 7, 9, 11 items) 42 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  45. 45. Experiment 2: results Angular range: • Differs largely among 162° 326° participants • Depends on item-set size Minimum and maximum Angular variation: angular range for 7 items • Constant variation between two contiguous items 43 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  46. 46. Experiment 2: results Angular range: 400 • Differs largely among participants ∆ angle (in degrees) 300 • Depends on item-set size 200 100 Angular variation: • Constant variation between two 0 0 1 2 3 4 6 7 8 9 10 11 contiguous items number of items in set 44 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  47. 47. Experiment 2: results Angular range: Mean angular varation for 7 items • Differs largely among 200 participants 160 120 • Depends on item-set size 80 40 0 Angular variation: -40 -80 • Constant variation between two -120 contiguous items -160 -200 -3 -2 -1 0 1 2 3 Left Center Right 45 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  48. 48. Experiment 2: results Recognition: • KNN algorithm • euclidian distance on angular variation • cross-validation technique Results: • Very good for 5 items (96.3%) • Insufficient for 9 and 11 items • 7 items: • 87.7% for cross-validation • 96% with user specific training 46 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  49. 49. Experiment 2: results 100 Recognition: 96.3 • KNN algorithm 75 accuracy (in %) • euclidian distance on angular variation 50 • cross-validation technique 25 Results: • Very good for 5 items (96.3%) 0 5 items 7 items 9 items 11 items • Insufficient for 9 and 11 items mean accuracy rate • 7 items: • 87.7% for cross-validation • 96% with user specific training 47 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  50. 50. Experiment 2: results 100 Recognition: • KNN algorithm 75 77.1 accuracy (in %) • euclidian distance on angular 63.1 variation 50 • cross-validation technique 25 Results: • Very good for 5 items (96.3%) 0 5 items 7 items 9 items 11 items • Insufficient for 9 and 11 items mean accuracy rate • 7 items: • 87.7% for cross-validation • 96% with user specific training 48 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  51. 51. Experiment 2: results 100 Recognition: 87.7 • KNN algorithm 75 accuracy (in %) • euclidian distance on angular variation 50 • cross-validation technique 25 Results: • Very good for 5 items (96.3%) 0 5 items 7 items 9 items 11 items • Insufficient for 9 and 11 items mean accuracy rate • 7 items: • 87.7% for cross-validation • 96% with user specific training 49 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  52. 52. 1. Interactive television2. Related work3. Observing spontaneous gestures4. Exploring roll gestures5. Designing new interaction techniques6. Conclusion 50 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  53. 53. Designing interaction techniques Goals: • two-level circular menus • that combine gestural modalities • Pitch & Yaw • Roll • Directional pad buttons 51 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  54. 54. Designing interaction techniques Design space: • 1 modality per menu level • D-pad used as a baseline • Unpromising techniques dismissed at pretest 52 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  55. 55. Designing interaction techniques Design space: 2nd Level Pitch & Yaw Roll Directional Pad • 1 modality per menu 1st Level level • D-pad used as a Pitch & Yaw baseline • Unpromising techniques dismissed Roll at pretest Directional Pad 53 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  56. 56. Designing interaction techniques Design space: 2nd Level Pitch & Yaw Roll Directional Pad • 1 modality per menu 1st Level level • D-pad used as a Pitch & Yaw baseline • Unpromising techniques dismissed Roll at pretest Directional Pad Baseline 54 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  57. 57. Designing interaction techniques Design space: 2nd Level Pitch & Yaw Roll Directional Pad • 1 modality per menu 1st Level level • D-pad used as a Pitch & Yaw Gesture trigger penalty baseline • Unpromising Biomechanical Poor results in Gesture trigger techniques dismissed Roll constraint pretests penalty at pretest Directional Pad Baseline 55 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  58. 58. Designing interaction techniques Design space: 2nd Level Pitch & Yaw Roll Directional Pad • 1 modality per menu 1st Level level • D-pad used as a Pitch & Yaw 3D multi-strokes menu 3D gestures parallelization Gesture trigger penalty baseline • Unpromising Biomechanical Poor results in Gesture trigger techniques dismissed Roll constraint pretests penalty at pretest • 5 techniques to Directional Pad 2D/3D gestures 2D/3D gestures Baseline evaluate parallelization parallelization 56 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  59. 59. Experiment 3: interaction techniques Procedure: • Menu with 4 * 4 items • Combination of modalities • User feedback • 13 participants Task: • Perform the gesture corresponding to the symbolic stimulus 57 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  60. 60. Experiment 3: results Best modality combinations Mean accuracy rate by technique for accuracy: 100,0 97.4 98.4 94.2 80,0 • Pad / Pad (baseline) Accuracy in % 60,0 • Pad / Pitch & Yaw (better than baseline) 40,0 • Pad / Roll 20,0 0 Pad/Pad Pad/P&Y Pad/Roll 58 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  61. 61. Experiment 3: results Best modality combinations Mean time selection by technique for speed: 1700 1650 • Similar results 1275 1400 1390 Time in ms • Pad / Roll is a bit slower 850 425 0 Pad/Pad Pad/P&Y Pad/Roll 59 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  62. 62. Experiment 3: results User satisfaction: Subjective preferences 7 • Pad / Pad 6 5 • then Pad / Pitch & Yaw 4 • then P&Y / P&Y 3 2 • then Pad / Roll 1 0 Speed Accuracy Pleasant Tiring Easy to learn Pad/Pad Pad/P&Y Pad/Roll P&Y/P&Y 60 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  63. 63. Multimedia system menu Novice mode: • 2-level circular menu • 2 modalities • Novice mode: • 1st level : Pitch and Yaw • 2nd level : Roll 61 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  64. 64. Multimedia system menu Novice mode: • 2-levels circular menu • 1st level selection • 2nd level selection 62 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  65. 65. Multimedia system menu Novice mode: • 2-levels circular menu • 1st level selection • 2nd level selection 63 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  66. 66. Multimedia system menu Expert mode: • Same gestures • But the menu does not appear • Implicit learning of gestures 64 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  67. 67. 1. Interactive television2. Related work3. Observing spontaneous gestures4. Exploring roll gestures5. Designing new interaction techniques6. Conclusion 65 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  68. 68. Conclusion What we have learned: • Experiment 1: • Users rather use rotations when interacting with handheld devices • With a remote control, they use pitch and yaw to interact with a distant screen • Experiment 2: Roll gestures without feedback work fine: • for 5 levels • for 7 levels with user specific training • Using button and 3D gestures can efficiently augment remote control expressiveness 66 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  69. 69. Conclusion What we have learned: • Experiment 3: • 3D gestures provide good performance • especially when combined with d-pad buttons • Conclusion: • Moving the remote control into space is indeed an effective way to improve expressiveness 67 Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011
  70. 70. Thank you for your attention! GESTURE-AWARE REMOTE CONTROLS: GUIDELINES & INTERACTION TECHNIQUES http://perso.telecom-paristech.fr/ via/VIA_Project.html dong-bach.vo@telecom-paristech.fr Dong-Bach Vo ‒ ICMI 2011mardi 15 novembre 2011

×