MAGIC:
     A Motion Gesture Design Tool

     Daniel Ashbrook              Thad Starner

Nokia Research Center Hollywood ...
gesture



   2
gesture design issues




          3
gesture design issues

support for
                  false positives
non-experts




              3
Allow non-expert use…




4
Allow non-expert use…




…but support experts

               4
B
Iterative design




                   5
B
Iterative design




          Support extensive testing


                      5
B
Iterative design                Retrospection




          Support extensive testing


                      5
goal:

make gesture design easier



            6
magic
(multiple action gesture interface creation tool)




                        7
sensing & recognition
       sensor:
wrist-mounted
accelerometer




                 8
sensing & recognition
       sensor:
wrist-mounted
accelerometer

   microphone
     “skinput”
    gyroscope
  muscle inpu...
sensing & recognition
       sensor:           gesture
wrist-mounted        recognition:
accelerometer             DTW

  ...
sensing & recognition
       sensor:           gesture
wrist-mounted        recognition:
accelerometer             DTW

  ...
gesture design issues

support for
                  false positives
non-experts




              9
gesture design issues

support for
                     false positives
non-experts

    magic
user interface


          ...
Gestures to define:

• Play/Pause          • Volume Up 10%
• Shuffle              • Volume Down 10%
• Next Track          • ...
gesture classes and
     examples




         11
gesture classes and
cut
      examples




         11
gesture classes and
cut
      examples




         11
gesture classes and
cut
      examples




         11
magic usage stages

                  Gesture
                  Creation




                             False Positive
G...
magic usage stages

                  Gesture
                  Creation




                             False Positive
G...
gesture creation




       13
intra-gesture consistency
inter-gesture differentiability
inter-gesture differentiability
16
Actual   Guess               Actual   Guess


              wrong
            } guesses

73% goodness                 100%...
example
comparison
graph




             17
example           1
comparison
graph             2


                  3


                  4


                  5
     ...
gesture class comparison




           18
gesture class comparison




           18
gesture class comparison




                match box

           18
magic usage stages

                  Gesture
                  Creation




                             False Positive
G...
magic usage stages

                  Gesture
                  Creation




                             False Positive
G...
gesture design issues

support for
                   false positives
non-experts




              20
gesture design issues

support for
                   false positives
non-experts




              20
define gesture:
 delete email

                 21
false positives




define gesture:         accidentally
 delete email          delete email

                 21
everyday gesture library



          EGL




           22
everyday gesture library



          EGL




           22
everyday gesture library

     ?                    ?

     ?
             EGL          ?
         ?            ?
        ...
everyday gesture library

    OK
     ?                   X
                         ?

    OK
     ?
             EGL    ...
EGL data collection




     accelerometer      contextual video
24
magic usage stages

                  Gesture
                  Creation




                             False Positive
G...
magic usage stages

                  Gesture
                  Creation




                             False Positive
G...
false positive testing




          26
magic evaluation
       experiment set up




       27
Gesture criteria:
               •   The gesture must reliably activate the desired function.

quantitative   •   Performi...
participants
             Num                 Num       UCD       UI design Pattern rec
Condition participants   Age
     ...
collected EGLs
     EGL age gender        occupation     hours collected
       1    32   M      PhD CS                   ...
collected EGLs
     EGL age genderActivities
               EGL      occupation      hours collected
       1    32    • P...
in-magic EGL

    19:18




    EGL 1


      33
in-magic EGL

 5:19           13:59


given to        testing
 users


               EGL 1


                 33
magic evaluation
       quantitative results




       34
user performance
            false positives




       35
user performance
                                    false positives

              12
         11        1
    10        ...
user performance
                                     false positives

              12                                   ...
user performance
                              false positives

           12                             12
      11     ...
user performance
                          gesture goodness

• Gesture goodness:
 μ = 86% / σ = 24%
 • 7 with goodness >= ...
user performance
 about 2h to
complete task




                37
user performance
  about 2h to
 complete task




  mostly linear
  progression:
create, test, EGL
                    37
user performance
  about 2h to
 complete task




  mostly linear     mostly linear progression:
  progression:      add c...
magic evaluation
        qualitative results




       38
feedback
               general

“This is really hard!” (but…) “I really like the
software and I really like the interface...
feedback
                   egl

“I just kinda feared the EGL”


(regarding video) “I
                 didn't care why I w...
feedback
            visualizations

Graphs were “something a little bit complicated
for me”

“It takes a while to build u...
retrospection




previous playlist A                 previous playlist B
  low goodness                        high goodn...
retrospection




previous playlist A                 previous playlist B
  low goodness                        high goodn...
retrospection




previous playlist A                 previous playlist B
  low goodness                        high goodn...
future work

• improve speed, responsiveness
• easier to understand visualizations
• test out other sensors, algorithms
• ...
see paper for:

• related work
• users’ gesture strategies
• statistics!
Thank you!
      questions?
ISWC'10
             IEEE International Symposium on Wearable
              Connected Smartware - Wearable
               ...
user performance
                                gesture goodness

               precision · recall          F-measure, o...
user performance
                                         gesture goodness

               precision · recall             ...
user performance
                                         gesture goodness

               precision · recall             ...
user performance
                                         gesture goodness

               precision · recall             ...
gesture design tool desiderata




              48
gesture design tool desiderata
                   Allow non-expert use




              48
gesture design tool desiderata
                                           Allow non-expert use




                       ...
gesture design tool desiderata
                                           Allow non-expert use




                       ...
gesture design tool desiderata




     Allow expert use
                 49
gesture design tool desiderata



                   B
Iterative design




                   50
gesture design tool desiderata




          Retrospection


               51
gesture design tool desiderata




       Support further testing
                 52
gesture design tool desiderata




       Support further testing
                 52
EGL limitations

•   not guaranteed to span space of everyday life
•   EGL is defined by
    (sensor, situation, target use...
support for non-experts




           54
MAGIC: A Motion Gesture Design Tool
MAGIC: A Motion Gesture Design Tool
Upcoming SlideShare
Loading in...5
×

MAGIC: A Motion Gesture Design Tool

1,374

Published on

MAGIC: A Motion Gesture Design Tool

Daniel Ashbrook, Georgia Tech and Nokia Research Center Hollywood

Thad Starner, Georgia Tech

http://research.nokia.com/files/2010-Ashbrook-CHI10-MAGIC.pdf

Presented at the 28th Annual ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)

Abstract:
Devices capable of gestural interaction through motion sensing are increasingly becoming available to consumers; however, motion gesture control has yet to appear outside of game consoles. Interaction designers are frequently not expert in pattern recognition, which may be one reason for this lack of availability. Another issue is how to effectively test gestures to ensure that they are not unintentionally activated by a user’s normal movements during everyday usage. We present MAGIC, a gesture design tool that addresses both of these issues, and detail the results of an evaluation.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,374
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
58
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

  • motion, not pen/touch gesture
  • However, problems with designing gesture
  • However, problems with designing gesture
  • UI designers don’t have to know about USB protocols for mouse clicks, so gesture designers shouldn’t have to know math. But if they do, we want to support it!

    http://www.flickr.com/photos/chromewavesdotorg/528726488/sizes/m/
    http://www.flickr.com/photos/ervega/3709443341/sizes/o/
  • UI designers don’t have to know about USB protocols for mouse clicks, so gesture designers shouldn’t have to know math. But if they do, we want to support it!

    http://www.flickr.com/photos/chromewavesdotorg/528726488/sizes/m/
    http://www.flickr.com/photos/ervega/3709443341/sizes/o/
























  • auto start/stop record examples

    contextual video - play back for retrospection









  • goodness = harmonic mean of precision & recall; see paper for more

  • goodness = harmonic mean of precision & recall; see paper for more

  • goodness = harmonic mean of precision & recall; see paper for more

  • goodness = harmonic mean of precision & recall; see paper for more

  • goodness = harmonic mean of precision & recall; see paper for more

  • graphically locate outliers



































  • first: tutorial, then: task
  • criteria for realistic scenario - given to users



  • remainder is tested along with other volunteer-collected egls
  • remainder is tested along with other volunteer-collected egls
  • remainder is tested along with other volunteer-collected egls
  • remainder is tested along with other volunteer-collected egls
  • remainder is tested along with other volunteer-collected egls
  • remainder is tested along with other volunteer-collected egls
  • remainder is tested along with other volunteer-collected egls

  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!
  • noEGL is 27 times worse!

  • feedback: system is slow
  • feedback: system is slow
  • feedback: system is slow










  • be quick!

    predominantly HCI audience; will explain some ML
  • be quick!

    predominantly HCI audience; will explain some ML
  • be quick!

    predominantly HCI audience; will explain some ML
  • be quick!

    predominantly HCI audience; will explain some ML
  • non-expert: yes—most participants weren’t expert in pattern rec. In general, no stat. sig. diff. for patt. rec (see doc for more): PR experts didn’t do better, but that means non-pr didn’t do worse!
  • non-expert: yes—most participants weren’t expert in pattern rec. In general, no stat. sig. diff. for patt. rec (see doc for more): PR experts didn’t do better, but that means non-pr didn’t do worse!
  • non-expert: yes—most participants weren’t expert in pattern rec. In general, no stat. sig. diff. for patt. rec (see doc for more): PR experts didn’t do better, but that means non-pr didn’t do worse!
  • expert: somewhat; can pick which gestures to include, can adjust threshold; 3 participants self-rated as expert and seemed to have easier time understanding graphs.

    could do more: tweak algorithm parameters, switch out back-end recognition, see confusion matrix
  • Yes, definitely some. Lots of effort in creation == don’t want to change a lot if failure in EGL. But many did. System speed was big issue curbing iteration.
  • Yes! Retrospection was highly popular & heavily used.
  • EGL lets you test new gestures and new recognizers

    video can be used later as well for training or social acceptability testing
  • situation: jogging, sleeping, normal life?

    example target users:
    children vs adults
    or
    disabilities; people with different abilities need different EGLs (Parkinson’s vs able bodied)
  • other issue: interface designers are expert in interface design, not pattern recognition. Nor should they have to be!
  • MAGIC: A Motion Gesture Design Tool

    1. 1. MAGIC: A Motion Gesture Design Tool Daniel Ashbrook Thad Starner Nokia Research Center Hollywood Georgia Tech Georgia Tech
    2. 2. gesture 2
    3. 3. gesture design issues 3
    4. 4. gesture design issues support for false positives non-experts 3
    5. 5. Allow non-expert use… 4
    6. 6. Allow non-expert use… …but support experts 4
    7. 7. B Iterative design 5
    8. 8. B Iterative design Support extensive testing 5
    9. 9. B Iterative design Retrospection Support extensive testing 5
    10. 10. goal: make gesture design easier 6
    11. 11. magic (multiple action gesture interface creation tool) 7
    12. 12. sensing & recognition sensor: wrist-mounted accelerometer 8
    13. 13. sensing & recognition sensor: wrist-mounted accelerometer microphone “skinput” gyroscope muscle input … 8
    14. 14. sensing & recognition sensor: gesture wrist-mounted recognition: accelerometer DTW microphone “skinput” gyroscope muscle input … 8
    15. 15. sensing & recognition sensor: gesture wrist-mounted recognition: accelerometer DTW microphone HMMs “skinput” SVMs gyroscope neural networks muscle input $1 recognizer … … 8
    16. 16. gesture design issues support for false positives non-experts 9
    17. 17. gesture design issues support for false positives non-experts magic user interface 9
    18. 18. Gestures to define: • Play/Pause • Volume Up 10% • Shuffle • Volume Down 10% • Next Track • Next Playlist • Previous Track • Previous Playlist 10
    19. 19. gesture classes and examples 11
    20. 20. gesture classes and cut examples 11
    21. 21. gesture classes and cut examples 11
    22. 22. gesture classes and cut examples 11
    23. 23. magic usage stages Gesture Creation False Positive Gesture Testing Testing 12
    24. 24. magic usage stages Gesture Creation False Positive Gesture Testing Testing 12
    25. 25. gesture creation 13
    26. 26. intra-gesture consistency
    27. 27. intra-gesture consistency
    28. 28. inter-gesture differentiability
    29. 29. inter-gesture differentiability
    30. 30. 16
    31. 31. Actual Guess Actual Guess wrong } guesses 73% goodness 100% goodness 16
    32. 32. example comparison graph 17
    33. 33. example 1 comparison graph 2 3 4 5 17
    34. 34. gesture class comparison 18
    35. 35. gesture class comparison 18
    36. 36. gesture class comparison match box 18
    37. 37. magic usage stages Gesture Creation False Positive Gesture Testing Testing 19
    38. 38. magic usage stages Gesture Creation False Positive Gesture Testing Testing 19
    39. 39. gesture design issues support for false positives non-experts 20
    40. 40. gesture design issues support for false positives non-experts 20
    41. 41. define gesture: delete email 21
    42. 42. false positives define gesture: accidentally delete email delete email 21
    43. 43. everyday gesture library EGL 22
    44. 44. everyday gesture library EGL 22
    45. 45. everyday gesture library ? ? ? EGL ? ? ? ? 23
    46. 46. everyday gesture library OK ? X ? OK ? EGL OK ? X ? OK ? X ? 23
    47. 47. EGL data collection accelerometer contextual video 24
    48. 48. magic usage stages Gesture Creation False Positive Gesture Testing Testing 25
    49. 49. magic usage stages Gesture Creation False Positive Gesture Testing Testing 25
    50. 50. false positive testing 26
    51. 51. magic evaluation experiment set up 27
    52. 52. Gestures to define: • Play/Pause • Volume Up 10% • Shuffle • Volume Down 10% • Next Track • Next Playlist • Previous Track • Previous Playlist 28
    53. 53. Gesture criteria: • The gesture must reliably activate the desired function. quantitative • Performing the gesture must not activate other functions. • The functionality associated with a gesture must not be activated by a user’s everyday movements. • The gesture should be easy to remember. qualitative • The gesture should be easy to perform. • The gesture should be socially acceptable. 29
    54. 54. participants Num Num UCD UI design Pattern rec Condition participants Age female experience experience experience no EGL 7 29.0 2 5.9 6.7 4.0 EGL 7 31.6 2 6.6 5.6 3.0 (1-9) (1-9) (1-9) recruited participants with UI and/or UCD experience 30
    55. 55. collected EGLs EGL age gender occupation hours collected 1 32 M PhD CS 19:18 2 26 M MS CS 9:33 3 27 F Librarian 7:40 4 28 M Civil Engineer 6:53 5 30 F IT professional 4:50 6 37 M IT professional 4:08 7 28 F Writer 3:57 8 29 F Project manager 2:09 Total 29.6 4m/4f 58:28 31
    56. 56. collected EGLs EGL age genderActivities EGL occupation hours collected 1 32 • PhD CS M attending conference 19:18 2 26 • MS CS M brewing beer 9:33 3 27 • knitting F vacationing at beach Librarian 7:40 • 4 28 M vacationing at mountains • Civil Engineer 6:53 5 30 • IT professional F working at computer 4:50 6 37 • hiking M IT professional 4:08 • cooking 7 28 F home repair Food writer 3:57 • 8 29 F attending manager • Project work meetings 2:09 • Total 29.6 4m/4fmaking cheese 58:28 32
    57. 57. in-magic EGL 19:18 EGL 1 33
    58. 58. in-magic EGL 5:19 13:59 given to testing users EGL 1 33
    59. 59. magic evaluation quantitative results 34
    60. 60. user performance false positives 35
    61. 61. user performance false positives 12 11 1 10 2 9 3 8 4 7 5 6 EGL: 1.9/gesture/hour 35
    62. 62. user performance false positives 12 12 11 1 11 1 10 2 10 2 9 3 9 3 8 4 8 4 7 5 7 5 6 6 EGL: 1.9/gesture/hour noEGL: 52.1/gesture/hour 35
    63. 63. user performance false positives 12 12 11 1 11 1 10 2 10 2 9 User pattern recognition experience has 9 3 3 no effect on performance! 8 4 8 4 7 5 7 5 6 6 EGL: 1.9/gesture/hour noEGL: 52.1/gesture/hour 35
    64. 64. user performance gesture goodness • Gesture goodness: μ = 86% / σ = 24% • 7 with goodness >= 97% Actual Guess Actual Guess wrong } guesses 73% goodness 100% goodness 36
    65. 65. user performance about 2h to complete task 37
    66. 66. user performance about 2h to complete task mostly linear progression: create, test, EGL 37
    67. 67. user performance about 2h to complete task mostly linear mostly linear progression: progression: add class, make examples, create, test, EGL troubleshoot 37
    68. 68. magic evaluation qualitative results 38
    69. 69. feedback general “This is really hard!” (but…) “I really like the software and I really like the interface” “Gesture creation was easy” “It’s pretty awesome… it’s really fun” “I liked the task” … “I think it’s really interesting” 39
    70. 70. feedback egl “I just kinda feared the EGL” (regarding video) “I didn't care why I was hitting the [EGL]; I can't change what's in there” “It was really useful to compare both hat-mounted videos” (EGL video and gesture video) The EGL tab is “very important” 40
    71. 71. feedback visualizations Graphs were “something a little bit complicated for me” “It takes a while to build up an intuition about what [the graphs] mean” “I thought I needed to understand [the graphs] more” “[The graphs] give a good reflection of what I’ve done” 41
    72. 72. retrospection previous playlist A previous playlist B low goodness high goodness “retrospective realization” 42
    73. 73. retrospection previous playlist A previous playlist B low goodness high goodness “retrospective realization” 42
    74. 74. retrospection previous playlist A previous playlist B low goodness high goodness “retrospective realization” 42
    75. 75. future work • improve speed, responsiveness • easier to understand visualizations • test out other sensors, algorithms • gesture & example annotations 43
    76. 76. see paper for: • related work • users’ gesture strategies • statistics!
    77. 77. Thank you! questions?
    78. 78. ISWC'10 IEEE International Symposium on Wearable Connected Smartware - Wearable Computers Oct. 10th - 13th, 2010, COEX, Seoul, South Korea Topics include On-body, mobile Systems, usability, HCI and human factors in wearable computing, applications of wearable systems Submission deadline 21th, May, 2010 Papers, notes, posters, & design contest 30th, May, 2010 Videos, late breaking results, demonstrations, Ph.D. forum, & tutorials http://www.iswc.net
    79. 79. user performance gesture goodness precision · recall F-measure, or goodness = 2 · harmonic mean of precision + recall precision and recall 47
    80. 80. user performance gesture goodness precision · recall F-measure, or goodness = 2 · harmonic mean of precision + recall precision and recall 89% 95% goodness goodness p=80%, r = 100% p=100%, r = 90% 47
    81. 81. user performance gesture goodness precision · recall F-measure, or goodness = 2 · harmonic mean of precision + recall precision and recall 89% 95% 82% goodness goodness goodness p=80%, r = 100% p=100%, r = 90% 47 p=78%, r = 88%
    82. 82. user performance gesture goodness precision · recall F-measure, or goodness = 2 · harmonic mean of precision + recall precision and recall 89% 95% 82% 100% goodness goodness goodness goodness p=80%, r = 100% p=100%, r = 90% 47 p=78%, r = 88% p=100%, r = 100%
    83. 83. gesture design tool desiderata 48
    84. 84. gesture design tool desiderata Allow non-expert use 48
    85. 85. gesture design tool desiderata Allow non-expert use (1-9) Num UI design Pattern rec Condition participants Age Num female UCD experience experience experience noEGL 7 29.0 2 5.9 6.7 4.0 EGL 7 31.7 2 48 6.3 5.1 3.1
    86. 86. gesture design tool desiderata Allow non-expert use 87% mean goodness (1-9) Num UI design Pattern rec Condition participants Age Num female UCD experience experience experience noEGL 7 29.0 2 5.9 6.7 4.0 EGL 7 31.7 2 48 6.3 5.1 3.1
    87. 87. gesture design tool desiderata Allow expert use 49
    88. 88. gesture design tool desiderata B Iterative design 50
    89. 89. gesture design tool desiderata Retrospection 51
    90. 90. gesture design tool desiderata Support further testing 52
    91. 91. gesture design tool desiderata Support further testing 52
    92. 92. EGL limitations • not guaranteed to span space of everyday life • EGL is defined by (sensor, situation, target user population) 53
    93. 93. support for non-experts 54

    ×