Fishing or a Z?: Investigating the Effects of Error on
    Mimetic and Alphabet Device-based Gesture Interaction"Oct.	  23...
2
Outline"    I.     Introduc5on	      II.    Methods	      III.  Results	      IV.  Discussion	      V.     Future	  Work	  3
Introduction4
Introduction"             Device-­‐based	  gestures:	  	                 Gesturing	  by	  moving	  a	  smartphone	      ...
Motivation"        But	  errors	  are	  an	  inevitable	  part	  of	           interac5on	  with	  technology…	         ...
Related Work"             Gesture-­‐based	  Interac5on	                  Gesture-­‐Task	  mapping	  (Khnel	  et	  al.,	 ...
Iconic Gestures"        Mimetic / Pantomimic Gestures: natural, familiar, easy to learn, varied by         activities    ...
Research Question"    	  What	  are	  the	  effects	  of	  unrecognized	  gestures	  on	  user	  experience,	  and	        ...
Methods10
Mimetic Gesture Design"11
Alphabet Gesture Design"12
Study Design"               Qualita5ve	  Study;	  Automated	  Wizard-­‐of-­‐Oz	  method	  (Fabbrizio	  et	  al.,	  2005)	...
!"       !#                                                                                  !"#$%$&                      ...
Setup & Procedure"                                                                                          ≈1 hr         ...
#: ____               MENTAL DEMAND                 Low                        High               PHYSICAL DEMAND         ...
Results17
Workload"                                  !"#$%"&($)%*+$)                                                           !    ...
Workload"                               !"#$%&()*+(,-+                                 !                    !! !          ...
Workload"       Mime5c	  gestures	  are	  beder	  tolerated	  up	  to	  error	  rates	  of	  40%	  (cf.,	  Karam	  &	    ...
Observations"       Mime5c	  gestures	  evolve	  into	  real-­‐world	  counterparts	  under	  error,	  alphabet	         ...
User Feedback"       Perceived	  Canonical	  Varia5ons	            S9:	  “ The	  shaking,	  that	  was	  the	  hardest	 ...
Discussion23
Limitations"       Lab-­‐study	         Task	  Independence	         Random	  error	  distribu5on	  24
Implications for Gesture Recognition"       Mime5c	  gestures	  evolve	  into	  real-­‐world	  counterparts	  under	  err...
Implications for Gesture-based Interaction"       40%	  error	  tolerance	  in	  line	  with	  previous	  work	  (Karam	 ...
Future Work27
Future Work"       Quan5ta5ve	  assessment	  of	  how	  many	  errors	  precisely	  before	  canonical	  variant?	       ...
Questions29               h6p://staff.science.uva.nl/~elali/	  
References"     Brewster,	  S.	  Providing	  a	  structured	  method	  for	  integra5ng	  non-­‐speech	  audio	  into	  hu...
Upcoming SlideShare
Loading in …5
×

Fishing or a Z?: Investigating the Effects of Error on Mimetic and Alphabet Device-based Gesture Interaction

923
-1

Published on

Slides for the talk I gave at ICMI 2012, held in Santa Monica, CA, USA.

The full paper reference is:

El Ali, A., Kildal, J. & Lantz, V. (2012). Fishing or a Z?: Investigating the Effects of Error on Mimetic and Alphabet Device-based Gesture Interaction. In Proceedings of the 14th international conference on Multimodal Interaction (ICMI '12), 2012, Santa Monica, California.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
923
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Fishing or a Z?: Investigating the Effects of Error on Mimetic and Alphabet Device-based Gesture Interaction

  1. 1. Fishing or a Z?: Investigating the Effects of Error on
 Mimetic and Alphabet Device-based Gesture Interaction"Oct.  23,  2012 Abdallah  ‘Abdo’  El  Ali      Johan  Kildal   Vuokko  Lantz     h6p://staff.science.uva.nl/~elali/  
  2. 2. 2
  3. 3. Outline" I.  Introduc5on   II.  Methods   III.  Results   IV.  Discussion   V.  Future  Work  3
  4. 4. Introduction4
  5. 5. Introduction"   Device-­‐based  gestures:       Gesturing  by  moving  a  smartphone   device  in  3D  space     Research  seGngs,  home   environments,  everyday  mobile   interac5on     Alterna5ve  to  when  users  are   encumbered  (e.g.,  manual   mul5tasking)     Natural     Promising  alterna5ve  to  mobile   touchscreen/keyboard  input  under   situa5onal  impairments  5
  6. 6. Motivation"   But  errors  are  an  inevitable  part  of   interac5on  with  technology…     Many  gesture  classes  are  available   (e.g.,  iconic,  symbolic,  deic5c)  for  use   in  smartphones,  but  which  have   minimum  user  frustra5on  when   recogni5on  errors  occur?     We  inves5gate  user  error  tolerance   for  two  iconic  gesture  sets  used  in   HCI:  mime5c  and  alphabet  gestures  6
  7. 7. Related Work"   Gesture-­‐based  Interac5on     Gesture-­‐Task  mapping  (Khnel  et  al.,  2011;  Ruiz  et  al.,  2011)     Social  acceptability  (Rico  &  Brewster,  2010)     Gesture  Taxonomies:  Deic5c,  symbolic,  physical,  mime5c/pantomimic,  abstract  (‘  m/  ’)   (Rime  &  Schiaratura,  1991;  …)     Recogni5on  Errors     Speech:  Repeat  (Suhm  et  al.,  2001)  and  hyperar5culate  (Oviad  et  al.,  1998)     Mul5modal:  Modality-­‐switching  (“spiral  depth”  of  6)  (Oviad  &  van  Gent,  1996)     Touch-­‐less  Vision-­‐based  Gesture  Recogni5on:  How  many  errors  before  switching  to   keyboard  input?  40%  user  error  tolerance  before  modality  switching  (Karam  &   Schraefel,  2006)     Lidle  work  on  which  gesture  sets  are  most  robust  to  errors  during   gesture-­‐based  interac5on!  7
  8. 8. Iconic Gestures"   Mimetic / Pantomimic Gestures: natural, familiar, easy to learn, varied by activities   e.g., Fishing, calling, …   Alphabet Gestures: familiar, easy to learn, varied by stroke   e.g., letter “C”, letter “S”, …8
  9. 9. Research Question"  What  are  the  effects  of  unrecognized  gestures  on  user  experience,  and   what  are  the  differences  between  mime5c  and  alphabet  gestures  (under   varying  error  rates:  0-­‐20%,  20-­‐40%,  40-­‐60%)?    Hypotheses:        Mime5c  gestures  →  users  less  familiar  with  ideal  shape  →  more  gesture   varia5on  under  high  error  rates  →  but  lower  subjec5ve  workload  due  to   higher  degrees  of  freedom        Alphabet  gestures  →  users  more  familiar  with  ideal  shape  →  more  rigid   gestures  under  increasing  error  rates  →  but  higher  subjec5ve  workload  due   to  lower  degrees  of  freedom  9
  10. 10. Methods10
  11. 11. Mimetic Gesture Design"11
  12. 12. Alphabet Gesture Design"12
  13. 13. Study Design"   Qualita5ve  Study;  Automated  Wizard-­‐of-­‐Oz  method  (Fabbrizio  et  al.,  2005)     24  subjects  (16  male,  8  female)  aged  between  22-­‐41  (M=  29.6,  SD=  4.5)     Mixed  between-­‐  and  within  subject  factorial  design:  2  (gesture  type:  mime5c  vs.   alphabet)  x  3  (error  rate:  low  vs.  med  vs.  high)     Experiment  in  Presenta5on®,  Wii  Remote®  interac5on  using  GlovePie™     Random  error  distribu5on  across  trials  (20  prac5ce,  180  test)     Tutorial  &  videos  given  of  how  to  properly  perform  each  gesture     Data  collected:   1.  Modified  NASA-­‐TLX  workload  [0-­‐20  range]  ques5onnaire  data  (Hart  &  Wickens,  1990;  Brewster,   1994)   2.  Experiment  logs     3.  Video  recordings  of  subjects’  gesture  interac5on  13 4.  Post-­‐experiment  interviews  
  14. 14. !" !# !"#$%$& ()*$+, 8()*$+ 89:".#),2&"()*$+ !" !# !$ !% !& ! ;*<=""*" -./&%$ 0, 1)23#4&5 $*67.%*6 -&7=""*" >.?3=""*" @()*$+,2&"=""*"$*67.%*6 A&""*"$*67.%*6B4)*$+*"7&"$*C65&"4#)#6$&7D"#67*/.E#%*6F14
  15. 15. Setup & Procedure" ≈1 hr Reward Exit Interview x3 NASA-TLX Questionnaire Automated Wizard-of-Oz Test [...] Block Practice Block Instructions Perform & Tutorial Fishing Gesture Personal Information Form Perform Fishing Experiment Gesture Session Fishing Gesture t Test Block15
  16. 16. #: ____ MENTAL DEMAND Low High PHYSICAL DEMAND Low High TIME PRESSURE NASA-TLX Low High (Hart & Wickens, 1990) EFFORT EXPENDED Low High PERFORMANCE LEVEL ACHIEVED Poor Good FRUSTRATION EXPERIENCED Low High  ANNOYANCE EXPERIENCED Low High (Brewster, 1994) OVERALL PREFERENCE RATING16 Low High
  17. 17. Results17
  18. 18. Workload" !"#$%"&($)%*+$) ! ! !! ! ! ! !! !! !! %& & 0$ ," # " : ,% $+/ #, /% ../ $ "# %# *+ 0% 0" 9& - () ! 01 /0 $ ." 3* "8 ./ 0" 20 "0 +7 ,$ 6" !! 35 4 !18 ;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"
  19. 19. Workload" !"#$%&()*+(,-+ ! !! ! !! !! !! ! !! ! !! ! !! !! %& %& 0$ " # " : , $+/ #, /% ../ $ +, "# %# * $0% 0" 9& - () ! 1 /0 ." 3* 0 "8 ./ 0" 20 "0 $+7 !! , 6" 35 ! 4 ;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"19
  20. 20. Workload"   Mime5c  gestures  are  beder  tolerated  up  to  error  rates  of  40%  (cf.,  Karam  &   Schraefel,  2006),  compared  with  error  rates  of  up  to  only  20%  for  alphabet   gestures       From  a  usability  perspec5ve,  mime5c  gestures  more  robust  to  recogni5on   failures  than  alphabet  gestures  20
  21. 21. Observations"   Mime5c  gestures  evolve  into  real-­‐world  counterparts  under  error,  alphabet   gestures  tend  to  become  more  rigid  and  well  structured     Canonical  Varia5ons  via  posi5ve  reinforcement:  Survival  of  the  fidest  gesture   varia5ons     E.g.,  S12  exhibited  real-­‐world  varia5ons  on  both  the  Fishing  and  Trashing   gestures     Varia5ons  develop  as  low  as  spiral  depth  of  2  (i.e.,  min.  2  recogni5on  errors)  21
  22. 22. User Feedback"   Perceived  Canonical  Varia5ons     S9:  “ The  shaking,  that  was  the  hardest  one  because  you  couldn’t  just  shake  freely  [gestures   in  hand],  it  had  to  be  more  precise  shaking  [swing  to  the  leA,  swing  to  the  right]  so  not  just   any  sort  of  shaking  [shakes  hand  in  many  dimensions]”     Cultural  and  Individual  differences       S10:  “For  the  glass  filling,  there  are  many  ways  to  do  it.  SomeFmes  very  fast,  someFmes   slow  like  beer.”     Perceived  Performance     Ad  hoc  explana5ons  (e.g.,  fa5gue)  given  why  there  were  more  errors  in  some  blocks     S18:  “[Performance]  between  the  first  and  second  blocks  [baseline  and  low  error  rate   condiFons],  it  was  the  same...  10-­‐15%.”     Social  Acceptability     Alphabet  gestures  less  socially  acceptable  when  they  fail     S16:  “When  it  doesn’t  take  your  C,  you  keep  doing  it,  and  it  looks  ridiculous.”  22
  23. 23. Discussion23
  24. 24. Limitations"   Lab-­‐study     Task  Independence     Random  error  distribu5on  24
  25. 25. Implications for Gesture Recognition"   Mime5c  gestures  evolve  into  real-­‐world  counterparts  under  error,  alphabet   gestures  tend  to  become  more  rigid  and  well  structured        One  shot  recogni5on  for  mime5c  gestures  important!     Interes5ng  explana5ons  (e.g.,  canonical  varia5ons)  and  causes  (e.g.,  fa5gue)   given  why  there  were  more  errors  in  some  blocks      Transparency  in  gesture  recogni5on  technology  may  beder  support  users  in   error  handling  strategies  25
  26. 26. Implications for Gesture-based Interaction"   40%  error  tolerance  in  line  with  previous  work  (Karam  &  Schraefel,  2006),  which   shows  usability  of  gesture-­‐based  interac5on     Mime5c  gestures  overall  have  beder  user  experience,  more  use  cases,  and  thus   more  suitable  for  device-­‐based  gesture  interac5on  (even  under  high  recogni5on   error!)  26
  27. 27. Future Work27
  28. 28. Future Work"   Quan5ta5ve  assessment  of  how  many  errors  precisely  before  canonical  variant?     Other  classes  of  gestures  (e.g.,  manipula5ve)     Influence  of  device  form  factor  28
  29. 29. Questions29 h6p://staff.science.uva.nl/~elali/  
  30. 30. References" Brewster,  S.  Providing  a  structured  method  for  integra5ng  non-­‐speech  audio  into  human-­‐computer  interfaces.  PhD   thesis,  University  of  York,  1994.   Fabbrizio,  G.  D.,  Tur,  G.,  and  Hakkani-­‐Tur,  D.  Automated  wizard-­‐of-­‐oz  for  spoken  dialogue  systems.  In  Proc.   INTERSPEECH  2005  (2005),  1857–1860.   Karam,  M.,  and  Schraefel,  M.  C.  Inves5ga5ng  user  tolerance  for  errors  in  vision-­‐enabled  gesture-­‐based  interac5ons.   In  Proc.  AVI  ’06,  ACM  (NY,  USA,  2006),  225–232.   Khnel,  C.,  Westermann,  T.,  Hemmert,  F.,  Kratz,  S.,  Mller,  A.,  and  Muller,  S.  I’m  home:  Defining  and  evalua5ng  a   gesture  set  for  smart-­‐home  control.  Interna5onal  Journal  of  Human-­‐Computer  Studies  69,  11  (2011),  693  –  704.   Hart,  S.,  and  Wickens,  C.  Manprint:  an  Approach  to  Systems  Integra5on.  Van  Nostrand  Reinhold,  1990,  ch.  Workload   Assessment  and  Predic5on,  257–292.   Oviad,  S.,  Maceachern,  M.,  and  Anne  Levow,  G.  Predic5ng  hyperar5culate  speech  during  human-­‐computer  error   resolu5on.  Speech  Communica5on  24  (1998),  87–110.  17.   Oviad,  S.,  and  Van  Gent,  R.  Error  resolu5on  during  mul5modal  human-­‐computer  interac5on.  In  Proc.  ICSLP  ’96,  vol.   1  (oct  1996),  204–207.   Rico,  J.,  and  Brewster,  S.  Usable  gestures  for  mobile  interfaces:  evalua5ng  social  acceptability.  In  Proc.  CHI  ’10,  ACM   (NY,  USA,  2010),  887–896.   Rime,  B.,  and  Schiaratura,  L.  Fundamentals  of  Nonverbal  Behavior.  Cambridge  University  Press,  1991,  ch.  Gesture   and  speech,  239–281.   Ruiz,  J.,  Li,  Y.,  and  Lank,  E.  User-­‐defined  mo5on  gestures  for  mobile  interac5on.  In  Proc.  CHI  ’11,  ACM  (NY,  USA,   2011),  197–206.   Suhm,  B.,  Myers,  B.,  and  Waibel,  A.  Mul5modal  error  correc5on  for  speech  user  interfaces.  ACM  Trans.  Comput.-­‐ Hum.  Interact.  8  (2001),  60–98.  30
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×