Pen-Based Gesture Recognition

  • 1,864 views
Uploaded on

Lecture given at the Faculty of Informatics, University of Lugano, May 2007

Lecture given at the Faculty of Informatics, University of Lugano, May 2007

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,864
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
93
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Pen-Based Gesture Recognition Beat Signer Institute for Information Systems, ETH Zurich 8092 Zurich, Switzerland http://vub.academia.edu/BeatSigner USI Seminar, May 30, 2007
  • 2. Outline  Motivation  Digital Pen and Paper Functionality  Gesture recognition algorithms  Evaluation of algorithms  iGesture framework USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 2
  • 3. Pen-Based User Interfaces Light Pen Graphics Tablet PDAs and Tablet PCs Digital Pen and Paper USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 3
  • 4. Palm Graffiti Gestures USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 4
  • 5. Microsoft Application Gestures scratch-out erase content double-circle paste triangle insert left-semicircle undo square action item right-semicircle redo star action item caret past/insert check check-off inverted-caret insert curlicue cut chevron-left application-specific double-curlicue copy chevron-right application-specific circle application-specific arrow-up application-specific USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 5
  • 6. Customised Gestures USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 6
  • 7. Gesture Design Guidelines  Different gestures should not look too similar  better recognition results  Gestures should be user friendly  intuitivity and usability  Gesture set size  Reuse of gestures  same semantics for different applications  application-specific gestures USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 7
  • 8. Digital Pen and Paper  Developed by Anoto  Different pen manufacturers  Logitech, Maxel, Nokia, ...  Pattern space  60 million km2 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 8
  • 9. Digital Ink Processing  Parse proprietary pen formats  position, timestamp, force, tilt, …  Stroke detection  pen up / down events  time difference between succeeding points  distance between succeeding points, …  Interpolation / Filtering USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 9
  • 10. Note Representation <note> <trace> <point> <x>59.664</x><y>56.364</y> <timestamp>1103123178464</timestamp> <force>224.0</force> </point> <point> <x>59.671</x><y>56.378</y> <timestamp>1103123178484</timestamp> <force>227.0</force> </point> ... <trace> ... </note> USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 10
  • 11. Ink Markup Language (InkML), W3C  XML language for representing digital ink  Pen movement data  <trace> and <traceFormat> elements  x,y coordinates, force, angles  Device-specific information  resolution, sample rate, … USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 11
  • 12. Ink Markup Language (InkML) <ink> <trace>10 0, 9 14, 8 28, 7 42, 6 56, 6 70, 8 84, 8 98, 8 112, 9 126, 10 140, 13 154, 14 168, 17 182, 18 188 </trace> <trace>130 155, 144 159, 158 160, 170 154, 179 143, 179 129, 166 125, 152 128, 140 136, 131 149, 126 163 </trace> ... <inkSource manufacturer="magicomm" model="G303"> <sampleRate uniform="true" value="70"/> <srcProperty name="weight" value="100" units="g"/> </inkSource> ... </ink> USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 12
  • 13. Gesture Recognition Algorithms  Offline recognition algorithms  static image  Online recognition algorithms  spatio-temporal representation  Recognition methods  statistical classification, neural networks, …  Supported gesture types  single-stroke or multi-stroke USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 13
  • 14. SiGeR Algorithm  Simple Gesture Recogniser (Scott Swigart)  Gestures described by regular expressions  N, NE, E, SE, S, SW, W and NW W (E)+(N)+(W)+(S)+ S N (NE|E|SE)+(NW|N|NE)+ (SW|W|NW)+(SE|S|SW)+ E USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 14
  • 15. SiGeR Algorithm ...  Use additional statistical information Description = Directions [";" Constraints]; Directions = Direction ["," Directions]; Direction = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW"; Constraints = Constraint ["AND" Constraints]; Constraint = Operand Operator Operand; Operator = "EQ"|"NEQ"|"GT"|"GTE"|"LT"|"LTE"; Operand = "N"|"NE"|"E"|"SE"|"S"|"SW"|"W"|"NW"| "DIAGONAL"|"STRAIGHT"|"PROXIMITY"|"STOPPOINTS"; USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 15
  • 16. SiGeR Algorithm ... W  Examples E,N,W,S;STRAIGHT GT 0.8 S N AND PROXIMITY LT 0.2 E W,SW,SE,NE,N,S; PROXIMITY GT 0.7 AND STRAIGHT GT 0.3 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 16
  • 17. Rubine Algorithm, 1991  Statistical classification algorithm for single stroke gestures (training / classification)  A gesture G is represented as vector of P sample points G  s0 ,... sP1, with si  xi , yi , ti   Feature vector f extracted from G f   f1 ,... f F  USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 17
  • 18. Rubine Features ( x2  x0 ) f1  cos   ( x2  x0 ) 2  ( y2  y0 ) 2 ( y 2  y0 ) f 2  sin   ( x2  x0 ) 2  ( y2  y0 ) 2 f 3  ( xmax  xmin ) 2  ( ymax  ymin ) 2 ymax  ymin f 4  arctan xmax  xmin f5  xP 1  x0 2   yP 1  y0 2 ( xP 1  x0 ) f 6  cos   f5 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 18
  • 19. Rubine Features … f 7  sin    yP 1  y0  f5 Let xi  xi 1  xi yi  yi 1  yi P2 f 8   xi  yi 2 2 i 0 xi yi 1  xi 1yi Let  i  arctan xi xi 1  xi yi 1 P2 P 2 P2 f9  i f10    i f11    i 2 i 1 i 1 i i x  y P 2 2 2 Let ti  ti 1  ti f12  max i 2 i i 0 ti f13  t P 1  t0 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 19
  • 20. Rubine Training / Classification  Training phase gesture samples for class c Optimal Classifier wc  wc 0 ,..., wcF  ˆ ˆ ˆ  Recognition / classification phase F vcˆ  wcˆ 0   wcˆi f i i 1 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 20
  • 21. SiGrid Algorithm  Signature algorithm 000000 000001 000101 000100 100100 100101 100001 100000 000010 000011 000111 000110 100110 100111 100011 10010 001010 001011 001111 001110 101110 101111 101011 101010 001010 001001 001101 001100 101100 101101 101001 101000 Example 011000 011011 011111 011110 111100 111101 111001 111000 8 x 8 Grid 011010 011011 0111111 011110 111110 111111 111011 111010 010010 010011 010111 010110 110110 110111 110011 110010 010000 010001 010101 010100 110100 110101 110001 110000 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 21
  • 22. Evaluation Grafitti Numbers E-Rubine Rubine SiGrid SiGeR Correct 147 134 132 133 Error 3 15 15 4 Reject 0 1 3 13 Number of gesture classes: 10 Training: 15 examples for each gesture class (collected by 1 person) Test Samples: 150 (collected by 3 different persons) USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 22
  • 23. Evaluation TP TP : true positive Precision  TP  FP FP : false positive FN : false negative TP Recall  TP  FN 2  Precison  Recall F  Measure  Precision  Recall USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 23
  • 24. Evaluation Grafitti Numbers E-Rubine Rubine SiGrid SiGeR Correct 147 134 132 133 Error 3 15 15 4 Reject 0 1 3 13 Precision 0.980 0.899 0.898 0.970 Recall 1.000 0.993 0.978 0.911 F-Measure 0.990 0.944 0.936 0.940 Number of gesture classes: 10 Training: 15 examples for each gesture class (collected by 1 person) Test Samples: 150 (collected by 3 different persons) USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 24
  • 25. Evaluation Grafitti Letters (1) E-Rubine Rubine SiGrid Correct 334 280 273 Error 52 107 114 Reject 4 3 3 Precision 0.865 0.724 0.705 Recall 0.988 0.989 0.989 F-Measure 0.923 0.836 0.824 Number of gesture classes: 26 Training: 15 examples for each gesture class (collected by 1 person) Test Samples: 390 (collected by 3 persons) USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 25
  • 26. Evaluation Grafitti Letters (2) E-Rubine Rubine SiGrid Correct 342 305 297 Error 18 48 66 Reject 3 10 0 Precision 0.950 0.864 0.818 Recall 0.991 0.968 1.000 F-Measure 0.970 0.913 0.900 Number of gesture classes: 26 Training: 4x4 examples for each gesture class (collected by 4 persons) Test Samples: 363 (collected by 4 persons) USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 26
  • 27. Evaluation MS Application Gestures E-Rubine Rubine SiGrid Correct 196 178 145 Error 4 19 32 Reject 0 3 23 Precision 0.980 0.904 0.819 Recall 1.000 0.983 0.863 F-Measure 0.990 0.942 0.840 Number of gesture classes: 40 Training: 15 examples for each gesture class (collected by 1 person) Test Samples: 200 (collected by 1 person) USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 27
  • 28. Multi-Stroke Gestures USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 28
  • 29. Evaluation Multi-Stroke Gestures E-Rubine Rubine SiGrid Correct 75 72 73 Error 0 3 2 Reject 0 0 0 Precision 1.000 0.960 0.973 Recall 1.000 1.000 1.000 F-Measure 1.000 0.980 0.986 Number of gesture classes: 15 Training: 15 examples for each gesture class (collected by 1 person) Test Samples: 75 (collected by the same person) USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 29
  • 30. iGesture Framework  Exercise  download iGesture  analyse the design of the framework  investigate the existing algorithms  design and test your own algorithm  build a small application based on your recogniser USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 30
  • 31. iGesture Architecture Overview Management Evaluation Console Tools Recogniser Common Data Structures USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 31
  • 32. Gesture Representation USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 32
  • 33. Recognition Algorithm USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 33
  • 34. Persistence USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 34
  • 35. iGesture Test Bench Tab USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 35
  • 36. iGesture Admin Tab USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 36
  • 37. iGesture Test Data Tab USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 37
  • 38. Capture of Samples USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 38
  • 39. Evaluation Tools USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 39
  • 40. Recogniser API Configuration configuration = XMLTool.importConfiguration(new File(RUBINE)); GestureSet gestureSet = XMLTool.importGestureSet(new File(RUBINE_SET)); configuration.addGestureSet(gestureSet); recogniser = new Recogniser(configuration); ... ResultSet result = recogniser.recognise(note); USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 40
  • 41. Conclusions  Digital pen and paper technology  Gesture recognition algorithms  SiGeR, Rubine (E-Rubine), SiGrid  Evaluation  precision, recall and F-measure  General iGesture recognition framework USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 41
  • 42. References  Beat Signer, Ueli Kurmann and Moira C. Norrie, iGesture: A General Gesture Recognition Framework, Proc. of ICDAR 2007, 9th Intl. Conference on Document Analysis and Recognition, Curitiba, Brazil, September 2007 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 42
  • 43. References ...  Beat Signer, Moira C. Norrie and Ueli Kurmann, iGesture: A Java Framework for the Development and Deployment of Stroke-Based Online Gesture Recognition Algorithms, Technical Report ETH Zurich, TR561, September 2007 USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 43
  • 44. References ...  Dean Rubine, Specifying Gestures by Example, Proc. of ACM SIGGRAPH '91, 18th Intl. Conference on Computer Graphics and Interactive Techniques  iGesture Open Source Framework  http://www.igesture.org USI Seminar, May 30, 2007 Beat Signer, signer@inf.ethz.ch 44