— Several algorithms have been developed for pen-based gesture recognition. Yet, their integration in streamline engineering of interactive systems is bound to several shortcomings: they are hard to compare to each other, determining which one is the most suitable in which situation is a research problem, their performance largely vary depending on contextual parameters that are hard to predict, their fine-tuning in a real interactive application is a challenge. In order to address these shortcomings, we developed UsiGesture, an engineering method and a software support platform that accommodates multiple algorithms for pen-based gesture recognition in order to integrate them in a straightforward way into interactive computing systems. The method is aimed at providing designers and developers with support for the following steps: defining a dataset that is appropriate for an interactive system (e.g., made of com¬mands, symbols, characters), determining the most suitable gesture recognition algorithm depending on contextual variables (e.g., user, platform, environment), fine-tuning the parameters of this algorithm by multi-criteria optimization (e.g., system speed and recognition rate vs. human distinguishability and perception), and incorporation of the fine-tuned algorithm in an integrated development environment for engineering interactive systems.

1. François Beuvens and Jean Vanderdonckt
francois.beuvens@uclouvain.be
Researcher at LiLab, http://www.lilab.be
Université catholique de Louvain (Belgium)
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 1

2. - Context
- An open platform for benchmarking gesture
recognition algorithms
- Including the results in user interfaces
- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 2

3. - Context
- An open platform for benchmarking gesture
recognition algorithms
- Including the results in user interfaces
- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 3

4. - Scope: 2D pen-based gestures
- Needs of pen-based gesture recognition
- PDAs – Graffiti
- Mouse gestures plug-in for Firefox
- Games
- …
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 4

5. - No « One fits all » algorithm
- Needs of algorithms comparison
- Same data and conditions
- Data acquisition
- Enhancements of existing algorithms
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 5

6. - Context
- An open platform for benchmarking gesture
recognition algorithms
- Including the results in user interfaces
- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 6

7. 17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 7

8. - 30 participants
- 16 men and 14 women with average age of 28
- 60 gestures to reproduce
- 26 lower case letters, 10 digits, 16 action commands, 8
geometrical shapes
- 10 reproductions of each gesture by participant
- More information:
https://sites.google.com/site/comparativeevaluation/data-collection
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 8

9. - Training types
- User dependent, user independent (2 methods)
- Gesture types
- Letters, digits, actions, geometrical shapes
- Algorithms
- Rubine, One Dollar, Levenshtein, stochastic Levenshtein
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 9

10. - Training
- Compute mean vector of features and weights for learning
classes
- Recognition
- Compute vector of features
- Scalar product of weights
and vector
- Best score is chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 10

11. - Training
- Preprocessing: resampling, rotation, rescaling, translation
- Recognition
- Preprocessing
- Comparison to all training examples
- Finding optimal angle
- Evaluating point-to-point Euclidean distance
- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 11

12. - Training
- Transformation into strings
- Recognition
- Comparison to all training examples
- Computing Levenshtein distance
- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 12

13. - Training
- Transformation into strings
- Creation of a string pairs set
- Optimization of the primitive edition costs
- Expectation-maximization algorithm
- Recognition
- Comparison to all training examples
- Evaluating distance based on probabilities
- Smaller distance chosen
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 13

14. - Training types for Levenshtein on digits
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 14

15. - Gesture types for Rubine with user dependent
training and user independent training
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 15

16. - Algorithms on digits and actions with user
independent training
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 16

17. - Summary
User-dep. User-indep. 1 User-indep. 2
Digits $1 $1 $1
Letters $1 $1 $1
Actions Sto. Lev. Rub. Sto. Lev.
Shapes Rub. Rub. $1
- More Information:
https://sites.google.com/site/comparativeevaluation/results
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 17

18. - Rubine
- Preprocessings
- One Dollar
- kNN
- Levenshtein
- Preprocessings, kNN, normalisations, multi-strokes
gestures, other features handling, costs matrix
- Stochastic Levenshtein
- Idem except costs matrix
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 18

19. - Preprocessings: resampling, rotation, rescaling
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 19

20. - kNN
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 20

21. - Normalizations, pre-processings, kNN, multi-
strokes, pressure inclusion, costs
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 21

22. - Normalizations, preprocessings, kNN, multi-
strokes, pressure inclusion
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 22

23. - Summary Rub. $1 Lev. Sto. Lev.
Resampling + X ++ ++
Rescaling + X X X
Rotation - X - -
kNN X - - -
Multi-stroke X X + =
Multi-stroke +
X X ++ ++
resampling
Pressure X X -- -
Normalization 1 X X + -
Normalization 2 X X = -
Normalization 3 X X = -
Adapted costs
matrix X X + X
- More Information:
https://sites.google.com/site/comparativeevaluation/results
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 23

24. - Context
- An open platform for benchmarking gesture
recognition algorithms
- Including the results in user interfaces
- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 24

25. 17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 25

26. 17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 26

27. 17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 27

28. - Context
- An open platform for benchmarking gesture
recognition algorithms
- Including the results in user interfaces
- Ongoing and future works
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 28

29. - Theoretical
- Conceptual modeling of gesture-based UIs
- Methodological
- Integration into UsiXML
- Definition of development lifecycle (e.g. SPEM)
- Empirical
- User study on user satisfaction with gesture integration
- Development benefits estimation
17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 29

30. 17/05/2012 RCIS’12: Sixth International Conference on Research Challenges in Information Science 30