Paper presented at ACM EICS '22 Finger-based gesture input becomes a major interaction modality for surface computing. Due to the low precision of the finger and the variation in gesture production, multistroke gestures are still challenging to recognize in various setups. In this paper, we present µV, a multistroke gesture recognizer that addresses the properties of articulation, rotation, scaling, and translation invariance by combining $P+'s cloud-matching for articulation invariance with !FTL's local shape distance for RST-invariance. We evaluate µV against five competitive recognizers on MMG, an existing gesture set, and on two new versions for smartphones and tablets, MMG+ and RMMG+, a randomly rotated version on both platforms. µV is significantly more accurate than its predecessors when rotation invariance is required and not significantly inferior when it is not. µV is also significantly faster than others with many samples and not significantly slower with few samples

2. 𝜇𝑉: An Articulation, Rotation, Scaling, and Translation
Invariant (ARST) Multi-stroke Gesture Recognizer
Nathan Magrofuoco, Paolo Roselli, Jean Vanderdonckt

3. 1
About 2D Stroke Gestures
● Well-known examples on smartphones: “swipe” and “pinch”
● Often system-defined (not user-defined), specific to each OS
● No customization: gesture recognizers are unable to classify new gestures

4. 2
Machine Learning recognizers
● Highly accurate with complex gestures
● Many training templates
● Difficult-to- understand / modify / train / explain
Template Matching recognizers
● Highly accurate with simple gestures
● Few training templates that are pre-processed
and compared
● Easy-to- understand / integrate / train / explain

5. 3
How Template Matching Recognizers Work
● Pre-processing: convert each training template into a gesture model
● Classification: pre-process and compare a candidate to each training template by
matching the corresponding models and measuring their similarity.
Candidate vs. Letter V
= 0.07 (very similar)
Letter V Caret down
Candidate Candidate vs. Caret down
= 0.49 (not so similar)

6. 3
How Template Matching Recognizers Pre-process Training Templates
● Resampling: the original M points are transformed in N equidistantly spaced points:
N=64 is adequate, in general 32 ≤ N ≤ 256
● Rotation to 0°: compute the angle between the
centroïd and the first point, then rotate to 0°
Problem: the first point is not always the same
(different articulations)
● Rescaling: scale of a reference square
● Translation: move centroid (xM,yM) to (0,0)
Source: Radu-Daniel Vatavu. 2011. The effect of sampling rate on the performance of template-based gesture recognizers.
ICMI '11, pp. 271–278. DOI:https://doi.org/10.1145/2070481.2070531

7. 3
Example
Add a gesture Recognize a gesture
Source: https://www.linkedin.com/pulse/gesture-recognition-android-tej-partap-singh/

8. 7
Properties of Invariance
● Some recognizers remove variations along one or several dimensions (or properties).
● There are 5 major properties of invariance in gesture recognition for fast prototyping:
sampling, Articulation, Rotation, Scaling, and Translation (ARST-invariance)
1. Pre-processing
● Sampling
● Scaling
● Translation
2. Matching
● Articulation
● Rotation
● azeazaz
3. Similarity Metric (LSD)
● Rotation
● Scaling
● Translation

9. 7
When rotation invariance may be needed
Source: https://reves-d-espace.com/les-femmes-et-exploration-espace/

10. 11
Starting point
● $N and $N-Protractor satisfy ARST-invariant multi-stroke recognition
● Articulation-invariance is provided by generating all uni-stroke permutations, which
causes a combinatorial explosion if multi-strokes are constituted of 4+ strokes
Observation
● $P, $P+, and $Q implement cloud matching to satisfy Articulation-invariance.
● !FTL implements the Local Shape Distance (LSD) to satisfy RST-invariance.
Solution (µV or multi-Vectors)
● Combine cloud matching and the LSD to satisfy ARST-invariant recognition.

11. 13
Euclidean Distance
● The Euclidean distance between two points measures how similar they are.
● Provided that gesture points are pre-processed in scaling and translation:
we can measure the ST-invariant similarity between two gestures as the sum of their
between-points Euclidean distance.
Local Shape Distance
● The Euclidean distance between two shapes measures how similar they are.
● Provided that gesture points are converted into a series of shapes:
we can measure the RST-invariant similarity between two gestures as the sum of
their between-shapes Euclidean distance.

12. 15a
Example

13. 15b
Example

14. 15c
Example

15. 15d
Example

16. 15e
Example

17. 16
Accuracy on Uni-Stroke Datasets
● µV reached >90% in the largest conditions
● Significantly less accurate than the other recognizers
Accuracy on Multi-Stroke Datasets
● µV reached >90% on the MMG dataset
● Significantly less accurate than the other recognizers
Speed
● µV returned a result under the 50 ms time constraint in all conditions.
● Significantly faster than $P+ but not !FTL.

18. 17
Datasets Collection: MMG+ and RMMG+
● Two new datasets acquired with a smartphone and tablet with classes from MMG.
● 24 (participants) x 5 (samples per class) x 16 (classes) x 2 (devices) = 3840 gestures.
● RMMG+ consists of synthetically-rotated samples from MMG+ to evaluate Rotation-
invariant recognition.
Accuracy on the RMMG+ Dataset
● µV reached >90% in the user-dependent and the largest user-independent conditions.
● And it is significantly more accurate than the other recognizers, except $N-Protractor.
● Therefore, both $N-Protractor and µV are suitable for Rotation-invariant recognition.

19. 18
Conclusion on the µV Recognizer
● An ARST-invariant multi-stroke recognizer;
● Fast enough for interactive purpose (< 50ms);
● Accurate to classify uni-stroke gestures (> 90%);
● Significantly more accurate than its peers, except $N and $N-Protractor, to classify
rotation-invariant multi-strokes.
● Unlike $N and $N-Protractor, µV does not face a combinatorial explosion.