SlideShare a Scribd company logo
1 of 72
Lecture 1: Fundamentals of Gestural Interaction
Jean Vanderdonckt, UCLouvain
Vrije Universiteit Brussel, March 5, 2020, 2 pm-4 pm
Location: Room PL9.3.62, Pleinlaan 9, B-1050 Brussels
2
Definition and usage of gestures
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
• A gesture is any particular type of body
movement performed in 0D, 1D, 2D, 3D or
more
• e.g., Hand movement (supination, pronation, etc.)
• e.g., Head movement (lips, eyes, face, etc.)
• e.g., Full body movement (silhouette, posture, etc.)
x
0D
x
x
1D
x
y x
y
2D
x
y
z
x
y
z
3D
y
z
x
y z
u
4D
x
y
z
x
y z
u
v
5D
x
y
z
x
y z
u
vw
6D
3
Definition and usage of gestures
Sources: Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200.
Graham, J. A., Argyle, M. (1975). A cross-cultural study of the communication of extra-verbal meaning by gestures. International
Journal of Psychology 10, 57-67
• Benefits of gestures
• Gestures are rich sources of information and
expression
• Gestures convey information to listeners
• Gestures communicate attitudes and emotions both
voluntarily and involuntarily
• Any body limb can be used
• Hands and fingers, Head and shoulders
• Whole body,…
• Contact-based or contact-less
4
Definition and usage of gestures
• Benefits of gestures
• Gestures are fast to produce
• Shorter gestures are perceived as easier (93%)
• Gestures are a “natural” equivalent of keyboard
shortcuts
Definition and usage of gestures
• Benefits of gestures
• Gestures are fast to produce
• Gestures could even become menu shortcuts
Sources: C Appert, S Zhai, Using strokes as command shortcuts: cognitive benefits and toolkit support, Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems CHI 2009.
Dictionary Shortcuts Testing area
Definition and usage of gestures
• Benefits of gestures
• Gestures are fast to produce
• Gestures as shortcuts: cognitive
• advantages in learning and recall (3 times more)
Source: C Appert, S Zhai, Using strokes as command shortcuts: cognitive benefits and toolkit support, Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems CHI 2009.
7
Definition and usage of gestures
• Pen-based gestures (contact-based)
• are hand movements captured by a pointing
device (pen, mouse, stylus) on a constrained 2D
surface (tablet, whiteboard, wall screen,…)
enabling pen-based interaction and computing
8
Definition and usage of gestures
• Pen-based gestures (contact-based)
• Example: UI design by sketching (GAMBIT)
Source: Ugo Braga Sangiorgi, François Beuvens, Jean Vanderdonckt, User interface design by collaborative sketching. Conference on
Designing Interactive Systems 2012: 378-387.
9
Definition of strokes
• A stroke is a time series of points. The first point is
the starting point (often marked on a drawing), the last
point the ending point
• A pen-based gesture is a time series of strokes
• A shape is a stroke (iconic) gesture corresponding to
a geometrical figure, e.g., a square
• Shape recognition could be viewed as similar to gesture recognition
• A shape can be open or closed if the distance between the starting
and ending points falls below some threshold
• Circles, squares, or diamonds are closed shapes, while question
marks and brackets are open shapes
Stroke
Gesture
Shape
Motion
Stroke
10
Definition of gestures, which can be
• Unistroke or multistroke, depending on the number
of strokes produced by the user
• synonyms: single-stroke and multi-stroke)
• Unidirectional or multidirectional, depending on the
number of directions involved (one or many)
• ex.: a unistroke, unidirectional gesture is called a flick
• Verb: strike or propel (something) with a sudden quick
movement of the fingers; noun: a sudden quick
movement [Source: Merriam-Webster]
• ex.: a multi-orientation flick is called a mark
11
Definition of gestures
2 orientations 2 orientations 3 orientations
1 orientation
A gesture is said to be
• Unistroke
• Multistroke
• Unidirectional
• Multidirectional
• Flick
• Mark
12
Definition of shapes
The same shape can be drawn in many different ways
• Unistroke
• Multistroke
• Unidirectional
• Multidirectional
flick stroke
flick
flick
mark
mark mark
Invariance properties
1. Variation of strokes  stroke clustering
Stroke invariance = independence of any
combination of strokes
2. Variation of directions  direction interpretation
Direction invariance = independence of the
direction in which strokes are produced
Invariance properties
3. Variation of sampling  re-sampling needed
Sampling invariance = independence of the
sampling capability of the input device
Source: J.O. Wobbrock, A.D. Wilson, Y. Li, Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes, Proc.
of UIST’2007.
Invariance properties
Source: U. Sangiorgi, V. Genaro Motti, F. Beuvens, J. Vanderdonckt, Assessing lag perception in electronic sketching, Proc. of NordiCHI ’12.
4. Variation of lag  different perceptions by platform
Lag invariance = independence of any speed
The range below 20 FPS was rejected by users.
The difference between grades is not
significant above 24 FPS.
No conclusive evidence that subjects
perceived differences of 2, 5 and 10 FPS when
testing pairs of rates.
Invariance properties
5. Variation of position  different locations possible
Translation invariance = independence of the
location of articulation
Invariance properties
6. Variation of size  different sizes depending on the
surface of the platform: rescaling is perhaps needed
Scale invariance = independence of size/scale
Source: U. Sangiorgi, F. Beuvens, J. Vanderdonckt, User Interface Design by Collaborative Sketching, Proc. of DIS’2012
Invariance properties
7. Variation of angle  different orientations
depending on position
Rotation invariance = independence of any rotation
Invariance properties
8. Variation of skew => linear/bilinear transformations are
perhaps needed to recover the initial shape
Skew invariance = independence of any transformation
20
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
Example: xStroke recognizer
Principle: transform a gesture into a string by traversing
cells in a 3x3 matrix
Invariance properties
Horizontal flick = 123, 456, or 789
Vertical flick = 147, 258, 369
Main diagonal = 159
Second diagonal = 357
Variations of a uppercase ‘B’
123 654 456 987
741 23 654 456 987
147 741 23 654 456 987
21
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
Example: xStroke recognizer
- Stroke invariance: no (unistroke)
- Direction invariance: no, unless all variations incl.
Invariance properties
Horizontal flick = 123, 456, or 789
Vertical flick = 147, 258, 369
Main diagonal = 159
Second diagonal = 357
Right-to-Left (RTL) instead of Left-to-Right (LTR)
Horizontal flick = 321, 654, 987
Vertical flick = 741, 852, 963
Main diagonal = 951
Second diagonal = 753
Plus all sub-cases: 12, 23, 45, 56, 78, 89, etc.
22
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
Example: xStroke recognizer
- Stroke invariance: no (unistroke)
- Direction invariance: no, unless all variations incl.
Invariance properties
Problem: “Fuzzy” or imprecise traversals cause
false positive
Example: 12569 for main diagonal vs marking
23
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
Example: xStroke recognizer
- Sampling invariance: yes, but depends of the soft.
- Lag invariance: no, depends on the device
Invariance properties
Problem: forgetted traversals cause true
negative
A unistroke becomes a multi-stroke
24
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
Example: xStroke recognizer
- Translation invariance: yes, two modes
- Scale invariance: yes, bounding box
- Rotation invariance: no
Invariance properties
In Window mode In full screen mode
25
Rubine (1991)
Stroke gesture recognizers
• Statistical classification for unistroke gestures
• A gesture G is an array of P sampled points
• Features
– Incrementally computable
– Meaningful
– Enough, but not too many
26
Rubine (1991)
Stroke gesture recognizers
27
Rubine (1991)
Stroke gesture recognizers
• Training phase
• Recognition / classification phase
Gesture samples
for class c
Optimal
Classifier
x1
x2
0
C1
C3
C2
28
Rubine (1991)
Stroke gesture recognizers
• Stroke invariance: no (unistroke)
• Direction invariance: yes, by considering symmetrical gesture
• Sampling invariance: yes
• Lag invariance: yes, but depends on the device
• Translation invariance: yes, with translation to (0,0)
• Scale invariance: no
• Rotation invariance: no
29
Siger (2005)
Training phase
Vector string:
Stroke Vector direction
Left L
Right R
Up U
Down D
Regular expression:
(NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+.
LU,U,U,U,U,U,RU,RU,RU,RU,RU,RU,R,R,R,R,R,R,R,R,RD,RD,RD, RD,D,D,D,D,LD,LD,LD,LD,L,L,LD,LD,LD,D,D,D,D,D,D,D,D,D,L
Stroke gesture recognizers
30
Siger (2005)
Training phase Regular expression:
(NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+.
Stroke gesture recognizers
Dim strokeInfo as New StrokeInfo(InkPicture1.Ink.Strokes(0))
If strokeInfo.IsMatch(Vectors.Ups & Vectors.Rights & _ Vectors.Downs & Vectors.Lefts & Vectors.Downs)
Then MessageBox.Show("Question Mark")
End If
Code listing:
The SiGeR (Simple Gesture Recogniser) algorithm was
developed by Scott Swigart for the Microsoft Developer
Network to illustrate the implementation of custom
gesture recognizers for the Microsoft Tablet PC platform.
The algorithm classifies gestures on the basis of regular
expressions
31
Siger (2005)
Training phase Regular expression:
(NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+.
Stroke gesture recognizers
Dim strokeInfo as New StrokeInfo(InkPicture1.Ink.Strokes(0))
If strokeInfo.IsMatch(Vectors.Ups & Vectors.Rights & _ Vectors.Downs & Vectors.Lefts & Vectors.Downs)
Then MessageBox.Show("Question Mark")
End If
Code listing:
The SiGeR (Simple Gesture Recogniser) algorithm was
developed by Scott Swigart for the Microsoft Developer
Network to illustrate the implementation of custom
gesture recognizers for the Microsoft Tablet PC platform.
The algorithm classifies gestures on the basis of regular
expressions
32
LVS (Levenshtein-based, 2007)
• Training
• Transformation into a character chain
• Recognition
• Comparison to all training examples
• Calculating Edit-Distance
• Smaller distance is chosen
Source: Coyette, A., Kieffer, S., Vanderdonckt, J., Multi-Fidelity Prototyping of User Interfaces, Proc. of 11th IFIP TC 13 Int. Conf. on
Human-Computer Interaction Interact’2007
Stroke gesture recognizers
33
LVS (Levenshtein-based, 2007)
Stroke gesture recognizers
• Each gesture is superimposed with a grid
• Each sampling point is mapped to its closest grid node
• For adjacent nodes, a stroke direction (out of 8) is selected
• A string is generated from direction coding of input
• Complex task of comparison of drawing inputs is reduced to simple
calculation of string edit distances
34
LVS (Levenshtein-based, 2007)
Stroke gesture recognizers
• Distance is shortest sequence of edit commands that transform s to t
• Simplest set of operations:
– Copy character from s over t
– Delete a character in s (cost 1)
– Insert a character in t (cost 1)
– Substitute one character for another (cost 1)
35
LVS (Levenshtein-based, 2007)
Stroke gesture recognizers
• Distance is shortest sequence of edit commands that transform s to t
• Simplest set of operations:
– Copy character from s over t
– Delete a character in s (cost 1)
– Insert a character in t (cost 1)
– Substitute one character for another (cost 1)
36
LVS (Levenshtein-based, 2007)
Stroke gesture recognizers
• How to obtain rotation independence?
37
LVS (Levenshtein-based, 2007)
Stroke gesture recognizers
• Training types for Levenshtein on digits
– User-dependent: recognition system trained by the user tester
– User-independent (1): recognition system trained by a user different from the user
tester
– User-independent (2): recognition system trained by several users different from the
user tester
38
LVS (Levenshtein-based, 2007)
Stroke gesture recognizers
39
Many recognition techniques exist from AI/ML
• Hidden Markov Chains (HMM)
• Support Vector Machines (SVM)
• Feature based linear discriminants
• Neural Networks
• CNN
• …
Stroke gesture recognizers
40
Many recognition techniques exist from AI/ML
• Hidden Markov Chains (HMM)
• Support Vector Machines (SVM)
• Feature based linear discriminants
• Neural Networks
• CNN
• …
Challenges:
• They require advanced knowledge from machine learning &
pattern recognition
• Complex to implement, debug, and train
• Parameters that need tweaking
• Risk of overfitting
• Modifiability when new gestures need to be incorporated
Stroke gesture recognizers
41
Benefits:
• They require minimal programming knowledge, nothing from AI
• Easy to implement, debug, and train
• No calibration
• No need to retrain/remodel when new gestures appear
• Straightforward geometric interpretation
• Performance
• Recognition rate >90%
• Execution time < ms
• Computational complexity reasonable
• Limited amount of LOC
Stroke gesture recognizers: recent recognizers in HCI
42
Several recognizers:
• Members of the $-family
• $1: unistroke
• $N: multi-stroke
• $P: cloud-point multi-stroke
• $P+: optimized for users with special needs (the most accurate
today)
• $Q: optimized for low-end devices (the fastest one today)
• Other recognizers
• Protractor
• $N-Protractor
• $3, Protractor3D
• 1ȼ
• Penny Pincher
• Match-Up & Conquer
• …
• Common ground: the all use the Nearest-Neighbor-Classification
(NNC) technique from pattern matching
Stroke gesture recognizers: recent recognizers in HCI
43
Nearest-Neighbor-Classification (NNC): basic principle
1. Build a training set (or template set) =
set of classes of gestures captured from
users to serve as reference
2. Define a (dis)similarity function =
function F that compares two gestures:
- a candidate gesture
- any training/reference gesture
from the training set
3. Compare the candidate with every
reference and return the class of the
most similar reference
Stroke gesture recognizers: recent recognizers in HCI
F( , ) = 1.51
F( , ) = 0.98
F( , ) = 0.12
F( , ) = 0.12
44
Nearest-Neighbor-Classification (NNC) for 2D strokes
Y
0 0.25 0.50 0.75 1
X
0
0.25
0.50
0.75
1
= candidate point
x = reference point
x
x
x
x x
x
x
x
x
x
x
x
x
x
k nearest neighbors
1 nearest
neighbor distance
0 1
x x x x x
x x x
x x x
Reference gestures
Training set
candidate
gesture
p2
p3
p4
p1
k-NN
k nearest neighbors
1-NN
Single nearest neighbor
applied to gesture recognition
q3
q1
q2
q4
reference
gesture
Stroke gesture recognizers: recent recognizers in HCI
45
Nearest-Neighbor-Classification (NNC)
• Pre-processing steps to ensure invariance
• Re-sampling
• Points with same space between: isometricity
• Points with same timestamp between: isochronicity
• Same amount of points: isoparameterization
• Re-Scaling
• Normalisation of the bounding box into [0..1]x[0..1] square
• Rotation to reference angle
• Rotate to 0°
• Re-rotating and distance computation
• Distance computed between candidate gesture and
reference gestures (1-NN)
Stroke gesture recognizers: recent recognizers in HCI
46
The $1 recognizer
• Unistroke gesture recognizer
• Euclidean distance as dissimilarity
• implementable in 100 LOC
• 1-to-1 point matching;
• 99% with 3+ samples per gesture
• Accessible at https://depts.washington.edu/aimgroup/proj/dollar/
Stroke gesture recognizers: recent recognizers in HCI
Source: Jacob O. Wobbrock, Andrew D. Wilson, Yang Li. (2007). Gestures without libraries, toolkits or training: a $1 recognizer for user
interface prototypes. In Proc. UIST '07, 159-168
47
The $N recognizer
• Multistroke gesture recognizer
• Euclidean distance as dissimilarity
• implementable in 200 LOC
• generates all stroke permutations
• Uses $1 and stroke order permutations
• Accessible at http://depts.washington.edu/aimgroup/proj/dollar/ndollar.html
Stroke gesture recognizers: recent recognizers in HCI
Source: Anthony, L., Wobbrock, J.O. (2010). A lightweight multistroke recognizer for user interface prototypes. In Proc. of Graphics
Interface 2010, 245-252, 2010
48
The $P recognizer
• Multistroke gesture recognizer
• Should become articulation-independent: removes all the
temporal information associated to gesture points
• Users perform gestures in various ways, not easily accountable in
training sets
• 442 possible ways to draw a square with 1, 2, 3, and 4 strokes
Stroke gesture recognizers: recent recognizers in HCI
Source: Vatavu, R.D., Anthony, L., Wobbrock, J.O. (2012). Gestures as Point Clouds: A $P Recognizer for User Interface Prototypes. Int.
Conf. on Multimodal Interaction 2012, ICMI’12
49
Comparison of some $* recognizers
Stroke gesture recognizers: recent recognizers in HCI
Source: Vatavu, R.D., Anthony, L., Wobbrock, J.O. (2012). Gestures as Point Clouds: A $P Recognizer for User Interface Prototypes. Int.
Conf. on Multimodal Interaction 2012, ICMI’12
50
Other gesture recognizers
• Protractor (Li, CHI 2010)
• $3 (Kratz and Rohs, IUI 2010)
• Protractor3D (Kratz and Rohs, IUI 2011)
• $N-Protractor (Anthony and Wobbrock, GI 2012)
• 1ȼ (Herold and Stahovich, SBIM 2012)
• Match-Up & Conquer (Rekik, Vatavu, Grisoni, AVI 2014)
• Penny Pincher (Taranta and LaViola, GI 2015)
• Jackknife (Taranta et al., CHI 2017)
• Other dissimilarity functions
• Dynamic Time Warping (DTW), Hausdorff, modified Hausdorff,
Levenshtein + chain coding,…
• Techniques for optimizing training sets
• 1F (Vatavu, IUI 2012)
• metaheuristics (Pittman, Taranta, LaViola, IUI 2016).
Stroke gesture recognizers: recent recognizers in HCI
51
Example: Match-up & Conquer
• Robust recognition of multi-touch gestures
• Users perform multi-touch gestures in various ways, not easily
accountable in training sets
Stroke gesture recognizers: recent recognizers in HCI
Source: Yosra Rekik, Radu-Daniel Vatavu, Laurent Grisoni. 2014. Match-Up & Conquer: A Two-Step Technique for Recognizing
Unconstrained Bimanual and Multi-Finger Touch Input. In Proc. of AVI’14, 201-208
52
Nearest-Neighbor-Classification (NNC)
• Two families of approaches
• “Between points” distance
• $-Family recognizers: $1, $3, $N, $P, $P+,
$V, $Q,…
• Variants and optimizations: ProTractor,
Protractor3D,…
• “Vector between points” distance
• PennyPincher, JackKnife,…
[Vatavu R.-D. et al, ICMI ’12]
[Taranta E.M. et al, C&G ’16]
Software dimension: Which algorithm?
53
Nearest-Neighbor-Classification (NNC)
• Two families of approaches
• “Between points” distance
• $-Family recognizers: $1, $3, $N, $P, $P+,
$V, $Q,…
• Variants and optimizations: ProTractor,
Protactor3D,…
• “Vector between points” distance
• PennyPincher, JackKnife,…
• A third new family of approaches
• “Vector between vectors” distance:
our approach
Software dimension: Which algorithm?
54
• Local Shape Distance between 2 triangles based on
similarity (Roselli’s distance)
𝑎
𝑏
𝑢
𝑣
𝑎 + 𝑏
𝑢 + 𝑣
Paolo Roselli
Università degli Studi di Roma, Italy
Software dimension: Which algorithm?
Source: Lorenzo Luzzi & Paolo Roselli, The shape of planar smooth gestures and the convergence of a gesture recognizer, Aequationes
mathematicae volume 94, 219–233(2020).
55
• Step 1. Vectorization for each pair of vectors between
three consecutive points
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
56
• Step 1. Vectorization for each pair of vectors between
three consecutive points
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q4
q5
q6
Training
gesture
Candidate
gesture
q1
q2
q3
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
57
• Step 2. Mapping candidate’s triangles onto training
gesture’s triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
p1
p2
p3
p2
p3
p4
p3
p4
p5
p4
p5
p6
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
58
• Step 2. Mapping candidate’s triangles onto training
gesture’s triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
p1
p2
p3
p2
p3
p4
p3
p4
p5
p4
p5
p6
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
59
• Step 3. Computation of Local Shape Distance between
pairs of triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p3
p4
p5
p2
p3
p4
p4
p5
p6
Training
gesture
Candidate
gesture
p1p2p3,
q1q2q3
(N)LSD (
)
(
=0.02
p2p3p4,
q2q3q4 )
=0.04
(p3p4p5,
q3q4q5 )
=0.0001
)
p4p5p6,
q3q4q5
(
=0.03
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
60
• Step 4. Summing all individual figures into final one
• Step 5. Iterate for every training gesture
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p3
p4
p5
p2
p3
p4
p4
p5
p6
Training
gesture
Candidate
gesture
p1p2p3,
q1q2q3
(N)LSD (
)
p2p3p4,
q2q3q4 )
(
)
p4p5p6,
q3q4q5
(
(p3p4p5,
q3q4q5 )
=0.02 =0.04 =0.0001 =0.03
=0.02+0.04+0.0001+0.03=0.0901
(indicative figures)
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
• Physical demand depends on variables
• Gesture form: specifies which form of gesture is elicited.
Possible values are:
• S= stroke when the gesture only consists of taps and flicks
• T= static when the gesture is performed in only one location
• M= static with motion (when the gesture is performed with a
static pose while the rest is moving)
• D= dynamic when the gesture does capture any change or
motion
User experience dimension: evaluation criteria
• Physical demand depends on variables
• Laterality: characterizes how the two hands are
employed to produce gestures, with two categories, as
done in many studies. Possible values are:
• D= dominant unimanual, N= non-dominant unimanual,
S= symmetric bimanual, A= asymmetric bimanual
User experience dimension: evaluation criteria
Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426
D
(right handed)
N
(right handed)
S
(right handed)
A
(right handed)
• Aigner et al.’s taxonomy of mid-
air gestures
• P= Pointing gestures (= deictic
gestures) indicate people,
objects, directions
Psychological dimension: What is a gesture?
Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand
Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
• Aigner et al.’s taxonomy of mid-
air gestures
• Semaphoric gestures are hand
postures and movements
conveying specific meanings
• T= Static semaphorics are identified by a
specific hand posture. Example: a flat palm
facing from the actor means “stop”.
• D= Dynamic semaphorics convey information
through their temporal aspects. Example: a
circular hand motion means “rotate”
• S= Semaphoric strokes represent hand flicks
are single, stroke-like movements. Example: a
left flick of the hand means “dismiss this
object”
Psychological dimension: What is a gesture?
Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand
Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
• Aigner et al.’s taxonomy of mid-
air gestures
• M= Manipulation gestures a
guide movement in a short
feedback loop. Thus, they feature
a tight relationship between the
movements of the actor and the
movements of the object to be
manipulated. The actor waits for
the entity to “follow” before
continuing
Psychological dimension: What is a gesture?
Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human-
preferences-in-usage-of-gesture-types-for-hci/
• When to recognize?
Stroke gesture recognizers
Source: Olivier Bau, Wendy E. Mackay, OctoPocus: a dynamic guide for learning gesture-based command sets. UIST 2008: 37-46
• A Gesture Set is a set of potentially related gestures that could
be used together in one or several interactive applications
• A gesture set could be defined for a particular
- population of users, e.g., motor-impaired people
- set of tasks, e.g., annotation of X-rays
- platform/device, e.g., for an operating system, a style guide
- environment, e.g., machine verification
• A gesture set is said to be
- domain-dependent if used only in a particular application
domain (e.g., mammography, pediatrics, nuclear power plant)
- domain-independent if used across several applications (e.g.,
for an operating system)
Gesture sets: training and reference
• xStroke Gesture set: a set frequent gestures
• All printable ASCII characters: letters a-z, digits 0-9, and punctuation
Gesture sets: examples
• xStroke Gesture set: a set frequent gestures
• All additional gestures necessary to function as a full keyboard
replacement: Space, BackSpace, Return, Tab, Escape, Shift, Control,
Meta
Gesture sets: examples
• xStroke Gesture set: a set frequent gestures
• Advanced set of gestures:
• Numbers in letter mode
• Punctuation in letter m
Gesture sets: examples
1. Design applications with touch interaction as the primary
expected input method.
2. Provide visual feedback for interactions of all types (touch, pen,
stylus, mouse, etc.)
3. Optimize targeting by adjusting touch target size, contact
geometry, scrubbing and rocking.
4. Optimize accuracy through the use of snap points and
directional "rails".
5. Provide tooltips and handles to help improve touch accuracy for
tightly packed UI items.
6. Don't use timed interactions whenever possible (example of
appropriate use: touch and hold).
7. Don't use the number of fingers used to distinguish the
manipulation whenever possible
Gesture sets: guidelines
Thank you very much
for your attention

More Related Content

Similar to Fundamentals of Gestural Interaction

Deformable and Shape Changing Interfaces
Deformable and Shape Changing InterfacesDeformable and Shape Changing Interfaces
Deformable and Shape Changing InterfacesUniversity of Sussex
 
Converting Tatamis into Touch Sensors by Measuring Capacitance
Converting Tatamis into Touch Sensors by Measuring CapacitanceConverting Tatamis into Touch Sensors by Measuring Capacitance
Converting Tatamis into Touch Sensors by Measuring Capacitancesugiuralab
 
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...Jean Vanderdonckt
 
Human action recognition with kinect using a joint motion descriptor
Human action recognition with kinect using a joint motion descriptorHuman action recognition with kinect using a joint motion descriptor
Human action recognition with kinect using a joint motion descriptorSoma Boubou
 
Build Your Own VR Display Course - SIGGRAPH 2017: Part 2
Build Your Own VR Display Course - SIGGRAPH 2017: Part 2Build Your Own VR Display Course - SIGGRAPH 2017: Part 2
Build Your Own VR Display Course - SIGGRAPH 2017: Part 2StanfordComputationalImaging
 
Scratching the Surface with JavaFX
Scratching the Surface with JavaFXScratching the Surface with JavaFX
Scratching the Surface with JavaFXNLJUG
 
CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...
CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...
CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...Byungjoo Lee
 
연구실 소개(2014)
연구실 소개(2014)연구실 소개(2014)
연구실 소개(2014)NCLab_KAIST
 
GraphConnect Europe 2016 - Navigating All the Knowledge - James Weaver
GraphConnect Europe 2016 - Navigating All the Knowledge - James WeaverGraphConnect Europe 2016 - Navigating All the Knowledge - James Weaver
GraphConnect Europe 2016 - Navigating All the Knowledge - James WeaverNeo4j
 
COMP 4010: Lecture 4 - 3D User Interfaces for VR
COMP 4010: Lecture 4 - 3D User Interfaces for VRCOMP 4010: Lecture 4 - 3D User Interfaces for VR
COMP 4010: Lecture 4 - 3D User Interfaces for VRMark Billinghurst
 
Human-Computer Interaction: Adaptation, Distribution, Evaluation
Human-Computer Interaction: Adaptation, Distribution, EvaluationHuman-Computer Interaction: Adaptation, Distribution, Evaluation
Human-Computer Interaction: Adaptation, Distribution, EvaluationJean Vanderdonckt
 
COMP 4010 Lecture12 Research Directions in AR
COMP 4010 Lecture12 Research Directions in ARCOMP 4010 Lecture12 Research Directions in AR
COMP 4010 Lecture12 Research Directions in ARMark Billinghurst
 

Similar to Fundamentals of Gestural Interaction (20)

Designing Usable Interface
Designing Usable InterfaceDesigning Usable Interface
Designing Usable Interface
 
Deformable and Shape Changing Interfaces
Deformable and Shape Changing InterfacesDeformable and Shape Changing Interfaces
Deformable and Shape Changing Interfaces
 
Converting Tatamis into Touch Sensors by Measuring Capacitance
Converting Tatamis into Touch Sensors by Measuring CapacitanceConverting Tatamis into Touch Sensors by Measuring Capacitance
Converting Tatamis into Touch Sensors by Measuring Capacitance
 
Python Trifold2014
Python Trifold2014Python Trifold2014
Python Trifold2014
 
Pedestrian modelling
Pedestrian modellingPedestrian modelling
Pedestrian modelling
 
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
 
Human action recognition with kinect using a joint motion descriptor
Human action recognition with kinect using a joint motion descriptorHuman action recognition with kinect using a joint motion descriptor
Human action recognition with kinect using a joint motion descriptor
 
Build Your Own VR Display Course - SIGGRAPH 2017: Part 2
Build Your Own VR Display Course - SIGGRAPH 2017: Part 2Build Your Own VR Display Course - SIGGRAPH 2017: Part 2
Build Your Own VR Display Course - SIGGRAPH 2017: Part 2
 
Scratching the Surface with JavaFX
Scratching the Surface with JavaFXScratching the Surface with JavaFX
Scratching the Surface with JavaFX
 
CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...
CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...
CHI'16 Journal "A Mouse With Two Optical Sensors That Eliminates Coordinate D...
 
연구실 소개(2014)
연구실 소개(2014)연구실 소개(2014)
연구실 소개(2014)
 
Smart Room Gesture Control
Smart Room Gesture ControlSmart Room Gesture Control
Smart Room Gesture Control
 
GraphConnect Europe 2016 - Navigating All the Knowledge - James Weaver
GraphConnect Europe 2016 - Navigating All the Knowledge - James WeaverGraphConnect Europe 2016 - Navigating All the Knowledge - James Weaver
GraphConnect Europe 2016 - Navigating All the Knowledge - James Weaver
 
COMP 4010: Lecture 4 - 3D User Interfaces for VR
COMP 4010: Lecture 4 - 3D User Interfaces for VRCOMP 4010: Lecture 4 - 3D User Interfaces for VR
COMP 4010: Lecture 4 - 3D User Interfaces for VR
 
P J020
P J020P J020
P J020
 
report
reportreport
report
 
Human-Computer Interaction: Adaptation, Distribution, Evaluation
Human-Computer Interaction: Adaptation, Distribution, EvaluationHuman-Computer Interaction: Adaptation, Distribution, Evaluation
Human-Computer Interaction: Adaptation, Distribution, Evaluation
 
COMP 4010 Lecture12 Research Directions in AR
COMP 4010 Lecture12 Research Directions in ARCOMP 4010 Lecture12 Research Directions in AR
COMP 4010 Lecture12 Research Directions in AR
 
Communityday2013
Communityday2013Communityday2013
Communityday2013
 
Lecture#01.pptx
Lecture#01.pptxLecture#01.pptx
Lecture#01.pptx
 

More from Jean Vanderdonckt

To the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User InterfacesTo the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User InterfacesJean Vanderdonckt
 
Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Jean Vanderdonckt
 
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...Jean Vanderdonckt
 
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...Jean Vanderdonckt
 
Gesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOpsGesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOpsJean Vanderdonckt
 
Evaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and MeasuresEvaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and MeasuresJean Vanderdonckt
 
User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...Jean Vanderdonckt
 
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...Jean Vanderdonckt
 
Attach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you WorkAttach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you WorkJean Vanderdonckt
 
The Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV GesturesThe Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV GesturesJean Vanderdonckt
 
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper BodyHead and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper BodyJean Vanderdonckt
 
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for SmartphonesG-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for SmartphonesJean Vanderdonckt
 
Vector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture RecognitionVector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture RecognitionJean Vanderdonckt
 
An ontology for reasoning on body-based gestures
 An ontology for reasoning on body-based gestures An ontology for reasoning on body-based gestures
An ontology for reasoning on body-based gesturesJean Vanderdonckt
 
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design AlternativesAB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design AlternativesJean Vanderdonckt
 
Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
 Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
Gelicit: A Cloud Platform for Distributed Gesture Elicitation StudiesJean Vanderdonckt
 
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...Jean Vanderdonckt
 
Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...Jean Vanderdonckt
 
!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...
!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...
!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...Jean Vanderdonckt
 
Gestures for Smart Rings: Empirical Results, Insights, and Design Implications
Gestures for Smart Rings: Empirical Results, Insights, and Design ImplicationsGestures for Smart Rings: Empirical Results, Insights, and Design Implications
Gestures for Smart Rings: Empirical Results, Insights, and Design ImplicationsJean Vanderdonckt
 

More from Jean Vanderdonckt (20)

To the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User InterfacesTo the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User Interfaces
 
Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...
 
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
 
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
 
Gesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOpsGesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOps
 
Evaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and MeasuresEvaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and Measures
 
User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...
 
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
 
Attach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you WorkAttach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you Work
 
The Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV GesturesThe Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV Gestures
 
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper BodyHead and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
 
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for SmartphonesG-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
 
Vector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture RecognitionVector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture Recognition
 
An ontology for reasoning on body-based gestures
 An ontology for reasoning on body-based gestures An ontology for reasoning on body-based gestures
An ontology for reasoning on body-based gestures
 
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design AlternativesAB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
 
Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
 Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
 
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
 
Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...
 
!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...
!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...
!FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable P...
 
Gestures for Smart Rings: Empirical Results, Insights, and Design Implications
Gestures for Smart Rings: Empirical Results, Insights, and Design ImplicationsGestures for Smart Rings: Empirical Results, Insights, and Design Implications
Gestures for Smart Rings: Empirical Results, Insights, and Design Implications
 

Recently uploaded

Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfhans926745
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdfChristopherTHyatt
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 

Recently uploaded (20)

Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 

Fundamentals of Gestural Interaction

  • 1. Lecture 1: Fundamentals of Gestural Interaction Jean Vanderdonckt, UCLouvain Vrije Universiteit Brussel, March 5, 2020, 2 pm-4 pm Location: Room PL9.3.62, Pleinlaan 9, B-1050 Brussels
  • 2. 2 Definition and usage of gestures Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html • A gesture is any particular type of body movement performed in 0D, 1D, 2D, 3D or more • e.g., Hand movement (supination, pronation, etc.) • e.g., Head movement (lips, eyes, face, etc.) • e.g., Full body movement (silhouette, posture, etc.) x 0D x x 1D x y x y 2D x y z x y z 3D y z x y z u 4D x y z x y z u v 5D x y z x y z u vw 6D
  • 3. 3 Definition and usage of gestures Sources: Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200. Graham, J. A., Argyle, M. (1975). A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology 10, 57-67 • Benefits of gestures • Gestures are rich sources of information and expression • Gestures convey information to listeners • Gestures communicate attitudes and emotions both voluntarily and involuntarily • Any body limb can be used • Hands and fingers, Head and shoulders • Whole body,… • Contact-based or contact-less
  • 4. 4 Definition and usage of gestures • Benefits of gestures • Gestures are fast to produce • Shorter gestures are perceived as easier (93%) • Gestures are a “natural” equivalent of keyboard shortcuts
  • 5. Definition and usage of gestures • Benefits of gestures • Gestures are fast to produce • Gestures could even become menu shortcuts Sources: C Appert, S Zhai, Using strokes as command shortcuts: cognitive benefits and toolkit support, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 2009. Dictionary Shortcuts Testing area
  • 6. Definition and usage of gestures • Benefits of gestures • Gestures are fast to produce • Gestures as shortcuts: cognitive • advantages in learning and recall (3 times more) Source: C Appert, S Zhai, Using strokes as command shortcuts: cognitive benefits and toolkit support, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 2009.
  • 7. 7 Definition and usage of gestures • Pen-based gestures (contact-based) • are hand movements captured by a pointing device (pen, mouse, stylus) on a constrained 2D surface (tablet, whiteboard, wall screen,…) enabling pen-based interaction and computing
  • 8. 8 Definition and usage of gestures • Pen-based gestures (contact-based) • Example: UI design by sketching (GAMBIT) Source: Ugo Braga Sangiorgi, François Beuvens, Jean Vanderdonckt, User interface design by collaborative sketching. Conference on Designing Interactive Systems 2012: 378-387.
  • 9. 9 Definition of strokes • A stroke is a time series of points. The first point is the starting point (often marked on a drawing), the last point the ending point • A pen-based gesture is a time series of strokes • A shape is a stroke (iconic) gesture corresponding to a geometrical figure, e.g., a square • Shape recognition could be viewed as similar to gesture recognition • A shape can be open or closed if the distance between the starting and ending points falls below some threshold • Circles, squares, or diamonds are closed shapes, while question marks and brackets are open shapes Stroke Gesture Shape Motion Stroke
  • 10. 10 Definition of gestures, which can be • Unistroke or multistroke, depending on the number of strokes produced by the user • synonyms: single-stroke and multi-stroke) • Unidirectional or multidirectional, depending on the number of directions involved (one or many) • ex.: a unistroke, unidirectional gesture is called a flick • Verb: strike or propel (something) with a sudden quick movement of the fingers; noun: a sudden quick movement [Source: Merriam-Webster] • ex.: a multi-orientation flick is called a mark
  • 11. 11 Definition of gestures 2 orientations 2 orientations 3 orientations 1 orientation A gesture is said to be • Unistroke • Multistroke • Unidirectional • Multidirectional • Flick • Mark
  • 12. 12 Definition of shapes The same shape can be drawn in many different ways • Unistroke • Multistroke • Unidirectional • Multidirectional flick stroke flick flick mark mark mark
  • 13. Invariance properties 1. Variation of strokes  stroke clustering Stroke invariance = independence of any combination of strokes 2. Variation of directions  direction interpretation Direction invariance = independence of the direction in which strokes are produced
  • 14. Invariance properties 3. Variation of sampling  re-sampling needed Sampling invariance = independence of the sampling capability of the input device Source: J.O. Wobbrock, A.D. Wilson, Y. Li, Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes, Proc. of UIST’2007.
  • 15. Invariance properties Source: U. Sangiorgi, V. Genaro Motti, F. Beuvens, J. Vanderdonckt, Assessing lag perception in electronic sketching, Proc. of NordiCHI ’12. 4. Variation of lag  different perceptions by platform Lag invariance = independence of any speed The range below 20 FPS was rejected by users. The difference between grades is not significant above 24 FPS. No conclusive evidence that subjects perceived differences of 2, 5 and 10 FPS when testing pairs of rates.
  • 16. Invariance properties 5. Variation of position  different locations possible Translation invariance = independence of the location of articulation
  • 17. Invariance properties 6. Variation of size  different sizes depending on the surface of the platform: rescaling is perhaps needed Scale invariance = independence of size/scale Source: U. Sangiorgi, F. Beuvens, J. Vanderdonckt, User Interface Design by Collaborative Sketching, Proc. of DIS’2012
  • 18. Invariance properties 7. Variation of angle  different orientations depending on position Rotation invariance = independence of any rotation
  • 19. Invariance properties 8. Variation of skew => linear/bilinear transformations are perhaps needed to recover the initial shape Skew invariance = independence of any transformation
  • 20. 20 Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html Example: xStroke recognizer Principle: transform a gesture into a string by traversing cells in a 3x3 matrix Invariance properties Horizontal flick = 123, 456, or 789 Vertical flick = 147, 258, 369 Main diagonal = 159 Second diagonal = 357 Variations of a uppercase ‘B’ 123 654 456 987 741 23 654 456 987 147 741 23 654 456 987
  • 21. 21 Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html Example: xStroke recognizer - Stroke invariance: no (unistroke) - Direction invariance: no, unless all variations incl. Invariance properties Horizontal flick = 123, 456, or 789 Vertical flick = 147, 258, 369 Main diagonal = 159 Second diagonal = 357 Right-to-Left (RTL) instead of Left-to-Right (LTR) Horizontal flick = 321, 654, 987 Vertical flick = 741, 852, 963 Main diagonal = 951 Second diagonal = 753 Plus all sub-cases: 12, 23, 45, 56, 78, 89, etc.
  • 22. 22 Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html Example: xStroke recognizer - Stroke invariance: no (unistroke) - Direction invariance: no, unless all variations incl. Invariance properties Problem: “Fuzzy” or imprecise traversals cause false positive Example: 12569 for main diagonal vs marking
  • 23. 23 Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html Example: xStroke recognizer - Sampling invariance: yes, but depends of the soft. - Lag invariance: no, depends on the device Invariance properties Problem: forgetted traversals cause true negative A unistroke becomes a multi-stroke
  • 24. 24 Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html Example: xStroke recognizer - Translation invariance: yes, two modes - Scale invariance: yes, bounding box - Rotation invariance: no Invariance properties In Window mode In full screen mode
  • 25. 25 Rubine (1991) Stroke gesture recognizers • Statistical classification for unistroke gestures • A gesture G is an array of P sampled points • Features – Incrementally computable – Meaningful – Enough, but not too many
  • 27. 27 Rubine (1991) Stroke gesture recognizers • Training phase • Recognition / classification phase Gesture samples for class c Optimal Classifier x1 x2 0 C1 C3 C2
  • 28. 28 Rubine (1991) Stroke gesture recognizers • Stroke invariance: no (unistroke) • Direction invariance: yes, by considering symmetrical gesture • Sampling invariance: yes • Lag invariance: yes, but depends on the device • Translation invariance: yes, with translation to (0,0) • Scale invariance: no • Rotation invariance: no
  • 29. 29 Siger (2005) Training phase Vector string: Stroke Vector direction Left L Right R Up U Down D Regular expression: (NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+. LU,U,U,U,U,U,RU,RU,RU,RU,RU,RU,R,R,R,R,R,R,R,R,RD,RD,RD, RD,D,D,D,D,LD,LD,LD,LD,L,L,LD,LD,LD,D,D,D,D,D,D,D,D,D,L Stroke gesture recognizers
  • 30. 30 Siger (2005) Training phase Regular expression: (NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+. Stroke gesture recognizers Dim strokeInfo as New StrokeInfo(InkPicture1.Ink.Strokes(0)) If strokeInfo.IsMatch(Vectors.Ups & Vectors.Rights & _ Vectors.Downs & Vectors.Lefts & Vectors.Downs) Then MessageBox.Show("Question Mark") End If Code listing: The SiGeR (Simple Gesture Recogniser) algorithm was developed by Scott Swigart for the Microsoft Developer Network to illustrate the implementation of custom gesture recognizers for the Microsoft Tablet PC platform. The algorithm classifies gestures on the basis of regular expressions
  • 31. 31 Siger (2005) Training phase Regular expression: (NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+. Stroke gesture recognizers Dim strokeInfo as New StrokeInfo(InkPicture1.Ink.Strokes(0)) If strokeInfo.IsMatch(Vectors.Ups & Vectors.Rights & _ Vectors.Downs & Vectors.Lefts & Vectors.Downs) Then MessageBox.Show("Question Mark") End If Code listing: The SiGeR (Simple Gesture Recogniser) algorithm was developed by Scott Swigart for the Microsoft Developer Network to illustrate the implementation of custom gesture recognizers for the Microsoft Tablet PC platform. The algorithm classifies gestures on the basis of regular expressions
  • 32. 32 LVS (Levenshtein-based, 2007) • Training • Transformation into a character chain • Recognition • Comparison to all training examples • Calculating Edit-Distance • Smaller distance is chosen Source: Coyette, A., Kieffer, S., Vanderdonckt, J., Multi-Fidelity Prototyping of User Interfaces, Proc. of 11th IFIP TC 13 Int. Conf. on Human-Computer Interaction Interact’2007 Stroke gesture recognizers
  • 33. 33 LVS (Levenshtein-based, 2007) Stroke gesture recognizers • Each gesture is superimposed with a grid • Each sampling point is mapped to its closest grid node • For adjacent nodes, a stroke direction (out of 8) is selected • A string is generated from direction coding of input • Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances
  • 34. 34 LVS (Levenshtein-based, 2007) Stroke gesture recognizers • Distance is shortest sequence of edit commands that transform s to t • Simplest set of operations: – Copy character from s over t – Delete a character in s (cost 1) – Insert a character in t (cost 1) – Substitute one character for another (cost 1)
  • 35. 35 LVS (Levenshtein-based, 2007) Stroke gesture recognizers • Distance is shortest sequence of edit commands that transform s to t • Simplest set of operations: – Copy character from s over t – Delete a character in s (cost 1) – Insert a character in t (cost 1) – Substitute one character for another (cost 1)
  • 36. 36 LVS (Levenshtein-based, 2007) Stroke gesture recognizers • How to obtain rotation independence?
  • 37. 37 LVS (Levenshtein-based, 2007) Stroke gesture recognizers • Training types for Levenshtein on digits – User-dependent: recognition system trained by the user tester – User-independent (1): recognition system trained by a user different from the user tester – User-independent (2): recognition system trained by several users different from the user tester
  • 39. 39 Many recognition techniques exist from AI/ML • Hidden Markov Chains (HMM) • Support Vector Machines (SVM) • Feature based linear discriminants • Neural Networks • CNN • … Stroke gesture recognizers
  • 40. 40 Many recognition techniques exist from AI/ML • Hidden Markov Chains (HMM) • Support Vector Machines (SVM) • Feature based linear discriminants • Neural Networks • CNN • … Challenges: • They require advanced knowledge from machine learning & pattern recognition • Complex to implement, debug, and train • Parameters that need tweaking • Risk of overfitting • Modifiability when new gestures need to be incorporated Stroke gesture recognizers
  • 41. 41 Benefits: • They require minimal programming knowledge, nothing from AI • Easy to implement, debug, and train • No calibration • No need to retrain/remodel when new gestures appear • Straightforward geometric interpretation • Performance • Recognition rate >90% • Execution time < ms • Computational complexity reasonable • Limited amount of LOC Stroke gesture recognizers: recent recognizers in HCI
  • 42. 42 Several recognizers: • Members of the $-family • $1: unistroke • $N: multi-stroke • $P: cloud-point multi-stroke • $P+: optimized for users with special needs (the most accurate today) • $Q: optimized for low-end devices (the fastest one today) • Other recognizers • Protractor • $N-Protractor • $3, Protractor3D • 1ȼ • Penny Pincher • Match-Up & Conquer • … • Common ground: the all use the Nearest-Neighbor-Classification (NNC) technique from pattern matching Stroke gesture recognizers: recent recognizers in HCI
  • 43. 43 Nearest-Neighbor-Classification (NNC): basic principle 1. Build a training set (or template set) = set of classes of gestures captured from users to serve as reference 2. Define a (dis)similarity function = function F that compares two gestures: - a candidate gesture - any training/reference gesture from the training set 3. Compare the candidate with every reference and return the class of the most similar reference Stroke gesture recognizers: recent recognizers in HCI F( , ) = 1.51 F( , ) = 0.98 F( , ) = 0.12 F( , ) = 0.12
  • 44. 44 Nearest-Neighbor-Classification (NNC) for 2D strokes Y 0 0.25 0.50 0.75 1 X 0 0.25 0.50 0.75 1 = candidate point x = reference point x x x x x x x x x x x x x x k nearest neighbors 1 nearest neighbor distance 0 1 x x x x x x x x x x x Reference gestures Training set candidate gesture p2 p3 p4 p1 k-NN k nearest neighbors 1-NN Single nearest neighbor applied to gesture recognition q3 q1 q2 q4 reference gesture Stroke gesture recognizers: recent recognizers in HCI
  • 45. 45 Nearest-Neighbor-Classification (NNC) • Pre-processing steps to ensure invariance • Re-sampling • Points with same space between: isometricity • Points with same timestamp between: isochronicity • Same amount of points: isoparameterization • Re-Scaling • Normalisation of the bounding box into [0..1]x[0..1] square • Rotation to reference angle • Rotate to 0° • Re-rotating and distance computation • Distance computed between candidate gesture and reference gestures (1-NN) Stroke gesture recognizers: recent recognizers in HCI
  • 46. 46 The $1 recognizer • Unistroke gesture recognizer • Euclidean distance as dissimilarity • implementable in 100 LOC • 1-to-1 point matching; • 99% with 3+ samples per gesture • Accessible at https://depts.washington.edu/aimgroup/proj/dollar/ Stroke gesture recognizers: recent recognizers in HCI Source: Jacob O. Wobbrock, Andrew D. Wilson, Yang Li. (2007). Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proc. UIST '07, 159-168
  • 47. 47 The $N recognizer • Multistroke gesture recognizer • Euclidean distance as dissimilarity • implementable in 200 LOC • generates all stroke permutations • Uses $1 and stroke order permutations • Accessible at http://depts.washington.edu/aimgroup/proj/dollar/ndollar.html Stroke gesture recognizers: recent recognizers in HCI Source: Anthony, L., Wobbrock, J.O. (2010). A lightweight multistroke recognizer for user interface prototypes. In Proc. of Graphics Interface 2010, 245-252, 2010
  • 48. 48 The $P recognizer • Multistroke gesture recognizer • Should become articulation-independent: removes all the temporal information associated to gesture points • Users perform gestures in various ways, not easily accountable in training sets • 442 possible ways to draw a square with 1, 2, 3, and 4 strokes Stroke gesture recognizers: recent recognizers in HCI Source: Vatavu, R.D., Anthony, L., Wobbrock, J.O. (2012). Gestures as Point Clouds: A $P Recognizer for User Interface Prototypes. Int. Conf. on Multimodal Interaction 2012, ICMI’12
  • 49. 49 Comparison of some $* recognizers Stroke gesture recognizers: recent recognizers in HCI Source: Vatavu, R.D., Anthony, L., Wobbrock, J.O. (2012). Gestures as Point Clouds: A $P Recognizer for User Interface Prototypes. Int. Conf. on Multimodal Interaction 2012, ICMI’12
  • 50. 50 Other gesture recognizers • Protractor (Li, CHI 2010) • $3 (Kratz and Rohs, IUI 2010) • Protractor3D (Kratz and Rohs, IUI 2011) • $N-Protractor (Anthony and Wobbrock, GI 2012) • 1ȼ (Herold and Stahovich, SBIM 2012) • Match-Up & Conquer (Rekik, Vatavu, Grisoni, AVI 2014) • Penny Pincher (Taranta and LaViola, GI 2015) • Jackknife (Taranta et al., CHI 2017) • Other dissimilarity functions • Dynamic Time Warping (DTW), Hausdorff, modified Hausdorff, Levenshtein + chain coding,… • Techniques for optimizing training sets • 1F (Vatavu, IUI 2012) • metaheuristics (Pittman, Taranta, LaViola, IUI 2016). Stroke gesture recognizers: recent recognizers in HCI
  • 51. 51 Example: Match-up & Conquer • Robust recognition of multi-touch gestures • Users perform multi-touch gestures in various ways, not easily accountable in training sets Stroke gesture recognizers: recent recognizers in HCI Source: Yosra Rekik, Radu-Daniel Vatavu, Laurent Grisoni. 2014. Match-Up & Conquer: A Two-Step Technique for Recognizing Unconstrained Bimanual and Multi-Finger Touch Input. In Proc. of AVI’14, 201-208
  • 52. 52 Nearest-Neighbor-Classification (NNC) • Two families of approaches • “Between points” distance • $-Family recognizers: $1, $3, $N, $P, $P+, $V, $Q,… • Variants and optimizations: ProTractor, Protractor3D,… • “Vector between points” distance • PennyPincher, JackKnife,… [Vatavu R.-D. et al, ICMI ’12] [Taranta E.M. et al, C&G ’16] Software dimension: Which algorithm?
  • 53. 53 Nearest-Neighbor-Classification (NNC) • Two families of approaches • “Between points” distance • $-Family recognizers: $1, $3, $N, $P, $P+, $V, $Q,… • Variants and optimizations: ProTractor, Protactor3D,… • “Vector between points” distance • PennyPincher, JackKnife,… • A third new family of approaches • “Vector between vectors” distance: our approach Software dimension: Which algorithm?
  • 54. 54 • Local Shape Distance between 2 triangles based on similarity (Roselli’s distance) 𝑎 𝑏 𝑢 𝑣 𝑎 + 𝑏 𝑢 + 𝑣 Paolo Roselli Università degli Studi di Roma, Italy Software dimension: Which algorithm? Source: Lorenzo Luzzi & Paolo Roselli, The shape of planar smooth gestures and the convergence of a gesture recognizer, Aequationes mathematicae volume 94, 219–233(2020).
  • 55. 55 • Step 1. Vectorization for each pair of vectors between three consecutive points p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 56. 56 • Step 1. Vectorization for each pair of vectors between three consecutive points p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q4 q5 q6 Training gesture Candidate gesture q1 q2 q3 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 57. 57 • Step 2. Mapping candidate’s triangles onto training gesture’s triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture p1 p2 p3 p2 p3 p4 p3 p4 p5 p4 p5 p6 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 58. 58 • Step 2. Mapping candidate’s triangles onto training gesture’s triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture p1 p2 p3 p2 p3 p4 p3 p4 p5 p4 p5 p6 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 59. 59 • Step 3. Computation of Local Shape Distance between pairs of triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p3 p4 p5 p2 p3 p4 p4 p5 p6 Training gesture Candidate gesture p1p2p3, q1q2q3 (N)LSD ( ) ( =0.02 p2p3p4, q2q3q4 ) =0.04 (p3p4p5, q3q4q5 ) =0.0001 ) p4p5p6, q3q4q5 ( =0.03 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 60. 60 • Step 4. Summing all individual figures into final one • Step 5. Iterate for every training gesture p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p3 p4 p5 p2 p3 p4 p4 p5 p6 Training gesture Candidate gesture p1p2p3, q1q2q3 (N)LSD ( ) p2p3p4, q2q3q4 ) ( ) p4p5p6, q3q4q5 ( (p3p4p5, q3q4q5 ) =0.02 =0.04 =0.0001 =0.03 =0.02+0.04+0.0001+0.03=0.0901 (indicative figures) Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 61. • Physical demand depends on variables • Gesture form: specifies which form of gesture is elicited. Possible values are: • S= stroke when the gesture only consists of taps and flicks • T= static when the gesture is performed in only one location • M= static with motion (when the gesture is performed with a static pose while the rest is moving) • D= dynamic when the gesture does capture any change or motion User experience dimension: evaluation criteria
  • 62. • Physical demand depends on variables • Laterality: characterizes how the two hands are employed to produce gestures, with two categories, as done in many studies. Possible values are: • D= dominant unimanual, N= non-dominant unimanual, S= symmetric bimanual, A= asymmetric bimanual User experience dimension: evaluation criteria Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426 D (right handed) N (right handed) S (right handed) A (right handed)
  • 63. • Aigner et al.’s taxonomy of mid- air gestures • P= Pointing gestures (= deictic gestures) indicate people, objects, directions Psychological dimension: What is a gesture? Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
  • 64. • Aigner et al.’s taxonomy of mid- air gestures • Semaphoric gestures are hand postures and movements conveying specific meanings • T= Static semaphorics are identified by a specific hand posture. Example: a flat palm facing from the actor means “stop”. • D= Dynamic semaphorics convey information through their temporal aspects. Example: a circular hand motion means “rotate” • S= Semaphoric strokes represent hand flicks are single, stroke-like movements. Example: a left flick of the hand means “dismiss this object” Psychological dimension: What is a gesture? Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
  • 65. • Aigner et al.’s taxonomy of mid- air gestures • M= Manipulation gestures a guide movement in a short feedback loop. Thus, they feature a tight relationship between the movements of the actor and the movements of the object to be manipulated. The actor waits for the entity to “follow” before continuing Psychological dimension: What is a gesture? Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human- preferences-in-usage-of-gesture-types-for-hci/
  • 66. • When to recognize? Stroke gesture recognizers Source: Olivier Bau, Wendy E. Mackay, OctoPocus: a dynamic guide for learning gesture-based command sets. UIST 2008: 37-46
  • 67. • A Gesture Set is a set of potentially related gestures that could be used together in one or several interactive applications • A gesture set could be defined for a particular - population of users, e.g., motor-impaired people - set of tasks, e.g., annotation of X-rays - platform/device, e.g., for an operating system, a style guide - environment, e.g., machine verification • A gesture set is said to be - domain-dependent if used only in a particular application domain (e.g., mammography, pediatrics, nuclear power plant) - domain-independent if used across several applications (e.g., for an operating system) Gesture sets: training and reference
  • 68. • xStroke Gesture set: a set frequent gestures • All printable ASCII characters: letters a-z, digits 0-9, and punctuation Gesture sets: examples
  • 69. • xStroke Gesture set: a set frequent gestures • All additional gestures necessary to function as a full keyboard replacement: Space, BackSpace, Return, Tab, Escape, Shift, Control, Meta Gesture sets: examples
  • 70. • xStroke Gesture set: a set frequent gestures • Advanced set of gestures: • Numbers in letter mode • Punctuation in letter m Gesture sets: examples
  • 71. 1. Design applications with touch interaction as the primary expected input method. 2. Provide visual feedback for interactions of all types (touch, pen, stylus, mouse, etc.) 3. Optimize targeting by adjusting touch target size, contact geometry, scrubbing and rocking. 4. Optimize accuracy through the use of snap points and directional "rails". 5. Provide tooltips and handles to help improve touch accuracy for tightly packed UI items. 6. Don't use timed interactions whenever possible (example of appropriate use: touch and hold). 7. Don't use the number of fingers used to distinguish the manipulation whenever possible Gesture sets: guidelines
  • 72. Thank you very much for your attention

Editor's Notes

  1. Gesture-based interaction without the support of speech input Tailor-made for interaction design
  2. Gesture-based interaction without the support of speech input Tailor-made for interaction design
  3. Gesture-based interaction without the support of speech input Tailor-made for interaction design