Gestural Interaction, Is it Really Natural?

Francqui Chair 2020, Inaugural Lesson:
Gestural Interaction, Is it Really Natural?
Jean Vanderdonckt, UCLouvain
Vrije Universiteit Brussel, February 20, 2020, 4 pm-6 pm
Location: Room I.0.02, Pleinlaan 2, B-1050 Brussels
Presented by
Prof. Dr. Beat Signer

3
Jean Vanderdonckt
Université catholique de Louvain (UCLouvain)
Louvain School of Management (LSM)
Louvain Research Institute in Management and Organizations
(LouRIM)
Institute of Information and Communication Technologies,
Electronics and Applied Mathematics (ICTEAM)
Director of Louvain Interaction Lab
Place des Doyens, 1 – B-1348 Louvain-la-Neuve,
Belgium

Gestural Interaction (Francqui Chair, VUB, Brussels, February 20, 2020) 4
Some links
• Web site
https://wise.vub.ac.be/news/francqui-chair-2020-prof-jean-
vanderdonckt
• Join me on
• SlideShare: https://www.slideshare.net/jeanvdd
• LinkedIn: https://www.linkedin.com/in/jeanvdd/
• YouTube: https://www.youtube.com/user/jeanvdd
• Amazon: https://www.amazon.com/Jean-
Vanderdonckt/e/B01640UKYK

Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension

7
• The intersection between the 5 human senses gives rise to
many possible interaction modalities
Psychological dimension: What is a gesture?
Gesture

8
• Gesture communication emerges in young children even
before development of language
Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of
Nonverbal Behavior 24, 81-103

9
• Innate gestures (natural?)
• Gestures that the user intuitively knows or that make sense,
based on the person’s understanding of the world
• Examples
• Pointing to aim a target
• Grabbing to pick an object
(MS Kinect)
• Pushing to select something
• Learned gestures (less natural => memorability?)
• Gestures the user needs to learn before
• Examples
• Waving to engage
• Making a specific pose
to cancel an action

10
• Blind people gesture as they speak just as much as
sighted individuals do, even when they know their listener
is also blind.
Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of
Nonverbal Behavior 24, 81-103

11
• Blind people gesture as they speak just as much as
sighted individuals do, even when they know their listener
is also blind
• People gesture without a visual model
• Gestures therefore require neither a model nor an
observant partner
Source: J. M. Iverson, S. Goldin-Meadow. (1998). Why people gesture when they speak. Nature, 396:228

12
• Kendon’s classification of gestures (1972)
Gesticulation
Spontaneous movements of the hands
and arms that accompany speech
Speech-framed gestures
Gesticulation that is integrated into a spoken
utterance, replacing a particular word
Pantomimes
Gestures that depict objects or actions, with or
without an accompanying speech
Emblems Familiar gestures accepted as a standard
Signs Complete linguistic system

• McNeil’s interpretation of Kendon’s continuum (1994)
Gesticulation
Pantomimes
Emblems
Signs
“Adam Kendon once
distinguished gestures
of different kinds along
a continuum that I
named “Kendon's
Continuum”, in his
honor.” [McNeill, 1992]
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992
Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200
Mandatory presence
of speech
Optional presence
of speech
Mandatory presence of
speech frames
Optional absence
of speech
Mandatory absence
of speech

Gesticulation
Pantomimes
Emblems
Signs
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992
Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200
Mandatory presence
of speech
Optional presence
of speech
Mandatory presence of
speech frames
Optional absence
of speech
Mandatory absence
of speech
“As one moves along Kendon’s
Continuum, two kinds of
reciprocal changes occur. First,
the degree to which speech is an
obligatory accompaniment of
gesture decreases from
gesticulation to signs. Second, the
degree to which a gesture shows
the properties of a language
increases.”
[McNeill, 1992]

Gesticulation
Pantomimes
Emblems
Signs
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992.
“Gestures enhance,
complement, and
sometimes even
replace speech.”

• Gesture (…) are communicative movements of the hands
and arms which express — just as language — speakers’
attitudes, ideas, feelings and intentions…” (Müller, 1998)

• Saffer’s definition: “a gesture (…) is any physical
movement that a digital system can sense and
respond to without the aid of a traditional pointing
devices, such as a mouse or stylus”
Source: Saffer, D., Designing Gestural Interfaces, O'Reilly Media, November 2008.
Sensor
Gesture
recognizer
Actuator
Context of use =
(User, Platform/device, Environment)
Disturbances
Feedback
feeds drives
operates on
is sensed by
produces

• Turk’s definition in Human-Computer Interaction
(2002)
• ”…expressive, meaningful body motions –i.e. physical
movements of the fingers, hands, arms, head, face or
body with the intent to convey information or interact
with the environment.”
Sources: Turk, M. (2002). Gesture Recognition. In K. M. Stanney (Ed.), Handbook of Virtual Environments (pp. 223–237). London:
Lawrence Erlbaum Associates, Publishers.
Isabel Benavente Rodriguez, Nicolai Marquardt, Gesture Elicitation Study on How to Opt-in & Opt-out from Interactions
with Public Displays, Proc. of ISS ‘17, pp. 32-41.

• Aigner et al.’s taxonomy of mid-
air gestures
• P= Pointing gestures (= deictic
gestures) indicate people,
objects, directions
Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand
Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.

air gestures
• Semaphoric gestures are hand
postures and movements
conveying specific meanings
• T= Static semaphorics are identified by a
specific hand posture. Example: a flat palm
facing from the actor means “stop”.
• D= Dynamic semaphorics convey information
through their temporal aspects. Example: a
circular hand motion means “rotate”
• S= Semaphoric strokes represent hand flicks
are single, stroke-like movements. Example: a
left flick of the hand means “dismiss this
object”
Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand
Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.

air gestures
• M= Manipulation gestures a
guide movement in a short
feedback loop. Thus, they feature
a tight relationship between the
movements of the actor and the
movements of the object to be
manipulated. The actor waits for
the entity to “follow” before
continuing
Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human-
preferences-in-usage-of-gesture-types-for-hci/

• A gesture is any particular type of body
movement performed in 1D, 2D, or 3D.
• e.g., Hand movement (supination, pronation, etc.)
• e.g., Head movement (lips, eyes, face, etc.)
• e.g., Full body movement (silhouette, posture, etc.)
Source: Kendon, A. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press, 2004.

• A gesture is any particular type of body
movement performed in 1D, 2D, or 3D
Individual body part Combined body parts

25
Hardware dimension: How to gesture?
Paradigms of gesture interaction
Contact-based interaction
(surface limitation?)

26
Contact-less interaction
With wearable

27
Contact-less interaction
With wearable Without wearable
Close Far

29
Software dimension: Which algorithm?
In Window mode In full screen mode
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
XStroke

30
Rubine (1991)

31
Siger (2005)
Training phase
Vector string:
Stroke Vector direction
Left L
Right R
Up U
Down D
Regular expression:
(NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+.
LU,U,U,U,U,U,RU,RU,RU,RU,RU,RU,R,R,R,R,R,R,R,R,RD,RD,RD, RD,D,D,D,D,LD,LD,LD,LD,L,L,LD,LD,LD,D,D,D,D,D,D,D,D,D,L

32
LVS (2007)

33
Source: Beat Signer, U. Kurmann, Moira C. Norrie: iGesture: A General Gesture Recognition Framework. ICDAR 2007: 954-958

34

35
Nearest-Neighbor-Classification (NNC) for 2D strokes
Y
0 0.25 0.50 0.75 1
X
0
0.25
0.50
0.75
1
= candidate point
x = reference point
x
x
x
x x
x
x
x
x
x
x
x
x
x
k nearest neighbors
1 nearest
neighbor distance
0 1
x x x x x
x x x
x x x
Reference gestures
Training set
candidate
gesture
p2
p3
p4
p1
k-NN
k nearest neighbors
1-NN
Single nearest neighbor
applied to gesture recognition
q3
q1
q2
q4
reference
gesture

36
Nearest-Neighbor-Classification (NNC)
• Pre-processing steps to ensure invariance
• Re-sampling
• Points with same space between: isometricity
• Points with same timestamp between: isochronicity
• Same amount of points: isoparameterization
• Re-Scaling
• Normalisation of the bounding box into [0..1]x[0..1] square
• Rotation to reference angle
• Rotate to 0°
• Re-rotating and distance computation
• Distance computed between candidate gesture and
reference gestures (1-NN)

37
• Two families of approaches
• “Between points” distance
• $-Family recognizers: $1, $3, $N, $P, $P+,
$V, $Q,…
• Variants and optimizations: ProTractor,
Protactor3D,…
• “Vector between points” distance
• PennyPincher, JackKnife,…
[Vatavu R.-D. et al, ICMI ’12]
[Taranta E.M. et al, C&G ’16]

38
• Two families of approaches
• “Between points” distance
• $-Family recognizers: $1, $3, $N, $P, $P+,
$V, $Q,…
• Variants and optimizations: ProTractor,
Protactor3D,…
• “Vector between points” distance
• PennyPincher, JackKnife,…
• A third new family of approaches
• “Vector between vectors” distance:
our approach

39
• Local Shape Distance between 2 triangles based on
similarity (Roselli’s distance)
𝑎
𝑏
𝑢
𝑣
𝑎 + 𝑏
𝑢 + 𝑣
Paolo Roselli
Università degli Studi di Roma, Italy
Source: Lorenzo Luzzi & Paolo Roselli, The shape of planar smooth gestures and the convergence of a gesture recognizer, Aequationes
mathematicae volume 94, 219–233(2020).

40
• Step 1. Vectorization for each pair of vectors between
three consecutive points
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134

41
• Step 1. Vectorization for each pair of vectors between
three consecutive points
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q4
q5
q6
Training
gesture
Candidate
gesture
q1
q2
q3

42
• Step 2. Mapping candidate’s triangles onto training
gesture’s triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
p1
p2
p3
p2
p3
p4
p3
p4
p5
p4
p5
p6

43
• Step 2. Mapping candidate’s triangles onto training
gesture’s triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
p1
p2
p3
p2
p3
p4
p3
p4
p5
p4
p5
p6

44
• Step 3. Computation of Local Shape Distance between
pairs of triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p3
p4
p5
p2
p3
p4
p4
p5
p6
Training
gesture
Candidate
gesture
p1p2p3,
q1q2q3
(N)LSD (
)
(
=0.02
p2p3p4,
q2q3q4 )
=0.04
(p3p4p5,
q3q4q5 )
=0.0001
)
p4p5p6,
q3q4q5
(
=0.03

45
• Step 4. Summing all individual figures into final one
• Step 5. Iterate for every training gesture
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p3
p4
p5
p2
p3
p4
p4
p5
p6
Training
gesture
Candidate
gesture
p1p2p3,
q1q2q3
(N)LSD (
)
p2p3p4,
q2q3q4 )
(
)
p4p5p6,
q3q4q5
(
(p3p4p5,
q3q4q5 )
=0.02 =0.04 =0.0001 =0.03
=0.02+0.04+0.0001+0.03=0.0901
(indicative figures)

3D Hand gesture recognition

Full body gesture recognition
See video at
https://youtu.be/RTEGMlDRDL0

• Many application domains
Usage dimension: Which application domains?

• Smart Home: TV, fridge, coffee machine,…
• Example: Samsung Smart TV

• Ring device: gesture elicitation study
Source: Bogdan-Florin Gheran, Jean Vanderdonckt, Radu-Daniel Vatavu, Gestures for Smart Rings: Empirical Results, Insights, and
Design Implications. Conference on Designing Interactive Systems 2018: 623-635

• Ring device at Home (Family management)

• History: chironomia
Social dimension: critical factors
Source: Gilbert Austin, Chironomia, or a Treatise on Rhetorical Delivery (1806). Ed. Mary Margaret Robb and Lester Thonssen.
Carbondale, IL: Southern Illinois UP, 1966.

56
Gestures in Movies
• 12 Angry Men
(dir.: S. Lumet)
The defense and the prosecution have rested and the
jury is filing into the jury room to decide if a young man
is guilty or innocent of murdering his father. What
begins as an open-and-shut case of murder soon
becomes a detective story that presents a succession
of clues creating doubt, and a mini-drama of each of the
jurors' prejudices and preconceptions about the trial,
the accused, and each other. Based on the play, all of
the action takes place on the stage of the jury room.

Gesture in Movies

Range of
motion
• Range of motion
• Relates the distance between the position of the human
body producing the gesture and the location of the
gesture
• Possible values are:
• C= Close intimate, I= Intimate, P= Personal, S=Social,
U= Public, R= Remote

• Cultural influence and interpretation
• The gesture ”The ring” has four major meanings:
OK/Good, Orifice, Zero, Threat
Source: Morris, Collett, Marsh, & O’Shaghnessy (1979)
• “Very tasty”: how to gesture that?

• Cultural influence and interpretation
• The gesture ”The ring” has four major meanings:
OK/Good, Orifice, Zero, Threat
Source: Morris, Collett, Marsh, & O’Shaghnessy (1979)
• “Very tasty”: how to gesture that?
• “Select”: how to gesture that?
Italy: pointing Sweden: open hand Turkey: two open hands

• Engagement by propagation:
only by careful consideration of social gestures
Source: Jean-Yves Lionel Lawson, Jean Vanderdonckt, Radu-Daniel Vatavu: Mass-Computer Interaction for Thousands of Users and
Beyond. CHI Extended Abstracts 2018
www.skemmi.com
See video at https://www.youtube.com/watch?v=IZaAl59AUk8

• Social acceptance or reluctance?
What do you think of the Itchy Nose?
See video at https://www.youtube.com/watch?v=IQ_LkPM_GHs

0.319
0.246 0.232 0.225
0.203
0.185 0.181 0.178
0.167 0.167 0.167
0.244 0.231
0.141
0.218
0.179
0.103
0.218 0.218
0.103
0.141
0.
0.509
0.236
0.273
0.291
0.182
0.4
0.109
0.182
0.236
0.145
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Start
Player
Go to
Previous
Item
Turn
Alarm Off
Increase
Volume
Turn Light
On
Decrease
Volume
Turn TV
Off
Turn TV
On
Dim Light Turn
Alarm On
Han
C
Global Male
Both side
Double tap
2
Center tap Left push
Both sides
Both hands
tap
Right push Left to right
flick
Right to left
flick
Continuous
rubbing
Top push Top to
bottom flick
Repe
cent
n
Agreement
rate
Source: Jorge Luis Pérez-Medina, Santiago Villarreal, Jean Vanderdonckt: A Gesture Elicitation Study of Nose-Based Gestures.
Sensors 20(24): 7118 (2020)

• Compatibility
• Imposition: each OS imposes its own set of gestures
• Some are natural to use
• Some others are not natural at all and remain unused
• Lack of acceptability:
• Some gestures can be simple to be recognized by the system,
yet hard to remember and reproduce for end users
• Some gestures can be accepted by end users, but harder to be
recognized
• Gestures are often system-defined, sometimes
designer-defined, rarely not user-defined
• Need for Gesture Elicitation Study (GES)
• Is a study for asking end-users to elicit their own gestures for a
set of predefined functions through referents and reach a
consensus
User experience dimension: evaluation criteria

• Compatibility
• Need for a natural conceptual model: Virtual Library
Source: https://www.youtube.com/watch?v=ls5kj7oVwto

• Consistency
• Imposition: each OS imposes its own set of gestures
• A few gestures are common (hopefully, natural)
• Most other gestures are inconsistent
Source: Ryan Lee, www.gesturecons.com

• Consistency: standardization?
• Not really, but ongoing

• Consistency: standardization?
• Gesture example “Shake”: Wake up, Update, Reset, Next
track, Shuffle, Unlock, Enter a comment (SnappView)

• Discoverability
• GUI interaction is based on
• action exploration: eg by menu
• recognition (best)
• Gestures are not easy
to discover
• Solutions appear: feedforward
Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010
Olivier Bau, Wendy E. Mackay, OctoPocus: a dynamic guide for learning gesture-based command sets. UIST 2008: 37-46

• Control: explicit, mixed, not implicit
• “When users think they did one thing but actually did
something else, they lose their sense of controlling the
system because they don't understand the connection
between actions and results.”
Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010.
Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for
sound design. Tangible and Embedded Interaction 2008: 3-10
Commands AudioCube action(s)
Grab and set
DOF=0
(discrete)
DOF=1
(linearly
correlated)
Move cube in 2D, almost 3D, 3D

DOF=1
(linearly
correlated)
Move cube in 2D, almost 3D, 3D

DOF=1
(rotationally
correlated)
Rotate in 2D, 3D
DOF=2
(freeform)
2D, 3D gestures

• Physical demand depends on variables
• Gesture form: specifies which form of gesture is elicited.
Possible values are:
• S= stroke when the gesture only consists of taps and flicks
• T= static when the gesture is performed in only one location
• M= static with motion (when the gesture is performed with a
static pose while the rest is moving)
• D= dynamic when the gesture does capture any change or
motion

• Physical demand depends on variables
• Laterality: characterizes how the two hands are
employed to produce gestures, with two categories, as
done in many studies. Possible values are:
• D= dominant unimanual, N= non-dominant unimanual,
S= symmetric bimanual, A= asymmetric bimanual
Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426
D
(right handed)
N
(right handed)
S
(right handed)
A
(right handed)

• Agreement among end users
• Agreement Rate = the number of pairs of participants in
agreement with each other divided by the total number
of pairs of participants that could be in agreement
• Compute co-agreement for pairs, groups (eg male vs
female), categories of referents (eg basic vs. advanced)
agreement rate disagreement rate co-agreement rate
Source: Radu-Daniel Vatavu, Jacob O. Wobbrock, Between-Subjects Elicitation Studies: Formalization and Tool Support. CHI 2016:
3390-3402.

• FUN!
• In games, all gestures are permitted (body)
• In professional contexts, a gesture could be
considered as awkward, inappropriate
Example: MiniEurope (Alterface)

Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
Conclusion: is it really natural?

81
• Gesture interaction is suitable for
• Natural interactions: interact directly with objects in physical way
• Less cumbersome or visible hardware
• Flexibility in hardware
• Fun
• Gesture interaction is NOT suitable for
• Heavy data input (use keyboards instead)
• Absence of visual feedback (e.g., a system without a screen or
targeting users with visual impairments)
• Unmet physical demands (e.g., swipe to receive a phone call in
winter)
• Constrained contexts of use (e.g., privacy, embarrassment)
• User and task
• Platform/device
• Environment
Conclusion: is it really natural?

Thank you very much
for your attention

Gestural Interaction, Is it Really Natural?

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Gestural Interaction, Is it Really Natural?

Similar to Gestural Interaction, Is it Really Natural? (20)

More from Jean Vanderdonckt

More from Jean Vanderdonckt (20)

Recently uploaded

Recently uploaded (20)

Gestural Interaction, Is it Really Natural?

Editor's Notes