SlideShare a Scribd company logo
1 of 82
Francqui Chair 2020, Inaugural Lesson:
Gestural Interaction, Is it Really Natural?
Jean Vanderdonckt, UCLouvain
Vrije Universiteit Brussel, February 20, 2020, 4 pm-6 pm
Location: Room I.0.02, Pleinlaan 2, B-1050 Brussels
Presented by
Prof. Dr. Beat Signer
3
Jean Vanderdonckt
Université catholique de Louvain (UCLouvain)
Louvain School of Management (LSM)
Louvain Research Institute in Management and Organizations
(LouRIM)
Institute of Information and Communication Technologies,
Electronics and Applied Mathematics (ICTEAM)
Director of Louvain Interaction Lab
Place des Doyens, 1 – B-1348 Louvain-la-Neuve,
Belgium
Gestural Interaction (Francqui Chair, VUB, Brussels, February 20, 2020) 4
Some links
• Web site
https://wise.vub.ac.be/news/francqui-chair-2020-prof-jean-
vanderdonckt
• Join me on
• SlideShare: https://www.slideshare.net/jeanvdd
• LinkedIn: https://www.linkedin.com/in/jeanvdd/
• YouTube: https://www.youtube.com/user/jeanvdd
• Amazon: https://www.amazon.com/Jean-
Vanderdonckt/e/B01640UKYK
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
7
• The intersection between the 5 human senses gives rise to
many possible interaction modalities
Psychological dimension: What is a gesture?
Gesture
8
• The intersection between the 5 human senses gives rise to
many possible interaction modalities
• Gesture communication emerges in young children even
before development of language
Psychological dimension: What is a gesture?
Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of
Nonverbal Behavior 24, 81-103
9
• Innate gestures (natural?)
• Gestures that the user intuitively knows or that make sense,
based on the person’s understanding of the world
• Examples
• Pointing to aim a target
• Grabbing to pick an object
(MS Kinect)
• Pushing to select something
• Learned gestures (less natural => memorability?)
• Gestures the user needs to learn before
• Examples
• Waving to engage
• Making a specific pose
to cancel an action
Psychological dimension: What is a gesture?
10
• The intersection between the 5 human senses gives rise to
many possible interaction modalities
• Gesture communication emerges in young children even
before development of language
• Blind people gesture as they speak just as much as
sighted individuals do, even when they know their listener
is also blind.
Psychological dimension: What is a gesture?
Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of
Nonverbal Behavior 24, 81-103
11
• The intersection between the 5 human senses gives rise to
many possible interaction modalities
• Gesture communication emerges in young children even
before development of language
• Blind people gesture as they speak just as much as
sighted individuals do, even when they know their listener
is also blind
• People gesture without a visual model
• Gestures therefore require neither a model nor an
observant partner
Psychological dimension: What is a gesture?
Source: J. M. Iverson, S. Goldin-Meadow. (1998). Why people gesture when they speak. Nature, 396:228
12
• Kendon’s classification of gestures (1972)
Psychological dimension: What is a gesture?
Gesticulation
Spontaneous movements of the hands
and arms that accompany speech
Speech-framed gestures
Gesticulation that is integrated into a spoken
utterance, replacing a particular word
Pantomimes
Gestures that depict objects or actions, with or
without an accompanying speech
Emblems Familiar gestures accepted as a standard
Signs Complete linguistic system
• McNeil’s interpretation of Kendon’s continuum (1994)
Psychological dimension: What is a gesture?
Gesticulation
Speech-framed gestures
Pantomimes
Emblems
Signs
“Adam Kendon once
distinguished gestures
of different kinds along
a continuum that I
named “Kendon's
Continuum”, in his
honor.” [McNeill, 1992]
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992
Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200
Mandatory presence
of speech
Optional presence
of speech
Mandatory presence of
speech frames
Optional absence
of speech
Mandatory absence
of speech
• McNeil’s interpretation of Kendon’s continuum (1994)
Psychological dimension: What is a gesture?
Gesticulation
Speech-framed gestures
Pantomimes
Emblems
Signs
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992
Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200
Mandatory presence
of speech
Optional presence
of speech
Mandatory presence of
speech frames
Optional absence
of speech
Mandatory absence
of speech
“As one moves along Kendon’s
Continuum, two kinds of
reciprocal changes occur. First,
the degree to which speech is an
obligatory accompaniment of
gesture decreases from
gesticulation to signs. Second, the
degree to which a gesture shows
the properties of a language
increases.”
[McNeill, 1992]
• McNeil’s interpretation of Kendon’s continuum (1994)
Psychological dimension: What is a gesture?
Gesticulation
Speech-framed gestures
Pantomimes
Emblems
Signs
Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992.
“Gestures enhance,
complement, and
sometimes even
replace speech.”
• Gesture (…) are communicative movements of the hands
and arms which express — just as language — speakers’
attitudes, ideas, feelings and intentions…” (Müller, 1998)
Psychological dimension: What is a gesture?
• Saffer’s definition: “a gesture (…) is any physical
movement that a digital system can sense and
respond to without the aid of a traditional pointing
devices, such as a mouse or stylus”
Psychological dimension: What is a gesture?
Source: Saffer, D., Designing Gestural Interfaces, O'Reilly Media, November 2008.
Sensor
Gesture
recognizer
Actuator
Context of use =
(User, Platform/device, Environment)
Disturbances
Feedback
feeds drives
operates on
is sensed by
produces
• Turk’s definition in Human-Computer Interaction
(2002)
• ”…expressive, meaningful body motions –i.e. physical
movements of the fingers, hands, arms, head, face or
body with the intent to convey information or interact
with the environment.”
Psychological dimension: What is a gesture?
Sources: Turk, M. (2002). Gesture Recognition. In K. M. Stanney (Ed.), Handbook of Virtual Environments (pp. 223–237). London:
Lawrence Erlbaum Associates, Publishers.
Isabel Benavente Rodriguez, Nicolai Marquardt, Gesture Elicitation Study on How to Opt-in & Opt-out from Interactions
with Public Displays, Proc. of ISS ‘17, pp. 32-41.
• Aigner et al.’s taxonomy of mid-
air gestures
• P= Pointing gestures (= deictic
gestures) indicate people,
objects, directions
Psychological dimension: What is a gesture?
Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand
Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
• Aigner et al.’s taxonomy of mid-
air gestures
• Semaphoric gestures are hand
postures and movements
conveying specific meanings
• T= Static semaphorics are identified by a
specific hand posture. Example: a flat palm
facing from the actor means “stop”.
• D= Dynamic semaphorics convey information
through their temporal aspects. Example: a
circular hand motion means “rotate”
• S= Semaphoric strokes represent hand flicks
are single, stroke-like movements. Example: a
left flick of the hand means “dismiss this
object”
Psychological dimension: What is a gesture?
Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand
Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
• Aigner et al.’s taxonomy of mid-
air gestures
• M= Manipulation gestures a
guide movement in a short
feedback loop. Thus, they feature
a tight relationship between the
movements of the actor and the
movements of the object to be
manipulated. The actor waits for
the entity to “follow” before
continuing
Psychological dimension: What is a gesture?
Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human-
preferences-in-usage-of-gesture-types-for-hci/
• A gesture is any particular type of body
movement performed in 1D, 2D, or 3D.
• e.g., Hand movement (supination, pronation, etc.)
• e.g., Head movement (lips, eyes, face, etc.)
• e.g., Full body movement (silhouette, posture, etc.)
Psychological dimension: What is a gesture?
Source: Kendon, A. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press, 2004.
• A gesture is any particular type of body
movement performed in 1D, 2D, or 3D
Psychological dimension: What is a gesture?
Individual body part Combined body parts
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
25
Hardware dimension: How to gesture?
Paradigms of gesture interaction
Contact-based interaction
(surface limitation?)
26
Hardware dimension: How to gesture?
Paradigms of gesture interaction
Contact-based interaction
(surface limitation?)
Contact-less interaction
With wearable
27
Hardware dimension: How to gesture?
Paradigms of gesture interaction
Contact-based interaction
(surface limitation?)
Contact-less interaction
With wearable Without wearable
Close Far
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
29
Software dimension: Which algorithm?
In Window mode In full screen mode
Source:
https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html
XStroke
30
Software dimension: Which algorithm?
Rubine (1991)
31
Software dimension: Which algorithm?
Siger (2005)
Training phase
Vector string:
Stroke Vector direction
Left L
Right R
Up U
Down D
Regular expression:
(NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+.
LU,U,U,U,U,U,RU,RU,RU,RU,RU,RU,R,R,R,R,R,R,R,R,RD,RD,RD, RD,D,D,D,D,LD,LD,LD,LD,L,L,LD,LD,LD,D,D,D,D,D,D,D,D,D,L
32
Software dimension: Which algorithm?
LVS (2007)
33
Software dimension: Which algorithm?
Source: Beat Signer, U. Kurmann, Moira C. Norrie: iGesture: A General Gesture Recognition Framework. ICDAR 2007: 954-958
34
Software dimension: Which algorithm?
35
Nearest-Neighbor-Classification (NNC) for 2D strokes
Y
0 0.25 0.50 0.75 1
X
0
0.25
0.50
0.75
1
= candidate point
x = reference point
x
x
x
x x
x
x
x
x
x
x
x
x
x
k nearest neighbors
1 nearest
neighbor distance
0 1
x x x x x
x x x
x x x
Reference gestures
Training set
candidate
gesture
p2
p3
p4
p1
k-NN
k nearest neighbors
1-NN
Single nearest neighbor
applied to gesture recognition
q3
q1
q2
q4
reference
gesture
Software dimension: Which algorithm?
36
Nearest-Neighbor-Classification (NNC)
• Pre-processing steps to ensure invariance
• Re-sampling
• Points with same space between: isometricity
• Points with same timestamp between: isochronicity
• Same amount of points: isoparameterization
• Re-Scaling
• Normalisation of the bounding box into [0..1]x[0..1] square
• Rotation to reference angle
• Rotate to 0°
• Re-rotating and distance computation
• Distance computed between candidate gesture and
reference gestures (1-NN)
Software dimension: Which algorithm?
37
Nearest-Neighbor-Classification (NNC)
• Two families of approaches
• “Between points” distance
• $-Family recognizers: $1, $3, $N, $P, $P+,
$V, $Q,…
• Variants and optimizations: ProTractor,
Protactor3D,…
• “Vector between points” distance
• PennyPincher, JackKnife,…
[Vatavu R.-D. et al, ICMI ’12]
[Taranta E.M. et al, C&G ’16]
Software dimension: Which algorithm?
38
Nearest-Neighbor-Classification (NNC)
• Two families of approaches
• “Between points” distance
• $-Family recognizers: $1, $3, $N, $P, $P+,
$V, $Q,…
• Variants and optimizations: ProTractor,
Protactor3D,…
• “Vector between points” distance
• PennyPincher, JackKnife,…
• A third new family of approaches
• “Vector between vectors” distance:
our approach
Software dimension: Which algorithm?
39
• Local Shape Distance between 2 triangles based on
similarity (Roselli’s distance)
𝑎
𝑏
𝑢
𝑣
𝑎 + 𝑏
𝑢 + 𝑣
Paolo Roselli
Università degli Studi di Roma, Italy
Software dimension: Which algorithm?
Source: Lorenzo Luzzi & Paolo Roselli, The shape of planar smooth gestures and the convergence of a gesture recognizer, Aequationes
mathematicae volume 94, 219–233(2020).
40
• Step 1. Vectorization for each pair of vectors between
three consecutive points
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
41
• Step 1. Vectorization for each pair of vectors between
three consecutive points
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q4
q5
q6
Training
gesture
Candidate
gesture
q1
q2
q3
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
42
• Step 2. Mapping candidate’s triangles onto training
gesture’s triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
p1
p2
p3
p2
p3
p4
p3
p4
p5
p4
p5
p6
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
43
• Step 2. Mapping candidate’s triangles onto training
gesture’s triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
Training
gesture
Candidate
gesture
p1
p2
p3
p2
p3
p4
p3
p4
p5
p4
p5
p6
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
44
• Step 3. Computation of Local Shape Distance between
pairs of triangles
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p3
p4
p5
p2
p3
p4
p4
p5
p6
Training
gesture
Candidate
gesture
p1p2p3,
q1q2q3
(N)LSD (
)
(
=0.02
p2p3p4,
q2q3q4 )
=0.04
(p3p4p5,
q3q4q5 )
=0.0001
)
p4p5p6,
q3q4q5
(
=0.03
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
45
• Step 4. Summing all individual figures into final one
• Step 5. Iterate for every training gesture
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p4
p5
p6
q1
q2
q3
q4
q5
q6
p1
p2
p3
p3
p4
p5
p2
p3
p4
p4
p5
p6
Training
gesture
Candidate
gesture
p1p2p3,
q1q2q3
(N)LSD (
)
p2p3p4,
q2q3q4 )
(
)
p4p5p6,
q3q4q5
(
(p3p4p5,
q3q4q5 )
=0.02 =0.04 =0.0001 =0.03
=0.02+0.04+0.0001+0.03=0.0901
(indicative figures)
Software dimension: Which algorithm?
Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with
Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
3D Hand gesture recognition
Software dimension: Which algorithm?
Full body gesture recognition
Software dimension: Which algorithm?
See video at
https://youtu.be/RTEGMlDRDL0
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
• Many application domains
Usage dimension: Which application domains?
• Smart Home: TV, fridge, coffee machine,…
• Example: Samsung Smart TV
Usage dimension: Which application domains?
• Ring device: gesture elicitation study
Usage dimension: Which application domains?
Source: Bogdan-Florin Gheran, Jean Vanderdonckt, Radu-Daniel Vatavu, Gestures for Smart Rings: Empirical Results, Insights, and
Design Implications. Conference on Designing Interactive Systems 2018: 623-635
• Ring device at Home (Family management)
Usage dimension: Which application domains?
• Ring device at Home (Family management)
Usage dimension: Which application domains?
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
• History: chironomia
Social dimension: critical factors
Source: Gilbert Austin, Chironomia, or a Treatise on Rhetorical Delivery (1806). Ed. Mary Margaret Robb and Lester Thonssen.
Carbondale, IL: Southern Illinois UP, 1966.
Social dimension: critical factors
56
Gestures in Movies
• 12 Angry Men
(dir.: S. Lumet)
The defense and the prosecution have rested and the
jury is filing into the jury room to decide if a young man
is guilty or innocent of murdering his father. What
begins as an open-and-shut case of murder soon
becomes a detective story that presents a succession
of clues creating doubt, and a mini-drama of each of the
jurors' prejudices and preconceptions about the trial,
the accused, and each other. Based on the play, all of
the action takes place on the stage of the jury room.
Gesture in Movies
Social dimension: critical factors
Range of
motion
• Range of motion
• Relates the distance between the position of the human
body producing the gesture and the location of the
gesture
• Possible values are:
• C= Close intimate, I= Intimate, P= Personal, S=Social,
U= Public, R= Remote
Social dimension: critical factors
• Cultural influence and interpretation
• The gesture ”The ring” has four major meanings:
OK/Good, Orifice, Zero, Threat
Social dimension: critical factors
Source: Morris, Collett, Marsh, & O’Shaghnessy (1979)
• “Very tasty”: how to gesture that?
• Cultural influence and interpretation
• The gesture ”The ring” has four major meanings:
OK/Good, Orifice, Zero, Threat
Social dimension: critical factors
Source: Morris, Collett, Marsh, & O’Shaghnessy (1979)
• “Very tasty”: how to gesture that?
• Cultural influence and interpretation
• The gesture ”The ring” has four major meanings:
OK/Good, Orifice, Zero, Threat
Social dimension: critical factors
Source: Morris, Collett, Marsh, & O’Shaghnessy (1979)
• “Very tasty”: how to gesture that?
• “Select”: how to gesture that?
Italy: pointing Sweden: open hand Turkey: two open hands
• Engagement by propagation:
only by careful consideration of social gestures
Social dimension: critical factors
Source: Jean-Yves Lionel Lawson, Jean Vanderdonckt, Radu-Daniel Vatavu: Mass-Computer Interaction for Thousands of Users and
Beyond. CHI Extended Abstracts 2018
www.skemmi.com
See video at https://www.youtube.com/watch?v=IZaAl59AUk8
• Social acceptance or reluctance?
Social dimension: critical factors
What do you think of the Itchy Nose?
See video at https://www.youtube.com/watch?v=IQ_LkPM_GHs
• Social acceptance or reluctance?
Social dimension: critical factors
0.319
0.246 0.232 0.225
0.203
0.185 0.181 0.178
0.167 0.167 0.167
0.244 0.231
0.141
0.218
0.179
0.103
0.218 0.218
0.103
0.141
0.
0.509
0.236
0.273
0.291
0.182
0.4
0.109
0.182
0.236
0.145
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Start
Player
Go to
Previous
Item
Turn
Alarm Off
Increase
Volume
Turn Light
On
Decrease
Volume
Turn TV
Off
Turn TV
On
Dim Light Turn
Alarm On
Han
C
Global Male
Both side
Double tap
2
Center tap Left push
Both sides
Both hands
tap
Right push Left to right
flick
Right to left
flick
Continuous
rubbing
Top push Top to
bottom flick
Repe
cent
n
Agreement
rate
Source: Jorge Luis Pérez-Medina, Santiago Villarreal, Jean Vanderdonckt: A Gesture Elicitation Study of Nose-Based Gestures.
Sensors 20(24): 7118 (2020)
• Social acceptance or reluctance?
Social dimension: critical factors
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
• Compatibility
• Imposition: each OS imposes its own set of gestures
• Some are natural to use
• Some others are not natural at all and remain unused
• Lack of acceptability:
• Some gestures can be simple to be recognized by the system,
yet hard to remember and reproduce for end users
• Some gestures can be accepted by end users, but harder to be
recognized
• Gestures are often system-defined, sometimes
designer-defined, rarely not user-defined
• Need for Gesture Elicitation Study (GES)
• Is a study for asking end-users to elicit their own gestures for a
set of predefined functions through referents and reach a
consensus
User experience dimension: evaluation criteria
• Compatibility
• Need for a natural conceptual model: Virtual Library
User experience dimension: evaluation criteria
Source: https://www.youtube.com/watch?v=ls5kj7oVwto
• Consistency
• Imposition: each OS imposes its own set of gestures
• A few gestures are common (hopefully, natural)
• Most other gestures are inconsistent
User experience dimension: evaluation criteria
Source: Ryan Lee, www.gesturecons.com
• Consistency: standardization?
• Not really, but ongoing
User experience dimension: evaluation criteria
• Consistency: standardization?
• Gesture example “Shake”: Wake up, Update, Reset, Next
track, Shuffle, Unlock, Enter a comment (SnappView)
User experience dimension: evaluation criteria
• Discoverability
• GUI interaction is based on
• action exploration: eg by menu
• recognition (best)
• Gestures are not easy
to discover
• Solutions appear: feedforward
User experience dimension: evaluation criteria
Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010
Olivier Bau, Wendy E. Mackay, OctoPocus: a dynamic guide for learning gesture-based command sets. UIST 2008: 37-46
• Control: explicit, mixed, not implicit
• “When users think they did one thing but actually did
something else, they lose their sense of controlling the
system because they don't understand the connection
between actions and results.”
User experience dimension: evaluation criteria
Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010.
Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for
sound design. Tangible and Embedded Interaction 2008: 3-10
Commands AudioCube action(s)
Grab and set
DOF=0
(discrete)
DOF=1
(linearly
correlated)
Move cube in 2D, almost 3D, 3D
• Control: explicit, mixed, not implicit
• “When users think they did one thing but actually did
something else, they lose their sense of controlling the
system because they don't understand the connection
between actions and results.”
User experience dimension: evaluation criteria
Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010.
Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for
sound design. Tangible and Embedded Interaction 2008: 3-10
Commands AudioCube action(s)
DOF=1
(linearly
correlated)
Move cube in 2D, almost 3D, 3D
• Control: explicit, mixed, not implicit
• “When users think they did one thing but actually did
something else, they lose their sense of controlling the
system because they don't understand the connection
between actions and results.”
User experience dimension: evaluation criteria
Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010.
Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for
sound design. Tangible and Embedded Interaction 2008: 3-10
Commands AudioCube action(s)
DOF=1
(rotationally
correlated)
Rotate in 2D, 3D
DOF=2
(freeform)
2D, 3D gestures
• Physical demand depends on variables
• Gesture form: specifies which form of gesture is elicited.
Possible values are:
• S= stroke when the gesture only consists of taps and flicks
• T= static when the gesture is performed in only one location
• M= static with motion (when the gesture is performed with a
static pose while the rest is moving)
• D= dynamic when the gesture does capture any change or
motion
User experience dimension: evaluation criteria
• Physical demand depends on variables
• Laterality: characterizes how the two hands are
employed to produce gestures, with two categories, as
done in many studies. Possible values are:
• D= dominant unimanual, N= non-dominant unimanual,
S= symmetric bimanual, A= asymmetric bimanual
User experience dimension: evaluation criteria
Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426
D
(right handed)
N
(right handed)
S
(right handed)
A
(right handed)
• Agreement among end users
• Agreement Rate = the number of pairs of participants in
agreement with each other divided by the total number
of pairs of participants that could be in agreement
• Compute co-agreement for pairs, groups (eg male vs
female), categories of referents (eg basic vs. advanced)
User experience dimension: evaluation criteria
agreement rate disagreement rate co-agreement rate
Source: Radu-Daniel Vatavu, Jacob O. Wobbrock, Between-Subjects Elicitation Studies: Formalization and Tool Support. CHI 2016:
3390-3402.
• FUN!
• In games, all gestures are permitted (body)
• In professional contexts, a gesture could be
considered as awkward, inappropriate
User experience dimension: evaluation criteria
Example: MiniEurope (Alterface)
Psychological
dimension
Hardware
dimension
Software
dimension
Usage
dimension
Social
dimension
User experience
dimension
Conclusion: is it really natural?
81
• Gesture interaction is suitable for
• Natural interactions: interact directly with objects in physical way
• Less cumbersome or visible hardware
• Flexibility in hardware
• Fun
• Gesture interaction is NOT suitable for
• Heavy data input (use keyboards instead)
• Absence of visual feedback (e.g., a system without a screen or
targeting users with visual impairments)
• Unmet physical demands (e.g., swipe to receive a phone call in
winter)
• Constrained contexts of use (e.g., privacy, embarrassment)
• User and task
• Platform/device
• Environment
Conclusion: is it really natural?
Thank you very much
for your attention

More Related Content

What's hot

What's hot (20)

Haptic Technology #Manoj_Rockstar
Haptic Technology #Manoj_RockstarHaptic Technology #Manoj_Rockstar
Haptic Technology #Manoj_Rockstar
 
A Dynamic hand gesture recognition for human computer interaction
A Dynamic hand gesture recognition for human computer interactionA Dynamic hand gesture recognition for human computer interaction
A Dynamic hand gesture recognition for human computer interaction
 
Haptic technology
Haptic technologyHaptic technology
Haptic technology
 
Haptic technology-sense of touch(connect)
Haptic technology-sense of touch(connect)Haptic technology-sense of touch(connect)
Haptic technology-sense of touch(connect)
 
Gesture recognition
Gesture recognitionGesture recognition
Gesture recognition
 
Haptics touch the virtual
Haptics touch the virtualHaptics touch the virtual
Haptics touch the virtual
 
Implementation of augmented reality(ar) technology for business beneficiary
Implementation of augmented reality(ar) technology for business beneficiaryImplementation of augmented reality(ar) technology for business beneficiary
Implementation of augmented reality(ar) technology for business beneficiary
 
AMBIENT INTELLIGENCE by Bhagyasri Matta
AMBIENT INTELLIGENCE by Bhagyasri MattaAMBIENT INTELLIGENCE by Bhagyasri Matta
AMBIENT INTELLIGENCE by Bhagyasri Matta
 
History of Artificial Intelligence.pptx
History of Artificial Intelligence.pptxHistory of Artificial Intelligence.pptx
History of Artificial Intelligence.pptx
 
Haptic technology ppt
Haptic technology pptHaptic technology ppt
Haptic technology ppt
 
Hand gesture recognition
Hand gesture recognitionHand gesture recognition
Hand gesture recognition
 
Exergame Fitness Presentation
Exergame  Fitness  PresentationExergame  Fitness  Presentation
Exergame Fitness Presentation
 
Jeff Johnson: Psych 101: The Psychological Basis for UI Design Rules
Jeff Johnson: Psych 101: The Psychological Basis for UI Design RulesJeff Johnson: Psych 101: The Psychological Basis for UI Design Rules
Jeff Johnson: Psych 101: The Psychological Basis for UI Design Rules
 
Hand gesture recognition
Hand gesture recognitionHand gesture recognition
Hand gesture recognition
 
Ppt final
Ppt finalPpt final
Ppt final
 
Gesture recognition technology
Gesture recognition technologyGesture recognition technology
Gesture recognition technology
 
Augmented Reality - Connecting Physical and Virtual Worlds
Augmented Reality - Connecting Physical and Virtual WorldsAugmented Reality - Connecting Physical and Virtual Worlds
Augmented Reality - Connecting Physical and Virtual Worlds
 
Haptic technology
Haptic technologyHaptic technology
Haptic technology
 
A Framework For Dynamic Hand Gesture Recognition Using Key Frames Extraction
A Framework For Dynamic Hand Gesture Recognition Using Key Frames ExtractionA Framework For Dynamic Hand Gesture Recognition Using Key Frames Extraction
A Framework For Dynamic Hand Gesture Recognition Using Key Frames Extraction
 
HAPTIC TECHNOLOGY
HAPTIC TECHNOLOGY HAPTIC TECHNOLOGY
HAPTIC TECHNOLOGY
 

Similar to Gestural Interaction, Is it Really Natural?

embodied interaction
embodied interactionembodied interaction
embodied interaction
chienta
 
Theoretical underpinnings
Theoretical underpinningsTheoretical underpinnings
Theoretical underpinnings
Grainne Conole
 

Similar to Gestural Interaction, Is it Really Natural? (20)

CDE CONFERENCE POSTER Metaphor & elearning
CDE CONFERENCE POSTER Metaphor & elearningCDE CONFERENCE POSTER Metaphor & elearning
CDE CONFERENCE POSTER Metaphor & elearning
 
2010 mobilelearning workshopsctr5
2010 mobilelearning workshopsctr52010 mobilelearning workshopsctr5
2010 mobilelearning workshopsctr5
 
Participatory design fieldwork. Dealing with emotions
Participatory design fieldwork. Dealing with emotionsParticipatory design fieldwork. Dealing with emotions
Participatory design fieldwork. Dealing with emotions
 
Smiljana Antonijevic - Second Life, Second Body
Smiljana Antonijevic - Second Life, Second BodySmiljana Antonijevic - Second Life, Second Body
Smiljana Antonijevic - Second Life, Second Body
 
Introduction to language
Introduction to language Introduction to language
Introduction to language
 
Designing with Immigrants. When emotions run high
Designing with Immigrants. When emotions run high Designing with Immigrants. When emotions run high
Designing with Immigrants. When emotions run high
 
Designing with Inmigrants. When emotions run high
Designing with Inmigrants. When emotions run high Designing with Inmigrants. When emotions run high
Designing with Inmigrants. When emotions run high
 
embodied interaction
embodied interactionembodied interaction
embodied interaction
 
Tiii presentation by dr. ir S.A.G. Stephan Wensveen
Tiii presentation by dr. ir S.A.G. Stephan WensveenTiii presentation by dr. ir S.A.G. Stephan Wensveen
Tiii presentation by dr. ir S.A.G. Stephan Wensveen
 
VR in education
VR in educationVR in education
VR in education
 
Interaction between abstract agents: Increasing the readability of causal e...
Interaction between abstract agents:  Increasing the readability of causal  e...Interaction between abstract agents:  Increasing the readability of causal  e...
Interaction between abstract agents: Increasing the readability of causal e...
 
Chapter 10: Symbolic Interactionism and Social Constructionism-Toby Zhu
Chapter 10: Symbolic Interactionism and Social Constructionism-Toby ZhuChapter 10: Symbolic Interactionism and Social Constructionism-Toby Zhu
Chapter 10: Symbolic Interactionism and Social Constructionism-Toby Zhu
 
Designing with Immigrants. When emotions run high.pptx
Designing with Immigrants. When emotions run high.pptxDesigning with Immigrants. When emotions run high.pptx
Designing with Immigrants. When emotions run high.pptx
 
Interpersonal Communication V2
Interpersonal Communication V2Interpersonal Communication V2
Interpersonal Communication V2
 
Theoretical underpinnings
Theoretical underpinningsTheoretical underpinnings
Theoretical underpinnings
 
Understading communication ppt1
Understading communication  ppt1Understading communication  ppt1
Understading communication ppt1
 
2014-02-11-jd
2014-02-11-jd2014-02-11-jd
2014-02-11-jd
 
Designing Interactions / Experiences: Lecture #02
Designing Interactions / Experiences: Lecture #02Designing Interactions / Experiences: Lecture #02
Designing Interactions / Experiences: Lecture #02
 
Jwhshahaab
JwhshahaabJwhshahaab
Jwhshahaab
 
"Exciting": E-Learning and the Big "E"
"Exciting": E-Learning and the Big "E""Exciting": E-Learning and the Big "E"
"Exciting": E-Learning and the Big "E"
 

More from Jean Vanderdonckt

Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...
Jean Vanderdonckt
 
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
Jean Vanderdonckt
 

More from Jean Vanderdonckt (20)

To the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User InterfacesTo the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User Interfaces
 
Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...
 
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
 
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
 
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
 
Gesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOpsGesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOps
 
Engineering Slidable User Interfaces with Slime
Engineering Slidable User Interfaces with SlimeEngineering Slidable User Interfaces with Slime
Engineering Slidable User Interfaces with Slime
 
Evaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and MeasuresEvaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and Measures
 
Conducting a Gesture Elicitation Study: How to Get the Best Gestures From Peo...
Conducting a Gesture Elicitation Study: How to Get the Best Gestures From Peo...Conducting a Gesture Elicitation Study: How to Get the Best Gestures From Peo...
Conducting a Gesture Elicitation Study: How to Get the Best Gestures From Peo...
 
Designing Gestural Interaction: Challenges and Pitfalls
Designing Gestural Interaction: Challenges and PitfallsDesigning Gestural Interaction: Challenges and Pitfalls
Designing Gestural Interaction: Challenges and Pitfalls
 
Fundamentals of Gestural Interaction
Fundamentals of Gestural InteractionFundamentals of Gestural Interaction
Fundamentals of Gestural Interaction
 
User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...
 
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
 
Attach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you WorkAttach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you Work
 
The Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV GesturesThe Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV Gestures
 
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper BodyHead and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
 
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for SmartphonesG-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
 
Vector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture RecognitionVector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture Recognition
 
An ontology for reasoning on body-based gestures
 An ontology for reasoning on body-based gestures An ontology for reasoning on body-based gestures
An ontology for reasoning on body-based gestures
 
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design AlternativesAB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
 

Recently uploaded

Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
FIDO Alliance
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 

Recently uploaded (20)

Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 

Gestural Interaction, Is it Really Natural?

  • 1. Francqui Chair 2020, Inaugural Lesson: Gestural Interaction, Is it Really Natural? Jean Vanderdonckt, UCLouvain Vrije Universiteit Brussel, February 20, 2020, 4 pm-6 pm Location: Room I.0.02, Pleinlaan 2, B-1050 Brussels Presented by Prof. Dr. Beat Signer
  • 2.
  • 3. 3 Jean Vanderdonckt Université catholique de Louvain (UCLouvain) Louvain School of Management (LSM) Louvain Research Institute in Management and Organizations (LouRIM) Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM) Director of Louvain Interaction Lab Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium
  • 4. Gestural Interaction (Francqui Chair, VUB, Brussels, February 20, 2020) 4 Some links • Web site https://wise.vub.ac.be/news/francqui-chair-2020-prof-jean- vanderdonckt • Join me on • SlideShare: https://www.slideshare.net/jeanvdd • LinkedIn: https://www.linkedin.com/in/jeanvdd/ • YouTube: https://www.youtube.com/user/jeanvdd • Amazon: https://www.amazon.com/Jean- Vanderdonckt/e/B01640UKYK
  • 7. 7 • The intersection between the 5 human senses gives rise to many possible interaction modalities Psychological dimension: What is a gesture? Gesture
  • 8. 8 • The intersection between the 5 human senses gives rise to many possible interaction modalities • Gesture communication emerges in young children even before development of language Psychological dimension: What is a gesture? Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of Nonverbal Behavior 24, 81-103
  • 9. 9 • Innate gestures (natural?) • Gestures that the user intuitively knows or that make sense, based on the person’s understanding of the world • Examples • Pointing to aim a target • Grabbing to pick an object (MS Kinect) • Pushing to select something • Learned gestures (less natural => memorability?) • Gestures the user needs to learn before • Examples • Waving to engage • Making a specific pose to cancel an action Psychological dimension: What is a gesture?
  • 10. 10 • The intersection between the 5 human senses gives rise to many possible interaction modalities • Gesture communication emerges in young children even before development of language • Blind people gesture as they speak just as much as sighted individuals do, even when they know their listener is also blind. Psychological dimension: What is a gesture? Source: S.W. Goodwyn, L.P. Acredolo, C. Brown. (2000). Impact of Symbolic Gesturing on Early Language Development. Journal of Nonverbal Behavior 24, 81-103
  • 11. 11 • The intersection between the 5 human senses gives rise to many possible interaction modalities • Gesture communication emerges in young children even before development of language • Blind people gesture as they speak just as much as sighted individuals do, even when they know their listener is also blind • People gesture without a visual model • Gestures therefore require neither a model nor an observant partner Psychological dimension: What is a gesture? Source: J. M. Iverson, S. Goldin-Meadow. (1998). Why people gesture when they speak. Nature, 396:228
  • 12. 12 • Kendon’s classification of gestures (1972) Psychological dimension: What is a gesture? Gesticulation Spontaneous movements of the hands and arms that accompany speech Speech-framed gestures Gesticulation that is integrated into a spoken utterance, replacing a particular word Pantomimes Gestures that depict objects or actions, with or without an accompanying speech Emblems Familiar gestures accepted as a standard Signs Complete linguistic system
  • 13. • McNeil’s interpretation of Kendon’s continuum (1994) Psychological dimension: What is a gesture? Gesticulation Speech-framed gestures Pantomimes Emblems Signs “Adam Kendon once distinguished gestures of different kinds along a continuum that I named “Kendon's Continuum”, in his honor.” [McNeill, 1992] Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992 Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200 Mandatory presence of speech Optional presence of speech Mandatory presence of speech frames Optional absence of speech Mandatory absence of speech
  • 14. • McNeil’s interpretation of Kendon’s continuum (1994) Psychological dimension: What is a gesture? Gesticulation Speech-framed gestures Pantomimes Emblems Signs Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992 Kendon, A., Do gestures communicate? A review. Research on Language and Social Interaction 27, 1994, 175-200 Mandatory presence of speech Optional presence of speech Mandatory presence of speech frames Optional absence of speech Mandatory absence of speech “As one moves along Kendon’s Continuum, two kinds of reciprocal changes occur. First, the degree to which speech is an obligatory accompaniment of gesture decreases from gesticulation to signs. Second, the degree to which a gesture shows the properties of a language increases.” [McNeill, 1992]
  • 15. • McNeil’s interpretation of Kendon’s continuum (1994) Psychological dimension: What is a gesture? Gesticulation Speech-framed gestures Pantomimes Emblems Signs Source: D. McNeill. Hand and Mind: What Gesture Reveals about Thought. University Chicago Press, 1992. “Gestures enhance, complement, and sometimes even replace speech.”
  • 16. • Gesture (…) are communicative movements of the hands and arms which express — just as language — speakers’ attitudes, ideas, feelings and intentions…” (Müller, 1998) Psychological dimension: What is a gesture?
  • 17. • Saffer’s definition: “a gesture (…) is any physical movement that a digital system can sense and respond to without the aid of a traditional pointing devices, such as a mouse or stylus” Psychological dimension: What is a gesture? Source: Saffer, D., Designing Gestural Interfaces, O'Reilly Media, November 2008. Sensor Gesture recognizer Actuator Context of use = (User, Platform/device, Environment) Disturbances Feedback feeds drives operates on is sensed by produces
  • 18. • Turk’s definition in Human-Computer Interaction (2002) • ”…expressive, meaningful body motions –i.e. physical movements of the fingers, hands, arms, head, face or body with the intent to convey information or interact with the environment.” Psychological dimension: What is a gesture? Sources: Turk, M. (2002). Gesture Recognition. In K. M. Stanney (Ed.), Handbook of Virtual Environments (pp. 223–237). London: Lawrence Erlbaum Associates, Publishers. Isabel Benavente Rodriguez, Nicolai Marquardt, Gesture Elicitation Study on How to Opt-in & Opt-out from Interactions with Public Displays, Proc. of ISS ‘17, pp. 32-41.
  • 19. • Aigner et al.’s taxonomy of mid- air gestures • P= Pointing gestures (= deictic gestures) indicate people, objects, directions Psychological dimension: What is a gesture? Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
  • 20. • Aigner et al.’s taxonomy of mid- air gestures • Semaphoric gestures are hand postures and movements conveying specific meanings • T= Static semaphorics are identified by a specific hand posture. Example: a flat palm facing from the actor means “stop”. • D= Dynamic semaphorics convey information through their temporal aspects. Example: a circular hand motion means “rotate” • S= Semaphoric strokes represent hand flicks are single, stroke-like movements. Example: a left flick of the hand means “dismiss this object” Psychological dimension: What is a gesture? Source: Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindlbauer, D., Ion, A., Zhao, S., et al. (2012). Understanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI.
  • 21. • Aigner et al.’s taxonomy of mid- air gestures • M= Manipulation gestures a guide movement in a short feedback loop. Thus, they feature a tight relationship between the movements of the actor and the movements of the object to be manipulated. The actor waits for the entity to “follow” before continuing Psychological dimension: What is a gesture? Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human- preferences-in-usage-of-gesture-types-for-hci/
  • 22. • A gesture is any particular type of body movement performed in 1D, 2D, or 3D. • e.g., Hand movement (supination, pronation, etc.) • e.g., Head movement (lips, eyes, face, etc.) • e.g., Full body movement (silhouette, posture, etc.) Psychological dimension: What is a gesture? Source: Kendon, A. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press, 2004.
  • 23. • A gesture is any particular type of body movement performed in 1D, 2D, or 3D Psychological dimension: What is a gesture? Individual body part Combined body parts
  • 25. 25 Hardware dimension: How to gesture? Paradigms of gesture interaction Contact-based interaction (surface limitation?)
  • 26. 26 Hardware dimension: How to gesture? Paradigms of gesture interaction Contact-based interaction (surface limitation?) Contact-less interaction With wearable
  • 27. 27 Hardware dimension: How to gesture? Paradigms of gesture interaction Contact-based interaction (surface limitation?) Contact-less interaction With wearable Without wearable Close Far
  • 29. 29 Software dimension: Which algorithm? In Window mode In full screen mode Source: https://www.usenix.org/legacy/publications/library/proceedings/usenix03/tech/freenix03/full_papers/worth/worth_html/xstroke.html XStroke
  • 30. 30 Software dimension: Which algorithm? Rubine (1991)
  • 31. 31 Software dimension: Which algorithm? Siger (2005) Training phase Vector string: Stroke Vector direction Left L Right R Up U Down D Regular expression: (NE|E|SE)+(NW|N|NE)+(SW|W|NW)+(SE|S|SW)+. LU,U,U,U,U,U,RU,RU,RU,RU,RU,RU,R,R,R,R,R,R,R,R,RD,RD,RD, RD,D,D,D,D,LD,LD,LD,LD,L,L,LD,LD,LD,D,D,D,D,D,D,D,D,D,L
  • 32. 32 Software dimension: Which algorithm? LVS (2007)
  • 33. 33 Software dimension: Which algorithm? Source: Beat Signer, U. Kurmann, Moira C. Norrie: iGesture: A General Gesture Recognition Framework. ICDAR 2007: 954-958
  • 35. 35 Nearest-Neighbor-Classification (NNC) for 2D strokes Y 0 0.25 0.50 0.75 1 X 0 0.25 0.50 0.75 1 = candidate point x = reference point x x x x x x x x x x x x x x k nearest neighbors 1 nearest neighbor distance 0 1 x x x x x x x x x x x Reference gestures Training set candidate gesture p2 p3 p4 p1 k-NN k nearest neighbors 1-NN Single nearest neighbor applied to gesture recognition q3 q1 q2 q4 reference gesture Software dimension: Which algorithm?
  • 36. 36 Nearest-Neighbor-Classification (NNC) • Pre-processing steps to ensure invariance • Re-sampling • Points with same space between: isometricity • Points with same timestamp between: isochronicity • Same amount of points: isoparameterization • Re-Scaling • Normalisation of the bounding box into [0..1]x[0..1] square • Rotation to reference angle • Rotate to 0° • Re-rotating and distance computation • Distance computed between candidate gesture and reference gestures (1-NN) Software dimension: Which algorithm?
  • 37. 37 Nearest-Neighbor-Classification (NNC) • Two families of approaches • “Between points” distance • $-Family recognizers: $1, $3, $N, $P, $P+, $V, $Q,… • Variants and optimizations: ProTractor, Protactor3D,… • “Vector between points” distance • PennyPincher, JackKnife,… [Vatavu R.-D. et al, ICMI ’12] [Taranta E.M. et al, C&G ’16] Software dimension: Which algorithm?
  • 38. 38 Nearest-Neighbor-Classification (NNC) • Two families of approaches • “Between points” distance • $-Family recognizers: $1, $3, $N, $P, $P+, $V, $Q,… • Variants and optimizations: ProTractor, Protactor3D,… • “Vector between points” distance • PennyPincher, JackKnife,… • A third new family of approaches • “Vector between vectors” distance: our approach Software dimension: Which algorithm?
  • 39. 39 • Local Shape Distance between 2 triangles based on similarity (Roselli’s distance) 𝑎 𝑏 𝑢 𝑣 𝑎 + 𝑏 𝑢 + 𝑣 Paolo Roselli Università degli Studi di Roma, Italy Software dimension: Which algorithm? Source: Lorenzo Luzzi & Paolo Roselli, The shape of planar smooth gestures and the convergence of a gesture recognizer, Aequationes mathematicae volume 94, 219–233(2020).
  • 40. 40 • Step 1. Vectorization for each pair of vectors between three consecutive points p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 41. 41 • Step 1. Vectorization for each pair of vectors between three consecutive points p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q4 q5 q6 Training gesture Candidate gesture q1 q2 q3 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 42. 42 • Step 2. Mapping candidate’s triangles onto training gesture’s triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture p1 p2 p3 p2 p3 p4 p3 p4 p5 p4 p5 p6 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 43. 43 • Step 2. Mapping candidate’s triangles onto training gesture’s triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 Training gesture Candidate gesture p1 p2 p3 p2 p3 p4 p3 p4 p5 p4 p5 p6 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 44. 44 • Step 3. Computation of Local Shape Distance between pairs of triangles p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p3 p4 p5 p2 p3 p4 p4 p5 p6 Training gesture Candidate gesture p1p2p3, q1q2q3 (N)LSD ( ) ( =0.02 p2p3p4, q2q3q4 ) =0.04 (p3p4p5, q3q4q5 ) =0.0001 ) p4p5p6, q3q4q5 ( =0.03 Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 45. 45 • Step 4. Summing all individual figures into final one • Step 5. Iterate for every training gesture p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p4 p5 p6 q1 q2 q3 q4 q5 q6 p1 p2 p3 p3 p4 p5 p2 p3 p4 p4 p5 p6 Training gesture Candidate gesture p1p2p3, q1q2q3 (N)LSD ( ) p2p3p4, q2q3q4 ) ( ) p4p5p6, q3q4q5 ( (p3p4p5, q3q4q5 ) =0.02 =0.04 =0.0001 =0.03 =0.02+0.04+0.0001+0.03=0.0901 (indicative figures) Software dimension: Which algorithm? Source: Jean Vanderdonckt, Paolo Roselli, Jorge Luis Pérez-Medina, !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances. ICMI 2018: 125-134
  • 46. 3D Hand gesture recognition Software dimension: Which algorithm?
  • 47. Full body gesture recognition Software dimension: Which algorithm? See video at https://youtu.be/RTEGMlDRDL0
  • 49. • Many application domains Usage dimension: Which application domains?
  • 50. • Smart Home: TV, fridge, coffee machine,… • Example: Samsung Smart TV Usage dimension: Which application domains?
  • 51. • Ring device: gesture elicitation study Usage dimension: Which application domains? Source: Bogdan-Florin Gheran, Jean Vanderdonckt, Radu-Daniel Vatavu, Gestures for Smart Rings: Empirical Results, Insights, and Design Implications. Conference on Designing Interactive Systems 2018: 623-635
  • 52. • Ring device at Home (Family management) Usage dimension: Which application domains?
  • 53. • Ring device at Home (Family management) Usage dimension: Which application domains?
  • 55. • History: chironomia Social dimension: critical factors Source: Gilbert Austin, Chironomia, or a Treatise on Rhetorical Delivery (1806). Ed. Mary Margaret Robb and Lester Thonssen. Carbondale, IL: Southern Illinois UP, 1966.
  • 56. Social dimension: critical factors 56 Gestures in Movies • 12 Angry Men (dir.: S. Lumet) The defense and the prosecution have rested and the jury is filing into the jury room to decide if a young man is guilty or innocent of murdering his father. What begins as an open-and-shut case of murder soon becomes a detective story that presents a succession of clues creating doubt, and a mini-drama of each of the jurors' prejudices and preconceptions about the trial, the accused, and each other. Based on the play, all of the action takes place on the stage of the jury room.
  • 57. Gesture in Movies Social dimension: critical factors
  • 58. Range of motion • Range of motion • Relates the distance between the position of the human body producing the gesture and the location of the gesture • Possible values are: • C= Close intimate, I= Intimate, P= Personal, S=Social, U= Public, R= Remote Social dimension: critical factors
  • 59. • Cultural influence and interpretation • The gesture ”The ring” has four major meanings: OK/Good, Orifice, Zero, Threat Social dimension: critical factors Source: Morris, Collett, Marsh, & O’Shaghnessy (1979) • “Very tasty”: how to gesture that?
  • 60. • Cultural influence and interpretation • The gesture ”The ring” has four major meanings: OK/Good, Orifice, Zero, Threat Social dimension: critical factors Source: Morris, Collett, Marsh, & O’Shaghnessy (1979) • “Very tasty”: how to gesture that?
  • 61. • Cultural influence and interpretation • The gesture ”The ring” has four major meanings: OK/Good, Orifice, Zero, Threat Social dimension: critical factors Source: Morris, Collett, Marsh, & O’Shaghnessy (1979) • “Very tasty”: how to gesture that? • “Select”: how to gesture that? Italy: pointing Sweden: open hand Turkey: two open hands
  • 62. • Engagement by propagation: only by careful consideration of social gestures Social dimension: critical factors Source: Jean-Yves Lionel Lawson, Jean Vanderdonckt, Radu-Daniel Vatavu: Mass-Computer Interaction for Thousands of Users and Beyond. CHI Extended Abstracts 2018 www.skemmi.com See video at https://www.youtube.com/watch?v=IZaAl59AUk8
  • 63. • Social acceptance or reluctance? Social dimension: critical factors What do you think of the Itchy Nose? See video at https://www.youtube.com/watch?v=IQ_LkPM_GHs
  • 64. • Social acceptance or reluctance? Social dimension: critical factors 0.319 0.246 0.232 0.225 0.203 0.185 0.181 0.178 0.167 0.167 0.167 0.244 0.231 0.141 0.218 0.179 0.103 0.218 0.218 0.103 0.141 0. 0.509 0.236 0.273 0.291 0.182 0.4 0.109 0.182 0.236 0.145 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Start Player Go to Previous Item Turn Alarm Off Increase Volume Turn Light On Decrease Volume Turn TV Off Turn TV On Dim Light Turn Alarm On Han C Global Male Both side Double tap 2 Center tap Left push Both sides Both hands tap Right push Left to right flick Right to left flick Continuous rubbing Top push Top to bottom flick Repe cent n Agreement rate Source: Jorge Luis Pérez-Medina, Santiago Villarreal, Jean Vanderdonckt: A Gesture Elicitation Study of Nose-Based Gestures. Sensors 20(24): 7118 (2020)
  • 65. • Social acceptance or reluctance? Social dimension: critical factors
  • 67. • Compatibility • Imposition: each OS imposes its own set of gestures • Some are natural to use • Some others are not natural at all and remain unused • Lack of acceptability: • Some gestures can be simple to be recognized by the system, yet hard to remember and reproduce for end users • Some gestures can be accepted by end users, but harder to be recognized • Gestures are often system-defined, sometimes designer-defined, rarely not user-defined • Need for Gesture Elicitation Study (GES) • Is a study for asking end-users to elicit their own gestures for a set of predefined functions through referents and reach a consensus User experience dimension: evaluation criteria
  • 68. • Compatibility • Need for a natural conceptual model: Virtual Library User experience dimension: evaluation criteria Source: https://www.youtube.com/watch?v=ls5kj7oVwto
  • 69. • Consistency • Imposition: each OS imposes its own set of gestures • A few gestures are common (hopefully, natural) • Most other gestures are inconsistent User experience dimension: evaluation criteria Source: Ryan Lee, www.gesturecons.com
  • 70. • Consistency: standardization? • Not really, but ongoing User experience dimension: evaluation criteria
  • 71. • Consistency: standardization? • Gesture example “Shake”: Wake up, Update, Reset, Next track, Shuffle, Unlock, Enter a comment (SnappView) User experience dimension: evaluation criteria
  • 72. • Discoverability • GUI interaction is based on • action exploration: eg by menu • recognition (best) • Gestures are not easy to discover • Solutions appear: feedforward User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010 Olivier Bau, Wendy E. Mackay, OctoPocus: a dynamic guide for learning gesture-based command sets. UIST 2008: 37-46
  • 73. • Control: explicit, mixed, not implicit • “When users think they did one thing but actually did something else, they lose their sense of controlling the system because they don't understand the connection between actions and results.” User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010. Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for sound design. Tangible and Embedded Interaction 2008: 3-10 Commands AudioCube action(s) Grab and set DOF=0 (discrete) DOF=1 (linearly correlated) Move cube in 2D, almost 3D, 3D
  • 74. • Control: explicit, mixed, not implicit • “When users think they did one thing but actually did something else, they lose their sense of controlling the system because they don't understand the connection between actions and results.” User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010. Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for sound design. Tangible and Embedded Interaction 2008: 3-10 Commands AudioCube action(s) DOF=1 (linearly correlated) Move cube in 2D, almost 3D, 3D
  • 75. • Control: explicit, mixed, not implicit • “When users think they did one thing but actually did something else, they lose their sense of controlling the system because they don't understand the connection between actions and results.” User experience dimension: evaluation criteria Sources: Donald A. Norman, J. Nielsen, Gestural interfaces: a step backward in usability, Interactions, V7N5, Sept. 2010. Bert Schiettecatte, Jean Vanderdonckt, AudioCubes: a distributed cube tangible interface based on interaction range for sound design. Tangible and Embedded Interaction 2008: 3-10 Commands AudioCube action(s) DOF=1 (rotationally correlated) Rotate in 2D, 3D DOF=2 (freeform) 2D, 3D gestures
  • 76. • Physical demand depends on variables • Gesture form: specifies which form of gesture is elicited. Possible values are: • S= stroke when the gesture only consists of taps and flicks • T= static when the gesture is performed in only one location • M= static with motion (when the gesture is performed with a static pose while the rest is moving) • D= dynamic when the gesture does capture any change or motion User experience dimension: evaluation criteria
  • 77. • Physical demand depends on variables • Laterality: characterizes how the two hands are employed to produce gestures, with two categories, as done in many studies. Possible values are: • D= dominant unimanual, N= non-dominant unimanual, S= symmetric bimanual, A= asymmetric bimanual User experience dimension: evaluation criteria Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426 D (right handed) N (right handed) S (right handed) A (right handed)
  • 78. • Agreement among end users • Agreement Rate = the number of pairs of participants in agreement with each other divided by the total number of pairs of participants that could be in agreement • Compute co-agreement for pairs, groups (eg male vs female), categories of referents (eg basic vs. advanced) User experience dimension: evaluation criteria agreement rate disagreement rate co-agreement rate Source: Radu-Daniel Vatavu, Jacob O. Wobbrock, Between-Subjects Elicitation Studies: Formalization and Tool Support. CHI 2016: 3390-3402.
  • 79. • FUN! • In games, all gestures are permitted (body) • In professional contexts, a gesture could be considered as awkward, inappropriate User experience dimension: evaluation criteria Example: MiniEurope (Alterface)
  • 81. 81 • Gesture interaction is suitable for • Natural interactions: interact directly with objects in physical way • Less cumbersome or visible hardware • Flexibility in hardware • Fun • Gesture interaction is NOT suitable for • Heavy data input (use keyboards instead) • Absence of visual feedback (e.g., a system without a screen or targeting users with visual impairments) • Unmet physical demands (e.g., swipe to receive a phone call in winter) • Constrained contexts of use (e.g., privacy, embarrassment) • User and task • Platform/device • Environment Conclusion: is it really natural?
  • 82. Thank you very much for your attention

Editor's Notes

  1. Kendon [12] added that an important part of ‘kinetics’ research shows that gesture phrases can be organized in relation to speech phrases. We can parallel his arguments and reasonings to relatively coincide with pen gestures (as natural human gestures) and the instantiated sketch-objects (as dictionised speech contents). He also stated that there is a consistent patterning in how gesture phrases are formed in relation to the phrases of speech – just as, in a continuous discourse, speakers group tone units into higher order groupings resembling a hierarchy, so gesture phrases may be similarly organized. Gestures that are put together to form phrases of bodily actions have the characteristics that permit them to be ‘recognized’ as components of willing communicative action
  2. Designed for speech-related gestures Not completely relevant for interaction design
  3. Gesture-based interaction without the support of speech input Tailor-made for interaction design
  4. Gesture-based interaction without the support of speech input Tailor-made for interaction design
  5. Gesture-based interaction without the support of speech input Tailor-made for interaction design
  6. Gesture-based interaction without the support of speech input Tailor-made for interaction design
  7. Chironomia is the art of using gesticulations or hand gestures to good effect in traditional rhetoric or oratory. Effective use of the hands, with or without the use of the voice, is a practice of great antiquity, which was developed and systematized by the Greeks and the Romans. Various gestures had conventionalized meanings which were commonly understood, either within certain class or professional groups, or broadly among dramatic and oratorical audiences.