SlideShare a Scribd company logo
1 of 80
Lecture 3: Conducting a Gesture Elicitation Study:
How to Get the Best Gestures From People?
Jean Vanderdonckt, UCLouvain
Vrije Universiteit Brussel, 20 April 2021, on-line
How to get the best gestures from people?
Case study: sketching a graphical UI
(Interact’07)
https://www.youtube.com/watch?v=SBNB1O-8pGw&t=5s
How to determine the best gestures for widgets?
How to determine the best gestures for widgets?
1: none
How to determine the best gestures for widgets?
1: none
2: low
How to determine the best gestures for widgets?
1: none
2: low
3: medium
How to determine the best gestures for widgets?
1: none
2: low
3: medium
4: high
Use de Borda Count method (Condorcet method)
• Each participant assigns a score equal to the number of
representations to whom he or she is preferred in
decreasing order: n-1, n-2, n-3,…
n=5
Participant #1 0 n-3=2 n-4=1 n-1=4 n-2=3
Participant #2 n-1=4 0 n-3=2 n-2=3 n-4=1
Score= 4
Score= 7
Score= 3
Score= 2
Score= 4
Use de Borda Count method (Condorcet method)
• Each participant assigns a score equal to the number of
representations to whom he or she is preferred in
decreasing order: n-1, n-2, n-3,…
Further classification: complexity of the representation
ST=sketching time
DEL=delete operations
Further classification: complexity of the representation
ST=sketching time
DEL=delete operations
Source: Suzanne Kieffer, Adrien Coyette, Jean Vanderdonckt, User interface design by sketching: a complexity analysis of widget representations.
EICS 2010: 57-66
How to get the best gestures from people?
Considered insofar:
• Human factors: preference, sketching time, delete
operations
• System factors: recognition rate
Other method for preference
• By elicitation
• By wizard of Oz
Gesture Elicitation Study (GES)
• To elicit
• To call forth or draw out something, such as information or a
response (Merriam Webster)
• To evoke or draw out a reaction, answer, or fact from someone
(Dictionary.com)
• To draw forth something that is latent or potential into existence
(Dictionary.com)
• Elicitation
• Acquisition of information from a person or a group of persons in a
manner that does not disclose the intent of the interview,
discussion or conversation
• Elicitation technique
• A technique used to discreetly extract information, not readily
available for a specific purpose
Source: Wobbrock et al., 2009
Gesture Elicitation Study (GES)
• Definition
• Is a study for asking end-users to elicit their own gestures for a set
of predefined functions through referents and reach a consensus
• Aims and goals
• To let end users elicit the gestures they want for a function
• To compute agreement between end users (=participants, subjects)
• To define a vocabulary of potential gestures
• To identify a consensus set, which is a sub-vocabulary of agreed
gestures. Output = Consensus Set
• For one context of use at a time: C= (U,P,E)
• User : any person, Functions: IoT functions
• Platform/device/sensor: an armband
• Environment: smart home, usability lab, controlled room
Gesture Elicitation Study (GES)
• Advantages
• Free from any form of stress or burden of working
• Information comes deliberately once the favorable context is
created
• No adverse effect
• Easy and cheap to run
• Disadvantages
• Legacy bias
• Previous experience with a similar system, another device
• No experience
• Lack of creativity
GES Step 1: Prepare setup C=(U,P,E)
• Create a representative
sampling of the user
population
GES Step 1: Prepare setup C=(U,P,E)
• Find a representative sampling of the user population
• Example of stratified sampling
Sampling Female Male
Age group % 32 51 49
18-24 8.9 3 2 1
25-34 15.4 5 3 2
35-44 16.5 5 3 2
45-54 18.3 6 3 3
55-64 18.7 6 3 3
65+ 22.2 7 4 3
Total 100 32 18 14
GES Step 1: Prepare setup C=(U,P,E)
• Find a representative sampling of the user population
• Find n=30 participants
• Ideally 50% female, 50% male
• Different ages (not all students!)
• Different backgrounds (not all in your discipline!)
• Different levels of experience (e.g., with touch or not)
• Each participant takes part in the experiment individually
• Define an experiment schedule
• Location (should be constant): place to choose
• Time: 30 min for each participant + resting time
• Schedule: organize a Doodle
• Test the whole protocol with a guinea pig
• Different from the participants!
GES Step 1: Prepare setup C=(U,P,E)
• Identify referents for the following tasks
• A referent is the expected result of an action (e.g., key
pressing, touch, gesture) for a function
• Distribution the definition of the 14 referents across groups
• Example: IoT functions:
(1) Turn the TV on/off (2) Start player
(3) Turn up the volume (4) Turn down the volume
(5) Go to the next item in a list (6) Go to the previous item
(7) Turn AC on/off (8) Turn lights on/off
(9) Brighten lights (10) Dim lights
(11)Turn heat on/off (12) Turn alarm on/off
(13)Answer phone call (14) End phone call
• Randomize the order of the referents before the study for each
participant (create lists with www.random.org)
Is symmetric to
GES Step 1: Prepare setup C=(U,P,E)
• What is a referent?
Referent:
- State before action
- Action description
- State after action
GES Step 1: Prepare setup C=(U,P,E)
• Referent: textual, graphical, animation, video (visual priming)
20. Permute files remotely
Before
After
My Network
My Disk
My Network
My Disk
Our Network
Partner Disk
Our Network
Partner Disk
Our Network
Partner Disk
Our Network
Partner Disk
Yellow Document
Red Document
Red Document
Yellow Document
Blue Document
Blue Document
GES Step 1: Prepare setup C=(U,P,E)
• Referent: textual, graphical, animation, video (visual priming)
https://www.youtube.com/watch?v=iVzE4EzfGoc&t=12s
GES Step 1: Prepare setup C=(U,P,E)
• Prepare the hardware, software
• Keep them constant throughout the experiment
• Determine a software for recording gestures in Log Files
• Example: Python software for Leap Motion
• Example: MyO Gesture Control
GES Step 1: Prepare setup C=(U,P,E)
• Identify the physical location: office, room, usability lab,…
• Keep it constant throughout the experiment
• Sometimes determined by the device
• Example: Usability lab
GES Step 2: Pre-test (before experiment)
• Each participant
• Comes to the experiment location according to schedule
• Signs a consent form
• Fills in a demographic questionnaire
• Notoriety and experience if any for the target device
• System experience in general (e.g., for a tablet, frequency)
• Performs a Motor-Skill test: to touch their thumb to each of
the other fingers of the same hand 2 times consecutively
without flaw
• Is presented with an introduction to the device (e.g., a video)
• Experimenter
• Assigns an ID to each participant on the consent form
• Checks that all forms are completely filled in, before and
after
GES Step 3: Test (conduct experiment)
• Each participant
• Is presented with the list of referents (randomly selected)
• Confirms she understood the tasks
• Is asked to think of a suitable gesture
• Says “I am ready”
• Gives a rating (Goodness-of-fit) between 1 and 10
• To describe how well the proposed gesture fits the referent
• 1 = very poor fit  10 = excellent fit
• Optionally gives a value between 1 and 7 for
• Complexity: 1= most complex  7= simplest gesture
• Memorability: 1=hardest to remember  7=easiest to remember
• Fun: 1=is the least funny  7=is the funniest
for
each
gesture
in
the
random
set
GES Step 3: Test (conduct experiment)
• The experimenter, on the gesture sheet
• Records the thinking time = time since the task started (i.e., when
the referent was presented to the participant) and the moment
when the participant knows what gesture to propose
• Fills in the gesture sheet: for each gesture collected
• Writes the referent ID
• Records the thinking time
• Write all scores: Goodness-of-fit, Complexity, Memorability, Fun
• Videotapes the whole session for further analysis
• Keep video files for the final report
• Takes some pictures of the room for some interesting gestures
• Records the gestures into files with the very right device
• Example: X, Y, Z, timestamp
GES Step 4: Post-test (after experiment)
• The participant
• Fills in the IBM PSSUQ questionnaire
• Is being asked a few open questions
• What did you like the most in the experiment?
• What did you hate the most in the experiment?
• The experimenter
• Checks that the questionnaire is properly filled in
• Asks questions to the participant if any
• Encourages the participant to answer the open questions
• All
• Encode all data into spreadsheet
• Template file available
GES Step 5: Gesture classification
• Give each individual collected gesture a consistent,
structured name
• Adopt a structure (action verb)+(limb)+(parameter)
• Examples:
• Swipe right with two fingers, swipe left with the dominant hand
• References
• Hand gestures (e.g., LeapMotion):
http://gestureml.org/doku.php/gestures/motion/gesture_index
• Arm gestures (see in body motion gestures section above)
• arm_raised_forward (left/right), arm_raised_side (left/right),
• arm_point (left/right)
• arm_wave (left/right), arm_push (left/right), arm_throw (left/right)
• arm_punch (left/right), arm_folded (left/right), arm_on_hip (left/right)
GES Step 5: Gesture classification
• Assign each individual gesture to a category ID
• Example:
• Swipe right with two, three fingers, swipe right with the dominant hand
are assigned to the category “1: Swipe right” however the gesture is
performed
• Create your own classification
• You can draw inspiration from existing classifications
• You may depart from existing classifications by
• Adding, deleting, modifying your own categories
• Refining, generalizing existing categories
• Detail enough: minimum 10 categories
• Do not overdetail (e.g., by creating a category for each
gesture): maximum 20 gestures
• In the Excel file, tab “Elicited Gesture”, fill in the light grey area
• Input other data, like Thinking time, Goodness-of-fit, etc.
GES Step 5: Gesture classification
• Descriptive labeling: Gesture Cards
• Examples
• Touch the ring once, twice, or multiple times in a row. Tap a
rhythmic pattern on the ring’s surface
• Rotate the ring on the finger. Rotate the finger wearing the ring.
Rotate the hand wearing the ring
• Slide the ring along the finger. Pull out the ring. Place the ring
back on the finger. Change the ring to a different finger,
GES Step 5: Gesture classification
• Gesture type: describes the underlying meaning of a
gesture. Possible values are:
• P= Pointing gestures (= deictic gestures) indicate people, objects,
directions
• Semaphoric gestures are hand postures and movements conveying
specific meanings
• T= Static semaphorics are identified by a specific hand posture. Examples:
thumbs-up means “okay”, a flat palm facing from the actor means “stop”.
• D= Dynamic semaphorics convey information through their temporal aspects.
Example: a circular hand motion means “rotate”
• S= Semaphoric strokes represent hand flicks are single, stroke-like movements.
Example: a left flick of the hand means “dismiss this object”
• A = Pantomimic gestures demonstrate a specific task to be performed or
imitated, which mostly involves motion and particular hand postures.
• Examples: filling an imaginary glass with water, by tilting an imaginary bucket.
They often consist of multiple low-level gestures, grabbing an object, moving it,
and releasing it again
GES Step 5: Gesture classification
• Gesture type: describes the
underlying meaning of a gesture.
Possible values are (cont’d):
• Iconic gestures communicate information about objects
or entities, such as specific sizes, shapes, and motion
paths:
• I= Static iconics are performed by spontaneous
static hand postures. Example: Draw an “O” with
index finger and thumb means a “circle”
• Y= Dynamic iconics are often used to describe
paths or shapes, such as moving the hand in
circles, meaning “the circle”.
• - M= Manipulation gestures a guide movement in a
short feedback loop. Thus, they feature a tight
relationship between the movements of the actor and
the movements of the object to be manipulated. The
actor waits for the entity to “follow” before continuing
Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human-preferences-in-usage-of-gesture-types-for-hci/
GES Step 5: Gesture classification
• Gesture form: specifies which form of gesture is elicited.
Possible values are:
• S= stroke when the gesture only consists of taps and flicks
• T= static when the gesture is performed in only one location
• M= static with motion (when the gesture is performed with a
static pose while the rest is moving)
• D= dynamic when the gesture does capture any change or
motion
GES Step 5: Gesture classification
• Range of motion: relates the distance between the
position of the human body producing the gesture and
the location of the gesture. Possible values are
• C= Close intimate, I= Intimate, P= Personal, S=Social,
U= Public, R= Remote
Range of
motion
GES Step 5: Gesture classification
• Laterality: characterizes how the two hands are
employed to produce gestures, with two categories, as
done in many studies. Possible values are:
• D= dominant unimanual, N= non-dominant unimanual,
S= symmetric bimanual, A= asymmetric bimanual
Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426
D
(right handed)
N
(right handed)
S
(right handed)
A
(right handed)
GES Step 5: Gesture classification
• Classify each gesture category according to
the following criteria and enter corresponding
code
• Use other classification criteria depending on
• User
• Platform/device
• Environment
GES Step 5: Compute agreement among gestures
• Download AGaTE from
http://depts.washington.edu/acelab/proj/dollar/agate.html
• Prepare a CSV file with participant ID (column), referent
name (line), Gesture category ID (cell)
Participant ID
Referent
name
Gesture category
ID
GES Step 5: Compute agreement among gestures
• Compute Agreement Rate (AR) for all referents
• Agreement Rate = the number of pairs of participants in
agreement with each other divided by the total number
of pairs of participants that could be in agreement
• Compute co-agreement for pairs, groups (eg male vs
female), categories of referents (eg basic vs. advanced)
agreement rate disagreement rate co-agreement rate
Source: https://dl.acm.org/citation.cfm?id=2669511
GES Step 5: Compute agreement among gestures
• Example: agreement rate for one referent with 5
participants proposing 2 gestures A and B.
• Connected pairs represent how two participants
performed the same gesture
Source: https://dl.acm.org/citation.cfm?id=2669511
• Ring Gestures
• 41
24 participants
= 672 gesture proposals
x 2 rings (Ring Zero)
x 14 referents
Source: B. Gheran, J. Vanderdonckt, R.-D. Vatavu, Gestures for Smart Rings: Empirical Results, Insights, and Design Implications.
Proc. of ACM DIS’18, 623-635.
Example
• Ring Gestures: samples
Example
https://www.youtube.com/watch?v=FHT-5aFNhsA
• Ring Gestures: agreement scores
Example
• Ring Gestures: consensus set
Example
GES Step 6: Analyze agreement among gestures
• Plot all referents in decreasing order of their AR, with error bars
denoting confidence interval (95%) and gesture category
45
0.193 0.173 0.157 0.157 0.14 0.13 0.117 0.11 0.11 0.107 0.107 0.097 0.08 0.07 0.067 0.06 0.053 0.053 0.043
0.107
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Turn Light
On
Turn TV
On
Begin
Player
Turn Light
Off
Turn TV
Off
Go to
previous
item
Go to next
item
Turn Heat
On
Answer a
Call
Increase
Volume
Turn Heat
Off
Increase
Light
Decrease
Volume
Hang Up
Call
Decrease
Light
Turn AC
On
Turn
Alarm On
Turn
Alarm Off
Turn AC
Off
Average
Agreement
rate
Low agreement
Medium agreement
Most
elicited
gesture
15: Splay 27: Button 27: Button
3: Point 14: Fist 10: Swipe 10: Swipe 8: Rotate 30: Phone 30: Phone
8: Rotate
28: Dimmer 28: Dimmer
15: Splay 6: Flat 10: Swipe 27: Button 27: Button 10: Swipe
Referent
Category
Source: https://dl.acm.org/citation.cfm?id=2702223
GES Step 7: Analyze Gestures
• Provide a detailed analysis of gestures elicited
• By criteria
• By agreement rate
• By observation (e.g., field notes, video notes, gesture sheet)
• By interview
• By IBM PSSUQ questionnaire
• Identify potential correlations between data (e.g., t test,
ANOVA, etc.)
• Decide Consensus set = set of agreed gestures based
on results
• Example: https://ieeexplore.ieee.org/document/8890919
Example of Gesture Elicitation Study
Head and Shoulders Gestures:
Exploring User-Defined Gestures with Upper Body
Source: Jean Vanderdonckt, Nathan Magrofuoco, Suzanne Kieffer, Jorge Pérez, Ysabelle Rase, Paolo Roselli, Santiago Villarreal:
Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body. HCI (19) 2019: 192-213
48
What are head and shoulders gestures?
• A head gesture is any movement of the head leaving the rest
of the body unaffected (stationary)
• A head gesture could occur in any plane (sagittal, transverse,
frontal)
• A shoulder gesture is any movement of the shoulder joint that
leaves the rest of the arm unaffected (stationary).
• A shoulder gesture occurs in any plane of motion (sagittal,
transverse, frontal) or direction (forward, backward, or circular)
49
Y
Yaw
Z
Roll
X
Pitch
50
Why are head and shoulders gestures interesting?
• Hands-free interaction is made possible
• Fixed-Gaze Head Movement is appropriate when
• No device needed
• Both hands should be free
• No need to move the gaze
• Capability to trigger actions from a moderate set
• Medium command duration
• Accurate recognizers for head and shoulders gestures start to
appear
M13 M31
M65
M56
M42
M24
[Qinjie et al., 2019]
51
Why are head and shoulders gestures interesting?
• They are used in physical exercises
52
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Head Label Alias Movement
(frontal,
transversal,
sagittal)
X trans-
lation
Move the head
left, right
Face left, face
right
Lateral translation
(v,c,c)
Y trans-
lation
Move the head
up, down
Face up, face
down
Neck elevation,
depression (c,v,c)
Z trans-
lation
Move the head
forward,
backward
Thrust,
retreat
Protraction,
retraction (c,c,v)
53
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Head Label Alias Movement
(frontal,
transversal,
sagittal)
Frontal
tilting
Tilt the head to
the left, right
Bend left,
right
Lateral flexion
(v,v,c)
Transvers
al tilting
Tilt the head up,
down
Bend up,
down
Extension, flexion
(v,c,v)
Saggital
tilting
Tilt the head
forward,
backward
Bend
forward,
backward
Extension, flexion
(c,v,v)
54
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Head Label Alias Movement
(frontal,
transversal,
sagittal)
X rotation Turn the head
up, down
Uphead,
downhead
Horizontal
rotation (c,v,v)
Y rotation Turn the head
left, right
Lefthead,
righthead
Vertical rotation
(v,c,v)
Z rotation Turn the head
forward,
backward
Forehead,
backhead
Facial rotation
(v,v,c)
55
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Shoulders Label Alias Movement (frontal, transversal,
sagittal)
X translation Move shoulder horizontally to left, right Decontract,
contract
Extension, flexion (v,c,c)
Y translation Raise shoulder, lower shoulder Raise, lower Shoulder elevation, depression
(c,v,c)
Z translation Move shoulder forward/backward Protract, retract Shoulder protraction, retraction
(c,c,v)
• There are also common head and shoulders gestures
• Larger design space than www.gestureml.org
Source: http://gestureml.org/doku.php/gestures/motion/gesture_index
“These head-motion based gestures can be great for adding subtle context cues to
game controls and metrics or even used to directly modify the way digital content is
presented on a display.”
58
Experiment: Gesture Elicitation Study (GES)
• Participants
• 10 females + 12 males = 22 participants
• Aged from 18 to 62y (M=29, SD=13)
• Various occupations: secretary, teacher, employee,...
• Device usage frequencies
• Creativity score
y = 0.1467x + 52.482
R² = 0.0775
0
10
20
30
40
50
60
70
80
18 28 38 48 58
Creativity
[score]
Age [years]
6.09 6.05
2.68
1.73
1.05
0.00
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Computer
Smartphone
Tablet
Game
console
MS
Kinect
Studied
device
Average
fequency
of
usage
Device
59
Experiment: Gesture Elicitation Study (GES)
• Stimuli: 14 referents for IoT tasks:
Turn the TV On/Off, Start Player, Turn the Volume up, Turn the volume
down, Go to the next channel, Go to the previous channel, Turn Air
Conditioning On/Off, Turn Lights On/Off, Brighten Lights, Dim Lights,
Turn Heating system On/Off, Turn Alarm On/Off, Answer a phone call,
and End Phone Call.
Example: 3. INCREASE: Brighten lights
Before After
60
Experiment: Results
• 22 participants X 14 referents = 308 elicited gestures
resulting into 10 categories
Head single gesture, 102
Concurrent compound
gesture, 70
Sequential
compound
gesture, 44
Both shoulders single
gesture, 29
Dominant shoulder single
gesture, 19
Non-dominant shoulder
single gesture, 14
Head repeated gesture, 10
Both shoulders
repeated
gestures, 9
Dominant shoulder
repeated gesture, 4
Non-dominant shoulder
repeated gesture, 3
Other, 26
61
Experiment: Results
• Aggregated measures per referent
62
Experiment: Results
• Evolution of aggregated measures per referent
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
0.00
5.00
10.00
15.00
20.00
25.00
Value
Time
[Sec]
Referent
Average thinking time Goodness-of-fit
Linear (Average thinking time) Linear (Goodness-of-fit)
63
Experiment: Results
• Breakdown per criteria
Upface/downface, 3%
Thrust, 1% Bend, 31%
Nod,
4%
Rotate,
4%
Left/right, 12%
Backhead, 0%
Raise
11%
Lower,
4%
Shrug, 11% Clog, 9%
Protract
6%
Retract,
3%
Head (%), 50.87 Shoulders (%), 30.62
Head and Shoulders (%),
18.51
One stroke
68.09%
Two strokes
22.37%
Three or more strokes
9.54%
Body part
Elicited gestures
Amount of strokes
0.390 0.390
0.316
0.283
0.267 0.260
0.263
0.250 0.250 0.250 0.248
0.229
0.215
0.188
0.138
0.368 0.368
0.286
0.251
0.238 0.234
0.232
0.221 0.221 0.216 0.215
0.195
0.182
0.156
0.104
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Go to Next
Channel
Go to
Previous
Channel
Answer
Phone Call
End Phone
Call
Play/pause Turn TV
On/Off
Average Turn Lights
On/Off
Turn Alarm
On/Off
Decrease
Volume
Brighten
Lights
Increase
Volume
Dim Lights Turn Air
Conditioning
On/Off
Turn Heating
System
On/Off
High Medium
64
Experiment: 14 Consensus gestures
Agreement score [Vatavu & Wobbrock, 2015]
Agreement rate [Vatavu & Wobbrock, 2016]
65
Conclusion
• Contributions
• Design space for head and shoulders gestures
• Corpus of 308 elicited gestures with measures
• Classification into 10 categories
• Consensus set of 14 head and shoulders gestures
• Design guidelines
• Use bending gestures as a first-class citizen
• Use Upface/downface for infrequent tasks
• Use thrust only for play/pause
• Forehead and backhead gestures should not be used,
apart for exceptional assignation
Gesture Elicitation Study
Shortcomings and variants
• Shortcomings
• Legacy bias (Morris et al, 2010)
Source: M. Morris et al., Reducing Legacy Bias in Gesture Elicitation Studies, 2010
https://interactions.acm.org/archive/view/may-june-2014/reducing-legacy-bias-in-gesture-elicitation-studies
Priming. Priming users to think about the
capabilities of a new form factor or sensing
technology is another approach that may
reduce the impact of legacy bias
Partners. Inviting users to participate in
elicitation studies in groups, rather than
individually, can be another approach to
overcoming legacy bias
• Shortcomings
• Manual (variable) classification of gestures
Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI
2019: 224
“It is possible to design a highly guessable symbol set by acquiring guesses from participants.”
“Participants are first recruited to propose symbols for specified referents within a given domain.
The more participants, the more likely the resulting symbol set will be guessable to external users.”
The goal is to obtain a rich set of symbols from which to create the resultant symbol set.”
• Shortcomings
• Manual (variable) classification of gestures
N=30 children, Referent: “Scratch like a cat!”
 Grouping criteria #1: same hand  39% agreement
 Grouping criteria #2: same hand & body pose  19.6%
 Grouping criteria #3: same hand & body pose & pattern of
movement  12.2%
Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI
2019: 224
• Shortcomings
• Automatic classification of gestures
1. A dissimilarity function for gestures Δ(𝑔𝑖,𝑔𝑗)
 for example, the Euclidean distance
 the Dynamic Time Warping cost function
 or any distance function that takes two gestures as
input and returns a real, positive value of how
dissimilar they are
2. A threshold (τ, tau) for values computed by Δ
Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI
2019: 224
• Shortcomings
• Automatic classification of gestures
Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI
2019: 224
• Shortcomings and variants
• Open vs Closed elicitation
Open
Closed
• Shortcomings and variants
• Gesture elicitation distributed in time and space
GestMAN
Source: GestMan: a cloud-based tool for stroke-gesture datasets
• Shortcomings and variants
• Referent-free elicitation
Open
• Shortcomings and variants
• Some limbs are privileged
Source: Villarreal et al., A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies?
Body parts:
1.- The
Individual
Frequency of
Body Parts
(IFBP)
2. The
Combination
Frequency of
Body Parts
(CFBP)
• Shortcomings and variants
• Eliciting other symbols than gestures, other modalities
Source: Villarreal et al., A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies?
• Shortcomings and variants
• Eliciting more than one symbol
Source: Villarreal et al., A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies?
Multiple symbols could
be combined to trigger a
function by relying on
hierarchical structure and
congruence
• Conclusion
• Elicitation study is a practical tool for eliciting (symbol)
proposals from participants for commands, icons,
shortcuts, gestures, vocal commands, etc.
• Efficient, natural
• Subject to manual classification
• Not always sure for agreement rates (other scores)
• Legacy bias
• Continuity from elicitation to recognition
• Subject to Context variability (U,P,E)
• Conclusion
• Many GES already exist, but no consolidation of this
knowledge!
y = 3.8909x - 4.1636
R² = 0.9052
0
5
10
15
20
25
30
35
40
45
2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
Number
of
studies
Year of publication
Frequency of Gesture ElicitationStudies
How to get the best gestures from people?
Considered insofar:
• Human factors: preference, sketching time, delete
operations, agreement, (dis)similarity
• System factors: recognition rate
More to come and to consider:
• Human factors: hedonic value, memorability,
naturalness, discoverability, consistency, congruence, …
• System factors: recognition rate, execution time,
computational complexity, more measures…

More Related Content

Similar to Conducting a Gesture Elicitation Study: How to Get the Best Gestures From People?

Advanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise ARAdvanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise ARMark Billinghurst
 
The 5-step Approach to Controlled Experiment Design for Human Computer Intera...
The 5-step Approach to Controlled Experiment Design for Human Computer Intera...The 5-step Approach to Controlled Experiment Design for Human Computer Intera...
The 5-step Approach to Controlled Experiment Design for Human Computer Intera...National University of Singapore
 
Wellness at hand: Exploring interactive technology to support smokers
Wellness at hand: Exploring interactive technology to support smokersWellness at hand: Exploring interactive technology to support smokers
Wellness at hand: Exploring interactive technology to support smokersUniversity of Melbourne, Australia
 
Technology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_applicationTechnology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_applicationBarrow County Schools
 
U Penn Wharton design challenge '17
U Penn Wharton design challenge '17U Penn Wharton design challenge '17
U Penn Wharton design challenge '17HJ Kwon
 
IOC 2004 - Learning Styles and Student Performance in an E-Learning Environment
IOC 2004 - Learning Styles and Student Performance in an E-Learning EnvironmentIOC 2004 - Learning Styles and Student Performance in an E-Learning Environment
IOC 2004 - Learning Styles and Student Performance in an E-Learning EnvironmentMichael Barbour
 
Analytic emperical Mehods
Analytic emperical MehodsAnalytic emperical Mehods
Analytic emperical MehodsM Surendar
 
Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...
Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...
Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...Facultad de Informática UCM
 
Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...
Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...
Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...MIT
 
Usability testing / Nearly everything you need to know to get started
Usability testing / Nearly everything you need to know to get startedUsability testing / Nearly everything you need to know to get started
Usability testing / Nearly everything you need to know to get startedRebecca Destello
 
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...Barry Ryan
 
Session 3 Research Methods - Data Analysis - Case Study Example
Session 3 Research Methods - Data Analysis - Case Study ExampleSession 3 Research Methods - Data Analysis - Case Study Example
Session 3 Research Methods - Data Analysis - Case Study Examplemilolostinspace
 
Learning Analytics Dashboards
Learning Analytics DashboardsLearning Analytics Dashboards
Learning Analytics DashboardsSten Govaerts
 
Design, Create, Evaluate Process (1).pptx
Design, Create, Evaluate Process (1).pptxDesign, Create, Evaluate Process (1).pptx
Design, Create, Evaluate Process (1).pptxLe Hung
 
PH-HISTO-PPT.FINAL DEF.pptx
PH-HISTO-PPT.FINAL DEF.pptxPH-HISTO-PPT.FINAL DEF.pptx
PH-HISTO-PPT.FINAL DEF.pptxJcModelo
 
Ud the motion en_jtaboada
Ud the motion en_jtaboadaUd the motion en_jtaboada
Ud the motion en_jtaboadaJordi Taboada
 
Design and Application of Experiments and User Studies
Design and Application of Experiments and User StudiesDesign and Application of Experiments and User Studies
Design and Application of Experiments and User StudiesVictor Adriel Oliveira
 

Similar to Conducting a Gesture Elicitation Study: How to Get the Best Gestures From People? (20)

Tests
TestsTests
Tests
 
Lean Ethnography
Lean EthnographyLean Ethnography
Lean Ethnography
 
Advanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise ARAdvanced Methods for User Evaluation in Enterprise AR
Advanced Methods for User Evaluation in Enterprise AR
 
The 5-step Approach to Controlled Experiment Design for Human Computer Intera...
The 5-step Approach to Controlled Experiment Design for Human Computer Intera...The 5-step Approach to Controlled Experiment Design for Human Computer Intera...
The 5-step Approach to Controlled Experiment Design for Human Computer Intera...
 
Wellness at hand: Exploring interactive technology to support smokers
Wellness at hand: Exploring interactive technology to support smokersWellness at hand: Exploring interactive technology to support smokers
Wellness at hand: Exploring interactive technology to support smokers
 
Technology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_applicationTechnology intergrationplan kellieouzts_application
Technology intergrationplan kellieouzts_application
 
U Penn Wharton design challenge '17
U Penn Wharton design challenge '17U Penn Wharton design challenge '17
U Penn Wharton design challenge '17
 
Lecture-2.pdf
Lecture-2.pdfLecture-2.pdf
Lecture-2.pdf
 
IOC 2004 - Learning Styles and Student Performance in an E-Learning Environment
IOC 2004 - Learning Styles and Student Performance in an E-Learning EnvironmentIOC 2004 - Learning Styles and Student Performance in an E-Learning Environment
IOC 2004 - Learning Styles and Student Performance in an E-Learning Environment
 
Analytic emperical Mehods
Analytic emperical MehodsAnalytic emperical Mehods
Analytic emperical Mehods
 
Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...
Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...
Aplicando Analítica de Aprendizaje para la Evaluación de Competencias y Compo...
 
Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...
Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...
Learning Analytics for the Evaluation of Competencies and Behaviors in Seriou...
 
Usability testing / Nearly everything you need to know to get started
Usability testing / Nearly everything you need to know to get startedUsability testing / Nearly everything you need to know to get started
Usability testing / Nearly everything you need to know to get started
 
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
Barry Ryan_Alternative Laboratories: Near Peer Constructed Technology Enhanc...
 
Session 3 Research Methods - Data Analysis - Case Study Example
Session 3 Research Methods - Data Analysis - Case Study ExampleSession 3 Research Methods - Data Analysis - Case Study Example
Session 3 Research Methods - Data Analysis - Case Study Example
 
Learning Analytics Dashboards
Learning Analytics DashboardsLearning Analytics Dashboards
Learning Analytics Dashboards
 
Design, Create, Evaluate Process (1).pptx
Design, Create, Evaluate Process (1).pptxDesign, Create, Evaluate Process (1).pptx
Design, Create, Evaluate Process (1).pptx
 
PH-HISTO-PPT.FINAL DEF.pptx
PH-HISTO-PPT.FINAL DEF.pptxPH-HISTO-PPT.FINAL DEF.pptx
PH-HISTO-PPT.FINAL DEF.pptx
 
Ud the motion en_jtaboada
Ud the motion en_jtaboadaUd the motion en_jtaboada
Ud the motion en_jtaboada
 
Design and Application of Experiments and User Studies
Design and Application of Experiments and User StudiesDesign and Application of Experiments and User Studies
Design and Application of Experiments and User Studies
 

More from Jean Vanderdonckt

To the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User InterfacesTo the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User InterfacesJean Vanderdonckt
 
Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Jean Vanderdonckt
 
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...Jean Vanderdonckt
 
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...Jean Vanderdonckt
 
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...Jean Vanderdonckt
 
Gesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOpsGesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOpsJean Vanderdonckt
 
Engineering Slidable User Interfaces with Slime
Engineering Slidable User Interfaces with SlimeEngineering Slidable User Interfaces with Slime
Engineering Slidable User Interfaces with SlimeJean Vanderdonckt
 
Evaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and MeasuresEvaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and MeasuresJean Vanderdonckt
 
User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...Jean Vanderdonckt
 
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...Jean Vanderdonckt
 
Attach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you WorkAttach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you WorkJean Vanderdonckt
 
The Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV GesturesThe Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV GesturesJean Vanderdonckt
 
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper BodyHead and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper BodyJean Vanderdonckt
 
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for SmartphonesG-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for SmartphonesJean Vanderdonckt
 
Vector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture RecognitionVector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture RecognitionJean Vanderdonckt
 
An ontology for reasoning on body-based gestures
 An ontology for reasoning on body-based gestures An ontology for reasoning on body-based gestures
An ontology for reasoning on body-based gesturesJean Vanderdonckt
 
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design AlternativesAB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design AlternativesJean Vanderdonckt
 
Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
 Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
Gelicit: A Cloud Platform for Distributed Gesture Elicitation StudiesJean Vanderdonckt
 
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...Jean Vanderdonckt
 
Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...Jean Vanderdonckt
 

More from Jean Vanderdonckt (20)

To the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User InterfacesTo the end of our possibilities with Adaptive User Interfaces
To the end of our possibilities with Adaptive User Interfaces
 
Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...Engineering the Transition of Interactive Collaborative Software from Cloud C...
Engineering the Transition of Interactive Collaborative Software from Cloud C...
 
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
UsyBus: A Communication Framework among Reusable Agents integrating Eye-Track...
 
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Mult...
 
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowle...
 
Gesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOpsGesture-based information systems: from DesignOps to DevOps
Gesture-based information systems: from DesignOps to DevOps
 
Engineering Slidable User Interfaces with Slime
Engineering Slidable User Interfaces with SlimeEngineering Slidable User Interfaces with Slime
Engineering Slidable User Interfaces with Slime
 
Evaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and MeasuresEvaluating Gestural Interaction: Models, Methods, and Measures
Evaluating Gestural Interaction: Models, Methods, and Measures
 
User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...User-centred Development of a Clinical Decision-support System for Breast Can...
User-centred Development of a Clinical Decision-support System for Breast Can...
 
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...Simplifying the Development of  Cross-Platform Web User Interfaces by Collabo...
Simplifying the Development of Cross-Platform Web User Interfaces by Collabo...
 
Attach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you WorkAttach Me, Detach Me, Assemble Me like you Work
Attach Me, Detach Me, Assemble Me like you Work
 
The Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV GesturesThe Impact of Comfortable Viewing Positions on Smart TV Gestures
The Impact of Comfortable Viewing Positions on Smart TV Gestures
 
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper BodyHead and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body
Head and Shoulders Gestures: Exploring User-De fined Gestures with Upper Body
 
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for SmartphonesG-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
G-Menu: A Keyword-by-Gesture based Dynamic Menu Interface for Smartphones
 
Vector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture RecognitionVector-based, Structure Preserving Stroke Gesture Recognition
Vector-based, Structure Preserving Stroke Gesture Recognition
 
An ontology for reasoning on body-based gestures
 An ontology for reasoning on body-based gestures An ontology for reasoning on body-based gestures
An ontology for reasoning on body-based gestures
 
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design AlternativesAB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
 
Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
 Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
Gelicit: A Cloud Platform for Distributed Gesture Elicitation Studies
 
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
MoCaDiX: Designing Cross-Device User Interfaces of an Information System base...
 
Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...Specification of a UX process reference model towards the strategic planning ...
Specification of a UX process reference model towards the strategic planning ...
 

Recently uploaded

IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024Stephen Perrenod
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandIES VE
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...CzechDreamin
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftshyamraj55
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...marcuskenyatta275
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Hiroshi SHIBATA
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024Stephanie Beckett
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfFIDO Alliance
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfFIDO Alliance
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...FIDO Alliance
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfSrushith Repakula
 

Recently uploaded (20)

IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 

Conducting a Gesture Elicitation Study: How to Get the Best Gestures From People?

  • 1. Lecture 3: Conducting a Gesture Elicitation Study: How to Get the Best Gestures From People? Jean Vanderdonckt, UCLouvain Vrije Universiteit Brussel, 20 April 2021, on-line
  • 2. How to get the best gestures from people? Case study: sketching a graphical UI (Interact’07) https://www.youtube.com/watch?v=SBNB1O-8pGw&t=5s
  • 3. How to determine the best gestures for widgets?
  • 4. How to determine the best gestures for widgets? 1: none
  • 5. How to determine the best gestures for widgets? 1: none 2: low
  • 6. How to determine the best gestures for widgets? 1: none 2: low 3: medium
  • 7. How to determine the best gestures for widgets? 1: none 2: low 3: medium 4: high
  • 8. Use de Borda Count method (Condorcet method) • Each participant assigns a score equal to the number of representations to whom he or she is preferred in decreasing order: n-1, n-2, n-3,… n=5 Participant #1 0 n-3=2 n-4=1 n-1=4 n-2=3 Participant #2 n-1=4 0 n-3=2 n-2=3 n-4=1 Score= 4 Score= 7 Score= 3 Score= 2 Score= 4
  • 9. Use de Borda Count method (Condorcet method) • Each participant assigns a score equal to the number of representations to whom he or she is preferred in decreasing order: n-1, n-2, n-3,…
  • 10. Further classification: complexity of the representation ST=sketching time DEL=delete operations
  • 11. Further classification: complexity of the representation ST=sketching time DEL=delete operations Source: Suzanne Kieffer, Adrien Coyette, Jean Vanderdonckt, User interface design by sketching: a complexity analysis of widget representations. EICS 2010: 57-66
  • 12. How to get the best gestures from people? Considered insofar: • Human factors: preference, sketching time, delete operations • System factors: recognition rate Other method for preference • By elicitation • By wizard of Oz
  • 13. Gesture Elicitation Study (GES) • To elicit • To call forth or draw out something, such as information or a response (Merriam Webster) • To evoke or draw out a reaction, answer, or fact from someone (Dictionary.com) • To draw forth something that is latent or potential into existence (Dictionary.com) • Elicitation • Acquisition of information from a person or a group of persons in a manner that does not disclose the intent of the interview, discussion or conversation • Elicitation technique • A technique used to discreetly extract information, not readily available for a specific purpose Source: Wobbrock et al., 2009
  • 14. Gesture Elicitation Study (GES) • Definition • Is a study for asking end-users to elicit their own gestures for a set of predefined functions through referents and reach a consensus • Aims and goals • To let end users elicit the gestures they want for a function • To compute agreement between end users (=participants, subjects) • To define a vocabulary of potential gestures • To identify a consensus set, which is a sub-vocabulary of agreed gestures. Output = Consensus Set • For one context of use at a time: C= (U,P,E) • User : any person, Functions: IoT functions • Platform/device/sensor: an armband • Environment: smart home, usability lab, controlled room
  • 15. Gesture Elicitation Study (GES) • Advantages • Free from any form of stress or burden of working • Information comes deliberately once the favorable context is created • No adverse effect • Easy and cheap to run • Disadvantages • Legacy bias • Previous experience with a similar system, another device • No experience • Lack of creativity
  • 16. GES Step 1: Prepare setup C=(U,P,E) • Create a representative sampling of the user population
  • 17. GES Step 1: Prepare setup C=(U,P,E) • Find a representative sampling of the user population • Example of stratified sampling Sampling Female Male Age group % 32 51 49 18-24 8.9 3 2 1 25-34 15.4 5 3 2 35-44 16.5 5 3 2 45-54 18.3 6 3 3 55-64 18.7 6 3 3 65+ 22.2 7 4 3 Total 100 32 18 14
  • 18. GES Step 1: Prepare setup C=(U,P,E) • Find a representative sampling of the user population • Find n=30 participants • Ideally 50% female, 50% male • Different ages (not all students!) • Different backgrounds (not all in your discipline!) • Different levels of experience (e.g., with touch or not) • Each participant takes part in the experiment individually • Define an experiment schedule • Location (should be constant): place to choose • Time: 30 min for each participant + resting time • Schedule: organize a Doodle • Test the whole protocol with a guinea pig • Different from the participants!
  • 19. GES Step 1: Prepare setup C=(U,P,E) • Identify referents for the following tasks • A referent is the expected result of an action (e.g., key pressing, touch, gesture) for a function • Distribution the definition of the 14 referents across groups • Example: IoT functions: (1) Turn the TV on/off (2) Start player (3) Turn up the volume (4) Turn down the volume (5) Go to the next item in a list (6) Go to the previous item (7) Turn AC on/off (8) Turn lights on/off (9) Brighten lights (10) Dim lights (11)Turn heat on/off (12) Turn alarm on/off (13)Answer phone call (14) End phone call • Randomize the order of the referents before the study for each participant (create lists with www.random.org) Is symmetric to
  • 20. GES Step 1: Prepare setup C=(U,P,E) • What is a referent? Referent: - State before action - Action description - State after action
  • 21. GES Step 1: Prepare setup C=(U,P,E) • Referent: textual, graphical, animation, video (visual priming) 20. Permute files remotely Before After My Network My Disk My Network My Disk Our Network Partner Disk Our Network Partner Disk Our Network Partner Disk Our Network Partner Disk Yellow Document Red Document Red Document Yellow Document Blue Document Blue Document
  • 22. GES Step 1: Prepare setup C=(U,P,E) • Referent: textual, graphical, animation, video (visual priming) https://www.youtube.com/watch?v=iVzE4EzfGoc&t=12s
  • 23. GES Step 1: Prepare setup C=(U,P,E) • Prepare the hardware, software • Keep them constant throughout the experiment • Determine a software for recording gestures in Log Files • Example: Python software for Leap Motion • Example: MyO Gesture Control
  • 24. GES Step 1: Prepare setup C=(U,P,E) • Identify the physical location: office, room, usability lab,… • Keep it constant throughout the experiment • Sometimes determined by the device • Example: Usability lab
  • 25. GES Step 2: Pre-test (before experiment) • Each participant • Comes to the experiment location according to schedule • Signs a consent form • Fills in a demographic questionnaire • Notoriety and experience if any for the target device • System experience in general (e.g., for a tablet, frequency) • Performs a Motor-Skill test: to touch their thumb to each of the other fingers of the same hand 2 times consecutively without flaw • Is presented with an introduction to the device (e.g., a video) • Experimenter • Assigns an ID to each participant on the consent form • Checks that all forms are completely filled in, before and after
  • 26. GES Step 3: Test (conduct experiment) • Each participant • Is presented with the list of referents (randomly selected) • Confirms she understood the tasks • Is asked to think of a suitable gesture • Says “I am ready” • Gives a rating (Goodness-of-fit) between 1 and 10 • To describe how well the proposed gesture fits the referent • 1 = very poor fit  10 = excellent fit • Optionally gives a value between 1 and 7 for • Complexity: 1= most complex  7= simplest gesture • Memorability: 1=hardest to remember  7=easiest to remember • Fun: 1=is the least funny  7=is the funniest for each gesture in the random set
  • 27. GES Step 3: Test (conduct experiment) • The experimenter, on the gesture sheet • Records the thinking time = time since the task started (i.e., when the referent was presented to the participant) and the moment when the participant knows what gesture to propose • Fills in the gesture sheet: for each gesture collected • Writes the referent ID • Records the thinking time • Write all scores: Goodness-of-fit, Complexity, Memorability, Fun • Videotapes the whole session for further analysis • Keep video files for the final report • Takes some pictures of the room for some interesting gestures • Records the gestures into files with the very right device • Example: X, Y, Z, timestamp
  • 28. GES Step 4: Post-test (after experiment) • The participant • Fills in the IBM PSSUQ questionnaire • Is being asked a few open questions • What did you like the most in the experiment? • What did you hate the most in the experiment? • The experimenter • Checks that the questionnaire is properly filled in • Asks questions to the participant if any • Encourages the participant to answer the open questions • All • Encode all data into spreadsheet • Template file available
  • 29. GES Step 5: Gesture classification • Give each individual collected gesture a consistent, structured name • Adopt a structure (action verb)+(limb)+(parameter) • Examples: • Swipe right with two fingers, swipe left with the dominant hand • References • Hand gestures (e.g., LeapMotion): http://gestureml.org/doku.php/gestures/motion/gesture_index • Arm gestures (see in body motion gestures section above) • arm_raised_forward (left/right), arm_raised_side (left/right), • arm_point (left/right) • arm_wave (left/right), arm_push (left/right), arm_throw (left/right) • arm_punch (left/right), arm_folded (left/right), arm_on_hip (left/right)
  • 30. GES Step 5: Gesture classification • Assign each individual gesture to a category ID • Example: • Swipe right with two, three fingers, swipe right with the dominant hand are assigned to the category “1: Swipe right” however the gesture is performed • Create your own classification • You can draw inspiration from existing classifications • You may depart from existing classifications by • Adding, deleting, modifying your own categories • Refining, generalizing existing categories • Detail enough: minimum 10 categories • Do not overdetail (e.g., by creating a category for each gesture): maximum 20 gestures • In the Excel file, tab “Elicited Gesture”, fill in the light grey area • Input other data, like Thinking time, Goodness-of-fit, etc.
  • 31. GES Step 5: Gesture classification • Descriptive labeling: Gesture Cards • Examples • Touch the ring once, twice, or multiple times in a row. Tap a rhythmic pattern on the ring’s surface • Rotate the ring on the finger. Rotate the finger wearing the ring. Rotate the hand wearing the ring • Slide the ring along the finger. Pull out the ring. Place the ring back on the finger. Change the ring to a different finger,
  • 32. GES Step 5: Gesture classification • Gesture type: describes the underlying meaning of a gesture. Possible values are: • P= Pointing gestures (= deictic gestures) indicate people, objects, directions • Semaphoric gestures are hand postures and movements conveying specific meanings • T= Static semaphorics are identified by a specific hand posture. Examples: thumbs-up means “okay”, a flat palm facing from the actor means “stop”. • D= Dynamic semaphorics convey information through their temporal aspects. Example: a circular hand motion means “rotate” • S= Semaphoric strokes represent hand flicks are single, stroke-like movements. Example: a left flick of the hand means “dismiss this object” • A = Pantomimic gestures demonstrate a specific task to be performed or imitated, which mostly involves motion and particular hand postures. • Examples: filling an imaginary glass with water, by tilting an imaginary bucket. They often consist of multiple low-level gestures, grabbing an object, moving it, and releasing it again
  • 33. GES Step 5: Gesture classification • Gesture type: describes the underlying meaning of a gesture. Possible values are (cont’d): • Iconic gestures communicate information about objects or entities, such as specific sizes, shapes, and motion paths: • I= Static iconics are performed by spontaneous static hand postures. Example: Draw an “O” with index finger and thumb means a “circle” • Y= Dynamic iconics are often used to describe paths or shapes, such as moving the hand in circles, meaning “the circle”. • - M= Manipulation gestures a guide movement in a short feedback loop. Thus, they feature a tight relationship between the movements of the actor and the movements of the object to be manipulated. The actor waits for the entity to “follow” before continuing Source: https://www.microsoft.com/en-us/research/publication/understanding-mid-air-hand-gestures-a-study-of-human-preferences-in-usage-of-gesture-types-for-hci/
  • 34. GES Step 5: Gesture classification • Gesture form: specifies which form of gesture is elicited. Possible values are: • S= stroke when the gesture only consists of taps and flicks • T= static when the gesture is performed in only one location • M= static with motion (when the gesture is performed with a static pose while the rest is moving) • D= dynamic when the gesture does capture any change or motion
  • 35. GES Step 5: Gesture classification • Range of motion: relates the distance between the position of the human body producing the gesture and the location of the gesture. Possible values are • C= Close intimate, I= Intimate, P= Personal, S=Social, U= Public, R= Remote Range of motion
  • 36. GES Step 5: Gesture classification • Laterality: characterizes how the two hands are employed to produce gestures, with two categories, as done in many studies. Possible values are: • D= dominant unimanual, N= non-dominant unimanual, S= symmetric bimanual, A= asymmetric bimanual Source: https://www.tandfonline.com/doi/abs/10.1080/00222895.1987.10735426 D (right handed) N (right handed) S (right handed) A (right handed)
  • 37. GES Step 5: Gesture classification • Classify each gesture category according to the following criteria and enter corresponding code • Use other classification criteria depending on • User • Platform/device • Environment
  • 38. GES Step 5: Compute agreement among gestures • Download AGaTE from http://depts.washington.edu/acelab/proj/dollar/agate.html • Prepare a CSV file with participant ID (column), referent name (line), Gesture category ID (cell) Participant ID Referent name Gesture category ID
  • 39. GES Step 5: Compute agreement among gestures • Compute Agreement Rate (AR) for all referents • Agreement Rate = the number of pairs of participants in agreement with each other divided by the total number of pairs of participants that could be in agreement • Compute co-agreement for pairs, groups (eg male vs female), categories of referents (eg basic vs. advanced) agreement rate disagreement rate co-agreement rate Source: https://dl.acm.org/citation.cfm?id=2669511
  • 40. GES Step 5: Compute agreement among gestures • Example: agreement rate for one referent with 5 participants proposing 2 gestures A and B. • Connected pairs represent how two participants performed the same gesture Source: https://dl.acm.org/citation.cfm?id=2669511
  • 41. • Ring Gestures • 41 24 participants = 672 gesture proposals x 2 rings (Ring Zero) x 14 referents Source: B. Gheran, J. Vanderdonckt, R.-D. Vatavu, Gestures for Smart Rings: Empirical Results, Insights, and Design Implications. Proc. of ACM DIS’18, 623-635. Example
  • 42. • Ring Gestures: samples Example https://www.youtube.com/watch?v=FHT-5aFNhsA
  • 43. • Ring Gestures: agreement scores Example
  • 44. • Ring Gestures: consensus set Example
  • 45. GES Step 6: Analyze agreement among gestures • Plot all referents in decreasing order of their AR, with error bars denoting confidence interval (95%) and gesture category 45 0.193 0.173 0.157 0.157 0.14 0.13 0.117 0.11 0.11 0.107 0.107 0.097 0.08 0.07 0.067 0.06 0.053 0.053 0.043 0.107 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 Turn Light On Turn TV On Begin Player Turn Light Off Turn TV Off Go to previous item Go to next item Turn Heat On Answer a Call Increase Volume Turn Heat Off Increase Light Decrease Volume Hang Up Call Decrease Light Turn AC On Turn Alarm On Turn Alarm Off Turn AC Off Average Agreement rate Low agreement Medium agreement Most elicited gesture 15: Splay 27: Button 27: Button 3: Point 14: Fist 10: Swipe 10: Swipe 8: Rotate 30: Phone 30: Phone 8: Rotate 28: Dimmer 28: Dimmer 15: Splay 6: Flat 10: Swipe 27: Button 27: Button 10: Swipe Referent Category Source: https://dl.acm.org/citation.cfm?id=2702223
  • 46. GES Step 7: Analyze Gestures • Provide a detailed analysis of gestures elicited • By criteria • By agreement rate • By observation (e.g., field notes, video notes, gesture sheet) • By interview • By IBM PSSUQ questionnaire • Identify potential correlations between data (e.g., t test, ANOVA, etc.) • Decide Consensus set = set of agreed gestures based on results • Example: https://ieeexplore.ieee.org/document/8890919
  • 47. Example of Gesture Elicitation Study Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body Source: Jean Vanderdonckt, Nathan Magrofuoco, Suzanne Kieffer, Jorge Pérez, Ysabelle Rase, Paolo Roselli, Santiago Villarreal: Head and Shoulders Gestures: Exploring User-Defined Gestures with Upper Body. HCI (19) 2019: 192-213
  • 48. 48 What are head and shoulders gestures? • A head gesture is any movement of the head leaving the rest of the body unaffected (stationary) • A head gesture could occur in any plane (sagittal, transverse, frontal) • A shoulder gesture is any movement of the shoulder joint that leaves the rest of the arm unaffected (stationary). • A shoulder gesture occurs in any plane of motion (sagittal, transverse, frontal) or direction (forward, backward, or circular)
  • 50. 50 Why are head and shoulders gestures interesting? • Hands-free interaction is made possible • Fixed-Gaze Head Movement is appropriate when • No device needed • Both hands should be free • No need to move the gaze • Capability to trigger actions from a moderate set • Medium command duration • Accurate recognizers for head and shoulders gestures start to appear M13 M31 M65 M56 M42 M24 [Qinjie et al., 2019]
  • 51. 51 Why are head and shoulders gestures interesting? • They are used in physical exercises
  • 52. 52 Why are head and shoulders gestures interesting? • They exhibit some potential for a novel vocabulary Head Label Alias Movement (frontal, transversal, sagittal) X trans- lation Move the head left, right Face left, face right Lateral translation (v,c,c) Y trans- lation Move the head up, down Face up, face down Neck elevation, depression (c,v,c) Z trans- lation Move the head forward, backward Thrust, retreat Protraction, retraction (c,c,v)
  • 53. 53 Why are head and shoulders gestures interesting? • They exhibit some potential for a novel vocabulary Head Label Alias Movement (frontal, transversal, sagittal) Frontal tilting Tilt the head to the left, right Bend left, right Lateral flexion (v,v,c) Transvers al tilting Tilt the head up, down Bend up, down Extension, flexion (v,c,v) Saggital tilting Tilt the head forward, backward Bend forward, backward Extension, flexion (c,v,v)
  • 54. 54 Why are head and shoulders gestures interesting? • They exhibit some potential for a novel vocabulary Head Label Alias Movement (frontal, transversal, sagittal) X rotation Turn the head up, down Uphead, downhead Horizontal rotation (c,v,v) Y rotation Turn the head left, right Lefthead, righthead Vertical rotation (v,c,v) Z rotation Turn the head forward, backward Forehead, backhead Facial rotation (v,v,c)
  • 55. 55 Why are head and shoulders gestures interesting? • They exhibit some potential for a novel vocabulary Shoulders Label Alias Movement (frontal, transversal, sagittal) X translation Move shoulder horizontally to left, right Decontract, contract Extension, flexion (v,c,c) Y translation Raise shoulder, lower shoulder Raise, lower Shoulder elevation, depression (c,v,c) Z translation Move shoulder forward/backward Protract, retract Shoulder protraction, retraction (c,c,v)
  • 56. • There are also common head and shoulders gestures
  • 57. • Larger design space than www.gestureml.org Source: http://gestureml.org/doku.php/gestures/motion/gesture_index “These head-motion based gestures can be great for adding subtle context cues to game controls and metrics or even used to directly modify the way digital content is presented on a display.”
  • 58. 58 Experiment: Gesture Elicitation Study (GES) • Participants • 10 females + 12 males = 22 participants • Aged from 18 to 62y (M=29, SD=13) • Various occupations: secretary, teacher, employee,... • Device usage frequencies • Creativity score y = 0.1467x + 52.482 R² = 0.0775 0 10 20 30 40 50 60 70 80 18 28 38 48 58 Creativity [score] Age [years] 6.09 6.05 2.68 1.73 1.05 0.00 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 Computer Smartphone Tablet Game console MS Kinect Studied device Average fequency of usage Device
  • 59. 59 Experiment: Gesture Elicitation Study (GES) • Stimuli: 14 referents for IoT tasks: Turn the TV On/Off, Start Player, Turn the Volume up, Turn the volume down, Go to the next channel, Go to the previous channel, Turn Air Conditioning On/Off, Turn Lights On/Off, Brighten Lights, Dim Lights, Turn Heating system On/Off, Turn Alarm On/Off, Answer a phone call, and End Phone Call. Example: 3. INCREASE: Brighten lights Before After
  • 60. 60 Experiment: Results • 22 participants X 14 referents = 308 elicited gestures resulting into 10 categories Head single gesture, 102 Concurrent compound gesture, 70 Sequential compound gesture, 44 Both shoulders single gesture, 29 Dominant shoulder single gesture, 19 Non-dominant shoulder single gesture, 14 Head repeated gesture, 10 Both shoulders repeated gestures, 9 Dominant shoulder repeated gesture, 4 Non-dominant shoulder repeated gesture, 3 Other, 26
  • 61. 61 Experiment: Results • Aggregated measures per referent
  • 62. 62 Experiment: Results • Evolution of aggregated measures per referent 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 0.00 5.00 10.00 15.00 20.00 25.00 Value Time [Sec] Referent Average thinking time Goodness-of-fit Linear (Average thinking time) Linear (Goodness-of-fit)
  • 63. 63 Experiment: Results • Breakdown per criteria Upface/downface, 3% Thrust, 1% Bend, 31% Nod, 4% Rotate, 4% Left/right, 12% Backhead, 0% Raise 11% Lower, 4% Shrug, 11% Clog, 9% Protract 6% Retract, 3% Head (%), 50.87 Shoulders (%), 30.62 Head and Shoulders (%), 18.51 One stroke 68.09% Two strokes 22.37% Three or more strokes 9.54% Body part Elicited gestures Amount of strokes
  • 64. 0.390 0.390 0.316 0.283 0.267 0.260 0.263 0.250 0.250 0.250 0.248 0.229 0.215 0.188 0.138 0.368 0.368 0.286 0.251 0.238 0.234 0.232 0.221 0.221 0.216 0.215 0.195 0.182 0.156 0.104 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 Go to Next Channel Go to Previous Channel Answer Phone Call End Phone Call Play/pause Turn TV On/Off Average Turn Lights On/Off Turn Alarm On/Off Decrease Volume Brighten Lights Increase Volume Dim Lights Turn Air Conditioning On/Off Turn Heating System On/Off High Medium 64 Experiment: 14 Consensus gestures Agreement score [Vatavu & Wobbrock, 2015] Agreement rate [Vatavu & Wobbrock, 2016]
  • 65. 65 Conclusion • Contributions • Design space for head and shoulders gestures • Corpus of 308 elicited gestures with measures • Classification into 10 categories • Consensus set of 14 head and shoulders gestures • Design guidelines • Use bending gestures as a first-class citizen • Use Upface/downface for infrequent tasks • Use thrust only for play/pause • Forehead and backhead gestures should not be used, apart for exceptional assignation
  • 67. • Shortcomings • Legacy bias (Morris et al, 2010) Source: M. Morris et al., Reducing Legacy Bias in Gesture Elicitation Studies, 2010 https://interactions.acm.org/archive/view/may-june-2014/reducing-legacy-bias-in-gesture-elicitation-studies Priming. Priming users to think about the capabilities of a new form factor or sensing technology is another approach that may reduce the impact of legacy bias Partners. Inviting users to participate in elicitation studies in groups, rather than individually, can be another approach to overcoming legacy bias
  • 68. • Shortcomings • Manual (variable) classification of gestures Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI 2019: 224 “It is possible to design a highly guessable symbol set by acquiring guesses from participants.” “Participants are first recruited to propose symbols for specified referents within a given domain. The more participants, the more likely the resulting symbol set will be guessable to external users.” The goal is to obtain a rich set of symbols from which to create the resultant symbol set.”
  • 69. • Shortcomings • Manual (variable) classification of gestures N=30 children, Referent: “Scratch like a cat!”  Grouping criteria #1: same hand  39% agreement  Grouping criteria #2: same hand & body pose  19.6%  Grouping criteria #3: same hand & body pose & pattern of movement  12.2% Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI 2019: 224
  • 70. • Shortcomings • Automatic classification of gestures 1. A dissimilarity function for gestures Δ(𝑔𝑖,𝑔𝑗)  for example, the Euclidean distance  the Dynamic Time Warping cost function  or any distance function that takes two gestures as input and returns a real, positive value of how dissimilar they are 2. A threshold (τ, tau) for values computed by Δ Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI 2019: 224
  • 71. • Shortcomings • Automatic classification of gestures Source: Radu-Daniel Vatavu, The Dissimilarity-Consensus Approach to Agreement Analysis in Gesture Elicitation Studies. CHI 2019: 224
  • 72. • Shortcomings and variants • Open vs Closed elicitation Open Closed
  • 73. • Shortcomings and variants • Gesture elicitation distributed in time and space GestMAN Source: GestMan: a cloud-based tool for stroke-gesture datasets
  • 74. • Shortcomings and variants • Referent-free elicitation Open
  • 75. • Shortcomings and variants • Some limbs are privileged Source: Villarreal et al., A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies? Body parts: 1.- The Individual Frequency of Body Parts (IFBP) 2. The Combination Frequency of Body Parts (CFBP)
  • 76. • Shortcomings and variants • Eliciting other symbols than gestures, other modalities Source: Villarreal et al., A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies?
  • 77. • Shortcomings and variants • Eliciting more than one symbol Source: Villarreal et al., A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies? Multiple symbols could be combined to trigger a function by relying on hierarchical structure and congruence
  • 78. • Conclusion • Elicitation study is a practical tool for eliciting (symbol) proposals from participants for commands, icons, shortcuts, gestures, vocal commands, etc. • Efficient, natural • Subject to manual classification • Not always sure for agreement rates (other scores) • Legacy bias • Continuity from elicitation to recognition • Subject to Context variability (U,P,E)
  • 79. • Conclusion • Many GES already exist, but no consolidation of this knowledge! y = 3.8909x - 4.1636 R² = 0.9052 0 5 10 15 20 25 30 35 40 45 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Number of studies Year of publication Frequency of Gesture ElicitationStudies
  • 80. How to get the best gestures from people? Considered insofar: • Human factors: preference, sketching time, delete operations, agreement, (dis)similarity • System factors: recognition rate More to come and to consider: • Human factors: hedonic value, memorability, naturalness, discoverability, consistency, congruence, … • System factors: recognition rate, execution time, computational complexity, more measures…