The document discusses head and shoulders gestures as a means of hands-free interaction, providing a framework for user-defined gestures with upper body movements. It presents findings from an experiment that elicited 308 gestures categorized into 10 types, revealing the potential for a novel vocabulary and design guidelines for effective gesture use. The results emphasize the utility of these gestures in various applications and suggest specific gestures for specific tasks.

1. Head and Shoulders Gestures:
Exploring User-Defined Gestures with Upper Body
HCI International ‘2019 (Orlando, FL, July 26th-31st, 2019)

2. Head and Shoulders Gestures:
Exploring User-Defined Gestures with Upper Body
HCI International ‘2019 (Orlando, FL, July 26th-31st, 2019)
Jean Vanderdonckt1, Nathan Magrofuoco1, Suzanne Kieffer2,
Jorge Luis Perez-Medina1,3, Paolo Roselli4, Ysabelle Rase1,
Santiago Villarreal1
1LouRIM, Université catholique de Louvain, Belgium
2IL&C, Université catholique de Louvain, Belgium
3Universidad de Las Américas, Quito, Ecuador
4Università degli Studi di Roma “Tor Vergata”, Roma, Italy

3. 3
What are head and shoulders gestures?
• A head gesture is any movement of the head leaving the rest
of the body unaffected (stationary)
• A head gesture could occur in any plane (sagittal, transverse,
frontal)
• A shoulder gesture is any movement of the shoulder joint that
leaves the rest of the arm unaffected (stationary).
• A shoulder gesture occurs in any plane of motion (sagittal,
transverse, frontal) or direction (forward, backward, or circular)

4. 4
Y
Yaw
Z
Roll
X
Pitch

5. 5
Why are head and shoulders gestures interesting?
• Hands-free interaction is made possible
• Fixed-Gaze Head Movement is appropriate when
• No device needed
• Both hands should be free
• No need to move the gaze
• Capability to trigger actions from a moderate set
• Medium command duration
• Accurate recognizers for head and shoulders gestures start to
appear
M13 M31
M65M56
M42
M24
[Qinjie et al., 2019]

6. 6
Why are head and shoulders gestures interesting?
• They are used in physical exercises

7. 7
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Head Label Alias Movement
(frontal,
transversal,
sagittal)
X trans-
lation
Move the head
left, right
Face left, face
right
Lateral translation
(v,c,c)
Y trans-
lation
Move the head
up, down
Face up, face
down
Neck elevation,
depression (c,v,c)
Z trans-
lation
Move the head
forward,
backward
Thrust,
retreat
Protraction,
retraction (c,c,v)

8. 8
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Head Label Alias Movement
(frontal,
transversal,
sagittal)
Frontal
tilting
Tilt the head to
the left, right
Bend left,
right
Lateral flexion
(v,v,c)
Transvers
al tilting
Tilt the head up,
down
Bend up,
down
Extension, flexion
(v,c,v)
Saggital
tilting
Tilt the head
forward,
backward
Bend
forward,
backward
Extension, flexion
(c,v,v)

9. 9
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Head Label Alias Movement
(frontal,
transversal,
sagittal)
X rotation Turn the head
up, down
Uphead,
downhead
Horizontal
rotation (c,v,v)
Y rotation Turn the head
left, right
Lefthead,
righthead
Vertical rotation
(v,c,v)
Z rotation Turn the head
forward,
backward
Forehead,
backhead
Facial rotation
(v,v,c)

10. 10
Why are head and shoulders gestures interesting?
• They exhibit some potential for a novel vocabulary
Shoulders Label Alias Movement (frontal, transversal,
sagittal)
X translation Move shoulder horizontally to left, right Decontract,
contract
Extension, flexion (v,c,c)
Y translation Raise shoulder, lower shoulder Raise, lower Shoulder elevation, depression
(c,v,c)
Z translation Move shoulder forward/backward Protract, retract Shoulder protraction, retraction
(c,c,v)

11. • There are also common head and shoulders gestures

12. 12
Experiment: Gesture Elicitation Study (GES)
• Participants
• 10 females + 12 males = 22 participants
• Aged from 18 to 62y (M=29, SD=13)
• Various occupations: secretary, teacher, employee,...
• Device usage frequencies
• Creativity score
y = 0.1467x + 52.482
R² = 0.0775
0
10
20
30
40
50
60
70
80
18 28 38 48 58
Creativity[score]
Age [years]
6.09 6.05
2.68
1.73
1.05
0.00
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Computer
Smartphone
Tablet
Gameconsole
MSKinect
Studieddevice
Averagefequencyofusage
Device

13. 13
Experiment: Gesture Elicitation Study (GES)
• Stimuli: 14 referents for IoT tasks:
Turn the TV On/Off, Start Player, Turn the Volume up, Turn the volume
down, Go to the next channel, Go to the previous channel, Turn Air
Conditioning On/Off, Turn Lights On/Off, Brighten Lights, Dim Lights,
Turn Heating system On/Off, Turn Alarm On/Off, Answer a phone call,
and End Phone Call.
Example: 3. INCREASE: Brighten lights
Before After

14. 14
Experiment: GES main test
• The participant
• Is presented with the list of referents (randomly selected)
• Confirms she understood the tasks
• Is asked to think of a suitable gesture
• Says “I am ready”
• Gives a rating between 1 and 10 to describe how well the proposed
gesture fits the referent (10=excellent)
• The researcher
• Records the thinking time = time since the task started (i.e., when
the referent was presented to the participant) and the moment
when the participant knows what gesture to propose
• Fills in the gesture sheet: ID, sketch, time, comment
• Videotapes the whole session for further analysis

15. 15
Experiment: Results
• 22 participants X 14 referents = 308 elicited gestures
resulting into 10 categories
Head single gesture, 102
Concurrent compound
gesture, 70
Sequential
compound
gesture, 44
Both shoulders single
gesture, 29
Dominant shoulder single
gesture, 19
Non-dominant shoulder
single gesture, 14
Head repeated gesture, 10
Both shoulders
repeated
gestures, 9
Dominant shoulder
repeated gesture, 4
Non-dominant shoulder
repeated gesture, 3
Other, 26

16. 16
Experiment: Results
• Aggregated measures per referent

17. 17
Experiment: Results
• Evolution of aggregated measures per referent
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
0.00
5.00
10.00
15.00
20.00
25.00
Value
Time[Sec]
Referent
Average thinking time Goodness-of-fit
Linear (Average thinking time) Linear (Goodness-of-fit)

18. 18
Experiment: Results
• Breakdown per criteria
Upface/downface, 3%
Thrust, 1% Bend, 31%
Nod,4%
Rotate,4%
Left/right, 12%
Backhead, 0%
Raise
11%
Lower,4%
Shrug, 11% Clog, 9%
Protract
6%
Retract,3%
Head (%), 50.87 Shoulders (%), 30.62
Head and Shoulders (%),
18.51
One stroke
68.09%
Two strokes
22.37%
Three or more strokes
9.54%
Body part
Elicited gestures
Amount of strokes

19. 0.390 0.390
0.316
0.283
0.267 0.260
0.263
0.250 0.250 0.250 0.248
0.229
0.215
0.188
0.138
0.368 0.368
0.286
0.251
0.238 0.234
0.232
0.221 0.221 0.216 0.215
0.195
0.182
0.156
0.104
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Go to Next
Channel
Go to
Previous
Channel
Answer
Phone Call
End Phone
Call
Play/pause Turn TV
On/Off
Average Turn Lights
On/Off
Turn Alarm
On/Off
Decrease
Volume
Brighten
Lights
Increase
Volume
Dim Lights Turn Air
Conditioning
On/Off
Turn Heating
System
On/Off
High Medium
19
Experiment: 14 Consensus gestures
Agreement score [Vatavu & Wobbrock, 2015]
Agreement rate [Vatavu & Wobbrock, 2016]

20. 20
Conclusion
• Contributions
• Design space for head and shoulders gestures
• Corpus of 308 elicited gestures with measures
• Classification into 10 categories
• Consensus set of 14 head and shoulders gestures
• Design guidelines
• Use bending gestures as a first-class citizen
• Use Upface/downface for infrequent tasks
• Use thrust only for play/pause
• Forehead and backhead gestures should not be used,
apart for exceptional assignation

21. Thank you very much
for your attention