1. #IEEEGoto
Kelly Goto
@go2girl
#IEEEGoto
Thumbs Up to Natural
Gesture Interfaces?
Can We Create a Global Gesture Language?
Tuesday, August 12, 14

2. #IEEEGoto
Gestures their Origins and Distributions
Morris, Collette, Marsh, O’Shaughnessy
pregnancyandbaby.comparamount.com
Tuesday, August 12, 14

3. #IEEEGoto
Tuesday, August 12, 14

4. #IEEEGoto
Gestuno » International Sign Language
http://www.handspeak.com
Global Gestures
A native language Gestuno is not. But it is a barrier breaking vocabulary that allows a select few to
bridge an international language gap... the need to bridge that international barrier has aroused a
sort of lingua franca rooted deep in the trade industry of modern Europe.
http://www.deaflibrary.org
“Government” in different cultures:
Tuesday, August 12, 14

5. #IEEEGoto
Global Gestures
Unlike human-to-human gesturing,
human-machine gesturing does not
have a foundation of thousands of
years of history to serve as a context.
- Silicon Labs
Tuesday, August 12, 14

6. #IEEEGoto
Freehand Gestures Are Entering Our Lives
Tuesday, August 12, 14

7. #IEEEGoto
Freehand Gestures Are Entering Our Lives
Tuesday, August 12, 14

8. #IEEEGoto
Freehand Gestures Are Entering Our Lives
Tuesday, August 12, 14

9. #IEEEGoto
9,000
8,000
7,000
6,000
5,000
4,000
3,000
2,000
1,000
$Million
2012 2013 2014 2015 2016 2017 2018
Gesture
Recogni<on Touchless
Sensing
Global Touchless Sensing & Gesture Recognition
Market Revenue, $million, (2013-2018)
• Expected
to
reach
$15.02
billion
by
2018.
• Market
value
in
2012
was
approximately
$2
billion.
• Consumer
electronics
applica<on
contributes
to
more
than
99%
of
the
global
gesture
recogni<on
market.
Global Market Revenue Projections
Tuesday, August 12, 14

10. #IEEEGoto
Global Market Revenue Projections
#IEEEGoto
Tuesday, August 12, 14

11. #IEEEGoto
Pointing Gestures
Tuesday, August 12, 14

12. #IEEEGoto
The UX Fellows Gesture Study
#IEEEGoto#IEEEGoto
Tuesday, August 12, 14

13. #IEEEGoto
We Compared Gestures in 18 Countries.
Tuesday, August 12, 14

14. #IEEEGoto
The UX Fellows Gesture Study
Which natural gestures would people from
various cultures spontaneously use to control
consumer electronic devices?
• Country differences?
• Country clusters?
• Underlying metaphors?
• Acceptance?
Tuesday, August 12, 14

15. #IEEEGoto
Setting
15 min
Warm-up: everyday gestures
#IEEEGoto
Tuesday, August 12, 14

16. #IEEEGoto
Warmup: Everyday ‘Natural’ Gestures
C: I like!
B: I guess I ate
something wrong
A: I think I need to
see a dentist
What are these people trying to tell us?
D: That meal is really
tasty
D: That meal is really
tasty
Tuesday, August 12, 14

17. #IEEEGoto
Examined Commands
Switch on device
Volume up
Channel up
Context info for current program
Channel down
Share current program
Volume down
Mute volume
Pause current program
Homepage
EPG
P+
P-­‐
Backward navigation in menus
Confirm on-screen dialog (OK)
Deny on-screen dialog (cancel)
Pause video-on-demand (VOD)
Restart VOD
Fast forward VOD
Rewind VOD
Skip chapter VOD
Previous chapter VOD
Stop VOD
Switch off device
Tuesday, August 12, 14

18. #IEEEGoto
Participant Sample
Persons who are able to imagine a gesture-control
scenario for a TV.
• Regular usage of a recent HD flat TV (max. 5-6
years of age)
and other CE devices and services
• High interest in consumer electronics
• Regular usage of a touch device
• No Xbox Kinect and gesture-enabled TV users
• Potential buyers of international "premium" CE
brands
(no national, retailer or "no name" brands)
Tuesday, August 12, 14

19. #IEEEGoto
Participant Sample
sample
socio-economic
background
country
Tuesday, August 12, 14

20. #IEEEGoto
Participant Sample
51%49%
18 countries
x 20 subjects
= 360 participants
16
65 years
32
min
max
mean
Tuesday, August 12, 14

21. #IEEEGoto
The Results
#IEEEGoto#IEEEGoto
Tuesday, August 12, 14

22. #IEEEGoto
Difficulty of TV-Related Gestures
Self-­‐reported
difficulty,
5
=
very
difficult,
1
=
very
easy,
n
=
360
tech-­‐savvy
CE
consumers
in
18
countries
easydifficult
Tuesday, August 12, 14

23. #IEEEGoto
Commonality of Gestures
no.
of
countries
with
the
same
main
gesture
High Medium Low
Tuesday, August 12, 14

24. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Tuesday, August 12, 14

25. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 18
of
18
Difficulty
(1-­‐5) 1.8
Q:
How
do
you
turn
down
the
volume?
Tuesday, August 12, 14

26. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 18
of
18
Difficulty
(1-­‐5) 1.8
Q:
How
do
you
turn
up
the
volume?
Tuesday, August 12, 14

27. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 17
of
18
Difficulty
(1-­‐5) 1.5
Q:
How
do
you
pause
viewing
a
film?
NL
Tuesday, August 12, 14

28. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 17
of
18
Difficulty
(1-­‐5) 2.0
Q:
How
would
you
pause
the
current
TV
program?
NL
Tuesday, August 12, 14

29. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 15
of
18
Difficulty
(1-­‐5) 1.7
Q:
How
do
you
rewind?
(with
an
image
being
shown)
AU,
UK
NL
Tuesday, August 12, 14

30. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 15
of
18
Difficulty
(1-­‐5) 1.9
Q:
You
are
watching
a
boring
scene,
how
do
you
fast-­‐forward
(with
an
image
being
shown)?
AU,
UK
NL
Tuesday, August 12, 14

31. #IEEEGoto
Commands with High Commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 15
of
18
Difficulty
(1-­‐5) 1.9
Q::
You
have
chosen
a
film
from
an
online
video
library.
The
TV
asks
you
"do
you
really
want
to
buy
the
film?"
How
do
you
answer
"yes"?
CN,
IT,
KR
Tuesday, August 12, 14

32. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backward navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Tuesday, August 12, 14

33. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Commonality 13
of
18
Difficulty
(1-­‐5) 1.8
Q:
You
are
zapping
through
the
TV
programs:
How
do
you
change
from
channel
15
to
channel
16
for
example?
P+
AU,
ES,
KR,
US
CA
Tuesday, August 12, 14

34. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Commonality 12
of
18
Difficulty
(1-­‐5) 1.6
Q:
You
are
zapping
through
the
TV
programs:
How
do
you
change
from
channel
16
to
channel
15
for
example?
AU,
ES,
KR,
US
CA
P-­‐
Tuesday, August 12, 14

35. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Commonality 13
of
18
Difficulty
(1-­‐5) 2.1
Q:
How
would
you
stop
watching
the
film
altogether
(because
you
want
to
do
something
different)?
CN,
KR
FI,
RU
NL
Tuesday, August 12, 14

36. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Commonality 11
of
18
Difficulty
(1-­‐5) 2.0
Q:
How
do
you
set
the
volume
to
"mute"?
AU,
CN,
MX FI,
RU
BE
UK
Tuesday, August 12, 14

37. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Commonality 10
of
18
Difficulty
(1-­‐5) 2.3
Q:
Imagine
that
you
were
looking
for
a
film
in
an
online
video
library.
You
have
gone
to
the
comedies
sec<on
and
have
chosen
the
descrip<on
of
a
single
movie.
Now
you
want
to
return
to
the
previous
set
of
op<ons
(all
of
the
comedies).
Which
gesture
would
you
use
for
that?
AU,
BE,
CA
AR,
RU
ES,
MX,
US
Tuesday, August 12, 14

38. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Commonality 9
of
18
Difficulty
(1-­‐5) 1.8
Q:
How
do
you
express
that
you
don’t
want
to
buy
it?
AU,
CN,
RU,
TR
KR
AR,
BE,
IT,
MX
Tuesday, August 12, 14

39. #IEEEGoto
Commands with Medium Commonality
Channel up
Channel down
Stop VOD
Mute volume
Backwards navigation in menus
Deny on-screen dialog
(cancel)
Skip chapter VOD
Commonality 9
of
18
Difficulty
(1-­‐5) 2.2
Q:
How
would
you
skip
to
the
next
chapter
of
the
film?
BE,
CN,
ES,
FI,
FR,
IN,
MX,
UK
NL
Tuesday, August 12, 14

40. #IEEEGoto
Commands with Low Commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
Tuesday, August 12, 14

41. #IEEEGoto
Commands with Low Commonality
Commonality 8
of
18
Difficulty
(1-­‐5) 2.0
Q:
How
would
you
switch
on
the
TV?
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
BE,
FR,
US
KR
DE,
IT
TR
CN
CA
FI
Tuesday, August 12, 14

42. #IEEEGoto
Commands with Low Commonality
Commonality 7
of
18
Difficulty
(1-­‐5) 2.9
Q:
How
would
you
switch
off
the
TV?
Note:
No
common
gesture
in
Italy
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
ES,
MX,
NL
IN,
KR
RU
TR
CA
AR
Tuesday, August 12, 14

43. #IEEEGoto
Commands with Low Commonality
Commonality 8
of
18
Difficulty
(1-­‐5) 2.0
Q:
How
would
you
return
to
the
previous
chapter
of
the
film?
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
BE,
CN,
ES,
FI,
FR,
MX,
NL
IN,
US
UK
Tuesday, August 12, 14

44. #IEEEGoto
Commands with Low Commonality
Commonality 7
of
18
Difficulty
(1-­‐5) 2.0
Q:
How
do
you
start
it
(film)
again?
Note:
No
common
gesture
in
UK
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
AU,
ES,
KR,
MX
BE
NL RU
US
AR,
DE
Tuesday, August 12, 14

45. #IEEEGoto
Commands with Low Commonality
Commonality 6
of
18
Difficulty
(1-­‐5) 3.2
Q:
How
would
you
share
with
your
friends
what
you
are
currently
watching
or
listening
to?
Note:
No
common
gesture
in
UK
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
CN,
FR,
IN
FI
AR,
TR
MX
NL
KR
RU
US
Tuesday, August 12, 14

46. #IEEEGoto
Commands with Low Commonality
Commonality 5
of
18
Difficulty
(1-­‐5) 2.8
Q:
You'd
like
to
see
an
overview
of
tonight's
TV
programs
of
the
main
channels
(EPG).
How
do
you
ac<vate
this
func<on
with
a
hand
gesture?
Note:
No
common
gesture
in
Italy
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
AU,
US CN,
FI,
IN,
KR,
RU
BE,
UK
TR
NL
MX
Tuesday, August 12, 14

47. #IEEEGoto
Commands with Low Commonality
Commonality 4
of
18
Difficulty
(1-­‐5) 2.8
Q:
How
would
you
get
more
informa<on
for
the
running
TV
program?
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
FR,
KR,
RU
AU,
FI
CN,
IN,
TR
AR
DE
BE
ES
US
UK
Tuesday, August 12, 14

48. #IEEEGoto
Commands with Low Commonality
Commonality 4
of
18
Difficulty
(1-­‐5) 2.9
Q:
Now
you
want
to
go
to
the
ini<al
set
of
op<ons
-­‐
that
means
the
first
op<ons
you
see
when
you
switch
on
the
device.
Note:
No
common
gesture
in
Italy
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current
program
Homepage
AR,
AU,
MX
DE
NL,
UK
FI
US
BE,
IN
CA,
ES,
TR
Tuesday, August 12, 14

49. #IEEEGoto
Some General Insights
#IEEEGoto
Tuesday, August 12, 14

50. #IEEEGoto
Direct vs Abstract Gestures
We observed two kinds of natural gestures.
● "Direct" gestures
¡ strongly influenced by touch devices
● Abstract gestures
¡ basis in language, iconography and metaphors
¡ associated with more complex tasks
¡ lower intercultural commonality
Tuesday, August 12, 14

51. #IEEEGoto
1.0
1.5
2.0
2.5
3.0
TR FR MX AR IN FI CN RU NL DE AU US BE ES CA IT KR UK
Everyday
gestures CE
gestures
difficulty
min
max
5.0
Average Gesture Difficulty by Country
…
Tuesday, August 12, 14

52. #IEEEGoto
Average Gesture Difficulty by Country
1.0
1.5
2.0
2.5
3.0
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR
CE
gestures
…
difficulty
min
max
5.0
Tuesday, August 12, 14

53. #IEEEGoto
General Acceptance
§ Our 360 tech-savvy participants responded very
positively to the idea of semantic gesture input for TVs.
"fun to use", "convenient", "coolness factor”
§ Participants would like to use gestures to control other
devices as well (e.g. audio).
Tuesday, August 12, 14

54. #IEEEGoto
General Acceptance
Can CE gestures be deducted from everyday gestures?
USA
Influenced by touch screens —
most thought gestures should be
customizable.
Canada
Personal hobbies eg: gestures
inspired by skiing, kick-boxing etc.
Sign language
Russia.
Not a lot of everyday gestures used.
Replicating alphabets, creating new
gestures and pointing used.
No.
Yes.
Mexico
Smart phones and touch screen
devices eg: sliding the menu up from
the bottom, swiping gestures etc.
Spain | Mexico | France | Netherlands
Everyday gestures and touch screen gestures
Australia | Germany | Italy
Everyday gestures
Argentina | China | Korea
Touch screen devices
India
Analog gestures, touch screen
gestures and icon based gestures
Begium
Everyday gestures, gestures while using
electronic devices and touch screen gestures
Finland
Everyday gestures and touch
screen gestures
Tuesday, August 12, 14

55. #IEEEGoto
Private vs Public Display
§ Considerations
§ Ergonomics
§ “Gorilla Arm”
§ Recognition
privatepublic
Tuesday, August 12, 14

56. #IEEEGoto
Operation
Many participants felt that voice and sound
could be a good support for gesture input.
waking
up
the
device
with
a
sound
Can the TV
hear me?
alterna4ve
op4on
based
on
context
and
need
Tuesday, August 12, 14

57. #IEEEGoto
Operation
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR
Seman<c
Poin<ng
Many users would prefer switching between gesture types.
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ
Seman<c l l l l
l l l
l l l l l l
l l 15
Poin<ng
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ
Seman<c l l l l
l l l
l l l l l l
l l 15
Poin<ng
l
l l l
l
l
l l l
9
associated
with
more
advanced
commands alterna4ve
op4on
based
on
context
and
need
Tuesday, August 12, 14

58. #IEEEGoto
Typical Concerns of Users
§ Learning and retention of gestures
(but: would be easy if based on implicit
behavior repertory)
§ Sensitivity and accuracy of devices
§ Interference by other gestures in the
room
§ Compatibility with lean-back disposition
Tuesday, August 12, 14

59. #IEEEGoto
Conclusions
1. For basic CE commands a common language for natural
gestures already exists across cultures.
2. More complex commands could be realized by either
a. customizing these gestures for each individual market
b. customizing individually
c. proposing a set of semantic gestures that need to be
learned (the study provides some suggestions)
d. switching to pointing gestures in combination with a
GUI
3. It is necessary to consider cultural differences at a
country level
Tuesday, August 12, 14

60. #IEEEGoto
Future of communication
Do we want to create a better remote
control, or do we want to create a 2-
way language of communication?
- go2girl
Tuesday, August 12, 14

61. #IEEEGoto
Future of communication
LG Magic Remote
Tuesday, August 12, 14

62. #IEEEGoto
The problem: no standard
gestures currently exist.
Many new devices in the world of
‘touchless tech’ are using completely
different gestures for the same
common functions.
Future of communication
Tuesday, August 12, 14

63. #IEEEGoto
Can we utilize this methodology to
start a conversation and bring the deaf
community, the large technology
giants and standards together to
develop a universal gesture language?
source: autocarhire.com
Future of communication
Tuesday, August 12, 14

64. #IEEEGoto
Future of communication
Tuesday, August 12, 14

65. #IEEEGoto
The governance
crisis point
Quantity & complexity of
information
Ability to deal with
quantity & complexity
of information
Zap Think LLC
TIME
(innovation) COMPLEXITY vs. COPING (desire)
Tuesday, August 12, 14

66. #IEEEGoto
Remember: People don’t advance as quickly
as technology.
BEYOND USABLE
Tuesday, August 12, 14

67. #IEEEGoto
CONTEXT
Tuesday, August 12, 14

68. #IEEEGotosource: wikipedia
the future?
Tuesday, August 12, 14

69. #IEEEGoto
Find all results on
www.uxfellows.com
Drop us a line:
kelly@gotomedia.com
Michael Woermann
m.woermann@facit-digital.com
www.3bridgesaustin.com
Tuesday, August 12, 14

70. #IEEEGoto
● UX design + research consulting
● San Francisco, CA
● Global mobile focus
#IEEEGoto
Tuesday, August 12, 14