UXPA 2013 Annual Conference Wednesday, July 10, 2013 1:30pm - 2:30pm ET by Kelly Goto, Michael Wörmann As a growing number of three-dimensional motion and natural gesture-controlled consumer electronics come to market, there are new challenges for user experience design for globally merchandized devices, as gestures are presumably influenced by local cultures. As specialists for culture-specific user experience research, UX Fellows wanted to find out which gestures people from various cultures would spontaneously use to control consumer electronics. We conducted 360 face-to-face interviews with consumer electronics-savvy users in 18 different countries. The most frequent gestures and their difficulty were identified for each country and then aggregated.

1. Thumbs up to natural gesture
interfaces?
Kelly Goto & Michael Wörmann
UXPA Conference 2013
Washington DC

2. Introducing UX Fellows
● Worldwide user experience research
● 23 partner companies
● Joint quality levels and processes
● UX expertise + local culture expertise

3. High quality UX research in 35+ countries

4. Introducing gotomedia
● UX design + research consulting
● San Francisco, CA
● Global mobile focus

5. Introducing Facit Digital
● Qual and quant UX
research & consulting
● Munich, Germany
● ITV, desktop & mobile
UX
● Specialized lab facilities

6. C: I like!
B: I guess I ate
something wrong
A: I think I need to
see a dentist
D: That meal is
really tasty
What are these people trying to tell us?
D: That meal is
really tasty

7. Freehand gesture technology is entering
our lives

8. Graphical User Interfaces vs
Natural User Interfaces
Graphical User Interfaces (GUI)
● Unintuitive, users must learn required
movements (typing, using a mouse)
● Does not correspond with natural
human movements
● Requires a collection of clunky
hardware
● Creates ergonomic problems and
repetitive motion injuries (RMI) due to
unnatural motions
Natural User Interface (NUI)
● Intuitive, users already use the required
gestures to different degrees
● Based on natural human movements
● Only hardware required is a screen
● Since gestures used are natural
movements for the body, users can
position themselves comfortably and
not experience ergonomic distress
Gestures used for NUIs should be designed based on what comes
naturally to the largest number of users and should not cause RMI’s
if done repeatedly.

9. Pointing gestures and semantic gestures

10. Pointing gestures

11. Semantic gestures
● Gestures with an associated meaning
● No need for an additional GUI
● Potentially more efficient
Challenges
● Presumably influenced by local cultures
● Some cultures might be more predisposed to the use of
gestures than othersSemantic gesture for
mission control on a
Mac

12. The UX Fellows Gesture Study

13. The UX Fellows Gesture Study
Which semantic gestures would people
from various cultures spontaneously use
to control consumer electronic devices?
● Country differences?
● Country clusters?
● Underlying metaphors?
● Acceptance?

14. We compared spontaneous gesture
behavior in 18 countries.
RU
CN
IN
AU
AR
CA
US
MX
ES
FR
UK
IT
FI
DE
NL
BE
KRTR

15. Setting
15 min
Warm-up: everyday gestures

16. Examined Commands
Switch on device
Volume up
Channel up
Context info for current program
Channel down
Share current program
Volume down
Mute volume
Pause current program
Homepage
EPG
backward navigation in menus
Confirm on-screen dialog (OK)
Deny on-screen dialog (cancel)
Pause video-on-demand (VOD)
Restart VOD
Fast forward VOD
Rewind VOD
Skip chapter VOD
Previous chapter VOD
Stop VOD
Switch off device
P+
P-

17. Sample
Persons who are able to imagine a gesture-control scenario for a TV.
● Regular usage of a recent HD flat TV (max. 5-6 years of age) and
other CE devices and services
● High interest in consumer electronics
● Regular usage of a touch device
● No Xbox Kinect and gesture-enabled TV users
● Potential buyers of international "premium" CE brands
(no national, retailer or "no name" brands)

18. Sample
sample
socio-economic
background
country

19. Sample
51%49%
16
65 years
32
18 countries
x 20 subjects
= 360 participants
min
max
mean

20. The Results

21. Elaboration

22. Commonality of gestures
0
2
4
6
8
10
12
14
16
18
no. of countries with the same main gesture
no.ofcountries
high medium low

23. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)

24. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 18 of 18
Difficulty (1-5) 1.8
Q: How do you turn down the volume?

25. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 18 of 18
Difficulty (1-5) 1.5
Q: How do you turn up the volume?

26. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 17 of 18
Difficulty (1-5) 1.5
NL
Q: How do you pause viewing a film?

27. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 17 of 18
Difficulty (1-5) 2.0
NL
Q: How would you pause the current TV program?

28. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 15 of 18
Difficulty (1-5) 1.7
AU, UK
NL
Q: How do you rewind? (with an image being shown)

29. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
AU, UK
NL
Q: You are watching a boring scene, how do you fast-forward (with an image being shown)?
Commonality 15 of 18
Difficulty (1-5) 1.9

30. Commands with high commonality
Volume down
Volume up
Pause VOD
Pause current program
Rewind VOD
Fast forward VOD
Confirm on-screen dialog (OK)
Commonality 15 of 18
Difficulty (1-5) 1.9
CN, IT, KR
Q: You have chosen a film from an online video library. The TV asks you "do you really want to buy the film?" How do you answer "yes"?

31. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD

32. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD
P+
AU, ES, KR, US
CA
Q: You are zapping through the TV programs: How do you change from channel 15 to channel 16 for example?
Commonality 13 of 18
Difficulty (1-5) 1.8

33. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD
AU, ES, KR, UK, US
CA
P-
Q: You are zapping through the TV programs: How do you change from channel 16 to channel 15 for example?
Commonality 12 of 18
Difficulty (1-5) 1.6

34. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD
CN, KR
FI, RU
NL
Q: How would you stop watching the film altogether (because you want to do something different)?
Commonality 13 of 18
Difficulty (1-5) 2.1

35. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD
AU, CN, MX
BE
UK
FI, RU
Q: How do you set the volume to "mute"?
Commonality 11 of 18
Difficulty (1-5) 2.0

36. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
Backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD
AU, BE, CA
AR, RU
ES, MX, US
Q: Imagine that you were looking for a film in an online video library. You have gone to the comedies section and have chosen the description of a
single movie. Now you want to return to the previous set of options (all of the comedies). Which gesture would you use for that?
Commonality 10 of 18
Difficulty (1-5) 2.3

37. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD
AU, CN, RU, TR
KR
AR, BE, IT, MX
Q: How do you express that you don’t want to buy it?
Commonality 9 of 18
Difficulty (1-5) 1.8

38. Commands with medium commonality
Channel up
Channel down
Stop VOD
Mute volume
backward navigation in menus
Deny on-screen dialog (cancel)
Skip chapter VOD
BE, CN, ES, FI, FR, IN, MX, UK
NL
Q: How would you skip to the next chapter of the film?
Commonality 9 of 18
Difficulty (1-5) 2.2

39. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage

40. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
BE, FR, US
KR
DE, IT
TR
CNCA
FI
Q: How would you switch on the TV?
Commonality 8 of 18
Difficulty (1-5) 2.0

41. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
ES, MX, NL
CA
IN, KR
RU
AR TR
Q: How would you switch off the TV? Note: No common gesture in Italy
Commonality 7 of 18
Difficulty (1-5) 2.9

42. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
BE, CN, ES, FI, FR,
MX, NL
UK
IN, US
Q: How would you return to the previous chapter of the film?
Commonality 8 of 18
Difficulty (1-5) 2.0

43. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
AU, ES, KR, MX AR, DE
BE
NL RU
US
Q: How do you start it (film) again? Note: No common gesture in UK
Commonality 7 of 18
Difficulty (1-5) 2.0

44. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
CN, FR, IN
FI
AR, TR
MX
NL
KR
RU
US
Q: How would you share with your friends what you are currently watching or listening to? Note: No common gesture in UK
Commonality 6 of 18
Difficulty (1-5) 3.2

45. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
AU, US
CN, FI, IN,
KR, RU
BE, UK TR
NL
MX
Q: You'd like to see an overview of tonight's TV programs of the main channels (EPG). How do you activate this function with a hand gesture?
Note: No common gesture in Italy
Commonality 5 of 18
Difficulty (1-5) 2.8

46. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
FR, KR, RU
AU, FI
CN, IN, TR
AR
DE
BE
ES US
UK
Q: How would you get more information for the running TV program?
Commonality 4 of 18
Difficulty (1-5) 2.8

47. Commands with low commonality
Switch on device
Switch off device
Previous chapter VOD
Restart VOD
Share current program
EPG
Context info for current program
Homepage
AR, AU, MX
DENL, UK
FI
BE, IN
US
CA, ES, TR
Q: Now you want to go to the initial set of options - that means the first options you see when you switch on the device.
Note: No common gesture in Italy
Commonality 4 of 18
Difficulty (1-5) 2.9

48. Direct vs. abstract gestures
We observed two kinds of gestures.
● "Direct" gestures
o strongly influenced by touch devices
● Abstract gestures
o basis in language, iconography and metaphors
o associated with more complex tasks
o lower intercultural commonality

49. Some general insights

50. Average gesture difficulty by country
1.0
1.5
2.0
2.5
3.0
TR FR MX AR IN FI CN RU NL DE AU US BE ES CA IT KR UK
Everyday gestures CE gestures
difficulty
min
max 5,0
…

51. Average gesture difficulty by country
1.0
1.5
2.0
2.5
3.0
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR
CE gestures
difficulty
min
max 5,0
…

52. General acceptance
● Our 360 tech-savvy participants responded very positively to
the idea of semantic gesture input for TVs.
"fun to use", "convenient", "coolness factor"
● Participants would like to use gestures to control other
devices as well (e.g. audio).

53. Operation
Many participants felt that voice and sound could be a good
support for gesture input.
waking up the device with a sound
Can the TV
hear me?

54. Gesture type preference
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR
Semantic
Pointing
Many users would prefer switching between gesture types.
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ
Semantic                15
Pointing
FR CN TR IN AR RU MX US IT DE AU FI CA BE ES NL UK KR Σ
Semantic                15
Pointing          9
associated
with more
advanced
commands

55. Typical concerns of users
● Learning and retention of gestures
(but: would be easy if based on implicit behavior repertory)
● Sensitivity and accuracy of devices
● Interference by other gestures in the room
● Compatibility with lean-back disposition

56. Conclusions

57. Conclusions
1. For basic CE commands a common language for semantic
gestures already exists across cultures.
2. More complex commands could be realized by either
a. customizing these gestures for each individual market
b. customizing individually
c. proposing a set of semantic gestures that need to be
learned (the study provides some suggestions)
d. switching to pointing gestures in combination with a GUI
3. It is necessary to consider cultural differences at a country
level when it comes to designing advanced gesture control
for globally marketed devices.

58. Thumbs up to the audience!
Find all results on www.uxfellows.com
Drop us a line: Michael.Woermann@facit-digital.com
Kelly@gotomedia.com