Instead of relying on graphical or vocal modalities for searching an item by keyword (called K-Menu), this paper presents the G-Menu exploiting gesture interaction and gesture recognition: when a user sketches a keyword by gesturing the first letters of its label, a menu with items related to the recognized letters is constructed dynamically and presented to the user for selection and auto-completion. The selection can be completed either gesturally by an appropriate gesture (called the G-Menu) or by touch only (called the T-Menu). This paper compares the three types of menu, i.e., by keyword, by gesture, and by touching, in a user study with twenty participants on their item selection time (for measuring task efficiency), their error rate (for measuring task effectiveness), and their subjective satisfaction (for measuring user satisfaction). Paper accessible at https://dial.uclouvain.be/pr/boreal/en/object/boreal%3A213790

1. 1 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Orange
G-Menu: A Keyword-by-
Gesture based Dynamic
Menu Interface for
Smartphones
Jean Vanderdonckt (UCLouvain)
Eric Petit (Orange)
26 juin 2019

2. 2 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Agenda
1. Starting point: K-Menu
2. Research question
3. Context
4. Experiment
Method
Results and Discussion
5. Conclusion

3. 3 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Starting point: K-Menu
[CHI’07]
Scenario
[Lee, 2007]
If a user selects a menu item M after entering a keyword K, it
is considered that the user inputs the keyword K when finding
the menu M.

4. 4 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Starting point: K-Menu
Experiment
[Lee, 2007]
T-Menu = touch-based menu
K-Menu = menu by keyword
K-Menu was faster than T-Menu in task
completion times.
Users started with T-Menu first,
then K-Menu
Users started with K-Menu first,
then T-Menu

5. 5 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Starting point: K-Menu
Experiment
[Lee, 2007]
K-Menu was more preferred by participants.

6. 6 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Starting point: K-Menu
today
iOS

7. 7 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Research question
K-Menu = keyword text-entry
menu
Would keyword entry without any
keyboard offer any benefit?
G-Menu = keyword by gesture menu
T-Menu = keyword by touch menu

8. 8 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Context
Orange « Tactile facile »
« Tactile facile »
Application to adapt the Android user interface
depending on user profile.
Demonstration
See https://boutique.orange.fr/informations/accessibilite-autonomie/tactile-facile.php
Facile+ Vision+ Vision++ Motor+ Micro-
gesture

9. 9 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Implementation of gesture recognition (DGIL)

10. 10 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Control condition
K-Menu
Search by keyboard
Experiment
Apparatus
T-Menu
Search by touch
G-Menu
Search by gesture

11. 11 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Hypotheses
H11 = The users select items in the K-Menu faster than in the G-Menu and the
T-Menu
H21 = The users select items in a T-menu faster than the other menus if the
target menu item is close to the location where users initiate the search
H31 = The users produce more errors with a G-menu than with the K-Menu
and the T-Menu

12. 12 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Stimuli
30 lists of 18 item from a pool of 130 menu
items were randomly generated with 6 items
for each category:
- 6 for K-Menu (K)
- 6 for T-Menu (T)
- 6 for G-Menu (G)
Number Item Categor
y
1 Rouss
el
K
2 Imbert K
3 Nicolas G
4 Sauva
ge
G
5 Koch T
6 Oliver T
7 Bonnet T
8 Michel G
9 Nguye
n
G
10 Herve T
List example (excerpt)

13. 13 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Task
- Sign a consent form
- Fill in a background
questionnaire
- Demonstration of K,
T, G
- 5 min. familiarization
- 18 item selections
(list)
- 5 statements to
evaluate
(5-point Likert scale):

14. 14 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Sampling
13 female and 17 male = 30
participants
- Different ages (min: 19, max: 70, M =
25.73,SD = 10.05)
- Different degrees (primary to master)
- Different occupations (student,
employee)
- Different domains
30 participants x 18 items = 540 trials
540 trials – 33 outliers = 517 final trials
(170 K, 174 T, 173 G)
Usage frequency
Computer : 6.16/7
Smartphone : 6.3/7
Tablet : 2.34/7
Game player : 3.33/7
Kinect : 1/7

15. 15 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
4.09 8.07 8.43
0.00
2.53
5.46
8.38
11.31
14.24
17.17
K-Menu T-Menu G-Menu
Averageselectiontime[sec]
Menu type
Experiment
Results: Selection time
***
***
K-Menu is the fastest one of all
T-Menu is not significantly faster
than G-Menu
H11 is supported

16. 16 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Results: Selection Time for T-Menu when item is
close 1. Close (displacement  41)
The T-Menu is faster (t = 2:86, p = :004) than the G-Menu
but not lower than the K-Menu condition (t = 4:80, p***=
.00003).
2. All items (all displacements)
The K-Menu stays the fastest
H21 is not fully supported

17. 17 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Results: Error rate
***
1.04
0.260.36
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
G-MenuK-MenuT-Menu
G-Menu K-Menu
T-Menu
Menu type
Averageamountoferrors
**
***
K-Menu has the lowest error rate
of all
T-Menu has a lower error rate
than G-Menu
H31 is supported

18. 18 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Experiment
Results: Statements
23
7
1
27
2 1
20
10
0
25
4
1
17
13
0
5
13
18
3
20
7
3
12
15
4
16
10 11 10 9
2
10 11
0
8
22
7 8
15
1
10
19
2
7
21
0
5
10
15
20
25
30
Amountofparticipants
Ranking
K-Menu T-Menu
Easyness Effectiveness Speed (selection time) Efficiency Learning curve
G-Menu
K-Menu
T-Menu
Winner K-Menu K-Menu K-Menu K-Menu

19. 19 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Conclusion
- K-Menu remains the fastest, easiest, least error-
prone
- G-Menu has advantages elsewhere:
- The G-Menu always remains at immediate availability and use
- The G-Menu is always very natural since it is based on stroke gesture
recognition of "naturally" produced gestures
- The G-Menu may be suitable when the menu items are numerous and
without any semantic meaning (e.g., contact names)
- The G-Menu is an enjoyable menu as an alternative to the most powerful
K-Menu when conditions imposed by the context of use are more
demanding
- Appropriate for eye-free interaction

20. 20 HCI International’ 2019 (Orlando, July 26th-31st, 2019)
Thank you very
much for your
attention