The document discusses the optimization of text input using mid-air hand gestures, exploring the challenges of assigning letters to gestures and evaluating design effectiveness. It emphasizes mathematical methods for efficient design space exploration, utilizing techniques like integer programming and heuristic algorithms to balance multiple objectives in text entry performance. The research aims to enhance user experience by integrating ergonomic and cognitive factors while accommodating diverse typing tasks.

1. Optimization ofText Input
Anna Feit, Doctoral student, Aalto University
Dagstuhl Seminar on Computational Interactivity, 06.06.2017

3. …
Mid-air hand gestures for text input

4. A
B
C
…
Z
space
?
…
27 letters32 gestures
1033 mappings
Which hand gesture to use for which letter?

5. Decisions:
For each letter and gesture,
assign the letter to the gesture
or not.
Constraints:
No letter / gesture is assigned
more than once.
Evaluation criteria:
Evaluate each design and pick
the best one.
1033 feasible designs

6. Mathematical or algorithmic method to find the best design in
the space
• Search very large design spaces
• Efficient and rigorous process
• Quantitative guarantees on the goodness of the outcome
• Explicitly trade-off different criteria and constraints
Optimization

7. Challenges
Formulation Objectives Optimization
Formulation of the design
problem and space,
identification of design
variables and constraints
argmin
𝐾
෍
𝑘𝜖𝐾
෍
𝑙𝜖𝐾
𝐶 𝐾(𝑘, 𝑙)
Modeling of evaluation criteria
and combination into a fast to
compute objective function
(interaction cost)
Mathematical solver or
approximation algorithm to
efficiently and thoroughly
search the design space
Objective 1
Objective2

8. The (quadratic) letter assignment problem
Given: n letters – 𝑖, 𝑗 𝜖 𝛴
m gestures – 𝑘, 𝑙 𝜖 𝐺
Let: 𝑥𝑖𝑘 = 1 if letter 𝑖 is assigned to gesture 𝑘, 𝑥𝑖𝑘 = 0 otherwise
𝐗 = {𝑥𝑖𝑘 | ∀ 𝑖 𝜖 𝛴, 𝑘 𝜖 𝐺, 𝑥𝑖𝑘 𝜖 0,1 } characterises the full design space
[Burkhard, 1977]
Formulation

9. Goal: Find the assignment of letters to gestures that minimize the cost of typing
one character after another
Formulation

10. [Zhai, Hunter & Smith, 2000]
[Light & Anderson, 1993]
Formulation
[Oulasvirta & Karrenbauer, 2014]

11. ” It is almost impossible to write correctly French
with a keyboard marketed in France”
French Ministry of Culture and Communications
[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Formulation

12. [Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Formulation
é è à ù ê Ê É È À ç Ç æ Æ œ Œ ß ẞ þ Þ ð Ð ŋ Ŋ ij IJ ə Ə ʒ Ʒ & θ ı İ @ ™ ® ©
ſ º ª · ´ ˋ ˆ ¨ ˉ ̲ ˘ ̑ ˇ ˜ ˙ ̣ ̊ ˝ ˵ ¸ ˛ ̦ ̵ ̷ + < > = ± × ÷ ≤ ≥ ≃ % ‰ √ ∞ ¼ ½ ¾
# / | . , ; : ! ? ¡ ¿ … - - — – _ * † ‡ § ( ) [ ] { } “ ” ‘ ’ « » ‚ „ ‹ › € $ £ ¢ ¤ ¥ ₩
?? ?
> 𝟏𝟎 𝟐𝟏𝟑
𝒂𝒔𝒔𝒊𝒈𝒏𝒎𝒆𝒏𝒕𝒔

13. [Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Formulation
è

14. Objectives
• Performance – Fitts’ law weighted by letter pair frequency
• QWERTY similarity [Dunlop & Levine 2012]
• Word or gesture clarity [Dunlop & Levine 2012, Smith, Bi & Zhai 2015]

15. Objectives
Text entry is a complex task involving cognitive and motor processes.
Fast performance involves more than quickly pointing from one key to
another
• Different performance factors
• Different tasks
• Different skill levels
• Different strategies

16. Objectives
[How we type,
Feit, Weir, Oulasvirta, CHI 2016]

17. Objectives
…
|C6| = 0.38
Middle vs. Ring, participant 2046
Non-instructed:Ring
Instructed: Middle
Gesture performance models
based on Fitts’ law and theories
of motor control
Anatomical comfort:
Individuation index for each
finger
[Investigating the Dexterity of Multi-Finger Input for Mid-AirText Entry,
Sridhar, Feit,Theobalt, Oulasvirta, CHI 2015]

18. Objectives
[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Standardization committee:
”The new keyboard should facilitate typing of correct french, should be easy to learn and intuitive to use”
• Performance and ergonomics of typing a
special character before or after a letter
• Intuitive and easy to learn:
• Grouping similar characters
• Position similar to QWERTY
• Language statistics take into account different
typing tasks, e.g. programming, social media
usage, formal writing, etc.

19. Optimization
Mathematical, exact methods
Linear or Integer Programming, Branch and Bound
methods
Pro:
• Explicit bounds and guarantees on optimality
• Fast solvers available, e.g. Gurobi, CPLEX (IBM)
Con:
• Objective function in closed mathematical form
• Not so flexible (e.g. noisy input data, interactive
optimization, multi-objectives, etc. )
Heuristic approximation algorithms
• Simulated annealing, Genetic algorithms, Biology
inspired algorithms etc.
Pro:
• Straightforward to implement and standard
implementations available
• Flexible, e.g. combine with simulation models
Con:
• No bounds or guarantee to find the global
optimum
• Potentially slow
• Formulation of design space and constraints

20. Optimization
[Feit, Nancel,Weir, John, Bailly,
Karrenbauer, Oulasvirta, upcoming]
Mathematical solver: Gurobi
• Guarantees to cover the full
design space
• Gives explicit bounds
• Nevertheless: cannot solve to
the global optimum
Challenge: integrate
optimization with stakeholders’
opinions

21. Challenges
• Multi-objective optimization: weighted sum versus pareto
optimization
• No ”one size fits all”: trade-off different tasks, skills, strategies, etc.
or optimally adapt
• Models and input data: efficient, mathematical models, noisy data

22. (App) Menus
(Web)
Layouts
UI elements
Gestures

23. www.annafeit.de @AnnaFeit
Anna Feit
Doctoral student,
finishing end 2017
Research topics:
• Text entry
• UI Optimization
I also know a bit about:
• User modeling
• Mid-air input
• Eye tracking