1. Optimizing Special
Character Entry
TheCase ofthe FrenchKeyboard Standard
Anna Feit, Aalto University, Helsinki
Joint work with
Mathieu Nancel, Maximilian John, Andreas Karrenbauer, Daryl Weir, and Antti Oulasvirta

2. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Fun facts about the French keyboard
• There is no way to type œuf, Été or FRANÇAIS (egg, Summer, FRENCH)
• où (where) is the only word that contains ù, yet it has it’s own key
• No french quotation marks « »
• Typing . requires Shift, so do all numbers, and try finding the @ key…
” It is almost impossible to write correctly French
with a keyboard marketed in France”
French Ministry of Culture and Communications
08.09.2017
2

3. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Design Task
08.09.2017
3
Assign 129 keyslots to 122
special characters
Normal
Shift
Alt Gr
Alt Gr + Shift
! ? " # $ % & ' ( ) [ ] { } * + , - . / : ; < =
> ≤ ≥ ÷ @ | ¿ ¡ ¢ £ ¤ ¥ § © ª « ® ° ± · º »
¼ ½ ¾ × ù Ù à æ Æ ç Ç è é ê œ Œ þ İ ı IJ
ij Ŋ ŋ ſ Ə Ʒ ə ʒ μ ð Ð Þ ß ẞ _ - – — ‘ ’ ‚ “
” „ † ‡ … ‰ ‹ › ₩ ™ √ ∞ ≃ · ´ ˋ ˆ ¨ ˉ ̲ ˘ ̑
ˇ ˜ ˙ ̣ ̊ ˝ ˵ ¸ ˛ ̦ ̵ ̷
> 𝟏𝟎 𝟐𝟏𝟑
𝒂𝒔𝒔𝒊𝒈𝒏𝒎𝒆𝒏𝒕𝒔

4. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Given: n characters – 𝑖, 𝑗 𝜖 Σ
m keyslots – 𝑘, 𝑙 𝜖 𝑆
Let: 𝑥𝑖𝑘 = 1 if character 𝑖 is assigned to keyslot 𝑘, 𝑥𝑖𝑘 = 0 otherwise
The (letter) assignment problem
[Burkhard, 1977]
08.09.2017
4

5. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Given: n characters – 𝑖, 𝑗 𝜖 Σ
m keyslots – 𝑘, 𝑙 𝜖 𝑆
Let: 𝑥𝑖𝑘 = 1 if character 𝑖 is assigned to keyslot 𝑘, 𝑥𝑖𝑘 = 0 otherwise
Goal: Find the assignment of characters to keyslots that minimizes the cost
of typing one character after another
The (letter) assignment problem
[Burkhard, 1977]
08.09.2017
5

6. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Given: n characters – 𝑖, 𝑗 𝜖 Σ
m keyslots – 𝑘, 𝑙 𝜖 𝑆
Let: 𝑥𝑖𝑘 = 1 if character 𝑖 is assigned to keyslot 𝑘, 𝑥𝑖𝑘 = 0 otherwise
Goal: Find the assignment of characters to keyslots that minimizes the cost
of typing one character after another
[Burkhard, 1977]
The (letter) assignment problem
Assign each symbol to exactly one input action
08.09.2017
6

7. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Given: n characters – 𝑖, 𝑗 𝜖 Σ
m keyslots – 𝑘, 𝑙 𝜖 𝑆
Let: 𝑥𝑖𝑘 = 1 if character 𝑖 is assigned to keyslot 𝑘, 𝑥𝑖𝑘 = 0 otherwise
Goal: Find the assignment of characters to keyslots that minimizes the cost
of typing one character after another
[Burkhard, 1977]
The (letter) assignment problem
Assign each character to exactly one keyslot
Assign each keyslot not more than once
08.09.2017
7

8. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Optimization of Text Entry methods
08.09.2017
8
[Zhai, Hunter, Smith, 2000] [Karrenbauer and Oulasvirta, 2014]
[Feit and Oulasvirta, 2014]
[Sridhar, Feit, Theobalt and
Oulasvirta, 2015]
…

9. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Special character entry
Quadratic assignment problem is NP hard, with 129 keyslots, 122
characters too complex to even approximate.
Observation:
• Special characters are infrequent in comparison to letters
• Mostly entered before or after a letter
We can simplify the quadratic problem to a linear assignment
problem plus a sparse quadratic part
08.09.2017
9

10. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Given: n characters – 𝑖, 𝑗 𝜖 Σ
m keyslots – 𝑘, 𝑙 𝜖 𝑆
Let: 𝑥𝑖𝑘 = 1 if character 𝑖 is assigned to keyslot 𝑘, 𝑥𝑖𝑘 = 0 otherwise
Goal: Find the assignment of characters to keyslots that minimizes the cost
of typing one character after another or before / after a letter
Assignment problem
08.09.2017
10

11. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Given: n characters – 𝑖, 𝑗 𝜖 Σ
m keyslots – 𝑘, 𝑙 𝜖 𝑆
Let: 𝑥𝑖𝑘 = 1 if character 𝑖 is assigned to keyslot 𝑘, 𝑥𝑖𝑘 = 0 otherwise
Goal: Find the assignment of characters to keyslots that minimizes the cost
of typing one character after another or before / after a letter
Assignment problem
08.09.2017
11
sparse

12. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Objectives
08.09.2017
12
Standardization committee:
” The new keyboard should
• facilitate typing of correct french
• allow quick and easy access to all French characters
• be easy to learn and intuitive to use ”

13. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Motor Performance: minimize time to move between frequent
character-letter pairs
Objectives
Note: linear function
08.09.2017
13

14. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Objectives
Note: linear function
Ergonomics: minimize frequent extreme movements of wrist and
fingers when entering a special character in combination with a letter
Motor Performance: minimize time to move between frequent
character-letter pairs
Note: linear function
08.09.2017
14

15. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Learnability (Intuitiveness): minimize the distance between similar
characters to ensure a consistent and intuitive layout that is easy to
use and learn.
Objectives
08.09.2017
15

16. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Learnability (Intuitiveness): minimize the distance between similar
characters to ensure a consistent and intuitive layout that is easy to
use and learn.
Objectives
Learnability (Familiarity): minimize the distance of a character to its
position on the standard keyboard layout.
Note: linear function
08.09.2017
16

17. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Optimization Problem
08.09.2017
17

18. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Assign each letter to exactly one key
Optimization Problem
08.09.2017
18

19. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Assign each letter to exactly one key
Assign each key to at most one letter
Optimization Problem
08.09.2017
19

20. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Assign each character to exactly one key
Assign each key to at most one character
Capital character must be on shifted key of
lower case character
Optimization Problem
08.09.2017
20

21. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Letter and letter pair frequency
Weighted combination of different corpora representing diverse tasks
• Formal corpus:
legal texts, newspaper articles, wikipedia, emails, facebook
10 different corpora averaged, largest coverage of symbols
• Twitter corpus:
10000 tweets from 10 different twitter accounts,
6 "classic" (e.g. newspaper) and 4 "popular" accounts
• Code corpus:
50 projects from github of most common programming languages (Java,
C++, python, CSS, javascript, HTML)
Input data
08.09.2017
21

22. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Formal: Twitter: Code:
Input data
08.09.2017
22

23. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Input data
Performance crowdsourced movement times of 7555 different key pairs
embedded in nonsense words and repeated 10 times.
08.09.2017
23

24. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Input data
Ergonomics: heuristic cost based on literature, equally punishing movement
toward outer right and left, wrist and finger extension, use of modifiers (Alt, Shift)
08.09.2017
24

25. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Input data
Similarity
each letter-character and character-character pair rated with a heuristic score,
maximum of multiple similarity criteria such as:
• visual similarity, e.g. à and a,
• semantic or functional similarity, e.g. $ and £
• phonetic similarity, e.g. s and ß
• etc.
Sparse input matrix
08.09.2017
25

26. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Input data
Distance
Manhattan distance quantifies number of rows and columns between two
keyslots, additional cost for difference in modifier keys
Similar to search behavior of users (row and column wise)
08.09.2017
26

27. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Optimization in the Iterative Design Process
Problem and design
space
Objectives and
constraints
Input data
Optimization Process
Expert
Comittee
Optimal
layouts
08.09.2017
27

28. Anna Feit
annafeit.de
userinterfaces.aalto.fi
One solution
08.09.2017
28

29. Anna Feit
annafeit.de
userinterfaces.aalto.fi
One solution
French characters easily and
quickly accessible
08.09.2017
29

30. Anna Feit
annafeit.de
userinterfaces.aalto.fi
One solution
Easy to learn: familiar position and groups of
similar characters
08.09.2017
30

31. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Comparison with Azerty
Our mapping
AZERTY
Difference of AZERTY to our
mapping (smaller is worse)
08.09.2017
31

32. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Comparison with manual design
Our mapping
Manually created mapping
Difference of manual mapping to
our mapping (smaller is worse)
08.09.2017
32

33. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Optimization in the Iterative Design Process
Problem and design
space
Objectives and
constraints
Input data
Optimization Process
Expert
Comittee
Optimal
layouts
Manual changes evaluated
by objective functions
08.09.2017
33

34. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Current status
Public feedback
Nearly 3000 comments from French people (47% positive, 37%
negative, 16% neutral)
Revision of optimization model and layout to integrate feedback
08.09.2017
34

35. Anna Feit
annafeit.de
userinterfaces.aalto.fi
Summary
08.09.2017
35
• Large real world instance of the (sparse) quadratic assignment problem
• Relevant for millions of users and applicable to other languages
• Problem models objectives for performance, ergonomics, and learnability
of typing special characters
• Optimization as part of an iterative design process