Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Optimization of Text Input

69 views

Published on

Invited talk at the Dagstuhl Seminar on Computational Interactivity. I presented my PhD work on the optimization of text input methods.
More information:
http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=17232
Also see: http://computationalinteraction.org

Published in: Science
  • Be the first to comment

  • Be the first to like this

Optimization of Text Input

  1. 1. Optimization ofText Input Anna Feit, Doctoral student, Aalto University Dagstuhl Seminar on Computational Interactivity, 06.06.2017
  2. 2. … Mid-air hand gestures for text input
  3. 3. A B C … Z space ? … 27 letters32 gestures 1033 mappings Which hand gesture to use for which letter?
  4. 4. Decisions: For each letter and gesture, assign the letter to the gesture or not. Constraints: No letter / gesture is assigned more than once. Evaluation criteria: Evaluate each design and pick the best one. 1033 feasible designs
  5. 5. Mathematical or algorithmic method to find the best design in the space • Search very large design spaces • Efficient and rigorous process • Quantitative guarantees on the goodness of the outcome • Explicitly trade-off different criteria and constraints Optimization
  6. 6. Challenges Formulation Objectives Optimization Formulation of the design problem and space, identification of design variables and constraints argmin 𝐾 ෍ 𝑘𝜖𝐾 ෍ 𝑙𝜖𝐾 𝐶 𝐾(𝑘, 𝑙) Modeling of evaluation criteria and combination into a fast to compute objective function (interaction cost) Mathematical solver or approximation algorithm to efficiently and thoroughly search the design space Objective 1 Objective2
  7. 7. The (quadratic) letter assignment problem Given: n letters – 𝑖, 𝑗 𝜖 𝛴 m gestures – 𝑘, 𝑙 𝜖 𝐺 Let: 𝑥𝑖𝑘 = 1 if letter 𝑖 is assigned to gesture 𝑘, 𝑥𝑖𝑘 = 0 otherwise 𝐗 = {𝑥𝑖𝑘 | ∀ 𝑖 𝜖 𝛴, 𝑘 𝜖 𝐺, 𝑥𝑖𝑘 𝜖 0,1 } characterises the full design space [Burkhard, 1977] Formulation
  8. 8. Goal: Find the assignment of letters to gestures that minimize the cost of typing one character after another Formulation
  9. 9. [Zhai, Hunter & Smith, 2000] [Light & Anderson, 1993] Formulation [Oulasvirta & Karrenbauer, 2014]
  10. 10. ” It is almost impossible to write correctly French with a keyboard marketed in France” French Ministry of Culture and Communications [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Formulation
  11. 11. [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Formulation é è à ù ê Ê É È À ç Ç æ Æ œ Œ ß ẞ þ Þ ð Ð ŋ Ŋ ij IJ ə Ə ʒ Ʒ & θ ı İ @ ™ ® © ſ º ª · ´ ˋ ˆ ¨ ˉ ̲ ˘ ̑ ˇ ˜ ˙ ̣ ̊ ˝ ˵ ¸ ˛ ̦ ̵ ̷ + < > = ± × ÷ ≤ ≥ ≃ % ‰ √ ∞ ¼ ½ ¾ # / | . , ; : ! ? ¡ ¿ … - - — – _ * † ‡ § ( ) [ ] { } “ ” ‘ ’ « » ‚ „ ‹ › € $ £ ¢ ¤ ¥ ₩ ?? ? > 𝟏𝟎 𝟐𝟏𝟑 𝒂𝒔𝒔𝒊𝒈𝒏𝒎𝒆𝒏𝒕𝒔
  12. 12. [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Formulation è
  13. 13. Objectives • Performance – Fitts’ law weighted by letter pair frequency • QWERTY similarity [Dunlop & Levine 2012] • Word or gesture clarity [Dunlop & Levine 2012, Smith, Bi & Zhai 2015]
  14. 14. Objectives Text entry is a complex task involving cognitive and motor processes. Fast performance involves more than quickly pointing from one key to another • Different performance factors • Different tasks • Different skill levels • Different strategies
  15. 15. Objectives [How we type, Feit, Weir, Oulasvirta, CHI 2016]
  16. 16. Objectives … |C6| = 0.38 Middle vs. Ring, participant 2046 Non-instructed:Ring Instructed: Middle Gesture performance models based on Fitts’ law and theories of motor control Anatomical comfort: Individuation index for each finger [Investigating the Dexterity of Multi-Finger Input for Mid-AirText Entry, Sridhar, Feit,Theobalt, Oulasvirta, CHI 2015]
  17. 17. Objectives [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Standardization committee: ”The new keyboard should facilitate typing of correct french, should be easy to learn and intuitive to use” • Performance and ergonomics of typing a special character before or after a letter • Intuitive and easy to learn: • Grouping similar characters • Position similar to QWERTY • Language statistics take into account different typing tasks, e.g. programming, social media usage, formal writing, etc.
  18. 18. Optimization Mathematical, exact methods Linear or Integer Programming, Branch and Bound methods Pro: • Explicit bounds and guarantees on optimality • Fast solvers available, e.g. Gurobi, CPLEX (IBM) Con: • Objective function in closed mathematical form • Not so flexible (e.g. noisy input data, interactive optimization, multi-objectives, etc. ) Heuristic approximation algorithms • Simulated annealing, Genetic algorithms, Biology inspired algorithms etc. Pro: • Straightforward to implement and standard implementations available • Flexible, e.g. combine with simulation models Con: • No bounds or guarantee to find the global optimum • Potentially slow • Formulation of design space and constraints
  19. 19. Optimization [Feit, Nancel,Weir, John, Bailly, Karrenbauer, Oulasvirta, upcoming] Mathematical solver: Gurobi • Guarantees to cover the full design space • Gives explicit bounds • Nevertheless: cannot solve to the global optimum Challenge: integrate optimization with stakeholders’ opinions
  20. 20. Challenges • Multi-objective optimization: weighted sum versus pareto optimization • No ”one size fits all”: trade-off different tasks, skills, strategies, etc. or optimally adapt • Models and input data: efficient, mathematical models, noisy data
  21. 21. (App) Menus (Web) Layouts UI elements Gestures
  22. 22. www.annafeit.de @AnnaFeit Anna Feit Doctoral student, finishing end 2017 Research topics: • Text entry • UI Optimization I also know a bit about: • User modeling • Mid-air input • Eye tracking

×