Successfully reported this slideshow.
Upcoming SlideShare
×

# Model-Based User Interface Optimization: Part II: LETTER ASSIGNMENT - At SICSA Summer School on Computational Interaction 2015

1,741 views

Published on

Tutorial on Model-Based User Interface Optimization. Part I: LETTER ASSIGNMENT by Anna Feit of Aalto University. Presented at SICSA Summer School on Computational Interaction in 2015 in Glasgow. Note: This one-day lecture is divided into multiple parts.

Published in: Design
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### Model-Based User Interface Optimization: Part II: LETTER ASSIGNMENT - At SICSA Summer School on Computational Interaction 2015

1. 1. SICSA Summer School on Computational Interaction Letter Assignment Anna Feit 23.06.2015
2. 2. The Assignment Problem Keyboard Optimization Heuristic Approaches Competition Integer Programming Overview
3. 3. The Assignment Problem On secret agents and the choice of the right weapon …
4. 4. Assignment Problem Anna Feit – SICSA Summer School on Computational Interaction - 4 [Image: www.classtools.net] [Image: paxsims.wordpress.com]
5. 5. Assignment Problem p11 p12 p13 p14 p21 p22 p23 p24 p31 p32 p33 p34 p41 p42 p43 p44 Assign each task to exactly one agent such that we get most profit. Anna Feit – SICSA Summer School on Computational Interaction - 5
6. 6. How to model the problem? One variable xij for each Agent-Task pair: xij = 1 if Agent i is assigned to Task j. x11=1 Agent 1 is assigned to task 1 x12=0 x13=0 x14=0 Anna Feit – SICSA Summer School on Computational Interaction - 6
7. 7. How to model the problem? x11=1 x12=0 x13=0 x14=0 Profit for Agent 1: Agent 1 is assigned to task 1 Anna Feit – SICSA Summer School on Computational Interaction - 7
8. 8. How to model the problem? Objective function: Maximize the profit for all agents Anna Feit – SICSA Summer School on Computational Interaction - 8
9. 9. How to model the problem? Objective function: Maximize the profit for all agents Constraints: •  Each task j must be assigned to exactly one agent Anna Feit – SICSA Summer School on Computational Interaction - 9
10. 10. How to model the problem? Objective function: Maximize the profit for all agents Constraints: •  Each task j must be assigned to exactly one agent •  Each agent i must be assigned to exactly one task Anna Feit – SICSA Summer School on Computational Interaction - 10
11. 11. Mathematical formulation Easy problem Linear assignment problem: only linear terms in the objective function. Anna Feit – SICSA Summer School on Computational Interaction - 11
12. 12. Application to real world problems •  Send taxis to customers: Profit = route price – cost to reach customer •  Assign lectures to rooms Cost = number of lectures that are assigned to the same room But: very limited problem description •  Same number of agents and tasks •  Assign only one task to each agent •  How about other criteria, e.g. preferences? Anna Feit – SICSA Summer School on Computational Interaction - 12
13. 13. Generalized Assignment problem cij pij i j ci1 ci2 ci3 Assigning a task i to an agent j gives a profit pij but has a cost cij An agent can work on multiple tasks but has a work capacity tici1 ci2 ci3 ti Anna Feit – SICSA Summer School on Computational Interaction - 13
14. 14. Mathematical formulation Assignment problem: Generalized Assignment Problem: Maximize profit The sum of task costs assigned to agent i can not be larger than his working capacity ti Assign each task to one agent Anna Feit – SICSA Summer School on Computational Interaction - 14
15. 15. Mathematical formulation Assignment problem: Generalized Assignment Problem: NP-hard problem Anna Feit – SICSA Summer School on Computational Interaction - 15
16. 16. Quadratic Assignment Problem Relations within the two sets
17. 17. Quadratic Assignment Problem Flow between Factories Anna Feit – SICSA Summer School on Computational Interaction - 17
18. 18. Quadratic Assignment Problem 2 4 3 1 Distance between locations Anna Feit – SICSA Summer School on Computational Interaction - 18
19. 19. How to model this? One variable xiv for each Factory-Location pair: xiv = 1 if Factory i is assigned to Location v 1 2 x13 x24 f12 d34 Anna Feit – SICSA Summer School on Computational Interaction - 19
20. 20. Mathematical formulation Linear assignment problem: Quadratic assignment problem: Assign each Factory to exactly one LocationMinimize the costs between locations and factories Anna Feit – SICSA Summer School on Computational Interaction - 20
21. 21. Mathematical formulation Linear assignment problem: Quadratic assignment problem: NP-hard problem Anna Feit – SICSA Summer School on Computational Interaction - 21
22. 22. More Variations •  Stable Marriage Problem –  Takes into account preferences –  Application: Assign students to hospitals •  Stable Roommates Problem –  No separation into two sets, assigns one element to any of the other elements •  Weapon-Target Assignment problem –  Model cooperation between agents –  Not all targets must be assigned to a weapon Anna Feit – SICSA Summer School on Computational Interaction - 22
23. 23. Keyboard optimization Challenging the Qwerty Keyboard
24. 24. Task: Design a smartphone keyboard that is optimized for performance [Image: www.theinq
25. 25. What is wrong with Qwerty?
26. 26. The Letter Assignment Problem ABCDEFGHIJKLMNOPQRSTUVWXYZ   ?! Goal: Optimize performance 26 letters, 26 keys = 26!>  ​10↑25  different mappings More than stars in the observable universe! Anna Feit – SICSA Summer School on Computational Interaction - 26
27. 27. How can we model this problem? ABCDEFGHIJKLMNOPQRSTUVWXYZ   ?! Anna Feit – SICSA Summer School on Computational Interaction - 27
28. 28. Letter Assignment Problem as QAP Remember: 1 2 x13 x24 f12 d34 T H 14 7 pt,h t14,7 Bigram probability Pointing time xt,14 xh,7 Anna Feit – SICSA Summer School on Computational Interaction - 28
29. 29. Letter Assignment Problem as QAP T H 14 7 pt,h t14,7 Bigram probability Pointing time xt,14 xh,7pi,j Bigram frequency based on a language corpus tv,w Pointing time as given by Fitts’ Law. Anna Feit – SICSA Summer School on Computational Interaction - 29
30. 30. Letter Assignment Problem as QAP T H 14 7 pt,h t14,7 Bigram probability Pointing time xt,14 xh,7 Each key assigned to exactly one letter Each letter assigned to exactly one key Anna Feit – SICSA Summer School on Computational Interaction - 30
31. 31. Letter Assignment Problem as QAP T H 14 7 pt,h t14,7 Bigram probability Pointing time xt,14 xh,7 NP-hard problem ​ 𝟏 𝟎↑ 𝟐 𝟓  mappings Anna Feit – SICSA Summer School on Computational Interaction - 31
32. 32. Heuristic approaches
33. 33. Heuristic Approaches •  Do not rely on mathematical formulation •  Approximate the global optimum Good choice to: •  find some ”good” solution in a short time •  solve problems that are too complex to be solved mathematically Anna Feit – SICSA Summer School on Computational Interaction - 33
34. 34. Heuristic Approaches Here: two classes of heuristics: •  Greedy / Constructive heuristics •  Local search heuristics Try to find a good solution within a given time, no guarantee on how good it is. Anna Feit – SICSA Summer School on Computational Interaction - 34
35. 35. General Principle Greedy approach 1.  Start with an empty solution S = Ø. 2.  Iteratively add elements to S that bring the highest value. 3.  Stop when no more elements can be added that would improve S. Local search 1.  Start with a feasible solution S. 2.  Iteratively move to a better ”neighbour” by changing S. 3.  Stop if no better solution can be found in the neighborhood. Heuristics that tell you which element to add next Heuristics on how to construct a neighbour Anna Feit – SICSA Summer School on Computational Interaction - 35
36. 36. Greedy Heuristics: Example Heuristic: What element should be added next? Minimize pointing time for frequent bigrams 1.  Start with the empty keyboard 2.  Take the next most frequent letter pair 3.  Assign the 2 letters such that the key distance is minimized 4.  Stop when all letters are assigned th 1.52 he 1.28 in 0.94 er 0.94 an 0.82 re 0.68 nd 0.63 at 0.59 on 0.57 nt 0.56 … Anna Feit – SICSA Summer School on Computational Interaction - 36
37. 37. Greedy Heuristics: Example t h th 1.52 he 1.28 in 0.94 er 0.94 an 0.82 re 0.68 nd 0.63 at 0.59 on 0.57 nt 0.56 … Heuristic: What element should be added next? Minimize pointing time for frequent bigrams 1.  Start with the empty keyboard 2.  Take the next most frequent letter pair 3.  Assign the 2 letters such that the key distance is minimized 4.  Stop when all letters are assigned Anna Feit – SICSA Summer School on Computational Interaction - 37
38. 38. Greedy Heuristics: Example t th 1.52 he 1.28 in 0.94 er 0.94 an 0.82 re 0.68 nd 0.63 at 0.59 on 0.57 nt 0.56 … h e Heuristic: What element should be added next? Minimize pointing time for frequent bigrams 1.  Start with the empty keyboard 2.  Take the next most frequent letter pair 3.  Assign the 2 letters such that the key distance is minimized 4.  Stop when all letters are assigned Anna Feit – SICSA Summer School on Computational Interaction - 38
39. 39. Greedy Heuristics: Example t th 1.52 he 1.28 in 0.94 er 0.94 an 0.82 re 0.68 nd 0.63 at 0.59 on 0.57 nt 0.56 … h e i n Heuristic: What element should be added next? Minimize pointing time for frequent bigrams 1.  Start with the empty keyboard 2.  Take the next most frequent letter pair 3.  Assign the 2 letters such that the key distance is minimized 4.  Stop when all letters are assigned Anna Feit – SICSA Summer School on Computational Interaction - 39
40. 40. Greedy Heuristics: Example t th 1.52 he 1.28 in 0.94 er 0.94 an 0.82 re 0.68 nd 0.63 at 0.59 on 0.57 nt 0.56 … h e i n Heuristic: What element should be added next? Minimize pointing time for frequent bigrams 1.  Start with the empty keyboard 2.  Take the next most frequent letter pair 3.  Assign the 2 letters such that the key distance is minimized 4.  Stop when all letters are assigned Anna Feit – SICSA Summer School on Computational Interaction - 40
41. 41. Local Search Heuristics Heuristic: how to choose a neighbourhood Anna Feit – SICSA Summer School on Computational Interaction - 41
42. 42. Local Search Heuristics Heuristic: how to choose a neighbourhood S N(S) Main components: 1.  Goal function f(S) to evaluate the solutions 2.  Rules how to modify S to obtain the neighbourhood N(S) Currently best: Note: f(S) should penalize infeasibility Anna Feit – SICSA Summer School on Computational Interaction - 42
43. 43. Local Search Heuristics: Example Keyboard Optimization 1.  Start from a random keyboard S 2.  Neighbourhood N(S): All keyboards obtained from S by swapping 2 keys 3.  Evaluation of neighborhood: Compute the average pointing time of S, sum of Fitts’ pointing time weighted by bigram frequency 4.  Choose the best neighbour S* of S. If f(S*) < f(S): go to 2. Else: Stop, S is the local optimum. Anna Feit – SICSA Summer School on Computational Interaction - 43
44. 44. Problem [Source: http://bit.ly/1SvePF6] The optimum of a neighborhood may not be the global optimum Neighborhood Anna Feit – SICSA Summer School on Computational Interaction - 44
45. 45. Metaheuristics •  High-level heuristic that tells you how to change the local search heuristic •  Strategy to overcome local optima •  Naive Example: Multistart local search –  Local search heuristic: Obtain neighbordhood by swapping 2 keys of a keyboard –  Metaheuristic: If a local optimum is found, start over with a completely new randomized layout Anna Feit – SICSA Summer School on Computational Interaction - 45
46. 46. Tabu Search Basic Principle: 1.  Store a tabu list of the last t solutions. 2.  Perform a local search where the solutions in the tabu list cannot be chosen. 3.  If a local minimum S is found proceed with the next best S* ∈ N(S) not in the tabu list and add S to the tabu list. 4.  At termination return the best solution found so far. N(S) S Tabu: N(S*) S Tabu: S* N(S)
47. 47. Tabu search Main components: •  How to define N(S) (Local search heuristic) •  Length of the Tabu list •  The stopping rule. For example: –  Total number of iterations –  Number of iterations without improvement Anna Feit – SICSA Summer School on Computational Interaction - 47
48. 48. Simulated Annealing Idea: •  accept a worse solution from N(S) with a probabilty 0 < p < 1 •  the worse the solution the smaller p •  p decreases at each iteration Inspired by metallurgy: Heat a metal to a temperature T and let it cool down slowly at rate r  to soften it.
49. 49. Simulated Annealing Basic principle: 1.  Start with a random solution S Init Temperature T and cooling ratio 0 < r < 1 2.  For L iterations: a)  Choose a random solution S* from N(S) b)  Let d = f(S*) – f(S) •  If d<0: proceed with S* (S* is better) •  Else: proceed with S* with probability p=​ 𝑒↑− 𝑑/ 𝑇  3.  Reduce the temperature: T = rT and go to 2. 4.  At termination return the best solution found so far. Anna Feit – SICSA Summer School on Computational Interaction - 49
50. 50. Simulated Annealing Main components: •  How to define N(S) (Local search heuristic) •  Temperature T, cooling ration r, loop length L •  The stopping rule. For example: –  Total number of iterations –  Number of iterations without improvement –  Small Temperature T < ε Anna Feit – SICSA Summer School on Computational Interaction - 50
51. 51. Genetic Algorithms Inspired by the natural evolution of biological systems Anna Feit – SICSA Summer School on Computational Interaction - 51
52. 52. Genetic Algorithms Inspired by the natural evolution of biological systems Idea: •  Start with a set of solutions (= population) •  At each iteration (= generation) create a new population by combining the best solutions Anna Feit – SICSA Summer School on Computational Interaction - 52
53. 53. Genetic Algorithms Basic principle: 1.  Start with a random set of solutions (population) 2.  Evaluate the fitness f(S) of each solution 3.  Repeat until a new population is created: a)  Select a pair of parent solutions based on their fitness b)  With probability pc crossover the parents to form two offspring solutions. Otherwise create two offsprings as copies of the parents c)  With probability pm mutate the offsprings 4.  Repeat from step 2. At termination report the best solution found so far. Anna Feit – SICSA Summer School on Computational Interaction - 53
54. 54. Genetic Algorithms Main components: 1.  Population size and termination criterium 2.  Fitness evaluation: f(S) similar to the goal function in local search (incl. penalty for infeasibility) 3.  Parent selection: probability proportional to its fitness. Variations: e.g. choose one parent randomly 4.  Crossover: implemented as string combinations. E.g. combine parents at random point xk in string: 5.  Mutation: random modifications, e.g. swap elements (also occurs in nature) Anna Feit – SICSA Summer School on Computational Interaction - 54
55. 55. A good heuristic ensures… Communication: Definition of neighborhood should allow to reach any optimal solution Anna Feit – SICSA Summer School on Computational Interaction - 55
56. 56. A good heuristic ensures… Communication: Definition of neighborhood should allow to reach any optimal solution S Anna Feit – SICSA Summer School on Computational Interaction - 56
57. 57. A good heuristic ensures… Diversification: e.g. through high initial temperature, large tabu list S Anna Feit – SICSA Summer School on Computational Interaction - 57
58. 58. A good heuristic ensures… Intensification: e.g. use bigger N(S) if S is good S Anna Feit – SICSA Summer School on Computational Interaction - 58
59. 59. Optimized keyboard The metropolis keyboard [Zhai, Hunter and Smith, 2000] Variation of simulated annealing •  Goal function: Fitts’ pointing time •  Neighbour construction: No fixed layout, move a random key in a random direction by a random amount •  Several annealing and cooling iterations until ”a sufficiently efficient keyboard was found” e b c h t Anna Feit – SICSA Summer School on Computational Interaction - 59
60. 60. Metropolis Keyboard Diversification and Communication: high temperature, keys move to random directions Intensification: Low temperature, movement toward optimal keyboard 43.1 wpm (vs. 36 for Qwerty) Anna Feit – SICSA Summer School on Computational Interaction - 60
61. 61. Optimizing for more than one Finger The Kalq keyboard [Oulasvirta et al. CHI 2013] Keyboard optimization for two-thumb text entry Goal function: •  Adapted Fitts’ Law modeling performance with two thumbs •  Thumb alternation: + Preparation gain - Recover cost after long time Optimization procedure is the same, only goal function exchanged Anna Feit – SICSA Summer School on Computational Interaction - 61
62. 62. The Kalq Keyboard Combination of Local Search and Simulated annealing: 1.  Diversification: Local optimization for 5000 random layouts 500 iterations 2.  Communication: Simulated annealing starting from the 100 best solutions, 3000 iterations 3.  Intensification: Local optimization for the 10 best layouts, 10.000 iterations 49 wpm (vs. 47 for split Qwerty) Anna Feit – SICSA Summer School on Computational Interaction - 62
63. 63. Optimizing for multiple criteria Multiobjective Keyboard Optimization [Dunlop and Levine, 2012] Multiple goal functions •  Fitts’ pointing time •  Spell correction •  Qwerty similarity in vs. on if vs. of … Anna Feit – SICSA Summer School on Computational Interaction - 63
64. 64. Optimizing for multiple criteria Multiobjective Keyboard Optimization [Dunlop and Levine, CHI 2012] Variant of Local search: Goal: find a set of optimal solutions that dominate each other in one of the 3 goal functions •  Start with locally optimized set of solutions •  Neighborhood: –  randomly swap two keys –  randomly swap extra keys with 25% probability •  Termination: 2000 iterations Anna Feit – SICSA Summer School on Computational Interaction - 64
65. 65. Optimizing for multiple criteria ~37 wpm (vs. 33 for Qwerty) Anna Feit – SICSA Summer School on Computational Interaction - 65
66. 66. Summary •  Heuristics approaches to find a good solution in reasonable time Ø Constructive vs. Local search •  Metaheuristics to overcome local minima Ø Tabu search Ø Simulated Annealing Ø Genetic algorithms •  Define main components: Ø Local search: Goal function, Neighborhood, Termination criterium, Ø Meta heuristics: length of tabu list, temperature, crossover Anna Feit – SICSA Summer School on Computational Interaction - 66
67. 67. COMPETITION
68. 68. Exact methods Integer programming
69. 69. Integer Programming (IP) Linear Assignment problem All decision variables are (binary) integers Anna Feit – SICSA Summer School on Computational Interaction - 69
70. 70. Integer Programming (IP) Integer variables can be used to model •  indivisibilities, e.g. number of to-be-ordered products •  on/off decisions (binary variables), e.g. hiring or assigning Anna Feit – SICSA Summer School on Computational Interaction - 70
71. 71. Integer Programming (IP) Integer programming is about solving IP problems. In comparison to Heuristic approaches they •  increase the chance to find the global optimum •  are mathematically exact: •  Proven to find the global optimum or •  Give a guarantee on how good the solution is in comparison to the global optimum •  But: need a lot of time and computing power, efficient solution only for few ”easy” problems. Anna Feit – SICSA Summer School on Computational Interaction - 71
72. 72. Linear vs. Integer programming Example Anna Feit – SICSA Summer School on Computational Interaction - 72
73. 73. Linear vs. Integer programming Linear program (LP) Easy! (1.95, 4.92) Example Anna Feit – SICSA Summer School on Computational Interaction - 73
74. 74. Linear vs. Integer programming Integer program (IP) Hard! (1.95, 4.92) (5,0) Example Anna Feit – SICSA Summer School on Computational Interaction - 74
75. 75. How to solve an IP Basic principle: •  Find lower and upper bounds for the problem. •  Iteratively compute tighter and tigher bounds by –  Finding good but approximate solutions –  Solving easier problem –  Refining the problems Anna Feit – SICSA Summer School on Computational Interaction - 75
76. 76. Linear vs. Integer programming (1.95, 4.92)Example Upper bound: Optimal LP solution Anna Feit – SICSA Summer School on Computational Interaction - 76
77. 77. Linear vs. Integer programming Example Upper bound: Optimal LP solution Lower bound: Any feasible integer solution (1,1) Optimal value: Anna Feit – SICSA Summer School on Computational Interaction - 77
78. 78. Branch and Bound Divide and Conquer approach Basic idea: •  Divide the problem into easier subproblems (Branch) •  Solve subproblems to obtain upper and lower bounds •  Use bounds to exclude subproblems that cannot lead to an optimal solution (Bound) Typically applied via an enumerative tree Anna Feit – SICSA Summer School on Computational Interaction - 78
79. 79. Example: Keyboard Optimization [Karrenbauer and Oulasvirta, UIST 2014] S S1 S2 xa,1 = 1 xa,1 = 0 a a Anna Feit – SICSA Summer School on Computational Interaction - 79
80. 80. Example: Keyboard Optimization [Karrenbauer and Oulasvirta, UIST 2014] S S1 S2 xa,1 = 1 xa,1 = 0 a a xb,1 = 1 xb,1 = 0 S11 S12 bb Anna Feit – SICSA Summer School on Computational Interaction - 80
81. 81. Example: Keyboard Optimization [Karrenbauer and Oulasvirta, UIST 2014] S S1 S2 xa,1 = 1 xa,1 = 0 xb,1 = 1 xb,1 = 0 S11 S12 ub = 60 wpm lb = 30 wpm … L 30 wpm Anna Feit – SICSA Summer School on Computational Interaction - 81
82. 82. Example: Keyboard Optimization [Karrenbauer and Oulasvirta, UIST 2014] S S1 S2 xa,1 = 1 xa,1 = 0 xb,1 = 1 xb,1 = 0 S11 S12 ub = 60 wpm lb = 30 wpm Solving the LP: upper bound = 28 wpm … L 30 wpm Best solution: 30 ≤ S ≤60 Anna Feit – SICSA Summer School on Computational Interaction - 82
83. 83. Example: Keyboard Optimization [Karrenbauer and Oulasvirta, UIST 2014] S S1 S2 xa,1 = 1 xa,1 = 0 xb,1 = 1 xb,1 = 0 S11 S12 ub = 60 wpm lb = 30 wpm xb,1 = 1 S22 Solving the LP: ub= 58 wpm … L 30 wpm Anna Feit – SICSA Summer School on Computational Interaction - 83
84. 84. Example: Keyboard Optimization [Karrenbauer and Oulasvirta, 2014] S S1 S2 xa,1 = 1 xa,1 = 0 xb,1 = 1 xb,1 = 0 S11 S12 ub = 58 wpm lb = 30 wpm xb,1 = 1 S22 Solving the LP: ub= 58 wpm … L 30 wpm Anna Feit – SICSA Summer School on Computational Interaction - 84
85. 85. IP solution to Keyboard Optimization [Karrenbauer and Oulasvirta, UIST 2014] 42.65 wpm, within 7% of optimum Metropolis: 43.1 wpm Qwerty: 36 wpm Exact but slow: no optimal solution after 15h But guranteed to find the optimal solution after exponential time Anna Feit – SICSA Summer School on Computational Interaction - 85
86. 86. Summary •  Many Optimization methods applied in Keyboard Optimization •  Heuristic approaches: reasonable solution in fast time •  Metaheuristics: non-zero probability to find the global optimum in infinite time •  Integer Programming: •  Slow and complex •  Guaranteed optimal solution after exponential time •  Bounds on how good the current best solution is
87. 87. Further readings •  Dunlop M. and Levine J. (2012). Multidimensional pareto optimization of touchscreen keyboards for speed, familiarity and improved spell checking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM •  Karrenbauer, A., and Oulasvirta, A. Improvements to keyboard optimization with integer programming. Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 2014. •  Oulasvirta A. et al. (2013). Improving two-thumb text entry on touchscreen devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM •  Wolsey, L. A. (1998). Integer programming •  Zhai, S. , Hunter, M. and Smith B. A. (2000). The metropolis keyboard - an exploration of quantitative techniques for virtual keyboard design. In Proceedings of the 13th annual ACM symposium on User interface software and technology (UIST '00). ACM Anna Feit – SICSA Summer School on Computational Interaction - 87