This document provides an overview of optimization for user interface design. It discusses how user interface design problems can be formulated as optimization tasks by defining design variables, constraints, and objective functions. Various optimization algorithms from fields like operations research, computer science, and engineering can then be applied to find optimal user interface designs. The document outlines some classic optimization problems and examples of optimization in engineering design. It also discusses model-based approaches to user interface optimization that involve representing interface design problems formally and using predictive models of user behavior together with optimization algorithms to generate improved interface designs.

1. Optimization
for User Interface Design
Antti Oulasvirta, Anna Feit
Aalto University
SICSA SUMMER SCHOOL ON COMPUTATIONAL INTERACTION
GLASGOW
JUNE 23, 2015

2. BIT.LY/1K9SJ6R

3. USERINTERFACES.AALTO.FI

5. UI DESIGN AS OPTIMIZATION
DEFINING DESIGN PROBLEMS
ALGORITHMIC SOLUTIONS
ADVANCEDTOPICS
TODAY

6. SCHEDULE
09:30 Introduction
10:30 Letter Assignment
11:30 Coffee
11:45 Competition!
13:00 Lunch
13:45 Results
14:00 Integer programming
14:30 Solving real problems
15:30 Coffee
15:45 Advanced topics
16:45 Discussion
17:30 Closing

7. SICSA SUMMER SCHOOL ON COMPUTATIONAL INTERACTION
DAY 2: OPTIMIZATION
INTRODUCTION

8. MOTIVATION

9. design
evaluate
User-Centered Design

10. Evaluative studies
Design heuristics
User and task analysis
User-Centered Design

12. A hierarchical menu with 50 items
can be ordered in 10

13. Perception
Attention
Memory, learning
Motor control

14. Designers struggle with
problem-solving [E.g., Nigel Cross]

15. Costly, slow
Unreliable
Lacks guarantees
Subjective
Non-accumulative
User-centered design

16. BACKGROUND

17. • Mathematics
• Operations research
• Algorithm theory
• Computational complexity theory
• Engineering optimization
• Artificial intelligence
• Economics and management science
• Software engineering
A MULTI-DISCIPLINARY FIELD
Rao 2009

18. CLASSIC PROBLEMS
• Assignment problem
• Closure problem
• Constraint satisfaction
problem
• Cutting stock problem
• Integer programming
• Knapsack problem
• Minimum spanning tree
• Nurse scheduling problem
• Traveling salesman problem
• Vehicle routing problem
• Vehicle rescheduling
problem
• Weapon target assignment
problem
Rao 2009

19. ENGINEERING OPTIMIZATION
1. Design of aircraft and aerospace structures for
minimum weight
2. Finding the optimal trajectories of space vehicles
3. Design of civil engineering structures such as
frames, foundations, bridges, towers, chimneys, and
dams for minimum cost
4. Minimum-weight design of structures for
earthquake, wind, and other types of random
loading
5. Design of water resources systems for maximum
benefit
6. Optimal plastic design of structures
7. Optimum design of linkages, cams, gears,
machine tools, and other mechanical components
8. Selection of machining conditions in metal-
cutting processes for minimum production cost
9. Design of material handling equipment, such as
conveyors, trucks, and cranes, for minimum cost
10. Design of pumps, turbines, and heat transfer
equipment for maximum efficiency
11. Optimum design of electrical machinery such
as motors, generators, and transformers
12. Optimum design of electrical networks
13. Shortest route taken by a salesperson visiting
various cities during one tour
14. Optimal production planning, controlling, and
scheduling
15.Analysis of statistical data and building empirical
models from experimental
results to obtain the most accurate representation
of the physical phenomenon
16. Optimum design of chemical processing
equipment and plants
17. Design of optimum pipeline networks for
process industries
18. Selection of a site for an industry
19. Planning of maintenance and replacement of
equipment to reduce operating costs
20. Inventory control
21.Allocation of resources or services among
several activities to maximize the benefit
22. Controlling the waiting and idle times and
queueing in production lines to reduce the costs
23. Planning the best strategy to obtain maximum
profit in the presence of a competitor
24. Optimum design of control systems
Rao 2009

20. DESIGN OPTIMIZATION
Ford Duratorq engine
• aerospace
• automotive
• chemical
• electronics
• construction
• traffic
• manufacturing
• logistics
• telecommunications

21. NUMEROUS METHODS
1.2 Historical Development 3
Table 1.1 Methods of Operations Research
Mathematical programming or Stochastic process
optimization techniques techniques Statistical methods
Calculus methods Statistical decision theory Regression analysis
Calculus of variations Markov processes Cluster analysis, pattern
recognitionNonlinear programming Queueing theory
Geometric programming Renewal theory Design of experiments
Quadratic programming Simulation methods Discriminate analysis
(factor analysis)Linear programming Reliability theory
Dynamic programming
Integer programming
Stochastic programming
Separable programming
Multiobjective programming
Network methods: CPM and PERT
Game theory
Modern or nontraditional optimization techniques
Genetic algorithms
Simulated annealing
Ant colony optimization
Particle swarm optimization
Neural networks
Fuzzy optimization
HISTORICAL DEVELOPMENT
The existence of optimization methods can be traced to the days of Newton, Lagrange,
and Cauchy. The development of differential calculus methods of optimization was
Rao 2009

22. SCOPE

23. USER INTERFACE DESIGN
Graphical user interfaces
Consumer electronics
Automotive interfaces
Input devices
Web user interfaces
Gestural interaction Mobile interfaces
Dialogue interfaces

24. MODERN METHODS FOR
DESIGN OPTIMIZATION
• Modern methods of engineering optimization
• Genetic algorithms: 1975
• Ant colony optimization:1992
• Particle swarm optimization: 1995
Rao 2009

25. COMBINATORIAL OPTIMIZATION
WITH BLACK-BOX METHODS
• High dimensional problems
• Exhaustive search impossible
• Expensive computations
• E.g., user simulations
• Multiple objectives
• Uncertainty, missing knowledge,changing conditions
• Complex or un-analyzed objective functions

26. GOALS AND APPROACH

27. GOALS
Confidence comes with experience
1.Ability to define UID problems as optimization tasks
2.Ability to apply heuristic methods
3.Knowledge about mathematical approaches
4.Knowledge about state-of-the-art in UI optimization

28. TEACHING APPROACH
• Breadth plus some depth
• Lectures on methods, applications, and issues
• One longer exercise and several pen-and-paper
exercises

29. FORMING WORK GROUPS

31. BY THIS EVENING, YOU WILL HAVE…
• Knowledge allowing you to replicate basically all
work until 2013
• Including CHI’12 Best paper award and our
UIST’13 paper
• Broader perspectives to methods, issues and
possibilities in UI optimization
• Skills to formulate new problems

32. YOU WILL BE TORN
• It’s easy!
• Anybody can optimize with black-box methods.
(The question is how well)
• It’s hard!
• Solving real problems require insight in problem
definition, modeling, and algorithms

33. PRELIMINARIES
“CRASH COURSE”

34. WHAT IS OPTIMIZATION?
• The act of obtaining the best result under given circumstances
• The process of finding the conditions that give the best value
!
• Minimization and maximization are equivalent
Rao 2009

35. GENERAL FORM
• The optimum is the design vector X with the lowest
value
• Constraints can be added
• E.g., g(X) < 1
Find X =
8
>><
>>:
x1
x2
· · ·
xn
9
>>=
>>;
which minimizes f(X)
Rao 2009

36. COMBINATORIAL PROBLEMS
P = (S, ⌦, f)
Problem
Finite set of
candidate solutions
Constraints
Evaluation function

37. !
!
Black-box
methods
(e.g., simulated
annealing)
Optimizer
Fitts’ lawBigrams
Objective function
literature. Our approach is directly usable in the available IP
solvers such as CPLEX and Gurobi.
BACKGROUND: THE LETTER ASSIGNMENT PROBLEM
Given n letters and n keyslots, the task of the letter assign-
ment problem is to minimize the average cost ck` of selecting
letter ` after k weighted by the bigram probability pk`:
min
X
k
X
`
pk` · ck` (1)
It is an instance of the Koopmans-Beckman Problem [15, 6],
which is NP-hard. As we will see later, this problem can be
modeled with a quadratic function on binary variables. Thus,
it belongs to a broad class of problems called quadratic as-
signment problem (QAP) where the goal is to minimize the
total pair-wise interaction cost [22, 24].
Except for a few papers using heuristic cost functions (e.g.,
[9]), the Fitts-bigram energy is used (see also [4]). Here, the
Task
n!##
Open#
Save##
Save#as...#
Close#
...#
About#
n!#
A B C D E F G H I J
K L M N O P Q R
S T U V W X Y
A
A B C D E F G H I
J K L M N O P Q R
S T U V W X Y Z
A B C D E F G H I
J K L M N O P Q R
S T U V W X Y Z
Letter assignment
Keyboard optimization
See Karrenbauer & Oulasvirta UIST 2014

38. KEY CONCEPTS
• Decision variables
• Constraints (rules)
• Objective function
• types: minimization, maximization, existence,
feasibility
• Names: loss function, cost function, utility function,
fitness function, energy function

39. OBJECTIVE FUNCTION
• The following operations on the objective function will not
change the optimum solution x*:
• multiplication or division of f(x) by a positive constant c
• addition or subtraction of a positive constant c to/from f(x)
Rao 2009
Figure 1.1 Minimum of f (x) is same as maximum of −f (x).
cf(x)
cf(x)
f(x)
f(x)
f(x)
f(x) f(x)
cf*
f* f*
x*
x
x*
x
c + f(x)
c + f*
Figure 1.2 Optimum solution of cf (x) or c + f (x) same as that of f (x).

40. DIMENSIONS OF PROBLEMS
• Existence of constraints
• Types of functions involved
• Linear, quadratic and other nonlinear
• Permissible values of design variables
• Binary, integer, continuous, mixed
• Deterministic vs. stochastic problems
• Separability of objective function
• Number of objectives
Rao 2009

41. SEARCH CONCEPTS
• Variable assignment
• Constraint satisfaction
• Search neighborhood
• N: S —> 2S
Rao 2009

42. SEARCH SPACE
• Size of search space
• Types of solutions
• Feasible solution
• Optimal solution / optimum set
• Local
• Global
• Convergence
Rao 2009

43. OPTIMIZATION LANDSCAPE
Global optimum
Local optimum

44. ORDER NOTATION FOR
COMPUTATIONAL COMPLEXITYIntroduction Algorithms Spanning Trees
Notation for Functions f 2 . . .
!"#$%&"'(')"&%*"+%,+"-'.%"*%*/&0')"&1
!"#$ %&'()*')
!"+&,n$ +&,*-.)/0.%
!"1+&,n2c$ 3&+4+&,*-.)/0.%
&"n$ (56+.'7*-
O"n$ +.'7*-
!"n+&,n$ 85*(.+.'7*-
!"n9$ 85*:-*).%
3&+4'&0.*+
!"cn$ 7;3&'7').*+
!"n<$ =*%)&-.*+ &->%&06.'*)&-.*+
Lecture 1: sheet 26 / 43 Marc Uetz Discrete Optimization
[Marc Uetz - lecture notes on discrete optimization]
Complexity is the number of basic instructions that the algorithm performs until termination

45. COMPUTATION TIMESSay, we can do 2.2 · 109 operations per second (2.2 GHz)
algorithm A’s computation timea
, for tA(n) =
n log n n log n n2
n3
2n
16 ⇡0 ⇡0 ⇡0 0.002 ms 0.03 ms
64 ⇡0 ⇡0 0.002 ms 0.12 ms 266 y
256 ⇡0 0.001 ms 0.06 ms 7.6 ms 1.6 · 1060
y
4.096 ⇡0 0.02 ms 7.6 ms 31 s ???
65.536 ⇡0 0.47 ms 2 s 78 h ???
16.7 Mio ⇡0 0.2 s 35 h 68066 y ???
a
s=second, ms=1/1000 s, h=hour, y=year
Lecture 1: sheet 33 / 43 Marc Uetz Discrete Optimization
[Marc Uetz - lecture notes on discrete optimization]

46. EXAMPLE FROM MENU OPTIMIZATION
• Convergence the best method at around 10 minutes
• Although the worse method still improved, catching up would take hours
Figure 5: Temporal performance of the optimizer system
for the Firefox and Seashore case. In our test setup, 10,000
iterations takes about 10 minutes.
Cost-
About 10 minsFrom Bailly et al. UIST’13

47. CONSTRAINT SURFACE
3. Bound and acceptable point
4. Bound and unacceptable point
All four types of points are shown in Fig. 1.4.
Figure 1.4 Constraint surfaces in a hypothetical two-dimensional design space. Rao 2009

48. PERMUTATION MATRIX
P =
2
4
1 0 0
0 0 1
0 1 0
3
5
Slot 1
Slot 2
Slot 3
Element 1
Element 2
Element 3
P = {1,3,2}
Rao 2009

49. APPLICATION
correct optimization algorithms, and how to combine
as the situation may demand.
real world problem
algorithm, model, solution technique
computer implementation
analysis validation,
sensitivity analysis
numerical
methods verification
Rao 2009

50. CRITERIA: CHOOSING AN ALGORITHM
1.The type of problem to be solved
2.The availability of a ready-made computer program
3.The calendar time required for the development of a program
4.The necessity of derivatives of the functions f and gj, j = 1,2,...,m
5.The available knowledge about the efficiency of the method
6.The accuracy of the solution desired
7.The programming language and quality of coding desired
8.The robustness and dependability in finding the optimum
9.The generality of the program for solving other problems
10.The ease with which the program can be used
Rao 2009

51. SUMMARY
• Basic concepts for single criterion combinatorial
optimization
• Objective function
• Constraints
• Global vs. local optima
• Algorithm a matter of success vs. fail

52. MODEL-BASED
OPTIMIZATION OF USER
INTERFACES

53. “The Design Oracle”
Guaranteed quality
Explicit criteria
Perfect psychologist
Fast, inexpensive

54. Design Space*
Knowledge
* In reality, multidimensional

55. Results from a study
Design heuristics
Psychophysics functions
Model or simulation

56. MODEL-BASED INTERFACE
OPTIMIZATION
Designer
Generate
Evaluate
Optimizer
Model
Behavioral sciences
Social sciences
…
Computer science
Design studies

57. The Psychology of
Human-Computer Interaction
1983
!
Cognitive simulation
!
Raised operations research as a
model discipline
!
“Tortoise of accumulative science
and the hare of intuitive design”

58. GOMS
Limited applicability
Hard to use
Task definition
Must be re-run for every design

59. Formal analysis of
UID problems
A B C D E F G H I J
K L M N O P Q R
S T U V W X Y
A
A B C D E F G H I
J K L M N O P Q R
S T U V W X Y Z
A B C D E F G H I
J K L M N O P Q R
S T U V W X Y Z
Representation
of problem in code
Optimized UI
Combinatorial
optimization Predictive model
of user behavior
Literature,
Experiments

60. APPLICATION PROCEDURE
1. Choose decision variables
2. Represent the design space in code
3. Formulate the objective function
4. Obtain input values and set weights
5. Choose algorithm
6. Execute
7. Evaluate / test hypothesis

61. USES OF OPTIMIZATION IN UID
1.Find the best design for a given task
2.Find the best task for a given design
3.Identify how far from the optimum a present UI is
4.Explore design space
5.Change a UI to maximally improve it
6.Find the best design for contradictory conditions

62. BENEFITS
• Derivation of UI designs from first principles
• A problem-solving tool
• Encourages exact formulation of design task
• All assumptions can be scrutinized
• All outcomes are testable “hypotheses”

63. TYPES OF MODELS
• If-then heuristics (e.g., design heuristics)
• Verbal theories (e.g., mental models theory)
• Conceptual models (e.g., task analysis)
• Statistical models (e.g., technology adoption model)
• Analytical models (e.g., Fitts’ law, models)
• Simulation models (e.g., cognitive models)