Tutorial on Model-Based User Interface Optimization. Part III: SOLVING REAL PROBLEMS. Presented by Antti Oulasvirta, Aalto University. Presented at SICSA Summer School on Computational Interaction 2015. Note: This one-day lecture is divided into multiple parts.
VVIP Pune Call Girls Hadapsar (7001035870) Pune Escorts Nearby with Complete ...
Model-Based User Interface Optimization: Part III: SOLVING REAL PROBLEMS - At SICSA Summer School on Computational Interaction 2015
1. Optimization for User Interface Design
Antti Oulasvirta, Anna Feit
Aalto University
SICSA SUMMER SCHOOL ON COMPUTATIONAL INTERACTION
GLASGOW
JUNE 23, 2015
SOLVING REAL PROBLEMS
3. GOALS
• Learn to formulate your user interface design
problems as an optimization task
• Get more acquainted with assignment
problems:A common problem type in interface
design
• Appreciate the nature of common design
problems
4. CONTENTS
• Assignment problems
• Linear assignment, quadratic assignment
• Linear menu, hierarchical menu,
menus with groups
• Other problem types in HCI
• Transfer
• Functionality selection
• Notification scheduling
5 “mini exercises”
6. WHY PROBLEM FORMULATION
IS VITAL
• If you don’t do it,“optimization” does not make sense
• There’s no way you can avoid it: Your code does it implicitly
• You can find links to known problems and exploit their
solutions
• You can find efficient re-formulations
• Currently, this is the single most important obstacle for
optimization approaches to user interfaces
• It challenges present notions of what user interface design is
10. DEFINITION
• Assign agents to tasks
• Find a bijection with minimum total cost
• Assuming real-valued cost matrix C:
min
X
a2A
Ca,f(a)
f : A ! T
11. AS A LINEAR PROGRAM
An agent can only be
assigned once
Each task can hold only
one agent
min
X
i2A
X
j2T
C(i, j)xij
s.t.
X
j2T
xij = 1 for i 2 A,
X
i2A
xij = 1 for j 2 T
13. DEFINITION
• Given P facilities and L locations, a weight function w,
and a distance function d, find the bijection
that minimizes
Intuition:We minimize average distance among
facilities.We try to “pack” together the facilities that
are commonly used with each other
f : P ! L
X
a,b2P
wa,bdf(a),f(b)
w weight
d distance
14. FORMULATION WITH BINARY
DECISION VARIABLES
• Identify a permulation matrix X of dimensions
n x n such that
!
• Observations:
• Large problems (n > 30) may be impossible
• Generalization of TSP and therefore NP-complete
min
nX
i,j=1
nX
h,k=1
dihfjkxijxhk
f frequency
d distance
15. IN-CLASS DISCUSSION
Which of the following involve an assignment problem?
Graphical user interfaces
Consumer electronics
Automotive interfaces
Keyboards
Web user interfaces
Gestural interaction Mobile interfaces
Dialogue interfaces
18. THE SIMPLEST MENU
slot 1!
slot 2!
…!
slot n
One-dimensional (linear) menu
a.k.a. array / row facility layout problem
commands
19. DEFINE THIS DESIGN TASK
Given a frequency distribution for n commands, find an
assignment of commands to menu such that it
minimizes expected search+selection time
Command frequencies
command!
command!
…!
command
Menu
Command
Frequency
You have to make an assumption
about search+selection time
20. ONE SOLUTION
• Can be solved with the linear program
min
X
i2A
X
j2T
C(i, j)xij
s.t.
X
j2T
xij = 1 for i 2 A,
X
i2A
xij = 1 for j 2 T
Probability x Search time
Search time from the SDP model
Cockburn et al. CHI’07]
election times of each item i weighted by their probability
T =
n
Â
i=1
piTi
Selection time Ti of item i is given by
Ti = Tdsi + Tpi
Pointing time Tpi is governed by the Mackenzie varian
aw and is
Tpi = ap + bplog2(Ai/Wi + 1)
where Ai is movement amplitude and Wi is height of item
ically organized menus). ap and bp are empirically deter
i=1
Selection time Ti of item i is given by
Ti = Tdsi + Tpi
Pointing time Tpi is governed by the Mackenzie
law and is
Tpi = ap + bplog2(Ai/Wi + 1)
where Ai is movement amplitude and Wi is height
tically organized menus). ap and bp are empirically
constants.
Decision/Search time Tdsi is linear interpolation
and visual search:
law and is
Tpi = ap + bplog2(Ai/Wi + 1)
where Ai is movement amplitude and Wi is height of
tically organized menus). ap and bp are empirically d
constants.
Decision/Search time Tdsi is linear interpolation b
and visual search:
Tdsi = (1 ei)Tvsi + eiThhi
where ei[0..1] is an expertise factor ei = 1 1/t i w
number of repetitions with i. As the user becomes m
with a menu, there is a shift from serial visual search
among competing elements Thhi.
and visual search:
Tdsi = (1 ei)Tvsi + eiThhi
where ei[0..1] is an expertise factor ei = 1 1/t i where
number of repetitions with i. As the user becomes more ex
with a menu, there is a shift from serial visual search Tvsi t
among competing elements Thhi.
Visual search time Tvsi is assumed to be linear function
number of n:
Tvsi = bvsn + avs
Hick-Hyman decision time Thhi is dependent on the entrop
item Hi = log2(1/pi). Here, pi is the probability of the ite
the history of selections, pi = ti/tt where tt is the number
selections in the menu. Now, Thhi is given by:
where ei[0..1] is an expertise factor ei = 1 1/t i where ti is t
number of repetitions with i. As the user becomes more experie
with a menu, there is a shift from serial visual search Tvsi to dec
among competing elements Thhi.
Visual search time Tvsi is assumed to be linear function of th
number of n:
Tvsi = bvsn + avs
Hick-Hyman decision time Thhi is dependent on the entropy of t
item Hi = log2(1/pi). Here, pi is the probability of the item in
the history of selections, pi = ti/tt where tt is the number of to
selections in the menu. Now, Thhi is given by:
Thhi = bhhHi + ahh
where ahh and bhh are empirically determined constaints.
The model has been validated for a few different types of me
including linear, morphing, split, and rectangular menus. Figur
Fast to solve but trivial: Organizes the elements
in decreasing order of frequency
21. T = f(L,T, H, P)
Selection time
Menu length
Target Position
Practice
The model is based on Fitt’s law and Hick-Hyman law
Entropy
THE SEARCH-DECIDE-POINT MODEL
23. DEFINE THIS DESIGN TASK
Given a bigram distribution for n commands, find an
assignment of commands to a menu hierarchy such that
it minimizes expected search+selection time
Bigrams
Menu system
Command
Command
You have to make an
assumption about the cost of
searching submenus
24. ONE SOLUTION
• k submenus and n items can be organized in
ways into a tree
• A QAP, but with much larger matrices for cost
(here:A) and permutations P(n, m)
!
• Practically impossible to solve to optimality when
n >20
(n + k)n+k+2
= [csr] is an m x m matrix and P = [pij] is an n x n
P(n,m) is the set of all 1–1 (injective) mappings
…, n} to {1, …, m}, corresponding to the place-
the letters in these locations; and, !(!) is the loca-
etter i in the current solution ! ∈ !(!, !). Now,
(!) is the cost contribution of assigning letter i to
!(!) and letter j to !(!). This problem can be for-
with a quadratic objective function [33]: Let xij be
variable that is 1 if letter i is assigned to location j,
herwise. Now, Eq. 4 is equivalent to
min !!"!!"!!"!!"
!
!!!
!
!!!
!
!!!
!
!!!
,!!!!!!!!!!!!!!!!!!!!(5)!
" is the cost of using keyslot l after keyslot k.
gn of menu systems is a QAP, too: n commands are
to slots in k submenus that form a tree. Three dif-
must be addressed for treating menu system design
The Ant
Althoug
rapidly (
search g
particula
(ACO)
colony f
produce
Impleme
To our k
We use
ensures
upper an
pheromo
add pher
Bailly et al. UIST’13
p probability
a cost (search time)
26. DEFINE A DESIGN TASK
Given a bigram distribution for n commands, find an
assignment of commands to groups in a menu such that
it minimizes expected search+selection time
Relatedness distribution Grouped menu
Command
Command
You have to make an
assumption about how groups
affect search+selection time
27. SOLUTION
• Add another objective to the QAP:
min
nX
i,j=1
nX
h,k=1
dihfjkxijxhk +
nX
i,j=1
nX
h,k=1
dihrjkxijxhk
Minimizes distance
among related
commands
Alternative: You can
formulate this as an additional
constraint
To decide where to put a
divider, you need some
heuristic to do that
28. SUMMARY
• Assignment tasks are of two kinds
• One-shot cost (linear)
• Transaction cost (quadratic)
• Powerful enough for many spatial design problems
• We considered only single objective: performance.
We will return to the multi-objective case
37. DEFINITION AS AN ASSIGNMENT TASK
• The objective minimizes distances among assigned
elements according to a given relatedness
distribution (r)
!
!
• O-oh!This does not work!
min
nX
i,j=1
nX
h,k=1
dihrjkxijxhk
38. ANOTHER TRY
• Have to consider both source and target domain
probabilities to make this work
max
X
a2A
X
b2A
X
s2T
X
t2T
pa,bps,txa,sxb,t
subject to
X
s2t
xa,s 1 8a 2 A
X
a2A
xa,s = 1 8s 2 T
40. EXAMPLE FUNCTIONALITIES
File Creation and organization
Close Close the current file
Create notebook Create notebooks to organize the notes
Delete file Delete a file
Delete notebook Delete notebook *
Exit Exit the application
Export Export the current file, e.g. to pdf
Move note to notebook Move a selected note to one of the notebooks.
New Create a new file
Open choose a previously created file to be opened
Open containing folder Open folder that contains the document.
Recently opened choose a recently opened file to be opened
Reload Reload the current file
Rename Rename file.
Rename notebook
Save Save the current file
Structural Elements
Bullet size Change size of bullets.
Bullet type Change bullet type.
Bullets Start bullet list.
Change number size
Change start number
Insert blank page
Insert date and time Insert date and time into the text.
Insert horizontal line
Number style Change style of numbering, e.g. 1. or 2) or iii or IV, etc.
Numbering Start numbered list.
Page break
Page layout
Line number color Customize color of line numbers
Line number continuation Continue line numbers or restart, e.g. at new page
Line number font Customize font of line numbers
Line number size Customize size of line numbers
Line numbers Show or hide numbering of lines
Margins Define the margins of the page (top, bottom, right, left)
Orientation Change the orientation of the page (landscape or portrait)
Page size
Change the page size, e.g. to predefined templates (A4, A5, ...)
or to a custom size.
Text columns Define the number of columns of the layout (e.g. 2-column text)
Link and share
Link to calendar entry Link note to event in calendar
Publish online Publish note online, e.g. on blog, facebook, etc.
Put on homescreen Put the note in form of a widget on desktop or homescreen
Share share notes via another application, e.g. Email, messenger, etc.
View
Close all windows In case of multiple open documents.
Compare documents Compare two documents and show differences.
Full screen Show in full screen mode, hide controls.
Line wrap Wrap text at the end of a line
Move document to other view In case of multiple documents per window (e.g. tabs) or split view.
Save all windows Save all open document.
Show grid
Show ruler
Split view Split view to show two documents at once.
Synchronize scrolling Synchronize scrolling of two open documents for better comparison.
Zoom Zoom in or out the document
Printing
Choose printer Choose the printer for printing the document
Number of copies Change the number of copies to be printed.
Print current page Print only the currently viewed page
Print custom range Define custom range of pages to be printed.
Print document
Print multiple pages per sheet
Print selection Print only selected text
Printer settings
Change setting of printer (e.g. color, quality, duplex etc.)
as given by operating system.
Search printer Search for a connected printer not displayed.
Controls
Copy
Cut
Delete element Delete selection
Find Find a certain text in the document
Mark all text
Mark text
Paste Paste copied or cut text.
Redo Redo last action.
Replace Replace selection by text.
Undo Undo last action.
Input
Change encoding Change encoding of text (e.g. UTF-8)
Draw freehand
Draw shape e.g. rectangle, line, triangle, etc.
Fill color Change fill color of shape
Handwriting recognition Enter text by recognition of handwriting
Insert link
Insert picture
Insert video
Line color Change line color of drawing.
Line style e.g. pen, pencil, crayon etc.
Record audio
Sort elements Sort selected elements alphabetically or numerically
Spell check Check for correct spelling
Type text Entering text via keyboard
Voice recognition Enter text by voice input.
Text style options
Align text Change text alignment, e.g. left, right, middle, etc
Character spacing Change space between characters
Font color Change color of font
Font size Change font size
Indent Change text indent.
Line spacing Change space between text lines
Paragraph spacing Change space between paragraphs.
Text background color Change color of text background.
Text font Change font of text
Text style Change style of text, e.g. bold, italic, underline, etc.
Table
Add column to table
Add row to table
Cell color Custom color of table cell
Cell margins Customize text margins of a cell in the table
Column size Change size of each column
Create table Insert a table with defined number of rows and columns
Merge cells Merge two or more cells into one.
Remove column from table
Remove row from table
Row size Change size of each row
Split cell Split a cell in the table into two.
Table border Customize table border lines
Text alignment in table
Notes application (106 functionalities)
42. DEFINE A DESIGN TASK
Given a set of functionalities, as well as their estimated
usefulness values & usability costs, how to select the best
subset of functionalities?
43. THE NEXT RELEASE PROBLEM
• A software engineering problem
the next release of software.
ormulated as 0/1 knapsack problem: Let x1, ...xn be 0/1 va
ence/presence of requirements in software. The goal is
cted features to customers wi while keeping the total c
max
nX
i=1
wixi
s.t.
nX
i=1
cixi B
xi 2 {0, 1}
at address a set of customers with different requiremen
xpressing dependencies in the implementation of featur
mplemented without feature A) [Bagnall et al. 2001].
Knapsack formulation
A burglar breaks into a jewelry store
He carries a backpack to store the jewels
Backpack has limited capacity, jewels have a monet
Goal: put in the backpack the combination of items th
max total value
Let’s design a heuristic
WS 2011
But: Inadequate for UI
designers
45. INTERVIEW: RESULTS
Functionality selection is
•Embedded in user-centered design process
•iterative, corrective, exploratory
•cooperative and collaborative, multiple stakeholders
•Informed by multiple types of data and outputs
•user profiles, scenarios, sketches, prototypes, wireframes
•but designers often focus mainly on one scenario & user profile
•Addressing multiple design objectives
•usability, usefulness, user experience, profitability, implementability,
learnability, …
46. OBJECTIVE FUNCTION
erms of four objectives called USEP: usefulness U, satisfa
profitability P. These terms form the core of the objectiv
ut our work, supplementing it with case-specific weights, o
hen needed.
maxG = !U U + !SS + !EE + !P P
denote a weight given to an objective,
P
!2⌦ ! = 1.0.
wing, we define each term. Throughout, we denote a function
and the full set of selected functionality . By ”normaliza
al bar, we refer to scaling a variable to range [0, 1].
s
e usefulness of a design U as the sum of usefulness u and
ch selected functionality:
¯U =
1
2
(
X
2
¯u + D )
P. These terms form the core of the objective function we
upplementing it with case-specific weights, objectives, and
axG = !U U + !SS + !EE + !P P (4)
t given to an objective,
P
!2⌦ ! = 1.0.
each term. Throughout, we denote a functionality selected
set of selected functionality . By ”normalization”, marked
r to scaling a variable to range [0, 1].
a design U as the sum of usefulness u and dependency
ctionality:
¯U =
1
2
(
X
2
¯u + D ) (5)
2 [0, 1] and
D =
P
2
P
⌧2 ,⌧
P
i
P
j i,j
(6)
j2[0,1] are directional dependency scores defined by the de-
Usefulness
Dependency
the set of functionality that maximizes ”goodness” G. We
jectives called USEP: usefulness U, satisfaction S, ease-
. These terms form the core of the objective function we
pplementing it with case-specific weights, objectives, and
xG = !U U + !SS + !EE + !P P (4)
t given to an objective,
P
!2⌦ ! = 1.0.
each term. Throughout, we denote a functionality selected
et of selected functionality . By ”normalization”, marked
to scaling a variable to range [0, 1].
a design U as the sum of usefulness u and dependency
ctionality:
¯U =
1
2
(
X
2
¯u + D ) (5)
2 [0, 1] and
D =
P
2
P
⌧2 ,⌧
P
i
P
j i,j
(6)
2[0,1] are directional dependency scores defined by the de-
Satisfaction
3. Users value the loss of an important function more negatively than its inclusion in a design.
tisfaction
fine user satisfaction as a nonlinear function that maps the presence/absence of
tionality to a satisfaction score. This score refers to subjective experience of the
onality, as opposed to the usefulness score. We offer two formulations.
first definition uses a linear sum:
S =
X
2
¯s (7)
mitation is that the absence of a functionality is not accounted for in the score.
second definition builds on prospect theory that predicts that the absence of an
tant functionality is appraised more critically than its presence. This is captured
following mathematical function for prospect ⇡ of item :
⇡i =
⇢ p
si ifxi = 1
2
p
si ifxi = 0
(8)
we reformulate as
Ease-of-use
Note that satisfaction can be negative if a design misses functionali
to have.
This yields a new S⇡:
S =
nX
i=1
¯⇡(i)
S is used in the objective function after normalization.
4.3. Ease-of-use
We provide two definitions for the ease-of-use of a design E. The
quicker computation but simplifies the effect of increasing function
is a decreasing function of the number of selected functionality.
Our first formulation is a
E =
X
62
c
where c is the estimated complexity of .
ACM Transactions on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: Ja
Profitability
Fig. 4. We assume the ease-of-use of an interface to decrease non-linearly with
Here, y = 1/(x 1).
Our second formulation captures the fact that ease-of-use is c
ing more and more functionalities. Including very few triviall
use. For example, designers group semantically to decrease sel
chical menu [Bailly et al. 2014]. Interface techniques for comma
functionality, toolbars, autocompletion, panels, and tool palette
effect. Ease-of-use would increase linearly only in the case the i
randomly. We also note that some interface techniques allow co
number of features. For example, hotkeys can make access t
but users typically learn only few hotkeys. While we use the re
following, we recognize that the exact shape of this function
including the skills of the design and the interface type. The re
shown in Figure 4 and given as
E =
1
| | + 1
We can also formulate this by introducing a quadratic constrai
4.4. Profitability
We define the profitability of a functionality p in terms of its
and costs c 2 R: p = v c . After normalization of p, we can
¯P =
X
2
p =
X
2
(v c )
Note that if c > v, profitability is negative. Sometimes inclu
negative in profitability is justifiable in the light of other object
48. BINARY QUADRATIC PROGRAMMING
!
• Find a binary vector of
length n to maximize
quantity q
• Global optimum can be
computed
• Fast!
LP file generator
Gurobi
Survey data (.csv)
Designers
Design exploration
! Result
50. NOTIFICATION SCHEDULING
• Assign incoming messages to a computer to one
of the statuses:
• Show immediately with a pop-up
• Push to list of notifications
• Don’t show
52. CHARACTERIZATION
• n jobs, each consisting of a chain of operations
• m machines, each handles at most one operation at a time
• Each operation must be processed during an uninterrupted
period of a given length on a given machine
• The objective is to find a schedule (an allocation of the jobs
to time intervals to machines) with minimum duration
• In static JSS, n jobs arrive at an idle shop
• In dynamic JSS, jobs arrive intermittently
54. TASK: A NOTIFICATION BROKER
A set of incoming messages is given, decide to
which modality to channel them and when to
present them.
!
Tip:What is being minimized/maximed here?
56. SUMMARY
• Design tasks can described in terms of objectives
and constraints
• Assignment tasks are common in user interface
design
• Many optimization tasks in operations research
have applications in HCI
57. SICSA SUMMER SCHOOL ON COMPUTATIONAL INTERACTION
DAY 2: OPTIMIZATION
EXAMPLE: MID-AIR GESTURE
OPTIMIZATION
[SRIDHAR ET AL. PROC CHI’15]