Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CAN COMPUTERS
Antti Oulasvirta
Associate Professor
userinterfaces.aalto.fi
design?
Hands-on demo of
Sketchplorer at CHI’16
...
IS A COMBINATION OF 

DESIGN THINKING AND
COMPUTATIONAL THINKING
COMPUTATIONAL INTERFACE DESIGN
This talk shows how algori...
26/02/16 08:38How Many of Your Daily Tasks Could Be Automated?
OPERATIONS
How Many of Your Daily Tasks Could Be
Automated?...
HOW MIGHT DESIGN BE AUTOMATED?
26/02/16 08:38How Many of Your Daily Tasks Could Be Automated?
Page 1 of 4https://hbr.org/2...
WHAT IF…
INTERFACE DESIGN WAS ENGINEERING DESIGN?
POTENTIAL BENEFITS
IMPROVED QUALITY AND RELIABILITY
GUARANTEES FOR OUTCOMES
SOLUTIONS TO HARD PROBLEMS
SCRUTINIZABLE PROCE...
AUGUST DVORAK
The first to use empirical findings to
drive the design of keyboard layouts
•Designed DSK (Dvorak Simplified...
EARLY ATTEMPTS TO OPTIMIZE INTERFACES
1983
HCI
Card, Newell, & Moran
suggested that HCI models
should drive UI design.
Led...
Design Issues, Volume 17, Number 4, 2001, pp. 44-50.
CAN A MACHINE DESIGN?
NIGEL CROSS
Department of Design and Innovation...
Harold Thimbleby (1998)
proposed formal definition of
interactive devices.
•UI specifications would drive
the generation o...
COORDINATED
EFFORT IS NEEDED
THE ELEMENTS ARE THERE
We need a method that is (i) generalizable, (ii) produces
good outcome...
can computers design?

antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS ...
can computers design?

antti oulasvirta
“Math is easy; design is hard.” — Jeffrey Veen
FOUNDATION / DESIGN AS SEARCH
HERBERT A. SIMON
Complex problem-solving activities such
as chess can be analyzed as search in a
problem space.
This found...
DESIGN AS SEARCH
• The optimum is the design
vector X with the lowest value
• Constraints can be added
• E.g., g(X) < 1
Fi...
Prohibitively large design spaces
Multiple design objectives
Multiple user groups and tasks
A hierarchical menu with 50 it...
MULTI-OBJECTIVE OPTIMIZATION
HOW COMPUTER UNDERSTANDS DESIGN
max
d2D
X
c2C
X
u2U
X
t2T
X
o2O
wcwuwtwoGc,u,t,o(d)
Find the ...
BECAUSE THE
MATH IS HARD
DESIGN IS HARD
Interface design entails very large, high-dimensional
search spaces and complex ev...
can computers design?

antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS ...
can computers design?

antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
AS PHYSICS IS TO ENGINEERING
OPTIMIZATION
PSYCHOLOGY IS TO INTERFACE
OPTIMIZATION…
The use of predictive models allows con...
MODEL-BASED INTERFACE OPTIMIZATION
OVERVIEW
Designer
Generate
Evaluate
Optimizer
Model
Psychology
Neurosciences
Social sci...
Black-box 

methods 

(e.g., simulated 

annealing)
Optimizer
Fitts’ lawBigrams
Objective function
literature. Our approac...
Figure 12. The Metropolis keyboard (43.1 wpm)
encapsulated the circle it replaced and filled the gaps
between the circles,...
CODE CAN EXPRESS DESIGN KNOWLEDGE
• If-then rules (e.g., design heuristics)
• Mechanistic models (e.g., mental models theo...
Model of Visual Search and Selection Time in Linear Menus
Gilles Bailly1
Antti Oulasvirta1
Duncan P. Brumby2
Andrew Howes3...
OPTIMIZATION METHODS
Questions we can ask
1.The best possible design
2.All designs x% from the optimum
3.Diverse but good ...
GUARANTEE A GREATER-
THAN-ZERO CHANCE OF
FINDING THE OPTIMUM
DESIGN
OPTIMIZATION METHODS
KEYBOARD LAYOUT OPTIMIZATION
We can optimise
keyboard layouts for any
terminal and any given
language. We can find
the opt...
Antti, you‘re right.
But … You’re not
dreaming BIG enough
can computers design?

antti oulasvirta
can computers design?

antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMISATION
FOUNDATION / DESIGN AS SEARCH
RESULTS ...
can computers design?

antti oulasvirta
RESULTS / BEYOND KEYBOARDS
Cohen-Or 2006Itti & Koch 1998 Rosenholtz 2007
MODELS WE USE (SELECTED)
Saliency Clutter perception Color harmony
Fitts 195...
WEB PAGE DESIGN
Heading goes here
www.firstlink.com
www.secondlink.io
www.thirdlink.org
Button Button Button
Important
Orig...
SINGLE-OBJECTIVE RESULTS DO NOT MAKE SENSE
BBB
Heading goes here
www.firstlink.com
www.secondlink.io
www.thirdlink.org
Butt...
WEB PAGE DESIGN
MULTI-OBJECTIVE RESULT
Button
Button
Heading goes here
Button
www.firstlink.com
www.secondlink.io
www.third...
can computers design?

antti oulasvirta
LEGEND
Optimize for
Marker size
Marker color
Aspect ratio
Transparency
Correlation...
APPLICATION MENU
CASE WINDOWS PHONE
Baseline
All All except first
Meanselectiontime(ms)
2000
2500
3000
3500 Optimised
Base...
Title
ATTENTION-OPTIMIZED SKIM READING
We determine
dynamically the content
and number of static
highlights using principl...
SCATTERPLOT OPTIMIZATION
Matplotlib default design
Optimize for
Marker size
Marker color
Aspect ratio
Transparency
Correla...
GAME LEVEL DESIGN
USING HUMAN PERFORMANCE MODELS
Optimize for
Pillar width
Pillar gap
Timing accuracy
EASY MEDIUM DIFFICULT
Real gaming performance by a colleague
Optimize for
Pillar width
Pillar gap
Timing accuracy
GAME LEV...
can computers design?

antti oulasvirta
IN-AIR GESTURE DESIGN
MODEL ACQUISITION
TARGET SELECTION TASK
Experimental setup
formance achievable after practice. Given a target of width
and...
48
Example design: Chorded text entry
[Proc. CHI’15]
We can now address interfaces
where primary emphasis on
perception, motor control, and
ergonomics. Our examples span
input...
FUNCTIONALITY SELECTION
can computers design?

antti oulasvirta
FUNCTIONALITY SELECTION
em independent of each other; (2) link each term to easily...
EMPIRICAL EVALUATION
Professional designers found
the optimized functionality sets
as good or better than the ones
they de...
can computers design?

antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS ...
can computers design?

antti oulasvirta
DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER
“tools might enforce user interfaces th...
DesignEvaluate
INITIALIZING INTERFACE OPTIMIZATION
designer
computer

scientist
social

scientist
Design Task
Optimizer
Pr...
DesignEvaluate
DESIGN TOOLS HIDE ALGORITHMIC COMPLEXITY
designer
Optimizer
Design Task
Heuristics

Models
Interactive 

op...
FULL CONTROL CONCEPT
Optimize for
Menu structure
Position of item
Selection time
Full interactive control
of design object...
“SKETCHPLORER”
ZERO-EFFORT INTERACTIVE OPTIMIZATION
Hands-on demo of
Sketchplorer at CHI’16
in San Jose; May 2-7
ZERO-EFFORT CONTROL CONCEPT
Optimize for
Item positions
Item colors
Motor performanceItem sizes
Alignment Grid quality
No ...
can computers design?

antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS ...
can computers design?

antti oulasvirta
CONCLUSION / A FUTURE
AS DESIGN FROM FIRST PRINCIPLES
COMPUTATIONAL INTERFACE DESIGN
Predictive models (principles of interaction) allow
algorit...
To improve the
positive effect that
optimization can have
on design quality, we
need to better
understand creativity
in hu...
Design
Superiority
An organization
emerges that
can present
guarantees for
their designs
Bulk Design
Automated
A design se...
COMPUTERS CAN design.
Funding: European Research Council Staring Grant (contract 637991); Academy of Finland
research proj...
Can Computers Design? Presented at interaction16, March 2, 2016, Helsinki by Antti Oulasvirta / Aalto University
Can Computers Design? Presented at interaction16, March 2, 2016, Helsinki by Antti Oulasvirta / Aalto University
Can Computers Design? Presented at interaction16, March 2, 2016, Helsinki by Antti Oulasvirta / Aalto University
Upcoming SlideShare
Loading in …5
×

Can Computers Design? Presented at interaction16, March 2, 2016, Helsinki by Antti Oulasvirta / Aalto University

1,723 views

Published on

This talk outlines how algorithms may change the way user interfaces are designed. I present foundations and results across a range of user interface types.

Published in: Science
  • It's genuinely changed my life. I have been sleeping in the spare room for 4 months - and let's just say my sex life had become pretty boring! My wife and I were becoming strangers living in the same house. Thanks to your strategies, I am now back in our bed and the closeness and intimacy have returned. Thank you so much for taking the time to put all this together. It has genuinely changed my life. ➤➤ http://ishbv.com/snoringno/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Intimacy has never been so much fun! Buy the clinically proven men's natural supplement that helped guys increase satisfaction by 71.43%! ■■■ https://bit.ly/30G1ZO1
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Secrets To Making Up These secrets will help you get back together with your ex. ■■■ http://scamcb.com/exback123/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Can Computers Design? Presented at interaction16, March 2, 2016, Helsinki by Antti Oulasvirta / Aalto University

  1. 1. CAN COMPUTERS Antti Oulasvirta Associate Professor userinterfaces.aalto.fi design? Hands-on demo of Sketchplorer at CHI’16 in San Jose; May 2-7
  2. 2. IS A COMBINATION OF 
 DESIGN THINKING AND COMPUTATIONAL THINKING COMPUTATIONAL INTERFACE DESIGN This talk shows how algorithms can use predictive models to design user interfaces. If successful, this program has significant implications to interaction design practice.
  3. 3. 26/02/16 08:38How Many of Your Daily Tasks Could Be Automated? OPERATIONS How Many of Your Daily Tasks Could Be Automated? by Michael Chui, James Manyika, and Mehdi Miremadi DECEMBER 14, 2015 A sea of ink and hours of video have been dedicated to the looming battle between humans and machines, often concluding that the machines win and human workers lose. It is a nearly irresistible narrative, which has been captured in books such as Martin Ford’s Rise of the Robots and The Second Machine Age, by Erik Brynjolfsson and Andrew McAfee. It has also has inspired scholarship by academics such as Carl Benedikt Frey and Michael Osborne of Oxford University, who estimate that 47% of occupations in the United States could be automated within 20 years, and David Autor of MIT, who argues that the ability of machines to take on human jobs is vastly overstated. At McKinsey, we are conducting our own research into the impact of automation on jobs and organizations and recently published a brief paper previewing some of our findings. The most important insight has been that it is far more useful to think about the activities that can be automated rather than entire occupations—handing over discrete bits to machines, like pulling out the most useful data from an analyst’s report, generating a report on the latest sales figures, or moving products around a warehouse. Smart machines have already demonstrated the ability to see patterns in information, understand what humans are saying (answering a question like “show me where sales rose the most last week”), and manipulate physical objects. Once these capabilities are applied to various work activities, few occupations or organizations will remain untouched. Let’s look at what happens if we approach automation at the level of activities, rather than occupations. Take, for example, the role of a  Want more out of HBR.org? Sign in to get more free content month and build your own personal library on HBR.org. O K , T H A N K S
  4. 4. HOW MIGHT DESIGN BE AUTOMATED? 26/02/16 08:38How Many of Your Daily Tasks Could Be Automated? Page 1 of 4https://hbr.org/2015/12/how-many-of-your-daily-tasks-could-be-automated OPERATIONS How Many of Your Daily Tasks Could Be Automated? by Michael Chui, James Manyika, and Mehdi Miremadi DECEMBER 14, 2015 A sea of ink and hours of video have been dedicated to the looming battle between humans and machines, often concluding that the machines win and human workers lose. It is a nearly irresistible narrative, which has been captured in books such as Martin Ford’s Rise of the Robots and The Second Machine Age, by Erik Brynjolfsson and Andrew McAfee. It has also has inspired scholarship by academics such as Carl Benedikt Frey and Michael Osborne of Oxford University, who estimate that 47% of occupations in the United States could be automated within 20 years, and David Autor of MIT, who argues that the ability of machines to take on human jobs is vastly overstated. At McKinsey, we are conducting our own research into the impact of automation on jobs and organizations and recently published a brief paper previewing some of our findings. The most important insight has been that it is far more useful to think about the activities that can be automated rather than entire occupations—handing over discrete bits to machines, like pulling out the most useful data from an analyst’s report, generating a report on the latest sales figures, or moving products around a warehouse. Smart machines have already demonstrated the ability to see patterns in information, understand what humans are saying (answering a question like “show me where sales rose the most last week”), and manipulate physical objects. Once these capabilities are applied to various work activities, few occupations or organizations will remain untouched. Let’s look at what happens if we approach automation at the level of activities, rather than occupations. Take, for example, the role of a marketing manager or executive in a consumer products company. A marketing manager is well educated and well compensated—he or she  Want more out of HBR.org? Sign in to get more free content month and build your own personal library on HBR.org. O K , T H A N K S loading activities to machines be used most effectively? When you’re bringing in automation you need to think about two kinds of payoff— returns you get by using machines rather than labor for activities (investments in automation can generate benefits worth three to ten times the cost, we estimate, much of it from better performance rather than reductions in labor costs), plus the value derived from activities that employees carry out in the time that formerly used for work that is now automated. In the case of the marketing manager, this could mean more time to work on new product ideas, supervise direct reports, collaborate with managers in other functional areas, or develop new strategies.
  5. 5. WHAT IF… INTERFACE DESIGN WAS ENGINEERING DESIGN?
  6. 6. POTENTIAL BENEFITS IMPROVED QUALITY AND RELIABILITY GUARANTEES FOR OUTCOMES SOLUTIONS TO HARD PROBLEMS SCRUTINIZABLE PROCESS
  7. 7. AUGUST DVORAK The first to use empirical findings to drive the design of keyboard layouts •Designed DSK (Dvorak Simplified Keyboard) • Demonstrated 5-10% improvement in typing speed over Qwerty “Typewriting Behavior” 1936 PIONEER OF RATIONAL DESIGN
  8. 8. EARLY ATTEMPTS TO OPTIMIZE INTERFACES 1983 HCI Card, Newell, & Moran suggested that HCI models should drive UI design. Led a revolution in HCI modeling 1993 Human Factors Donald L. Fisher: “A quiet revolution is in progress in the field of human factors.” Demonstrated in the design of control panels 1977 Optimization Rainer Burkard defined mathematical design of typewriter layouts (QAP) Theorists had no real interest in applications
  9. 9. Design Issues, Volume 17, Number 4, 2001, pp. 44-50. CAN A MACHINE DESIGN? NIGEL CROSS Department of Design and Innovation Faculty of Technology, The Open University Milton Keynes MK7 6AA, UK Abstract: One strand of my research has been concerned with the computer as a design tool; but a second strand has been concerned with design computing as a research tool for improving our understanding of the design process. Some of this latter research is based on the simulation of computer behavior by human beings - a reversal of the more usual approach - and some is based on comparisons of computational models with human design behavior. Despite recent doubts expressed by some authors, I suggest that the question, ‘Can a machine design?’ is still a useful question to ask. Introduction Asking ‘Can a machine design?’ is similar to asking ‘Can a machine think?’ The answer to the latter question seems to be, ‘It all depends on what you mean by “think”.’ Alan Turing (1950) attempted to resolve the question by his ‘Turing Test’ for artificial intelligence - if you could not distinguish, in a blind test, between answers to your questions provided by either a human being or a machine, then the machine could be said to be exhibiting intelligent behavior, i.e. ‘thinking’. In some of my research related to computers in design, I have used something like the Turing Test in reverse - getting human beings to respond to design tasks as though they were machines. There have been various intentions behind this strategy. One intention has been to simulate computer systems that do not yet exist; another has been to try to shed light on what it is that human designers do, by interpreting their behavior as though they were computers. My assumption throughout has been that asking ‘Can a machine design?’ is an appropriate research strategy, not simply for trying to replace human design by machine design, but for better understanding the cognitive processes of human design activity. However, this assumption has been challenged recently. In this paper I will first review some of my research, and then return to this challenge. For computers to design, we should first understand how designers design
  10. 10. Harold Thimbleby (1998) proposed formal definition of interactive devices. •UI specifications would drive the generation of manuals, helps etc. •Could not generate designs, but compare them Ranjitha Kumar et al. (2013) proposed WebZeitgeist, a pattern miner that suggests user interfaces to queries. RECENT ATTEMPTS IN HCI UI Specification Design Mining top nav bar layout query large text block large header Figure 13. Five of the 20 search results for the three-part DQL layout query visualized on the left. The query, which executed in 2.1 minutes, returns pages that share a common high-level layout, but exhibit different designs. Machine Learning Webzeitgeist also enables a new kind of design-based ma- chine learning. For the first time, applications can stream structured visual descriptors for page elements from a cen- tral repository. Moreover, Webzeitgeist’s extensible architec- ture allows new data to be collected and integrated with the repository for supervised learning applications, for instance comparison, Figure 15 shows nearest-neighbor results for the top query in Figure 14 using only the vision-based GIST de- scriptors. While these elements are visually reminiscent of the query, they bear little structural or semantic relation to it. DISCUSSION AND FUTURE WORK
  11. 11. COORDINATED EFFORT IS NEEDED THE ELEMENTS ARE THERE We need a method that is (i) generalizable, (ii) produces good outcomes, (iii) is compatible with design practice.
  12. 12. can computers design?
 antti oulasvirta METHOD / MODEL-BASED INTERFACE OPTIMIZATION FOUNDATION / DESIGN AS SEARCH RESULTS / BEYOND KEYBOARDS CONCLUSION / A FUTURE DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER
  13. 13. can computers design?
 antti oulasvirta “Math is easy; design is hard.” — Jeffrey Veen FOUNDATION / DESIGN AS SEARCH
  14. 14. HERBERT A. SIMON Complex problem-solving activities such as chess can be analyzed as search in a problem space. This foundational concept impacted computer science, design, and cognitive sciences. Problem-solving as search “THE SCIENCES OF THE ARTIFICIAL”
  15. 15. DESIGN AS SEARCH • The optimum is the design vector X with the lowest value • Constraints can be added • E.g., g(X) < 1 Find X = 8 >>< >>: x1 x2 · · · xn 9 >>= >>; which minimizes f(X) Design variable A Variable B Objective
  16. 16. Prohibitively large design spaces Multiple design objectives Multiple user groups and tasks A hierarchical menu with 50 items can be ordered in 10
  17. 17. MULTI-OBJECTIVE OPTIMIZATION HOW COMPUTER UNDERSTANDS DESIGN max d2D X c2C X u2U X t2T X o2O wcwuwtwoGc,u,t,o(d) Find the design that maximises the weighted sums of… …the contexts
 of use …the user 
 groups …the user tasks …the design objectives
  18. 18. BECAUSE THE MATH IS HARD DESIGN IS HARD Interface design entails very large, high-dimensional search spaces and complex evaluation functions.
  19. 19. can computers design?
 antti oulasvirta METHOD / MODEL-BASED INTERFACE OPTIMIZATION FOUNDATION / DESIGN AS SEARCH RESULTS / BEYOND KEYBOARDS CONCLUSION / A FUTURE DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER
  20. 20. can computers design?
 antti oulasvirta METHOD / MODEL-BASED INTERFACE OPTIMIZATION
  21. 21. AS PHYSICS IS TO ENGINEERING OPTIMIZATION PSYCHOLOGY IS TO INTERFACE OPTIMIZATION… The use of predictive models allows controlling the design process, scrutinizing assumptions, and explaining the outcomes.
  22. 22. MODEL-BASED INTERFACE OPTIMIZATION OVERVIEW Designer Generate Evaluate Optimizer Model Psychology Neurosciences Social sciences Computer science Design studies
  23. 23. Black-box 
 methods 
 (e.g., simulated 
 annealing) Optimizer Fitts’ lawBigrams Objective function literature. Our approach is directly usable in the available IP solvers such as CPLEX and Gurobi. BACKGROUND: THE LETTER ASSIGNMENT PROBLEM Given n letters and n keyslots, the task of the letter assign- ment problem is to minimize the average cost ck` of selecting letter ` after k weighted by the bigram probability pk`: min X k X ` pk` · ck` (1) It is an instance of the Koopmans-Beckman Problem [15, 6], which is NP-hard. As we will see later, this problem can be modeled with a quadratic function on binary variables. Thus, it belongs to a broad class of problems called quadratic as- signment problem (QAP) where the goal is to minimize the total pair-wise interaction cost [22, 24]. Except for a few papers using heuristic cost functions (e.g., [9]), the Fitts-bigram energy is used (see also [4]). Here, the OVERVIEW We formula lem in order and-bound. crete optimi lem was am text [21]. W present tech A detailed r per. The int et al. [18]. The basic id tions and in Integer Prog solution that gers to the v sibility Prob Task n!## Open# Save## Save#as...# Close# ...# About# n!# A B C D E F G H I J K L M N O P Q R S T U V W X Y A A B C D E F G H I J K L M N O P Q R S T U V W X Y Z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Letter assignment KEYBOARD LAYOUT OPTIMIZATION EXAMPLE
  24. 24. Figure 12. The Metropolis keyboard (43.1 wpm) encapsulated the circle it replaced and filled the gaps between the circles, making more efficient use of the total space. The performance of this layout is 43.1 wpm2 based on the Mayzner and Tresselt digraph table. This is over 40% improvement over QWERTY and more than 10% q m v u p k z xy w l d b cst i r o ng j a sp Figure 9. Early stage of Metropolis random walk. The system is at a high energy state, moving towards lower energy state. [Zhai et al. 2001] t = a + b log2(D/W + 1) Simulated annealing EXAMPLE: THE METROPOLIS KEYBOARD
  25. 25. CODE CAN EXPRESS DESIGN KNOWLEDGE • If-then rules (e.g., design heuristics) • Mechanistic models (e.g., mental models theory) • Conceptual models (e.g., task analysis) • Regression models (e.g., Fitts’ law) • Statistical models with structure (e.g., TAM) • Simulations (e.g., ACT-R) PREDICTIVE POWER IS THE PRIORITY
  26. 26. Model of Visual Search and Selection Time in Linear Menus Gilles Bailly1 Antti Oulasvirta1 Duncan P. Brumby2 Andrew Howes3 1 Max Planck Institute for Informatics and Saarland University 2 UCL Interaction Centre, University College London 3 School of Computer Science, University of Birmingham ABSTRACT This paper presents a novel mathematical model for visual search and selection time in linear menus. Assuming two visual search strategies, serial and directed, and a pointing sub-task, it captures the change of performance with five fac- tors: 1) menu length, 2) menu organization, 3) target posi- tion, 4) absence/presence of target, and 5) practice. The novel aspect is that the model is expressed as probability density distribution of gaze, which allows for deriving total selec- tion time. We present novel data that replicates and extends the Nielsen menu selection paradigm and uses eye-tracking and mouse tracking to confirm model predictions. The same parametrization yielded a high fit to both menu selection time and gaze distributions. The model has the potential to im- prove menu designs by helping designers identify more ef- fective solutions without conducting empirical studies. Author Keywords Linear menus; User Performance; Mathematical predictive models; Visual search; Eye-tracking. ACM Classification Keywords H.5.2. Information Interfaces and Presentation (e.g. HCI): Miscellaneous INTRODUCTION Menus are used on numerous applications and systems for presenting, organizing, and selecting commands. However, designers reportedly struggle with menu design [5]. While many research papers focus on novel interaction techniques (e.g. [38]), and others on human factors (e.g. [6, 9, 32, 33, 36]), surprisingly few develop predictive models of users’ performance with menus [7, 8, 10, 12, 14, 16, 21, 28]. Pre- dictive models are an efficient way of encapsulating scientific knowledge. They synthesize phenomena that are typically fragmented across several studies, and they capture subtle in- teractions among multiple factors in an empirically verifiable form. They can help designers to design more efficient menus by informing choices without expensive empirical studies. Once matured enough, models can be incorporated into in- teractive design tools [5, 16, 25]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full cita- tion on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- publish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. CHI’14, April 26–May 1, 2014, Toronto, Canada. Copyright c 2014 ACM ISBN/14/04...$15.00. http://dx.doi.org/10.1145/2556288.2557093 Models of menu performance can be divided into two cate- gories: cognitive simulations and mathematical models. Sim- ulation models such as ACT-R models [3, 8] and EPIC mod- els [19, 21, 27] explain the progression of search by reference to underlying cognitive processes, such as perception, atten- tion, and memory. Mathematical models such as the SDP model [12, 14] are nonlinear regression equations that pre- dict selection time as a function of external factors such as menu length, target position and practice. These models ex- pose fewer details about the process but may be less complex and more straightforward to apply. Our goal is to advance the scope and validity of mathematical models of linear menus, a widely used menu type in desk- top applications and consumer electronics. We assume that tasks, including both searching and pointing, are performed using one of a number of strategies. In practice, the strat- egy space for perceptual/motor tasks, even very simple ones, is known to be large [23, 39] but here we explore a few key points in the space. We focus on two searching strategies: 1) Directed search makes use of knowledge, including knowl- edge of locations acquired through practice, alphabetic and semantic knowledge, to programme saccadic eye movements that jump to the vicinity of the target location or directly to its location. 2) Serial search starts with the first menu item and moves from one item to the next until the target is located or the menu is exhausted. In practice a user would combine elements of these strategies depending on the reliability of available of knowledge. For example, directed search might be combined with serial search in proportion to the reliabil- ity of location memories and the available relevance scores. In some situations location knowledge will be highly reliable and in others it will be the relevance scores that are more re- liable. Finally, 3) we also explore two pointing strategies: Find-and-point and Track-and-point. In our current model these strategies are expressed as prob- ability distributions of gaze. A distinct advantage of this model formulation is that it allows deriving both distributions for gaze points on the menu as well as selection time from the same parametrization. These predictions can be checked against eye tracking data. To parametrize and evaluate the model, we collected an ex- tensive dataset. Although the constituent processes of menu selection have been heavily studied in psychology, the menu selection task has been hard to study for the purposes of modeling. Multiple factors must be controlled, the apparatus like eye-trackers are complex, and the studies must be long enough to gauge practice effects. Our model is informed by Bailly et al. Proc. CHI’14 nput variables: target position (1-16), menu organization (unordered, alphabet- umber of previous encounters with the el is in a mathematical form that allows nu selection time and gaze distribution. at the total time to select an item con- two consecutive subtasks–search and affected by learning (Figure 1): ruitment of eye gaze and memory recall tion of the target in the menu by scan- previous simulation models [20, 21] and strategies: consists of top-to-bottom reading of h practice, allows more skipping. This a uniform distribution from the begin- nu to the target. ch consists of a direct attempt at moving p of the target. At first such attempts are user tries to guess the location, but with they become more accurate. We model h as a normalized Gaussian distribution e target. in the menu P (practice). An original aspect is to explain where (e.g., on which item) the eyes spend time. The total time for selecting target t is the sum of the gaze time required for each item i of the menu. The relationship between total time and the gaze distribution for each component is: T(✓) = 1 R i=lX i=0 Gs(i, ✓) + Gd(i, ✓) + Gp(i, ✓) (2) where Gs, Gd, Gp estimate the numbers of gaze points (se- rial, directed, and pointing) captured using an eye-tracker with a sampling rate of R. In our studies, R = 50Hz. Serial search Serial search refers to the top-to-bottom inspection of items. Items after the target are not visited. With practice, users not only skip more items but also spend less time on each visited item. For this reason, the number of gaze points on each item in a menu is not constant but decreases as a function of rep- etitions. The number of gaze points for i when looking for target t is: Gs(i, ✓) = ⇢ as ⇥ exp( bs ⇥ pi) + cs if i  t 0 otherwise (3) ion is that the total time to select an item con- spent in two consecutive subtasks–search and d both are affected by learning (Figure 1): es the recruitment of eye gaze and memory recall the position of the target in the menu by scan- e follow previous simulation models [20, 21] and o search strategies: l search consists of top-to-bottom reading of that, with practice, allows more skipping. This odeled as a uniform distribution from the begin- of the menu to the target. cted search consists of a direct attempt at moving yes on top of the target. At first such attempts are om, as the user tries to guess the location, but with exposure they become more accurate. We model ted search as a normalized Gaussian distribution red on the target. T(✓) = 1 R i=lX i=0 Gs(i, ✓) + Gd(i, ✓) + Gp(i, ✓) (2) where Gs, Gd, Gp estimate the numbers of gaze points (se- rial, directed, and pointing) captured using an eye-tracker with a sampling rate of R. In our studies, R = 50Hz. Serial search Serial search refers to the top-to-bottom inspection of items. Items after the target are not visited. With practice, users not only skip more items but also spend less time on each visited item. For this reason, the number of gaze points on each item in a menu is not constant but decreases as a function of rep- etitions. The number of gaze points for i when looking for target t is: Gs(i, ✓) = ⇢ as ⇥ exp( bs ⇥ pi) + cs if i  t 0 otherwise (3) where as, bs, cs are the parameters of the learning model, while pi is the number of previous encounters with item i. Directed search In directed search the user tries to glance directly at the target. Attempts are initially more random, but the spread of gaze distribution decreases with more repetitions. As experienced users have memory about the general location of the target, they can glance at the neighborhood of the target and, even- tually, directly move the eyes to it [11]. As in serial search, we assume that experienced users also skip more items and spend less time reading them than novices. We thus model this distribution of gaze with a normal distri- bution centered on the target item t. Probability for item i in the menu is given by its probability density function t, 2 d (i) modulated by the power law of learning: Gd(i, ✓) = t, 2 d (i) ⇥ (ad ⇥ exp( bd ⇥ pi) + cd) (4) where d is the standard deviation reflecting the spread of the search area. ad, bd and bd are the empirical parameters of the learning model. Note that the sum in Eq. 2 ranges from the first item to the end of the menu (i 2 [1, l]), but not longer, because we assume that users only search inside the menu. and d0 the dista the trail strategy T We can thus est that users can se point, we revisit t,1.1(i) that cov the target: Gp(i, ✓) = where t,1.1 is a get t with a stan 95% of the gaze Absent items We also provide not in the menu pends on menu l (p = 0), they ins detecting missin Ta(✓ where aa, ba, ca are the parameters of the learning model, mber of previous encounters with item i. h the user tries to glance directly at the target. ially more random, but the spread of gaze eases with more repetitions. As experienced ory about the general location of the target, at the neighborhood of the target and, even- ove the eyes to it [11]. As in serial search, experienced users also skip more items and eading them than novices. his distribution of gaze with a normal distri- n the target item t. Probability for item i in and d0 the distance from the cursor to the gaze when using the trail strategy. We assume that d0 = 0. Tp = a + b ⇥ log2(1 + ↵ ⇥ t) (8) We can thus estimate Gp. However, to account for the fact that users can see 2-5 items in the neighborhood of the gaze point, we revisit the equation 8. We add a normal distribution t,1.1(i) that covers the dispersion of gaze in the items around the target: Gp(i, ✓) = ap + bp ⇥ log(1 + ↵ ⇥ t) ⇥ t,1.1(i) (9) where t,1.1 is a normal distribution centered around the tar- get t with a standard deviation equal to 1.1. This reflects that 95% of the gazes are in the users’ (extended) fovea. EXAMPLE: MATHEMATICAL MODEL MODEL OF MENU SELECTION
  27. 27. OPTIMIZATION METHODS Questions we can ask 1.The best possible design 2.All designs x% from the optimum 3.Diverse but good options to a design 4.How far are we from the optimum 5.The single best change to improve a design A POWERFUL TOOLBOX Design variable A Variable B Objective
  28. 28. GUARANTEE A GREATER- THAN-ZERO CHANCE OF FINDING THE OPTIMUM DESIGN OPTIMIZATION METHODS
  29. 29. KEYBOARD LAYOUT OPTIMIZATION We can optimise keyboard layouts for any terminal and any given language. We can find the optimum design with mathematical guarantees Status in 2016 CASE CLOSED? m K! -! ,! ?! !! :! J! V! Q! H! U! I! E! A! R! N! T! P! B! O! .! W! L! ;! C! M! space! Z! X! D! S! F! G!Y! (c) Physical 3-row standard: Pred 78.36 wpm three keyboard optimization tasks. hen, T., and Kan, M.-Y. Creating a live, public short message service +7% faster than Qwerty +2% faster than DSK Physical keyboard layout optimized for English
  30. 30. Antti, you‘re right. But … You’re not dreaming BIG enough
  31. 31. can computers design?
 antti oulasvirta
  32. 32. can computers design?
 antti oulasvirta METHOD / MODEL-BASED INTERFACE OPTIMISATION FOUNDATION / DESIGN AS SEARCH RESULTS / BEYOND KEYBOARDS CONCLUSION / A FUTURE DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER
  33. 33. can computers design?
 antti oulasvirta RESULTS / BEYOND KEYBOARDS
  34. 34. Cohen-Or 2006Itti & Koch 1998 Rosenholtz 2007 MODELS WE USE (SELECTED) Saliency Clutter perception Color harmony Fitts 1954 Target selection Kieras-Hornof 2014 Salvucci 2001 Visual search Reading J.R. Anderson 1998 Wickens 2002 Working memory Multitasking Chen et al. 2015; Guiard 2015 Strategy adaptation Balinsky 2006Shipley-Kellman 1992 Wertheimer 1938 Unit perception Gestalt grouping Grid quality PERCEPTION&
AESTHETICS Lee & Oulasvirta 2016 Internal clock HUMAN 
 PERFORMANCE ATTENTION & COGNITION
  35. 35. WEB PAGE DESIGN Heading goes here www.firstlink.com www.secondlink.io www.thirdlink.org Button Button Button Important Original design* *Estimated lower bound for the size of this design space is 1090
  36. 36. SINGLE-OBJECTIVE RESULTS DO NOT MAKE SENSE BBB Heading goes here www.firstlink.com www.secondlink.io www.thirdlink.org ButtonButtonButton Heading goes here www.firstlink.com www.secondlink.io www.thirdlink.org Button Button Heading goes here Button www.firstlink.com www.secondlink.io www.thirdlink.org Heading goes here www.firstlink.com www.secondlink.io www.thirdlink.org B B B Button Button Heading goes here Button www.firstlink.com www.secondlink.io www.thirdlink.org Selection time only Visual search only Clutter perception only Color harmony only Grid quality only
  37. 37. WEB PAGE DESIGN MULTI-OBJECTIVE RESULT Button Button Heading goes here Button www.firstlink.com www.secondlink.io www.thirdlink.org
  38. 38. can computers design?
 antti oulasvirta LEGEND Optimize for Marker size Marker color Aspect ratio Transparency Correlation estimation Cluster detection Outlier detection Distribution assessment design variables design objectives
  39. 39. APPLICATION MENU CASE WINDOWS PHONE Baseline All All except first Meanselectiontime(ms) 2000 2500 3000 3500 Optimised Baseline Optimised Baseline All All except first Meanselectiontime(ms) 2000 2500 3000 3500 Optimised Baseline Optimised Baseline All All except first Meanselectiontime(ms) 2000 2500 3000 3500 Optimised Baseline Optimize for Size Color xy-position Paging Motor performance Visual search performance Clutter perception Color harmony Users (N=16) were significantly faster with the optimized menu. While they rated it to be more cluttered, they liked the colors more. Evaluation
  40. 40. Title ATTENTION-OPTIMIZED SKIM READING We determine dynamically the content and number of static highlights using principles of visual attention that avoid overloading Spotlights Optimize for Number of objects Duration per object Gist comprehension
  41. 41. SCATTERPLOT OPTIMIZATION Matplotlib default design Optimize for Marker size Marker color Aspect ratio Transparency Correlation estimation Cluster detection Outlier detection Distribution assessment Optimized (preliminary)
  42. 42. GAME LEVEL DESIGN USING HUMAN PERFORMANCE MODELS Optimize for Pillar width Pillar gap Timing accuracy
  43. 43. EASY MEDIUM DIFFICULT Real gaming performance by a colleague Optimize for Pillar width Pillar gap Timing accuracy GAME LEVEL DESIGN USING HUMAN PERFORMANCE MODELS
  44. 44. can computers design?
 antti oulasvirta IN-AIR GESTURE DESIGN
  45. 45. MODEL ACQUISITION TARGET SELECTION TASK Experimental setup formance achievable after practice. Given a target of width and distance D, Fitts’ law states that the MT to reach the get is given by MT = a + b log2(D/W + 1). Fitts’ law also been used previously to quantify performance differ- ces in fingers, wrist, and forearm [3, 5, 25, 13, 15]. How- r, in this work we use angular motions at joints instead translation [12]. Considering the angular target width ↵W d distance D, we get: mt✓ = a✓ + b✓ log2 ✓ ↵D W + 1 ◆ (1) acquire the a and b parameters, we conduct an experi- nt that employs a unidimensional pointing task. We ad- ss speed–accuracy trade-off in this task by using effective dth ¯W and distance ¯D. Performance disregarded in HCI. In order to compute it for every finger, the position of the non-instructed digit is plotted as a function of the instructed digit’s position. The resulting trajectories are typically linear and the slope of a line fitted to these data points serves as a measure for the relative coactivation: the extent to which a non-instructed finger moves relative to the instructed finger. Given the coactivation Cij of finger i during the movement of finger j, the individuation index of j is Ij = 1 [( nX i=1 | Cij | 1)/(n 1)], (2) where n is the number of fingers (5). Ij = 1 indicates perfectly individuated movement, and Ij = 0 if all non- instructed fingers move simultaneously with j. The original study of individuation was reported for fingers, but it can be extended to multiple joints used in multi-finger input. EXPERIMENT: FINGER DEXTERITY Our experimental method is based on the reciprocal selec- tion task used in Fitts’ law studies [14]. As shown in Figure 3, users move a finger between two targets. Instead of ex- trinsic targets (e.g. button), the target here is a joint angle. Visual feedback is provided on a monitor with high refresh rate. In contrast to most Fitts’ law studies, we track not only the end-points of movements but the full motion of the hand. This allows us to quantify three aspects of the dexterity of fin- ger motion: performance (speed and accuracy), individuation (unwanted motion of uninstructed fingers), and comfortable motion ranges. In addition, the data allows us to look at the range of individual differences. a finger between two target angles indicated on a motion was tracked. The color coding for joints is section. Note that the CMC joint of the thumb is a sp be independently moved in two directions. in a neutral pose. Moreover, we included th thumb which was the only interphalangeal jo moved and tracked well. Figure 3 also sho convention and color coding used in the rest the thumb we use Thumb-Down and Thumb “up-down” and “left-right movement” of the Participants The study was conducted with 13 participant locations. They were 8 male and 5 female, and with age ranging from 22 to 32 (mean 2 nical difficulties, one of the participants coul 4 of the 7 joint conditions. The experiment 2 hours per participant. Participants from on compensated with cinema vouchers. The tri out in a room with controlled lighting and no Experimental Design The experiment followed a 7⇥4 within-subj 7 DOFs and 4 ID conditions. To minimize o joint and ID conditions were randomized for Pre-trial practice was employed and breaks w ter the trial for each joint. Task, Materials, and Procedure The task is a unidimensional target selectio pants had to move a pointer up and down betw Finger individuation volved in the gesture: CiG = maxj2G Cij. Then, following the original Equation 2, we compute the individuation index for any multi-joint gesture as IG = 1 [( nX i=1 |CiG| |G|)/n |G|), (4) where |G| denotes the number of joints actively involved in gesture G. Step 3: Objective Function Formulation Our design task is to maximize the usability U of a letter assignment, i.e. the mapping of each character in a char- acter set to a unique posture (gesture) of the hand. To characterize U, we formulate a multi-term objective func- tion for mid-air text entry called PALM. PALM addresses four factors affecting mid-air text entry with multiple fin- gers: Performance, Anatomical comfort (individuation), Learnability, and Mnemonics. In addition to performance and individuation, we formalize learnability and mnemonics based on existing literature. Usability U is thus defined as a weighted sum of four normal- ized (i.e. 2 [0, 1]) terms2 . Formally, we write our usability objective as max U = wp ˆP + wa ˆA + wl ˆL + wm ˆM, (5) where the positive weights wp, wa, wl, and wm set by a de- signer sum up to 1. The remaining terms in the objective collapsing it into a few dimensions. Practicing a complex gesture gradually increases hierarchical organization and de- creases reliance on feedback. This has two consequences. First, the fewer DOFs a gesture involves, the easier it will be to learn. For instance, gesturing with one finger is easier to learn than a gesture using three fingers. We name the number of involved DOFs udofs. Second, if the involved digits involve the same end posture, it will be easier to learn because the articulations can be represented with a single learning primi- tive. For example, it is easier to extend all digits by 40° than to extend some by 20° and others by 40°. We denote the num- ber of DOFs for which a target angle is defined in a gesture as by utargets. Our learnability score combines these two aspects: L = 1 X k (0.5 ˆutargets, k + 0.5 ˆudofs, k). (7) Mnemonics Term (M) Studies of human memory suggest that categorization, chunk- ing, and mnenomics help forming more durable long-term memory traces among otherwise unrelated materials [35]. Our mnemonics score M considers the memorability of a let- ter assignment as a whole. We call a mnemonic set a set of similar gestures, such as gestures that all have a neighboring finger. To identify finger mnemonics, we build on a recent study of multi-finger chord gestures that showed a positive effect on learning [35]. We take the mnemonic principles presented there and extend them from three fingers to five. In particular, we include the following mnemonics rules: neigh- Motor control literature Learnability
  46. 46. 48 Example design: Chorded text entry [Proc. CHI’15]
  47. 47. We can now address interfaces where primary emphasis on perception, motor control, and ergonomics. Our examples span input devices, interaction techniques, menu systems, graphical user interfaces, basic information visualizations, web pages, mobile apps, gestural interfaces MILESTONE IN 2016 More UI types
  48. 48. FUNCTIONALITY SELECTION
  49. 49. can computers design?
 antti oulasvirta FUNCTIONALITY SELECTION em independent of each other; (2) link each term to easily understandable pulable definitions supported by literature; and (3) find a formulation that is nough in code. ective is to choose the set of functionality that maximizes ”goodness” G. We in terms of four objectives called USEP: usefulness U, satisfaction S, ease- and profitability P. These terms form the core of the objective function we ghout our work, supplementing it with case-specific weights, objectives, and ts when needed. maxG = !U U + !SS + !EE + !P P (4) 2 ⌦ denote a weight given to an objective, P !2⌦ ! = 1.0. ollowing, we define each term. Throughout, we denote a functionality selected n as and the full set of selected functionality . By ”normalization”, marked zontal bar, we refer to scaling a variable to range [0, 1]. lness e the usefulness of a design U as the sum of usefulness u and dependency f each selected functionality: ¯U = 1 2 ( X 2 ¯u + D ) (5) efulness scores u 2 [0, 1] and D = P 2 P ⌧2 ,⌧ P i P j i,j (6) pendency scores i,j2[0,1] are directional dependency scores defined by the de- dent of each other; (2) link each term to easily understandable itions supported by literature; and (3) find a formulation that is e. hoose the set of functionality that maximizes ”goodness” G. We our objectives called USEP: usefulness U, satisfaction S, ease- ility P. These terms form the core of the objective function we ork, supplementing it with case-specific weights, objectives, and ded. maxG = !U U + !SS + !EE + !P P (4) weight given to an objective, P !2⌦ ! = 1.0. define each term. Throughout, we denote a functionality selected e full set of selected functionality . By ”normalization”, marked e refer to scaling a variable to range [0, 1]. ess of a design U as the sum of usefulness u and dependency ed functionality: ¯U = 1 2 ( X 2 ¯u + D ) (5) es u 2 [0, 1] and D = P 2 P ⌧2 ,⌧ P i P j i,j (6) res i,j2[0,1] are directional dependency scores defined by the de- on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111. Usefulness Dependency N ing an objective function were to (1) include commonly consid- as separate terms with own weights, allowing the designer to ent of each other; (2) link each term to easily understandable itions supported by literature; and (3) find a formulation that is . hoose the set of functionality that maximizes ”goodness” G. We our objectives called USEP: usefulness U, satisfaction S, ease- lity P. These terms form the core of the objective function we rk, supplementing it with case-specific weights, objectives, and ed. maxG = !U U + !SS + !EE + !P P (4) weight given to an objective, P !2⌦ ! = 1.0. define each term. Throughout, we denote a functionality selected e full set of selected functionality . By ”normalization”, marked e refer to scaling a variable to range [0, 1]. ess of a design U as the sum of usefulness u and dependency d functionality: ¯U = 1 2 ( X 2 ¯u + D ) (5) s u 2 [0, 1] and D = P 2 P ⌧2 ,⌧ P i P j i,j (6) res i,j2[0,1] are directional dependency scores defined by the de- on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111. Satisfaction Fig. 3. Users value the loss of an important function more negatively than its inclusion in a design. 4.2. Satisfaction We define user satisfaction as a nonlinear function that maps the presence/absence of a functionality to a satisfaction score. This score refers to subjective experience of the unctionality, as opposed to the usefulness score. We offer two formulations. The first definition uses a linear sum: S = X 2 ¯s (7) The limitation is that the absence of a functionality is not accounted for in the score. The second definition builds on prospect theory that predicts that the absence of an mportant functionality is appraised more critically than its presence. This is captured by the following mathematical function for prospect ⇡ of item : ⇡i = ⇢ p si ifxi = 1 2 p si ifxi = 0 (8) which we reformulate as ⇡i = 3 p si · xi 2 p si (9) Note that satisfaction can be negative if a design misses functionality users expect it o have. This yields a new S⇡: S = nX i=1 ¯⇡(i) (10) S is used in the objective function after normalization. 4.3. Ease-of-use Ease-of-use ⇡i = 2 p si ifxi = 0 (8) which we reformulate as ⇡i = 3 p si · xi 2 p si (9) Note that satisfaction can be negative if a design misses functionality users expect it to have. This yields a new S⇡: S = nX i=1 ¯⇡(i) (10) S is used in the objective function after normalization. 4.3. Ease-of-use We provide two definitions for the ease-of-use of a design E. The former is allows quicker computation but simplifies the effect of increasing functionality. The second is a decreasing function of the number of selected functionality. Our first formulation is a E = X 62 c (11) where c is the estimated complexity of . ACM Transactions on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111.Profitability Fig. 4. We assume the ease-of-use of an interface to decrease non-linearly with the number of functionality. Here, y = 1/(x 1). Our second formulation captures the fact that ease-of-use is compromised by includ- ing more and more functionalities. Including very few trivially ensures high ease-of- use. For example, designers group semantically to decrease selection time in a hierar- chical menu [Bailly et al. 2014]. Interface techniques for command selection like search functionality, toolbars, autocompletion, panels, and tool palettes can achieve the same effect. Ease-of-use would increase linearly only in the case the interface was organized randomly. We also note that some interface techniques allow constant time for a small number of features. For example, hotkeys can make access time virtually constant, but users typically learn only few hotkeys. While we use the reciprocal function in the following, we recognize that the exact shape of this function depends many factors, including the skills of the design and the interface type. The reciprocal formulation is shown in Figure 4 and given as E = 1 | | + 1 (12) We can also formulate this by introducing a quadratic constraint E + P i Exi <= 1. 4.4. Profitability We define the profitability of a functionality p in terms of its business value v 2 R and costs c 2 R: p = v c . After normalization of p, we can compute P: ¯P = X 2 p = X 2 (v c ) (13) Note that if c > v, profitability is negative. Sometimes including features that are negative in profitability is justifiable in the light of other objectives. 4.5. Further elaborations Different user groups Uncertain values Two or more design approaches? ACM Transactions on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111. A Usefulness Satisfaction Ease of use Profitability 24 functionalities Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete Delete_notebook Draw_freehand Draw_shape Exit Export Fill_color Find Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Put_on_homescreen Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete Delete_notebook Draw_freehand Draw_shape Exit Export Fill_colorFind Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition File creation and organization Link and share Printing Table Structural elements Input Text style options Page layout Controls View F Usefulness Satisfaction Ease of use Profitability 82 functionalities Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete Delete_notebook Draw_freehand Draw_shape Exit Export Fill_color Find Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Put_on_homescreen Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete Delete_notebook Draw_freehand Draw_shape Exit Export Fill_colorFind Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition File creation and organization Link and share Printing Table Structural elements Input Text style options Page layout Controls View … k Diverse designs R Usefulness Satisfaction Ease of use Profitability 106 functionalities Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete_element Delete_file Delete_notebook Draw_freehand Draw_shape Exit Export Fill_color Find Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Put_on_homescreen Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition Zoom Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete_element Delete_file Delete_notebook Draw_freehand Draw_shape Exit Export Fill_colorFind Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition Zoom File creation and organization Link and share Printing Table Structural elements Input Text style options Page layout Controls View Robust design
  50. 50. EMPIRICAL EVALUATION Professional designers found the optimized functionality sets as good or better than the ones they designed themselves. Result R Usefulness Satisfaction Ease of use Profitability 106 functionalities Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete_element Delete_file Delete_notebook Draw_freehand Draw_shape Exit Export Fill_color Find Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Put_on_homescreen Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition Zoom Add_column_to_table Add_row_to_table Align_text Bullet_size Bullet_type Bullets Cell_color Cell_margins Change_encoding Change_number_size Change_start_number Character_spacing Choose_printer Close Close_all_windows Column_size Compare_documents Copy Create_notebook Create_table Cut Delete_element Delete_file Delete_notebook Draw_freehand Draw_shape Exit Export Fill_colorFind Font_color Font_size Full_screen Handwriting_recognition Indent Insert_blank_page Insert_date_and_time Insert_horizontal_line Insert_link Insert_picture Insert_video Line_color Line_number_color Line_number_continuation Line_number_font Line_number_size Line_numbers Line_spacing Line_style Line_wrap Link_to_calendar_entry Margins Mark_all_text Mark_text Merge_cells Move_document_to_other_view Move_note_to_notebook New Number_of_copies Number_style Numbering Open Open_containing_folder Orientation Page_break Page_size Paragraph_spacing Paste Print_current_page Print_custom_range Print_document Print_multiple_pages_per_sheet Print_selection Printer_settings Publish_online Recently_opened Record_audio Redo Reload Remove_column_from_table Remove_row_from_table Rename Rename_notebook Replace Row_size Save Save_all_windows Search_printer Share Show_grid Show_ruler Sort_elements Spell_check Split_cell Split_view Synchronize_scrolling Table_border Text_alignment_in_table Text_background_color Text_columns Text_font Text_style Type_text Undo Voice_recognition Zoom File creation and organization Link and share Printing Table Structural elements Input Text style options Page layout Controls View Example optimized functionality for a
 note taking application
  51. 51. can computers design?
 antti oulasvirta METHOD / MODEL-BASED INTERFACE OPTIMIZATION FOUNDATION / DESIGN AS SEARCH RESULTS / BEYOND KEYBOARDS CONCLUSION / A FUTURE DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER
  52. 52. can computers design?
 antti oulasvirta DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER “tools might enforce user interfaces that were highly usable, rather than today’s stance that tools should be neutral…” - Brad Myers et al. (2000)
  53. 53. DesignEvaluate INITIALIZING INTERFACE OPTIMIZATION designer computer
 scientist social
 scientist Design Task Optimizer Predictive
 Models
  54. 54. DesignEvaluate DESIGN TOOLS HIDE ALGORITHMIC COMPLEXITY designer Optimizer Design Task Heuristics
 Models Interactive 
 optimization tool Defines Steers
  55. 55. FULL CONTROL CONCEPT Optimize for Menu structure Position of item Selection time Full interactive control of design objectives, input data. Computer suggestions never override designer decisions, so you can still design as normally. MenuOptimizer Grouping of items Shortcuts Familiarity Novice/expert performance
  56. 56. “SKETCHPLORER” ZERO-EFFORT INTERACTIVE OPTIMIZATION Hands-on demo of Sketchplorer at CHI’16 in San Jose; May 2-7
  57. 57. ZERO-EFFORT CONTROL CONCEPT Optimize for Item positions Item colors Motor performanceItem sizes Alignment Grid quality No need for the designer to control anything. The computer recognises the design task and provides suggestions accordingly. Sketchplorer Clutter perception Color harmony Visual search
  58. 58. can computers design?
 antti oulasvirta METHOD / MODEL-BASED INTERFACE OPTIMIZATION FOUNDATION / DESIGN AS SEARCH RESULTS / BEYOND KEYBOARDS CONCLUSION / A FUTURE DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER
  59. 59. can computers design?
 antti oulasvirta CONCLUSION / A FUTURE
  60. 60. AS DESIGN FROM FIRST PRINCIPLES COMPUTATIONAL INTERFACE DESIGN Predictive models (principles of interaction) allow algorithms to identify usable interfaces in multi- dimensional design spaces. All inputs are scrutinizable and empirically verifiable, thus contributing to accumulation of design knowledge.
  61. 61. To improve the positive effect that optimization can have on design quality, we need to better understand creativity in human design process. To extend the scope of optimizable problems, we are working on predictive models of learnability, user experience, aesthetics, social cognition, etc. To extend applications in interaction design, we must study designers and formulate their activities in code. RESEARCH CHALLENGES Modeling Design Tasks Creativity
  62. 62. Design Superiority An organization emerges that can present guarantees for their designs Bulk Design Automated A design service emerges that use interactive optimization to significantly improve cost- efficiency in a popular category Computer Outperforms Human Computer is shown to reliably outperform human designers in controlled studies with end- users Self-Designing Interfaces Log data is used to parametrize optimizers that can optimally adapt the interface without human intervention.
  63. 63. COMPUTERS CAN design. Funding: European Research Council Staring Grant (contract 637991); Academy of Finland research project COMPUTED; AKA-JST research project with Kochi University of Technology; Max Planck Institute for Informatics; Aalto University Graduate Schools; Saarland University Thanks to my brilliant colleagues: Kumaripaba Athukorala, Myroslav Bachynskyi, Duncan Brumby, Xiuli Chen, Gilles Bailly, Anna Feit, Giulio Jacucci, Eve Hoggan, Kasper Hornbæk, Andrew Howes, Jussi Jokinen, Andreas Karrenbauer, Janin Koch, Per Ola Kristensson, Byungjoo Lee, Janne Lindqvist, Perttu Lähteenlahti, Luana Micallef, Arttu Modig, Jörg Müller, Sayan Sarcar, Rod Murray-Smith, Mathieu Nancel, Gregorio Palmas, Xiangshi Ren, Olli Savisaari, Srinath Sridhar, Jürgen Steimle, Christian Theobalt, Jilles Vreeken, Tino Weinkauf, Daryl Weir Links: userinterfaces.aalto.fi, computationalinteraction.org Antti Oulasvirta Associate Professor userinterfaces.aalto.fi Hands-on demo of Sketchplorer at CHI’16 in San Jose; May 2-7

×