Interactively Evolving User Interfaces Juan Quiroz Committee: Dr. Sushil Louis Dr. Sergiu Dascalu Dr. Swatee Naik

Outline Motivation Related work: GAs, IGAs, UI Design User fatigue in IGAs UI evolution Experiments Results Future Work

Motivation

Related work: GAs, IGAs, UI Design

User fatigue in IGAs

UI evolution

Experiments

Results

Future Work

Motivation User interface design is a complex, expensive, time consuming process Iterative process Users and contexts of use are numerous Streamline and improve UI design End-user customization

User interface design is a complex, expensive, time consuming process

Iterative process

Users and contexts of use are numerous

Streamline and improve UI design

End-user customization

IGA for UI Evolution IGA to explore the space of UIs Creativity and insight Evolution is guided by both the user preferences and coded guideline metrics Pick best and worst

IGA to explore the space of UIs

Creativity and insight

Evolution is guided by both the user preferences and coded guideline metrics

Pick best and worst

Genetic Algorithms Population based search technique Natural selection Survival of the fittest

Population based search technique

Natural selection

Survival of the fittest

Interactive Genetic Algorithms (IGAs) Fashion design (Kim 2000) Micromachine design (Kamalian 2005) Music, editorial design (Takagi 2001) Traveling salesman problem (Louis 1999)

Fashion design (Kim 2000)

Micromachine design (Kamalian 2005)

Music, editorial design (Takagi 2001)

Traveling salesman problem (Louis 1999)

User Evaluation: Subjective Fitness 100 75 15 80 20 8 38 53 82

User Evaluation: Ranking 1 5 6 9 3 2 4 8 7

User Evaluation: Tournaments A B Fitness A > Fitness B

User Fatigue in IGAs GAs tend to rely on: Large populations Many generations Suboptimal solutions Noisy fitness landscapes

GAs tend to rely on:

Large populations

Many generations

Suboptimal solutions

Noisy fitness landscapes

Alleviating User Fatigue Use small population sizes Display a subset of population Accelerate convergence through prediction (Llora 2005) Just what I had in mind!

Use small population sizes

Display a subset of population

Accelerate convergence through prediction (Llora 2005)

UI Design Support GUI toolkits and libraries

GUI toolkits and libraries

Guidelines of Style Microsoft, Apple, Java, KDE, Gnome Define a common look and feel for applications Discuss the use of color, layout of widgets, the use of fonts Interpreting the guidelines is itself a challenge Too vague or too specific

Microsoft, Apple, Java, KDE, Gnome

Define a common look and feel for applications

Discuss the use of color, layout of widgets, the use of fonts

Interpreting the guidelines is itself a challenge

Too vague or too specific

Ambiguity in Guidelines “ Use color to enhance the visual impact of your widgets” – Apple’s Human Interface Guidelines

“ Use color to enhance the visual impact of your widgets” – Apple’s Human Interface Guidelines

XUL User Interfaces XML User Interface Language Mark-up language for UIs Buttons, textboxes, sliders menubars, toolbars

XML User Interface Language

Mark-up language for UIs

Buttons, textboxes, sliders

menubars, toolbars

Related Work: UI Evolution Evolution of style sheets (Monmarché et al) Font and links color, paragraph spacing, font family, font decoration We allow both the user and the coded guidelines to guide the evolution

Evolution of style sheets (Monmarché et al)

Font and links color, paragraph spacing, font family, font decoration

We allow both the user and the coded guidelines to guide the evolution

Related Work: User Fatigue SVMs to combat user fatigue (Llora 2005) Kamalian et al. (2005) User evaluation every t th generation Demote or promote reaction to individuals Validity constraint is used to determine viable and meaningful designs

SVMs to combat user fatigue (Llora 2005)

Kamalian et al. (2005)

User evaluation every t th generation

Demote or promote reaction to individuals

Validity constraint is used to determine viable and meaningful designs

Interactive Genetic Algorithms in UI Design

Interactive Genetic Algorithms in UI Design

Lagoon MoveTo Panel

UI Representation Two chromosomes Widget chromosome Layout chromosome

Two chromosomes

Widget chromosome

Layout chromosome

Genetic Operators for UI IGA Single point crossover Bit flip mutation PMX – partial mapped crossover Swap mutation

Single point crossover

Bit flip mutation

PMX – partial mapped crossover

Swap mutation

Widget Color RGB color model Red = (255, 0, 0), Green = (0, 255, 0), Blue = (0, 0, 255) 2 24 color space for each widget HSV Same gamut as RGB No significant efficiency difference in RGB and HSV

RGB color model

Red = (255, 0, 0), Green = (0, 255, 0), Blue = (0, 0, 255)

2 24 color space for each widget

HSV

Same gamut as RGB

No significant efficiency difference in RGB and HSV

Fitness Evaluation Ask the user to select the best and worst UIs from the subset displayed Interpolate the subjective fitness of individuals in population Compute the objective metrics taken from guidelines of style Fitness = w1 * subjective + w2 * objective

Ask the user to select the best and worst UIs from the subset displayed

Interpolate the subjective fitness of individuals in population

Compute the objective metrics taken from guidelines of style

Fitness = w1 * subjective

+ w2 * objective

1: Fitness Evaluation Best Worst

2: Subjective Fitness Interpolation Compare to best Compare to worst Best Worst

Compare to best

Compare to worst

3. Objective Fitness Computation High contrast between widget colors and background color Low contrast between widget colors

High contrast between widget colors and background color

Low contrast between widget colors

Fitness = w1 * subjective fitness + w2 * objective fitness

Fitness = w1 * subjective fitness

+

w2 * objective fitness

Research Questions Which selection type is the most effective for this problem? Who from the population do we display for user evaluation? How often should we ask for user input?

Which selection type is the most effective for this problem?

Who from the population do we display for user evaluation?

How often should we ask for user input?

Experimental Setup Greedy simulated user 30 independent runs, 200 generations Population size of 100 Roulette wheel vs. tournament Display method comparison: best 10, random 10, best 5 and worst 5 User input every 1, 5, 10, 20, 40, 80 generations

Greedy simulated user

30 independent runs, 200 generations

Population size of 100

Roulette wheel vs. tournament

Display method comparison: best 10, random 10, best 5 and worst 5

User input every 1, 5, 10, 20, 40, 80 generations

Which selection type is the most effective for this problem?

Who from the population do we display for user evaluation? Fitness Convergence Convergence to Blue UIs

How often do we ask for user input? User input every t th generation Fitness convergence High values of t Low values of t

How often do we ask for user input? User input every t th generation Convergence to blue UIs High values of t Low values of t

Experimental Setup: Actual Users Three users 30 generations Pick the one they like the best and the one they like the least 5 sessions User input every 1, 3, 5, 10, 15 generations 30, 10, 6, 3, and 2 user evaluations respectively

Three users

30 generations

Pick the one they like the best and the one they like the least

5 sessions

User input every 1, 3, 5, 10, 15 generations

30, 10, 6, 3, and 2 user evaluations respectively

Results

What leads to the drop in average performance? Two sessions with a user Comparison to user selected worst turned on Comparison to user selected worst turned off Always pick the same UI as the best Ask for user input every 3 generations

Two sessions with a user

Comparison to user selected worst turned on

Comparison to user selected worst turned off

Always pick the same UI as the best

Ask for user input every 3 generations

What leads to the drop in average performance?

Generated UIs: Simulated User

Generated UIs: Simulated User

Generated UIs: User3

Generated UIs: User3

Future Work Ask user to select the best/worst UI only Varying the frequency of user input during a session Convergence acceleration with neuroevolution or SVMs Integration with a GUI toolkit or library User studies! Task completion Explore color representations Specify the type of data that needs to be represented

Ask user to select the best/worst UI only

Varying the frequency of user input during a session

Convergence acceleration with neuroevolution or SVMs

Integration with a GUI toolkit or library

User studies!

Task completion

Explore color representations

Specify the type of data that needs to be represented

Contributions We can use IGAs to evolve UIs Our simulated user and actual users are able to effectively bias the evolution of UIs UIs reflect coded guidelines of style Reduce user fatigue Interpolation technique Asking for less user input

We can use IGAs to evolve UIs

Our simulated user and actual users are able to effectively bias the evolution of UIs

UIs reflect coded guidelines of style

Reduce user fatigue

Interpolation technique

Asking for less user input

Demo

Questions? www.cse.unr.edu/~quiroz [email_address]

www.cse.unr.edu/~quiroz

[email_address]