Combating User Fatigue and Contradictions in Subjective-based Optimization SchemesPresentation Transcript

Xavier Llorà
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
xllora@uiuc.edu

Subjective-based optimization schemes?
•
Examples
•
Affinova
The electronic sheep
Emotional prosody
A common subjective-based optimization scheme
•
Interactive genetic algorithms
Limitations
Combating user fatigue and contradictions
•
Worst-case scenario
Partial-ordering graphs, surrogate fitness, and support-vector machines
Conclusions and current directions
•
Acknowledgments
•
September 21, 2006 Xavier Llorà 2

The goal: “Find a good solution in a given search space”
•
The handicap: “No quantitative measure of goodness is available”
•
The search space needs to be explored based on subjective criteria
•
Qualitative fitness is provided by the user
• Sorting of several solutions
• Choosing one of solution out of a subset
• Based on global impression
Limitations of subjective-based optimization:
• The time user can expend doing repetitive tasks (user fatigue)
• The progress achieved during the repetitive tasks (user frustration)
• Criteria adaptation (user’s criteria may change during the process as a
results of the process itself)
Let’s see some real examples
•
September 21, 2006 Xavier Llorà 3

Subjective-based optimization schemes? ✔
•
Examples
•
Affinova
The electronic sheep
Emotional prosody
A common subjective-based optimization scheme
•
Interactive genetic algorithms
Limitations
Combating user fatigue and contradictions
•
Worst-case scenario
Partial-ordering graphs, surrogate fitness, and support-vector machines
Conclusions and current directions
•
Acknowledgments
•
September 21, 2006 Xavier Llorà 4

Concept design: Create new product candidates
•
Affinova (http://www.affinnova.com/)
•
1. Base Concept 2. Featurize 3. Create alternatives
4. Generate alternative, involve customers, and iterate
September 21, 2006 Xavier Llorà 5

Proposed by Scott Draves (http://electricsheep.org/)
•
A form of aesthetic evolution, a concept ﬁrst realized by Karl Sims
•
The goal:
•
Animate and evolve artificial life-forms know as sheep
Reference to Philip K. Dirk’s novel
Use screensavers to:
•
Create a distributed rendering farm
Collect user votes on favorite sheep
Popular sheep life longer and generate new sheep
•
Search space:
•
The space of parameters of fractal flames,a generalization and
refinement of the Iterated Function System (IFS).
Each sheep is defined by 240 floating point values to optimize
September 21, 2006 Xavier Llorà 6

Sheep #1700 Sheep #110345149
September 21, 2006 Xavier Llorà 7

Cecilia Alm and Xavier Llorà [Alm & Llora, 2006] (http://www.i-discus.org)
•
The problem:
•
Text-to-speech (TTS) synthesis (given a text the associated speech is synthesized)
The lack of emotion (neutrality) makes the TTS synthesis sound unnatural
The goal:
•
Adjust the TTS to incorporate emotional prosody
Spoken text should sound sad, angry, happy, etc.
Very useful for story telling or automated audio book generation
Users can easily discern the emotional prosody, but hard to explain why
•
Search space:
•
Several parameters can be tweak to modify the prosody (6 per word)
These parameters define the prosody search space
September 21, 2006 Xavier Llorà 8

Sad Angry
Strawberry Strawberry
Tan-tan-tan Tan-tan-tan
Bubhalos Bubhalos
September 21, 2006 Xavier Llorà 9

Subjective-based optimization schemes? ✔
•
Examples ✔
•
Affinova ✔
The electronic sheep ✔
Emotional prosody ✔
A common subjective-based optimization scheme
•
Interactive genetic algorithms
Limitations
Combating user fatigue and contradictions
•
Worst-case scenario
Partial-ordering graphs, surrogate fitness, and support-vector machines
Conclusions and current directions
•
Acknowledgments
•
September 21, 2006 Xavier Llorà 10

No quantitative fitness is available
•
Qualitative fitness is provided by the user
•
Sorting of several solutions
Choosing one of solution out of a subset
Based on global impressions
User fatigue
•
Short time periods (1 hour)
Repetitive task
Frustration
•
Repeated evaluation of similar solutions
No progress is appreciated
How can we help the user in the subjective-based optimization quest?
•
Hint: Big time-scale difference between user judgment and computer
•
response
September 21, 2006 Xavier Llorà 11

Subjective-based optimization schemes? ✔
•
Examples ✔
•
Affinova ✔
The electronic sheep ✔
Emotional prosody ✔
A common subjective-based optimization scheme ✔
•
Interactive genetic algorithms ✔
Limitations ✔
Combating user fatigue and contradictions
•
Worst-case scenario
Partial-ordering graphs, surrogate fitness, and support-vector machines
Conclusions and current directions
•
Acknowledgments
•
September 21, 2006 Xavier Llorà 13

✖ Users need a clear idea of the outcome
•
Need for a clear criteria of the goal to reach
✖ A good visualization goes a long way
•
A non-intuitive visualization may mislead user’s evaluations
✔ Lack of numerical fitness can be a problem
•
No numeric form that can be optimize is available
✔ User fatigue needs to be minimized
•
User may only be able to provide reliable evaluations for short
time periods (1 hour)
✔ Users tend to change their criteria along the way
•
Easy to maintain an unique criteria for short time periods
September 21, 2006 Xavier Llorà 14

Can we learn user preferences from the interaction?
•
If so, we can build models of its preferences
•
Evaluate solutions before presenting them to the user
•
Reduce the number of solutions (combat fatigue)
Show progress during the process (fight user frustration)
The question to answer:
•
How to learn a model from a simple like/don’t like scenario?
September 21, 2006 Xavier Llorà 15

A minimal iGA scenario
•
Two solutions are presented for user evaluation
Three possible evaluation outcomes:
1. The first is better than the second
2. The second is better than the first
3. Don’t know, don’t care
Evaluation by comparison
•
The GA equivalent
•
Tournament selection s=2
•
Can we learn a numerical model out of partial user
•
evaluations?
September 21, 2006 Xavier Llorà 16

• Solution-quality order should be maintained
Any synthetic fitness needs to maintain the solution ordering
provided by the user
If s1≥ s2 ≥ … ≥ sn then fs(s1) ≥ fs(s2) ≥ … ≥ fs(sn)
• Synthetic fitness should allow extrapolation
Any synthetic fitness needs to be able to be a good predictor of
the ordering relation beyond the available evaluations collected
• Let’s make some assumptions:
Imagine I have a few evaluated solutions :)
September 21, 2006 Xavier Llorà 17

Given a fs: X → Y
•
X: problem attributes
Y: numeric value
Set of evaluated solutions
•
New solution
•
Compute the k-nearest neighbor
Assign the fitness of the
weighted fitness of the k-nearest
neighbors
No accurate prediction beyond
•
the current limits
September 21, 2006 Xavier Llorà 18

• Regression model
• Extrapolates beyond the
evaluated solutions
• Provides a direction toward
improvements
• How many solutions we
need to have a reliable
model?
September 21, 2006 Xavier Llorà 19

• y = a*x + b
• a>1 guaranties the order
• Tournament only cares
about the partial order
among solutions!
• Low cost, high error models
• PAC-learning bounds tell us
how many instances I need
to properly train
September 21, 2006 Xavier Llorà 20

If we have a numerical fitness:
•
We can build a regression model
We can use such model for combating fitness fatigue
What is it available?
•
Partial ordering of solutions (binary tournament)
Incremental refinement (new evaluations)
The idea:
•
Binary tournament produce partial order relations
I can build a graph that contains all the user evaluations:
• Nodes = unique solutions
• Edges = user evaluations (I like the first one, the second one, or neither)
Compute a numeric fitness for each solution based on the graph
structure
September 21, 2006 Xavier Llorà 21

• Given a set of solutions
Start with the pair-wise comparisons on the leaves
As you move up, compare the first leaves reach on a DFS for
each of the branches
Obtain the partial ordering of the solutions
Such ordering can be expressed in a graph form
September 21, 2006 Xavier Llorà 22

Nodes are solutions
•
Edges represent the evaluation provided by the user
•
1. One solution is better than the other (→)
2. Don’t know, don’t care ()
Transformed to contain only type 1 relations
Property: cycles detect user contradictions in evaluations
September 21, 2006 Xavier Llorà 23

• Dominance measure
Dominates Dominated by Difference
Real Synthetic
September 21, 2006 Xavier Llorà 24

1. Collect the user evaluations
2. Build the partial order graph
3. Compute the numeric fitness using the partial order
graph
4. Use ε-SVM for creating a regression model
5. Use the learned regression as the synthetic fitness
September 21, 2006 Xavier Llorà 25

• Cut down the number of human evaluations
• Exploit synthetic fitness
Optimize it
Sample the best candidates
Show the best solutions to the user
• The sample best solutions help combating
Fatigue (educated guess of user preference)
Frustration (produce new eureka solutions)
September 21, 2006 Xavier Llorà 26

Evaluation
Partial order
update
Sampling Sampling
fs(x)
Optimizer
Synthetic fitness
September 21, 2006 Xavier Llorà 27

• Considerations in the design process
Clear goal definition
Impact of problem visualization
Persistence of user criteria
• Focus
Lack of numeric fitness
User fatigue
• A simple controlled task
One Max
September 21, 2006 Xavier Llorà 28

September 21, 2006 Xavier Llorà 29

• The system
A simple web application
OneMax task
No linkage learning needed
ε-SVM and a linear kernel
The compact genetic algorithm (Harik, Cantú-Paz, Goldberg, & Miller,
1999)
• Set up
One user with no relation to the research
Repeated series of 10 independent runs for different problem
sizes {4, 8, 12, 16, 20, 24, 28, and 32 variables}
Collect the data to compare it to a simple GA
September 21, 2006 Xavier Llorà 30

September 21, 2006 Xavier Llorà 31

• Subjective-based optimization is likely to be dynamic
• User criteria may vary as a results of:
Lack of attention
Difficulty to perform pair-wise accurate comparisons
The solutions seen so far may be changing the target!
• As user criteria changes it is like to materialize as
contradictions
• These contradictions can be easily detected as cycles in
the partial-order graph
September 21, 2006 Xavier Llorà 32

• Based on the partial-order graph
• A simple one: proportion of solutions in cycles
September 21, 2006 Xavier Llorà 34

Subjective-based optimization schemes? ✔
•
Examples ✔
•
Affinova ✔
The electronic sheep ✔
Emotional prosody ✔
A common subjective-based optimization scheme ✔
•
Interactive genetic algorithms ✔
Limitations ✔
Combating user fatigue and contradictions ✔
•
Worst-case scenario ✔
Partial-ordering graphs, surrogate fitness, and support-vector machines ✔
Conclusions and current directions
•
Acknowledgments
•
September 21, 2006 Xavier Llorà 35

• Address the lack of numerical fitness and fatigue
• Synthetic fitness model of user preferences
• Optimize such model to take the advantage of the
timescale difference
• Remarkable speedups
• Measure of user consistency (reliability of the evaluations)
• Real-world applications confirm the theoretical results:
Emotional text-to-speech synthesis (two research groups)
Marketing campaign and product design (advertisement company)
Tuning of text mining analysis tools (chance discovery consortium)
September 21, 2006 Xavier Llorà 36

• Hierarchical tournament impact
• Relation between graph structure and quality of the
regression method
• User consistency versus problem difficulty
• Managing large scale experimentation (metadata stores)
• Optimization involving multiple users
How can we reliably combine partial-order graphs?
Can we create combination of partially build synthetic fitnesses?
How can we deal with multiple targets?
• Users may have multiple criteria in mind
• Users may not realize that even common criteria may have different
subjective interpretations
September 21, 2006 Xavier Llorà 37

• Sponsors:
National Science Foundation
Air Force Office of Scientific Research
SALERO project funded by IST EU
• Not one man’s endeavor:
David E. Goldberg, Kumara Sastry, Lalitha Lakshmi (IlliGAL/UIUC)
Loretta Auvil, Michael Welge (ALG/NCSA)
Cecilia Alm (Linguistics/UIUC)
Francesc Alías, Lluís Formiga (URL)
IlliGAL labbies
Chance Discovery Consortium members
September 21, 2006 Xavier Llorà 38

DISCUS project web page
•
http://www.i-discus.org
IlliGAL web site
•
http://www-illigal.ge.uiuc.edu
IlliGAL blog
•
http://illigal.blogspot.com
Xavier’s blog
•
http://gal31.ge.uiuc.edu/xllora
September 21, 2006 Xavier Llorà 39

Full NameComment goes here.florian.groebel6 years agoJuan Quiroz, Researcher and Patent Agent at Bangkok Tagsigauserfatigue7 years ago