• Save

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Combating User Fatigue and Contradictions in Subjective-based Optimization Schemes

on

  • 2,145 views

Combating User Fatigue and Contradictions in Subjective-based Optimization Schemes

Combating User Fatigue and Contradictions in Subjective-based Optimization Schemes

Statistics

Views

Total Views
2,145
Slideshare-icon Views on SlideShare
2,135
Embed Views
10

Actions

Likes
2
Downloads
0
Comments
0

4 Embeds 10

http://www.illigal.uiuc.edu 4
http://engrich.liv.ac.uk 3
http://www.cse.unr.edu 2
http://www.slideshare.net 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Combating User Fatigue and Contradictions in Subjective-based Optimization Schemes Combating User Fatigue and Contradictions in Subjective-based Optimization Schemes Presentation Transcript

    • Xavier Llorà National Center for Supercomputing Applications University of Illinois at Urbana-Champaign xllora@uiuc.edu
    • Subjective-based optimization schemes? • Examples •  Affinova  The electronic sheep  Emotional prosody A common subjective-based optimization scheme •  Interactive genetic algorithms  Limitations Combating user fatigue and contradictions •  Worst-case scenario  Partial-ordering graphs, surrogate fitness, and support-vector machines Conclusions and current directions • Acknowledgments • September 21, 2006 Xavier Llorà 2
    • The goal: “Find a good solution in a given search space” • The handicap: “No quantitative measure of goodness is available” • The search space needs to be explored based on subjective criteria •  Qualitative fitness is provided by the user • Sorting of several solutions • Choosing one of solution out of a subset • Based on global impression  Limitations of subjective-based optimization: • The time user can expend doing repetitive tasks (user fatigue) • The progress achieved during the repetitive tasks (user frustration) • Criteria adaptation (user’s criteria may change during the process as a results of the process itself) Let’s see some real examples • September 21, 2006 Xavier Llorà 3
    • Subjective-based optimization schemes? ✔ • Examples •  Affinova  The electronic sheep  Emotional prosody A common subjective-based optimization scheme •  Interactive genetic algorithms  Limitations Combating user fatigue and contradictions •  Worst-case scenario  Partial-ordering graphs, surrogate fitness, and support-vector machines Conclusions and current directions • Acknowledgments • September 21, 2006 Xavier Llorà 4
    • Concept design: Create new product candidates • Affinova (http://www.affinnova.com/) • 1. Base Concept 2. Featurize 3. Create alternatives 4. Generate alternative, involve customers, and iterate September 21, 2006 Xavier Llorà 5
    • Proposed by Scott Draves (http://electricsheep.org/) • A form of aesthetic evolution, a concept first realized by Karl Sims • The goal: •  Animate and evolve artificial life-forms know as sheep  Reference to Philip K. Dirk’s novel Use screensavers to: •  Create a distributed rendering farm  Collect user votes on favorite sheep Popular sheep life longer and generate new sheep • Search space: •  The space of parameters of fractal flames,a generalization and refinement of the Iterated Function System (IFS).  Each sheep is defined by 240 floating point values to optimize September 21, 2006 Xavier Llorà 6
    • Sheep #1700 Sheep #110345149 September 21, 2006 Xavier Llorà 7
    • Cecilia Alm and Xavier Llorà [Alm & Llora, 2006] (http://www.i-discus.org) • The problem: •  Text-to-speech (TTS) synthesis (given a text the associated speech is synthesized)  The lack of emotion (neutrality) makes the TTS synthesis sound unnatural The goal: •  Adjust the TTS to incorporate emotional prosody  Spoken text should sound sad, angry, happy, etc.  Very useful for story telling or automated audio book generation Users can easily discern the emotional prosody, but hard to explain why • Search space: •  Several parameters can be tweak to modify the prosody (6 per word)  These parameters define the prosody search space September 21, 2006 Xavier Llorà 8
    • Sad Angry Strawberry Strawberry Tan-tan-tan Tan-tan-tan Bubhalos Bubhalos September 21, 2006 Xavier Llorà 9
    • Subjective-based optimization schemes? ✔ • Examples ✔ •  Affinova ✔  The electronic sheep ✔  Emotional prosody ✔ A common subjective-based optimization scheme •  Interactive genetic algorithms  Limitations Combating user fatigue and contradictions •  Worst-case scenario  Partial-ordering graphs, surrogate fitness, and support-vector machines Conclusions and current directions • Acknowledgments • September 21, 2006 Xavier Llorà 10
    • No quantitative fitness is available • Qualitative fitness is provided by the user •  Sorting of several solutions  Choosing one of solution out of a subset  Based on global impressions User fatigue •  Short time periods (1 hour)  Repetitive task Frustration •  Repeated evaluation of similar solutions  No progress is appreciated How can we help the user in the subjective-based optimization quest? • Hint: Big time-scale difference between user judgment and computer • response September 21, 2006 Xavier Llorà 11
    • Interactive Standard Genetic Algorithms Genetic Algorithms Computer Aided Human-Based Design (CAD) Genetic Algorithms [Kosorukoff & Goldberg, 2002] September 21, 2006 Xavier Llorà 12
    • Subjective-based optimization schemes? ✔ • Examples ✔ •  Affinova ✔  The electronic sheep ✔  Emotional prosody ✔ A common subjective-based optimization scheme ✔ •  Interactive genetic algorithms ✔  Limitations ✔ Combating user fatigue and contradictions •  Worst-case scenario  Partial-ordering graphs, surrogate fitness, and support-vector machines Conclusions and current directions • Acknowledgments • September 21, 2006 Xavier Llorà 13
    • ✖ Users need a clear idea of the outcome •  Need for a clear criteria of the goal to reach ✖ A good visualization goes a long way •  A non-intuitive visualization may mislead user’s evaluations ✔ Lack of numerical fitness can be a problem •  No numeric form that can be optimize is available ✔ User fatigue needs to be minimized •  User may only be able to provide reliable evaluations for short time periods (1 hour) ✔ Users tend to change their criteria along the way •  Easy to maintain an unique criteria for short time periods September 21, 2006 Xavier Llorà 14
    • Can we learn user preferences from the interaction? • If so, we can build models of its preferences • Evaluate solutions before presenting them to the user • Reduce the number of solutions (combat fatigue)  Show progress during the process (fight user frustration)  The question to answer: • How to learn a model from a simple like/don’t like scenario?  September 21, 2006 Xavier Llorà 15
    • A minimal iGA scenario • Two solutions are presented for user evaluation  Three possible evaluation outcomes:  1. The first is better than the second 2. The second is better than the first 3. Don’t know, don’t care Evaluation by comparison • The GA equivalent • Tournament selection s=2 • Can we learn a numerical model out of partial user • evaluations? September 21, 2006 Xavier Llorà 16
    • • Solution-quality order should be maintained  Any synthetic fitness needs to maintain the solution ordering provided by the user If s1≥ s2 ≥ … ≥ sn then fs(s1) ≥ fs(s2) ≥ … ≥ fs(sn) • Synthetic fitness should allow extrapolation  Any synthetic fitness needs to be able to be a good predictor of the ordering relation beyond the available evaluations collected • Let’s make some assumptions:  Imagine I have a few evaluated solutions :) September 21, 2006 Xavier Llorà 17
    • Given a fs: X → Y •  X: problem attributes  Y: numeric value Set of evaluated solutions • New solution •  Compute the k-nearest neighbor  Assign the fitness of the weighted fitness of the k-nearest neighbors No accurate prediction beyond • the current limits September 21, 2006 Xavier Llorà 18
    • • Regression model • Extrapolates beyond the evaluated solutions • Provides a direction toward improvements • How many solutions we need to have a reliable model? September 21, 2006 Xavier Llorà 19
    • • y = a*x + b • a>1 guaranties the order • Tournament only cares about the partial order among solutions! • Low cost, high error models • PAC-learning bounds tell us how many instances I need to properly train September 21, 2006 Xavier Llorà 20
    • If we have a numerical fitness: •  We can build a regression model  We can use such model for combating fitness fatigue What is it available? •  Partial ordering of solutions (binary tournament)  Incremental refinement (new evaluations) The idea: •  Binary tournament produce partial order relations  I can build a graph that contains all the user evaluations: • Nodes = unique solutions • Edges = user evaluations (I like the first one, the second one, or neither)  Compute a numeric fitness for each solution based on the graph structure September 21, 2006 Xavier Llorà 21
    • • Given a set of solutions  Start with the pair-wise comparisons on the leaves  As you move up, compare the first leaves reach on a DFS for each of the branches  Obtain the partial ordering of the solutions  Such ordering can be expressed in a graph form September 21, 2006 Xavier Llorà 22
    • Nodes are solutions • Edges represent the evaluation provided by the user • 1. One solution is better than the other (→) 2. Don’t know, don’t care () Transformed to contain only type 1 relations  Property: cycles detect user contradictions in evaluations  September 21, 2006 Xavier Llorà 23
    • • Dominance measure Dominates Dominated by Difference Real Synthetic September 21, 2006 Xavier Llorà 24
    • 1. Collect the user evaluations 2. Build the partial order graph 3. Compute the numeric fitness using the partial order graph 4. Use ε-SVM for creating a regression model 5. Use the learned regression as the synthetic fitness September 21, 2006 Xavier Llorà 25
    • • Cut down the number of human evaluations • Exploit synthetic fitness  Optimize it  Sample the best candidates  Show the best solutions to the user • The sample best solutions help combating  Fatigue (educated guess of user preference)  Frustration (produce new eureka solutions) September 21, 2006 Xavier Llorà 26
    • Evaluation Partial order update Sampling Sampling fs(x) Optimizer Synthetic fitness September 21, 2006 Xavier Llorà 27
    • • Considerations in the design process  Clear goal definition  Impact of problem visualization  Persistence of user criteria • Focus  Lack of numeric fitness  User fatigue • A simple controlled task  One Max September 21, 2006 Xavier Llorà 28
    • September 21, 2006 Xavier Llorà 29
    • • The system  A simple web application  OneMax task  No linkage learning needed  ε-SVM and a linear kernel  The compact genetic algorithm (Harik, Cantú-Paz, Goldberg, & Miller, 1999) • Set up  One user with no relation to the research  Repeated series of 10 independent runs for different problem sizes {4, 8, 12, 16, 20, 24, 28, and 32 variables}  Collect the data to compare it to a simple GA September 21, 2006 Xavier Llorà 30
    • September 21, 2006 Xavier Llorà 31
    • • Subjective-based optimization is likely to be dynamic • User criteria may vary as a results of:  Lack of attention  Difficulty to perform pair-wise accurate comparisons  The solutions seen so far may be changing the target! • As user criteria changes it is like to materialize as contradictions • These contradictions can be easily detected as cycles in the partial-order graph September 21, 2006 Xavier Llorà 32
    • [Alías & Llorà, 2006] • Iteration 1 Iteration 2 Iteration 3 September 21, 2006 Xavier Llorà 33
    • • Based on the partial-order graph • A simple one: proportion of solutions in cycles September 21, 2006 Xavier Llorà 34
    • Subjective-based optimization schemes? ✔ • Examples ✔ •  Affinova ✔  The electronic sheep ✔  Emotional prosody ✔ A common subjective-based optimization scheme ✔ •  Interactive genetic algorithms ✔  Limitations ✔ Combating user fatigue and contradictions ✔ •  Worst-case scenario ✔  Partial-ordering graphs, surrogate fitness, and support-vector machines ✔ Conclusions and current directions • Acknowledgments • September 21, 2006 Xavier Llorà 35
    • • Address the lack of numerical fitness and fatigue • Synthetic fitness model of user preferences • Optimize such model to take the advantage of the timescale difference • Remarkable speedups • Measure of user consistency (reliability of the evaluations) • Real-world applications confirm the theoretical results:  Emotional text-to-speech synthesis (two research groups)  Marketing campaign and product design (advertisement company)  Tuning of text mining analysis tools (chance discovery consortium) September 21, 2006 Xavier Llorà 36
    • • Hierarchical tournament impact • Relation between graph structure and quality of the regression method • User consistency versus problem difficulty • Managing large scale experimentation (metadata stores) • Optimization involving multiple users  How can we reliably combine partial-order graphs?  Can we create combination of partially build synthetic fitnesses?  How can we deal with multiple targets? • Users may have multiple criteria in mind • Users may not realize that even common criteria may have different subjective interpretations September 21, 2006 Xavier Llorà 37
    • • Sponsors:  National Science Foundation  Air Force Office of Scientific Research  SALERO project funded by IST EU • Not one man’s endeavor:  David E. Goldberg, Kumara Sastry, Lalitha Lakshmi (IlliGAL/UIUC)  Loretta Auvil, Michael Welge (ALG/NCSA)  Cecilia Alm (Linguistics/UIUC)  Francesc Alías, Lluís Formiga (URL)  IlliGAL labbies  Chance Discovery Consortium members September 21, 2006 Xavier Llorà 38
    • DISCUS project web page • http://www.i-discus.org IlliGAL web site • http://www-illigal.ge.uiuc.edu IlliGAL blog • http://illigal.blogspot.com Xavier’s blog • http://gal31.ge.uiuc.edu/xllora September 21, 2006 Xavier Llorà 39