Paper presented at the 35th International Conference in Software Engineering, San Francisco, USA. May 2013

1. On the Value of User Preferences in
Search-Based Software Engineering:
A Case Study in Software Product Lines
Abdel Salam Sayyad
Tim Menzies
Hany Ammar
West Virginia University, USA
International Conference on Software Engineering
May 23rd, 2013

2. Sound bites
Feature-based SE
Representation and validation: important
Multi-objective Configuration: interesting
2
Stop tinkering with small stuff!
Minor improvements welcome…
… but not insightful!
Enough with the usual suspects:
NSGA-II, SPEA2, etc.
If user preferences matter
Then the best optimizer understands preferences the best

3. Roadmap
① Feature-based SE
② Evolutionary Algorithms
③ Experiment & Results
④ Conclusion

4. Roadmap
① Feature-based SE
② Evolutionary Algorithms
③ Experiment & Results
④ Conclusion

5. Welcome to the new world!
Old world:
product-based SE
e.g. Microsoft office
New world:
app-based SE
e.g. Apple app store
5

6. The times, they are a changin’
Old world: product-based SE
e.g. Microsoft office
6
• Vendors tried to retain their
user base via some
complete ecologies
• One software solution for
all user needs (e.g.
Microsoft Office).
• Large, complex, software
platforms,
– very slow to change.

7. The times, they are a changin’
New worlde: app-based SE
• Smart phones and tablet-
based software
• Users choosing many
numbers of small apps from
different vendors,
– each performing a specific
small task.
• Vendors must quickly and
continually reconfigure apps
– To retain and extend their
customer base.
7
e.g. Apple app store

8. Feature–oriented domain analysis [Kang 1990]
• Feature models = a
lightweight method for
defining a space of options
• De facto standard for
modeling variability
8
Cross-Tree Constraints
Cross-Tree Constraints

9. What are the user preferences?
• Multi-objective optimization = navigating competing concerns
– Success criteria = choose features that achieve the user preferences!
• Suppose each feature had the following metrics:
1. Boolean USED_BEFORE?
2. Integer DEFECTS
3. Real COST
• Show me the space of “best options” according to the objectives:
1. That satisfies most domain constraints (0 ≤ #violations ≤ 100%)
2. That offers most features
3. That we have used most before
4. Using features with least known defects
5. Using features with least cost
9

10. Diving deeper
• Much prior work in SBSE (*)
--------------------------
(*) Sayyad and Ammar, RAISE’13
10
explored tiny objective spaces
(2 or 3 objectives)
Used NSGA-II
Didn’t state why!

11. Roadmap
① Feature-based SE
② Evolutionary Algorithms
③ Experiment & Results
④ Conclusion

12. Multi-objective Optimization
12
Higher-level
Decision Making
The Pareto Front
The Chosen Solution

13. Survival of the fittest
(according to NSGA-II [Deb et al. 2002])
13
Boolean dominance (x Dominates y, or does not):
- In no objective is x worse than y
- In at least one objective, x is better than y Crowd
pruning

14. Tinkering with small stuff…
ssNSGA-II: steady state [Durillo ‘09+
14
Other Algorithms: Ranking criteria:
NSGA-II
SPEA2 *Zitzler ‘01+ More focus on diversity, with a
new diversity measure.
FastPGA [Eskandari ‘07+
Borrowed criteria from
NSGA-II, and diversity
measure from SPEA2
MOCell (Cellular GA) [Nebro ‘09] NSGA-II
NSGA-IIMOCHC (re-designed selection, crossover,
and mutation) [Nebro ‘07+
Boolean Dominance
Boolean Dominance
Boolean Dominance
Boolean Dominance
Boolean Dominance

15. Indicator-Based Evolutionary
Algorithm (IBEA) [Zitzler and Kunzli ‘04+
15
1) For {old generation + new generation} do
– Add up every individual’s amount of dominance with
respect to everyone else
– Sort all instances by F
– Delete worst, recalculate, delete worst, recalculate, …
2) Then, standard GA (cross-over, mutation) on the
survivors  Create a new generation  Back to 1.
Continuous Dominance

16. An important property of IBEA
• Allows solutions to crowd close to the zero point.
16
• In our problem: Allows many solutions that achieve
violations equal or close to zero… then allows
optimization in the other 4 dimensions…
• Crowd pruning can hurt!
• Using real-valued benchmark functions, Wagner et al.
(EMO 2007) confirm that the performance of NSGA-II and
SPEA2 rapidly deteriorates with increasing dimension,
whereas indicator-based algorithms (e.g. IBEA) cope very
well with high-dimensional objective spaces.

17. Roadmap
① Feature-based SE
② Evolutionary Algorithms
③ Experiment & Results
④ Conclusion

18. 2 Case studies from SPLOT
18
Cross-tree
constraints

19. Algorithms
• 7 Algorithms from jMetal: http://jmetal.sourceforge.net/
– IBEA, NSGA-II, ssNSGA-II, SPEA2, FastPGA, MOCell, MOCHC.
19
• Features are bits in the “product” string.
• Same settings for all algorithms.

20. 4 studies:
Two, three, four, five- objectives
20
The five objectives:
1. satisfy most domain constraints (0 ≤ violations ≤ 100%)
2. offer most features
3. most used before
4. least known defects
5. least cost
Two objectives = 1,2
Three objectives = 1,2,5
Four objectives = 1,2,3,5
Five objectives = 1,2,3,4,5

21. 21
HV = hypervolume of dominated region
Spread = coverage of frontier
% correct = % of solutions that fully satisfy the constraints

22. 22Comment 1: all about the same for the 2-objective problem
HV = hypervolume of dominated region
Spread = coverage of frontier
% correct = % of solutions that fully satisfy the constraints

23. 23Comment 2: IBEA is better on HV.
HV = hypervolume of dominated region
Spread = coverage of frontier
% correct = % of solutions that fully satisfy the constraints

24. 24Comment 3: IBEA has no spread operators, but gets best spread
HV = hypervolume of dominated region
Spread = coverage of frontier
% correct = % of solutions that fully satisfy the constraints

25. 25Comment 4: All the non-IBEA algorithms are very similar
HV = hypervolume of dominated region
Spread = coverage of frontier
% correct = % of solutions that fully satisfy the constraints

26. HV = hypervolume of dominated region
Spread = coverage of frontier
% correct = % of solutions that fully satisfy the constraints
26Comment 5: IBEA does much, much better on constraints

27. HV = hypervolume of dominated region
Spread = coverage of frontier
% correct = % of solutions that fully satisfy the constraints
27Comment 6: IBEA’s advantage is more striking with E-Shop

28. Why is this interesting?
Other Algorithms
• The usual suspects are widely,
uncritically used in many
applications
– E.g. especially NSGA-II and SPEA2
• Focused on internal algorithmic
tricks
– Techniques for
• improving spread
• Improving HV
• Avoid overlaps in cross-over of
dominated space
• etc.
• And the net effect of all those
differences?
– Not much
IBEA
• Rather stupid on those
internal tricks
– Just does same old crossover
mutate GA
– Plus: aggressive exploration
of the preference space
• And the net effect of all
those differences
– Better spreads
– Better HV
– Fewer constraint violations
28
Conclusion:
preference is power

29. Roadmap
① Feature-based SE
② Evolutionary Algorithms
③ Experiment & Results
④ Conclusion

30. Current Achievements and Future Work
• ICSE’13: initial results proving IBEA’s advantage.
• CMSBSE’13: IBEA runtime improvement.
• (Under review):
– mutation operator tailored to feature tree. 2-3
orders-of-magnitude improvement in runtime.
– Scalability to 6000+ real feature model using
innovative “population seeding” method.
30
• Future Work:
– Further scalability.
– Interactive configuration.

31. Sound bites
The new age of the app
In this new world: use FODA
(feature-oriented domain analysis)
31
Stop tinkering with small stuff
Many MEOAs have strikingly
similar performance
Enough with the usual suspects:
NSGA-II, SPEA2, etc.
Too much uncritical application of these algorithms
If user preferences matter
Then the best optimizer understands preferences the best
IBEA: aggressive preference exploration
Acknowledgment
This research work was
funded by the Qatar
National Research Fund
(QNRF) under the National
Priorities Research Program
(NPRP)
Grant No.: 09-1205-2-470.