On the Value of User Preferences in Search-Based Software Engineering:
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

On the Value of User Preferences in Search-Based Software Engineering:

on

  • 860 views

Tim Menzies,

Tim Menzies,
Abdel Salam Sayyad,
Hany Ammar

Statistics

Views

Total Views
860
Views on SlideShare
860
Embed Views
0

Actions

Likes
1
Downloads
7
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

On the Value of User Preferences in Search-Based Software Engineering: Presentation Transcript

  • 1. On#the#Value#of#User#Preferences#in# Search4Based#So7ware#Engineering:##A#Case#Study#in#So7ware#Product#Lines## # Tim#Menzies,## Abdel#Salam#Sayyad,# Hany#Ammar# # WVU,#USA# Nov’12## #
  • 2. Sound#bites#•  The#new#age#of#the#app# #•  Stop#Nnkering#with#small#stuff# #•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# #•  If#preferences#maSer# –  Then#the##best#opNmizer#understands#preferences#the#best# 2#
  • 3. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 4. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 5. WELCOME,TO,,THE,NEW,WORLDE,,, 5#
  • 6. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,e.g.#Microso7#office# •  E.g.#Apple#app#store# 6#
  • 7. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,•  Vendors#tried#to#retain#their# •  Smart#phones#and#tablet4 user#base#via#some# based#so7ware# complete#ecologies# # # •  Users#choosing#many# numbers#of#small#apps#from#•  One#so7ware#soluNon#for# different#vendors,## all#user#needs###(e.g.# –  each#performing#a#specific# Microso7#Office).## small#task.# # #•  Large,#complex,#so7ware# •  Vendors#must#quickly#and# plaorms,## conNnually#reconfigure#apps## –  To#retain#and#extend#their# –  very#slow#to#change.## customer#base.# 7#
  • 8. Feature–oriented#domain#analysis#•  Feature#maps#=#a# lightweight#method#for# defining#a#space#of##opNons#•  Product4line#configuraNons#•  Defacto#standard#for# modeling#variability## 8#
  • 9. Original#FODA#paper#:#2700+#citaNons# Half#since# 2007# 9#
  • 10. hSp://www.splot4research.org/# 200+#models,##plus#an#instance#generator# 10#
  • 11. LINUX#kernel#=#6000+#features#86%#declare#constraints#of#some#sort,##Most#features#refer#to#244#other#features.### 11#
  • 12. Need#for#beSer#automaNon#•  Such#complexity#needs#automated#support# –  especially##feature#models##combined#with# •  #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.## Search#for#valid#products:## 9#state#of#the#art#theorem# provers##[Pohl,#ASE’11]# # Bad## scalability# And#these#were# “simple”#models# 12#
  • 13. Diving##deeper#•  Much#prior#work#explored## Nny#objecNve#spaces## –  Two#or#three#objecNves# –  Or,#higher#(but#only#for#small#models)# # # #•  So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)# The# usual# 2.  That#offers#most#features# suspects# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 13# 5.  Using#features#with#least#known#defects#
  • 14. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 15. MOEA=#MulN4objecNve### evoluNonary#algorithms#######################•  Repeat#Nll#happy#or#exhausted# –  SelecNon#(cull#the#herd)# –  Cross4over#(the#rude#bit)# –  MutaNon#(stochasNc#jiggle)# 15#
  • 16. Some#MOEA#ApplicaNon#Domain Application TypesControl gas pipeline, pole balancing, missile evasion, pursuitDesign semiconductor layout, aircraft design, keyboard configuration, communication networksScheduling manufacturing, facility scheduling, resource allocationRobotics trajectory planningMachine Learning designing neural networks, improving classification algorithms, classifier systemsSignal Processing filter designGame Playing poker, checkers, prisoner’s dilemmaCombinatorial set covering, travelling salesman, routing, bin packing,Optimization graph colouring and partitioning
  • 17. MOEA#for#Search4based#SE#TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams#Requirements## #Bagnall,#Mansouri,#Zhang#Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd##Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe#Heap#allocaNon #Cohen,#Kooi,#Srisa4an##Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer##SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani##Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS#Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,## ## # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,# ## # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins#Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7#Model#checking #Alba,#Chicano,#Godefroid#Probing # # #Cohen,#Elbaum##UIOs# # # #Derderian,#Guo,#Hierons# So#study#FODA4#Comprehension #Gold,#Li,#Mahdavi# to#learn#how#to#Protocols# # #Alba,#Clark,#Jacob,#Troya# improve#these#Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo# tasks.#Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis# 17#
  • 18. Much#increased#interest## in#Search4based#SE# 18#
  • 19. The#Pareto#FronNer#•  Mutants#=#<D,O>#=#<decisions,#objecNves># –  E.g.#car# •  Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels# •  ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)# –  E.g.#learning#formula# •  Decisions:#what#variables#and#constants#to#use# •  ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may# complete)#•  Pareto#fronNer:#select#the#non4dominated#mutants# –  X#dominates#Y## •  if#for#all#objecNves,##X#is#never#any#worse##than#Y# •  If#for#one#objecNve,#X#beSer#than#Y# 19#
  • 20. Once#you#know#fronNer# •#Select#from#here#Issues:# •#Ignore#here#1)  Spread#2)  Hypervolume#3)  ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)# 20#
  • 21. The#usual#suspects:# ##=#NSGA4II### #=#SPEA2# In#this#case,#NSGA4II# gets#more#spread# Combines#N#objecNves## to#one##with#some## weighNng#scheme# 21#
  • 22. Some#details#on#the#usual#suspects#NSGA(II,, SPEA2,,•  Is#a#geneNc#algorithm# •  Is#a#geneNc#algorithm# # # •  Changes#the#definiNon#of#•  Non4dominated#sort# “dominaNon”# –  HeurisNc#way#to#fast#group# –  SPEA#(version#1)#scored#mutants#by# mutants#into#bands# how#many#others#they#dominated# # –  Got#confused#by#overlaps#in#the# dominaNon#sets#•  Crowd#pruning##via#approximate# # hypercube#around#each#mutant:# •  SPEA#(version#2):# O(Onlogn)# –  Adds#a#“local#density#factor”#to#the# dominaNon#weight# 3# –  Mutants#in#dense#areas#valued## 2# more# ## 1# •  SPEA2#beSer#than#SPEA1# 2# 22#
  • 23. Any#number#of#opNmizaNons## to#tradiNonal#GAs#•  The#history#of#MOEAs#in#the#last# •  DifferenNal#evoluNon#(Storn#1996)# 15#years#is# –  Mutate#by#interpolaNons#between# –  OpNmize#via#hybrid#GA#+#other# exisNng#mutants# search#method# # –  For#x#in#mutants## ####y#=#any1#+#extrapolate(any3#–#any2)#•  Local#search:## ####if#y#dominates#x#then#x#=#y# –  before#select,#do#a#liSle#simulated# # annealing#on#X%#of#the#populaNon# # •  Cellular#automata,#•  ScaSer#search#(Glover’s#next# •  #Ant#colony#opNmizaNon,#### generaNon#tabu#search)# –  Includes#a#liSle#local#search# •  Bayesian#staNsNcs#to#bias#the# # mutaNon,##•  ParNcle#swam#opNmizaNon# •  Etc## –  May#do#as#well#as#scaSer#search##(Yin# •  etc# and#Glover#2007)# 23#
  • 24. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 25. Three#groups#of#Algorithms#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 25#
  • 26. IBEA#•  Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences#•  I(x1,x2):# –  Least#adjust#objecNve#scores#such#that#x1#dominates#x2#•  Repeat#Nll#just#a#few#le7# –  Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else# # K=# # 0.05# # # # # –  Sort#all#instances#by#F# –  Delete#worst#•  Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors# ## 26#
  • 27. Case#studies#Data#from#hSp://www.splot4research.org/#Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/## Cross4tree# constraints# 27#
  • 28. 4#studies:# Bi,#tri,#quad,#five4#objecNves#So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤# 100%)# 2.  That#offers#most#features# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 5.  Using#features#with#least## known#defects# # # Binary#objecNves#=#1,2# Tri4objecNve#########=#1,2,3# Quad4objecNve####=#1,2,3,4# Five4objecNve######=#1,2,3,4,5# 28#
  • 29. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied# 29#
  • 30. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,1:,all#about#the#same#for#the#24objecNve#problem# 30#
  • 31. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,2:,E4shop#is#a#nasty#problem:#needs#50M#evals# 31#
  • 32. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,3:,IBEA#has#no#spread#operators,#but#gets#best#spread#32#
  • 33. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,4:,IBEA#has#no#HV#operators,#but#usually#gets#best#HV#33#
  • 34. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,5:,All#the#non4IBEA#algorithms#are#very#similar# 34#
  • 35. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,6:,IBEA#does#much,#much##beSer#on#constraints# 35#
  • 36. Why#is#this#interesNng?#Other,MOEAs, IBEA,•  The#usual#suspects#are#widely,# •  Rather#stupid#on#those# uncriNcally#used#in#many#MOEA# applicaNons# internal#tricks# –  E.g.#especially#NSGA4II#and#SPEA2# –  Just#does#a#ye#olde#crossover#•  Focused#on#internal#algorithmic# mutate#GA# tricks# –  Plus:#aggressive#exploraNon# –  Techniques#for# of#the#preference#space# •  #improving#spread## •  Improving#HV# •  And#the#net#effect#of#all# •  Avoid#overlaps#in#cross4over#of# dominated#space# those#differences# •  etc# –  BeSer#spreads#•  And#the#net#effect#of#all#those# –  BeSer#HV# differences?# –  Not#much# –  Fewer#constraint#violaNons# Conclusion:# preference#is#power# 36#
  • 37. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 38. What#about#non4MOEA#soluNons?#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 38#
  • 39. Ethan’s#complaint#•  So7ware#engineer## designs#are#o7en## nested#hierarchical## constraints#•  Ethan#Jackson,#Microso7,## advocate#for#the#Z3#SMT#solver:# –  Why#mutate#at#random,#then#check#for# constraint?# –  BeSer#to#drive#the#mutaNons#by#the# constraints?# 39#
  • 40. Dump#MOEAs?## Move#to#more#logical#forms?#Pro:, Con:,move,to,,say,,SMT,solvers, stay,with,MOEA,•  Next#generaNon#of# •  ImplementaNon#complexity# algorithm# less# –  The#next#big#thing# •  More#tools#•  BeSer#to#drive#the# •  Easier#modificaNon#and# mutaNons#by#the# experimentaNon# constraints?# •  Models#can#be#expressed# anyway#you#like# 40#
  • 41. Tree#mutaNon#•  Work#in#progress#•  Simple#adaptaNon#of#current#MOEAs#for#systems# of#hierarchical#constraints# #if#rand(0,1)<#mutaNon_probability:############Dont#mutate#if#youre#violaNng#one#of#the#rules:# # #1)#if#deselecNng#root#feature# # #2)#if#selecNng#feature#whose############################parent#is#not#selected# # #3)#if#deselecNng#feature#that##########################another#selected#feature#requires# # #4)#if#group#cardinality#violaNon# #else:# # #flip#this#bit# # # ##### # # #if#selecNng#(turning#on)#a#feature#then# # # # #turn#on#children### IBEA# # # #else#if#deselecNng#(turning#off)# stabilizes#70########################################feature#then:# Nmes#faster# # # ######### #########turn#off#all#children# 41#
  • 42. Tree#mutaNon#preserves## domain#constraints# So#what## case#for# SMT?# 42#
  • 43. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 44. Sound#bites#•  The#new#age#of#the#app# –  In,this,new,worlde:,,use,FODA, (feature(oriented,domain,analysis),•  Stop#Nnkering#with#small#stuff## –  Many,MOEAs,have,strikingly,, similar,performance,•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# –  Too,much,uncri<cal,applica<on,of,these,algorithms,•  If#preferences#maSer# –  Then#the##best#opNmizer#understands##preferences#the#best# –  IBEA:,aggressive,preference,explora<on, –  Tree,muta<on:,respect,your,domain, 44#