On the Value of User Preferences in Search-Based Software Engineering:

  • 516 views
Uploaded on

Tim Menzies, …

Tim Menzies,
Abdel Salam Sayyad,
Hany Ammar

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
516
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
7
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. On#the#Value#of#User#Preferences#in# Search4Based#So7ware#Engineering:##A#Case#Study#in#So7ware#Product#Lines## # Tim#Menzies,## Abdel#Salam#Sayyad,# Hany#Ammar# # WVU,#USA# Nov’12## #
  • 2. Sound#bites#•  The#new#age#of#the#app# #•  Stop#Nnkering#with#small#stuff# #•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# #•  If#preferences#maSer# –  Then#the##best#opNmizer#understands#preferences#the#best# 2#
  • 3. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 4. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 5. WELCOME,TO,,THE,NEW,WORLDE,,, 5#
  • 6. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,e.g.#Microso7#office# •  E.g.#Apple#app#store# 6#
  • 7. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,•  Vendors#tried#to#retain#their# •  Smart#phones#and#tablet4 user#base#via#some# based#so7ware# complete#ecologies# # # •  Users#choosing#many# numbers#of#small#apps#from#•  One#so7ware#soluNon#for# different#vendors,## all#user#needs###(e.g.# –  each#performing#a#specific# Microso7#Office).## small#task.# # #•  Large,#complex,#so7ware# •  Vendors#must#quickly#and# plaorms,## conNnually#reconfigure#apps## –  To#retain#and#extend#their# –  very#slow#to#change.## customer#base.# 7#
  • 8. Feature–oriented#domain#analysis#•  Feature#maps#=#a# lightweight#method#for# defining#a#space#of##opNons#•  Product4line#configuraNons#•  Defacto#standard#for# modeling#variability## 8#
  • 9. Original#FODA#paper#:#2700+#citaNons# Half#since# 2007# 9#
  • 10. hSp://www.splot4research.org/# 200+#models,##plus#an#instance#generator# 10#
  • 11. LINUX#kernel#=#6000+#features#86%#declare#constraints#of#some#sort,##Most#features#refer#to#244#other#features.### 11#
  • 12. Need#for#beSer#automaNon#•  Such#complexity#needs#automated#support# –  especially##feature#models##combined#with# •  #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.## Search#for#valid#products:## 9#state#of#the#art#theorem# provers##[Pohl,#ASE’11]# # Bad## scalability# And#these#were# “simple”#models# 12#
  • 13. Diving##deeper#•  Much#prior#work#explored## Nny#objecNve#spaces## –  Two#or#three#objecNves# –  Or,#higher#(but#only#for#small#models)# # # #•  So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)# The# usual# 2.  That#offers#most#features# suspects# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 13# 5.  Using#features#with#least#known#defects#
  • 14. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 15. MOEA=#MulN4objecNve### evoluNonary#algorithms#######################•  Repeat#Nll#happy#or#exhausted# –  SelecNon#(cull#the#herd)# –  Cross4over#(the#rude#bit)# –  MutaNon#(stochasNc#jiggle)# 15#
  • 16. Some#MOEA#ApplicaNon#Domain Application TypesControl gas pipeline, pole balancing, missile evasion, pursuitDesign semiconductor layout, aircraft design, keyboard configuration, communication networksScheduling manufacturing, facility scheduling, resource allocationRobotics trajectory planningMachine Learning designing neural networks, improving classification algorithms, classifier systemsSignal Processing filter designGame Playing poker, checkers, prisoner’s dilemmaCombinatorial set covering, travelling salesman, routing, bin packing,Optimization graph colouring and partitioning
  • 17. MOEA#for#Search4based#SE#TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams#Requirements## #Bagnall,#Mansouri,#Zhang#Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd##Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe#Heap#allocaNon #Cohen,#Kooi,#Srisa4an##Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer##SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani##Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS#Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,## ## # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,# ## # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins#Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7#Model#checking #Alba,#Chicano,#Godefroid#Probing # # #Cohen,#Elbaum##UIOs# # # #Derderian,#Guo,#Hierons# So#study#FODA4#Comprehension #Gold,#Li,#Mahdavi# to#learn#how#to#Protocols# # #Alba,#Clark,#Jacob,#Troya# improve#these#Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo# tasks.#Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis# 17#
  • 18. Much#increased#interest## in#Search4based#SE# 18#
  • 19. The#Pareto#FronNer#•  Mutants#=#<D,O>#=#<decisions,#objecNves># –  E.g.#car# •  Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels# •  ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)# –  E.g.#learning#formula# •  Decisions:#what#variables#and#constants#to#use# •  ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may# complete)#•  Pareto#fronNer:#select#the#non4dominated#mutants# –  X#dominates#Y## •  if#for#all#objecNves,##X#is#never#any#worse##than#Y# •  If#for#one#objecNve,#X#beSer#than#Y# 19#
  • 20. Once#you#know#fronNer# •#Select#from#here#Issues:# •#Ignore#here#1)  Spread#2)  Hypervolume#3)  ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)# 20#
  • 21. The#usual#suspects:# ##=#NSGA4II### #=#SPEA2# In#this#case,#NSGA4II# gets#more#spread# Combines#N#objecNves## to#one##with#some## weighNng#scheme# 21#
  • 22. Some#details#on#the#usual#suspects#NSGA(II,, SPEA2,,•  Is#a#geneNc#algorithm# •  Is#a#geneNc#algorithm# # # •  Changes#the#definiNon#of#•  Non4dominated#sort# “dominaNon”# –  HeurisNc#way#to#fast#group# –  SPEA#(version#1)#scored#mutants#by# mutants#into#bands# how#many#others#they#dominated# # –  Got#confused#by#overlaps#in#the# dominaNon#sets#•  Crowd#pruning##via#approximate# # hypercube#around#each#mutant:# •  SPEA#(version#2):# O(Onlogn)# –  Adds#a#“local#density#factor”#to#the# dominaNon#weight# 3# –  Mutants#in#dense#areas#valued## 2# more# ## 1# •  SPEA2#beSer#than#SPEA1# 2# 22#
  • 23. Any#number#of#opNmizaNons## to#tradiNonal#GAs#•  The#history#of#MOEAs#in#the#last# •  DifferenNal#evoluNon#(Storn#1996)# 15#years#is# –  Mutate#by#interpolaNons#between# –  OpNmize#via#hybrid#GA#+#other# exisNng#mutants# search#method# # –  For#x#in#mutants## ####y#=#any1#+#extrapolate(any3#–#any2)#•  Local#search:## ####if#y#dominates#x#then#x#=#y# –  before#select,#do#a#liSle#simulated# # annealing#on#X%#of#the#populaNon# # •  Cellular#automata,#•  ScaSer#search#(Glover’s#next# •  #Ant#colony#opNmizaNon,#### generaNon#tabu#search)# –  Includes#a#liSle#local#search# •  Bayesian#staNsNcs#to#bias#the# # mutaNon,##•  ParNcle#swam#opNmizaNon# •  Etc## –  May#do#as#well#as#scaSer#search##(Yin# •  etc# and#Glover#2007)# 23#
  • 24. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 25. Three#groups#of#Algorithms#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 25#
  • 26. IBEA#•  Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences#•  I(x1,x2):# –  Least#adjust#objecNve#scores#such#that#x1#dominates#x2#•  Repeat#Nll#just#a#few#le7# –  Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else# # K=# # 0.05# # # # # –  Sort#all#instances#by#F# –  Delete#worst#•  Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors# ## 26#
  • 27. Case#studies#Data#from#hSp://www.splot4research.org/#Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/## Cross4tree# constraints# 27#
  • 28. 4#studies:# Bi,#tri,#quad,#five4#objecNves#So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤# 100%)# 2.  That#offers#most#features# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 5.  Using#features#with#least## known#defects# # # Binary#objecNves#=#1,2# Tri4objecNve#########=#1,2,3# Quad4objecNve####=#1,2,3,4# Five4objecNve######=#1,2,3,4,5# 28#
  • 29. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied# 29#
  • 30. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,1:,all#about#the#same#for#the#24objecNve#problem# 30#
  • 31. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,2:,E4shop#is#a#nasty#problem:#needs#50M#evals# 31#
  • 32. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,3:,IBEA#has#no#spread#operators,#but#gets#best#spread#32#
  • 33. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,4:,IBEA#has#no#HV#operators,#but#usually#gets#best#HV#33#
  • 34. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,5:,All#the#non4IBEA#algorithms#are#very#similar# 34#
  • 35. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,6:,IBEA#does#much,#much##beSer#on#constraints# 35#
  • 36. Why#is#this#interesNng?#Other,MOEAs, IBEA,•  The#usual#suspects#are#widely,# •  Rather#stupid#on#those# uncriNcally#used#in#many#MOEA# applicaNons# internal#tricks# –  E.g.#especially#NSGA4II#and#SPEA2# –  Just#does#a#ye#olde#crossover#•  Focused#on#internal#algorithmic# mutate#GA# tricks# –  Plus:#aggressive#exploraNon# –  Techniques#for# of#the#preference#space# •  #improving#spread## •  Improving#HV# •  And#the#net#effect#of#all# •  Avoid#overlaps#in#cross4over#of# dominated#space# those#differences# •  etc# –  BeSer#spreads#•  And#the#net#effect#of#all#those# –  BeSer#HV# differences?# –  Not#much# –  Fewer#constraint#violaNons# Conclusion:# preference#is#power# 36#
  • 37. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 38. What#about#non4MOEA#soluNons?#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 38#
  • 39. Ethan’s#complaint#•  So7ware#engineer## designs#are#o7en## nested#hierarchical## constraints#•  Ethan#Jackson,#Microso7,## advocate#for#the#Z3#SMT#solver:# –  Why#mutate#at#random,#then#check#for# constraint?# –  BeSer#to#drive#the#mutaNons#by#the# constraints?# 39#
  • 40. Dump#MOEAs?## Move#to#more#logical#forms?#Pro:, Con:,move,to,,say,,SMT,solvers, stay,with,MOEA,•  Next#generaNon#of# •  ImplementaNon#complexity# algorithm# less# –  The#next#big#thing# •  More#tools#•  BeSer#to#drive#the# •  Easier#modificaNon#and# mutaNons#by#the# experimentaNon# constraints?# •  Models#can#be#expressed# anyway#you#like# 40#
  • 41. Tree#mutaNon#•  Work#in#progress#•  Simple#adaptaNon#of#current#MOEAs#for#systems# of#hierarchical#constraints# #if#rand(0,1)<#mutaNon_probability:############Dont#mutate#if#youre#violaNng#one#of#the#rules:# # #1)#if#deselecNng#root#feature# # #2)#if#selecNng#feature#whose############################parent#is#not#selected# # #3)#if#deselecNng#feature#that##########################another#selected#feature#requires# # #4)#if#group#cardinality#violaNon# #else:# # #flip#this#bit# # # ##### # # #if#selecNng#(turning#on)#a#feature#then# # # # #turn#on#children### IBEA# # # #else#if#deselecNng#(turning#off)# stabilizes#70########################################feature#then:# Nmes#faster# # # ######### #########turn#off#all#children# 41#
  • 42. Tree#mutaNon#preserves## domain#constraints# So#what## case#for# SMT?# 42#
  • 43. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 44. Sound#bites#•  The#new#age#of#the#app# –  In,this,new,worlde:,,use,FODA, (feature(oriented,domain,analysis),•  Stop#Nnkering#with#small#stuff## –  Many,MOEAs,have,strikingly,, similar,performance,•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# –  Too,much,uncri<cal,applica<on,of,these,algorithms,•  If#preferences#maSer# –  Then#the##best#opNmizer#understands##preferences#the#best# –  IBEA:,aggressive,preference,explora<on, –  Tree,muta<on:,respect,your,domain, 44#