Your SlideShare is downloading. ×
  • Like
On the Value of User Preferences in Search-Based Software Engineering:
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

On the Value of User Preferences in Search-Based Software Engineering:

  • 529 views
Published

Tim Menzies, …

Tim Menzies,
Abdel Salam Sayyad,
Hany Ammar

Published in Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
529
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
7
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. On#the#Value#of#User#Preferences#in# Search4Based#So7ware#Engineering:##A#Case#Study#in#So7ware#Product#Lines## # Tim#Menzies,## Abdel#Salam#Sayyad,# Hany#Ammar# # WVU,#USA# Nov’12## #
  • 2. Sound#bites#•  The#new#age#of#the#app# #•  Stop#Nnkering#with#small#stuff# #•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# #•  If#preferences#maSer# –  Then#the##best#opNmizer#understands#preferences#the#best# 2#
  • 3. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 4. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 5. WELCOME,TO,,THE,NEW,WORLDE,,, 5#
  • 6. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,e.g.#Microso7#office# •  E.g.#Apple#app#store# 6#
  • 7. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,•  Vendors#tried#to#retain#their# •  Smart#phones#and#tablet4 user#base#via#some# based#so7ware# complete#ecologies# # # •  Users#choosing#many# numbers#of#small#apps#from#•  One#so7ware#soluNon#for# different#vendors,## all#user#needs###(e.g.# –  each#performing#a#specific# Microso7#Office).## small#task.# # #•  Large,#complex,#so7ware# •  Vendors#must#quickly#and# plaorms,## conNnually#reconfigure#apps## –  To#retain#and#extend#their# –  very#slow#to#change.## customer#base.# 7#
  • 8. Feature–oriented#domain#analysis#•  Feature#maps#=#a# lightweight#method#for# defining#a#space#of##opNons#•  Product4line#configuraNons#•  Defacto#standard#for# modeling#variability## 8#
  • 9. Original#FODA#paper#:#2700+#citaNons# Half#since# 2007# 9#
  • 10. hSp://www.splot4research.org/# 200+#models,##plus#an#instance#generator# 10#
  • 11. LINUX#kernel#=#6000+#features#86%#declare#constraints#of#some#sort,##Most#features#refer#to#244#other#features.### 11#
  • 12. Need#for#beSer#automaNon#•  Such#complexity#needs#automated#support# –  especially##feature#models##combined#with# •  #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.## Search#for#valid#products:## 9#state#of#the#art#theorem# provers##[Pohl,#ASE’11]# # Bad## scalability# And#these#were# “simple”#models# 12#
  • 13. Diving##deeper#•  Much#prior#work#explored## Nny#objecNve#spaces## –  Two#or#three#objecNves# –  Or,#higher#(but#only#for#small#models)# # # #•  So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)# The# usual# 2.  That#offers#most#features# suspects# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 13# 5.  Using#features#with#least#known#defects#
  • 14. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 15. MOEA=#MulN4objecNve### evoluNonary#algorithms#######################•  Repeat#Nll#happy#or#exhausted# –  SelecNon#(cull#the#herd)# –  Cross4over#(the#rude#bit)# –  MutaNon#(stochasNc#jiggle)# 15#
  • 16. Some#MOEA#ApplicaNon#Domain Application TypesControl gas pipeline, pole balancing, missile evasion, pursuitDesign semiconductor layout, aircraft design, keyboard configuration, communication networksScheduling manufacturing, facility scheduling, resource allocationRobotics trajectory planningMachine Learning designing neural networks, improving classification algorithms, classifier systemsSignal Processing filter designGame Playing poker, checkers, prisoner’s dilemmaCombinatorial set covering, travelling salesman, routing, bin packing,Optimization graph colouring and partitioning
  • 17. MOEA#for#Search4based#SE#TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams#Requirements## #Bagnall,#Mansouri,#Zhang#Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd##Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe#Heap#allocaNon #Cohen,#Kooi,#Srisa4an##Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer##SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani##Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS#Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,## ## # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,# ## # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins#Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7#Model#checking #Alba,#Chicano,#Godefroid#Probing # # #Cohen,#Elbaum##UIOs# # # #Derderian,#Guo,#Hierons# So#study#FODA4#Comprehension #Gold,#Li,#Mahdavi# to#learn#how#to#Protocols# # #Alba,#Clark,#Jacob,#Troya# improve#these#Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo# tasks.#Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis# 17#
  • 18. Much#increased#interest## in#Search4based#SE# 18#
  • 19. The#Pareto#FronNer#•  Mutants#=#<D,O>#=#<decisions,#objecNves># –  E.g.#car# •  Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels# •  ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)# –  E.g.#learning#formula# •  Decisions:#what#variables#and#constants#to#use# •  ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may# complete)#•  Pareto#fronNer:#select#the#non4dominated#mutants# –  X#dominates#Y## •  if#for#all#objecNves,##X#is#never#any#worse##than#Y# •  If#for#one#objecNve,#X#beSer#than#Y# 19#
  • 20. Once#you#know#fronNer# •#Select#from#here#Issues:# •#Ignore#here#1)  Spread#2)  Hypervolume#3)  ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)# 20#
  • 21. The#usual#suspects:# ##=#NSGA4II### #=#SPEA2# In#this#case,#NSGA4II# gets#more#spread# Combines#N#objecNves## to#one##with#some## weighNng#scheme# 21#
  • 22. Some#details#on#the#usual#suspects#NSGA(II,, SPEA2,,•  Is#a#geneNc#algorithm# •  Is#a#geneNc#algorithm# # # •  Changes#the#definiNon#of#•  Non4dominated#sort# “dominaNon”# –  HeurisNc#way#to#fast#group# –  SPEA#(version#1)#scored#mutants#by# mutants#into#bands# how#many#others#they#dominated# # –  Got#confused#by#overlaps#in#the# dominaNon#sets#•  Crowd#pruning##via#approximate# # hypercube#around#each#mutant:# •  SPEA#(version#2):# O(Onlogn)# –  Adds#a#“local#density#factor”#to#the# dominaNon#weight# 3# –  Mutants#in#dense#areas#valued## 2# more# ## 1# •  SPEA2#beSer#than#SPEA1# 2# 22#
  • 23. Any#number#of#opNmizaNons## to#tradiNonal#GAs#•  The#history#of#MOEAs#in#the#last# •  DifferenNal#evoluNon#(Storn#1996)# 15#years#is# –  Mutate#by#interpolaNons#between# –  OpNmize#via#hybrid#GA#+#other# exisNng#mutants# search#method# # –  For#x#in#mutants## ####y#=#any1#+#extrapolate(any3#–#any2)#•  Local#search:## ####if#y#dominates#x#then#x#=#y# –  before#select,#do#a#liSle#simulated# # annealing#on#X%#of#the#populaNon# # •  Cellular#automata,#•  ScaSer#search#(Glover’s#next# •  #Ant#colony#opNmizaNon,#### generaNon#tabu#search)# –  Includes#a#liSle#local#search# •  Bayesian#staNsNcs#to#bias#the# # mutaNon,##•  ParNcle#swam#opNmizaNon# •  Etc## –  May#do#as#well#as#scaSer#search##(Yin# •  etc# and#Glover#2007)# 23#
  • 24. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 25. Three#groups#of#Algorithms#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 25#
  • 26. IBEA#•  Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences#•  I(x1,x2):# –  Least#adjust#objecNve#scores#such#that#x1#dominates#x2#•  Repeat#Nll#just#a#few#le7# –  Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else# # K=# # 0.05# # # # # –  Sort#all#instances#by#F# –  Delete#worst#•  Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors# ## 26#
  • 27. Case#studies#Data#from#hSp://www.splot4research.org/#Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/## Cross4tree# constraints# 27#
  • 28. 4#studies:# Bi,#tri,#quad,#five4#objecNves#So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤# 100%)# 2.  That#offers#most#features# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 5.  Using#features#with#least## known#defects# # # Binary#objecNves#=#1,2# Tri4objecNve#########=#1,2,3# Quad4objecNve####=#1,2,3,4# Five4objecNve######=#1,2,3,4,5# 28#
  • 29. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied# 29#
  • 30. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,1:,all#about#the#same#for#the#24objecNve#problem# 30#
  • 31. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,2:,E4shop#is#a#nasty#problem:#needs#50M#evals# 31#
  • 32. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,3:,IBEA#has#no#spread#operators,#but#gets#best#spread#32#
  • 33. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,4:,IBEA#has#no#HV#operators,#but#usually#gets#best#HV#33#
  • 34. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,5:,All#the#non4IBEA#algorithms#are#very#similar# 34#
  • 35. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,6:,IBEA#does#much,#much##beSer#on#constraints# 35#
  • 36. Why#is#this#interesNng?#Other,MOEAs, IBEA,•  The#usual#suspects#are#widely,# •  Rather#stupid#on#those# uncriNcally#used#in#many#MOEA# applicaNons# internal#tricks# –  E.g.#especially#NSGA4II#and#SPEA2# –  Just#does#a#ye#olde#crossover#•  Focused#on#internal#algorithmic# mutate#GA# tricks# –  Plus:#aggressive#exploraNon# –  Techniques#for# of#the#preference#space# •  #improving#spread## •  Improving#HV# •  And#the#net#effect#of#all# •  Avoid#overlaps#in#cross4over#of# dominated#space# those#differences# •  etc# –  BeSer#spreads#•  And#the#net#effect#of#all#those# –  BeSer#HV# differences?# –  Not#much# –  Fewer#constraint#violaNons# Conclusion:# preference#is#power# 36#
  • 37. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 38. What#about#non4MOEA#soluNons?#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 38#
  • 39. Ethan’s#complaint#•  So7ware#engineer## designs#are#o7en## nested#hierarchical## constraints#•  Ethan#Jackson,#Microso7,## advocate#for#the#Z3#SMT#solver:# –  Why#mutate#at#random,#then#check#for# constraint?# –  BeSer#to#drive#the#mutaNons#by#the# constraints?# 39#
  • 40. Dump#MOEAs?## Move#to#more#logical#forms?#Pro:, Con:,move,to,,say,,SMT,solvers, stay,with,MOEA,•  Next#generaNon#of# •  ImplementaNon#complexity# algorithm# less# –  The#next#big#thing# •  More#tools#•  BeSer#to#drive#the# •  Easier#modificaNon#and# mutaNons#by#the# experimentaNon# constraints?# •  Models#can#be#expressed# anyway#you#like# 40#
  • 41. Tree#mutaNon#•  Work#in#progress#•  Simple#adaptaNon#of#current#MOEAs#for#systems# of#hierarchical#constraints# #if#rand(0,1)<#mutaNon_probability:############Dont#mutate#if#youre#violaNng#one#of#the#rules:# # #1)#if#deselecNng#root#feature# # #2)#if#selecNng#feature#whose############################parent#is#not#selected# # #3)#if#deselecNng#feature#that##########################another#selected#feature#requires# # #4)#if#group#cardinality#violaNon# #else:# # #flip#this#bit# # # ##### # # #if#selecNng#(turning#on)#a#feature#then# # # # #turn#on#children### IBEA# # # #else#if#deselecNng#(turning#off)# stabilizes#70########################################feature#then:# Nmes#faster# # # ######### #########turn#off#all#children# 41#
  • 42. Tree#mutaNon#preserves## domain#constraints# So#what## case#for# SMT?# 42#
  • 43. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 44. Sound#bites#•  The#new#age#of#the#app# –  In,this,new,worlde:,,use,FODA, (feature(oriented,domain,analysis),•  Stop#Nnkering#with#small#stuff## –  Many,MOEAs,have,strikingly,, similar,performance,•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# –  Too,much,uncri<cal,applica<on,of,these,algorithms,•  If#preferences#maSer# –  Then#the##best#opNmizer#understands##preferences#the#best# –  IBEA:,aggressive,preference,explora<on, –  Tree,muta<on:,respect,your,domain, 44#