Your SlideShare is downloading. ×
On the Value of User Preferences in Search-Based Software Engineering:
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

On the Value of User Preferences in Search-Based Software Engineering:

536
views

Published on

Tim Menzies, …

Tim Menzies,
Abdel Salam Sayyad,
Hany Ammar

Published in: Education

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
536
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. On#the#Value#of#User#Preferences#in# Search4Based#So7ware#Engineering:##A#Case#Study#in#So7ware#Product#Lines## # Tim#Menzies,## Abdel#Salam#Sayyad,# Hany#Ammar# # WVU,#USA# Nov’12## #
  • 2. Sound#bites#•  The#new#age#of#the#app# #•  Stop#Nnkering#with#small#stuff# #•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# #•  If#preferences#maSer# –  Then#the##best#opNmizer#understands#preferences#the#best# 2#
  • 3. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 4. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 5. WELCOME,TO,,THE,NEW,WORLDE,,, 5#
  • 6. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,e.g.#Microso7#office# •  E.g.#Apple#app#store# 6#
  • 7. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,,•  Vendors#tried#to#retain#their# •  Smart#phones#and#tablet4 user#base#via#some# based#so7ware# complete#ecologies# # # •  Users#choosing#many# numbers#of#small#apps#from#•  One#so7ware#soluNon#for# different#vendors,## all#user#needs###(e.g.# –  each#performing#a#specific# Microso7#Office).## small#task.# # #•  Large,#complex,#so7ware# •  Vendors#must#quickly#and# plaorms,## conNnually#reconfigure#apps## –  To#retain#and#extend#their# –  very#slow#to#change.## customer#base.# 7#
  • 8. Feature–oriented#domain#analysis#•  Feature#maps#=#a# lightweight#method#for# defining#a#space#of##opNons#•  Product4line#configuraNons#•  Defacto#standard#for# modeling#variability## 8#
  • 9. Original#FODA#paper#:#2700+#citaNons# Half#since# 2007# 9#
  • 10. hSp://www.splot4research.org/# 200+#models,##plus#an#instance#generator# 10#
  • 11. LINUX#kernel#=#6000+#features#86%#declare#constraints#of#some#sort,##Most#features#refer#to#244#other#features.### 11#
  • 12. Need#for#beSer#automaNon#•  Such#complexity#needs#automated#support# –  especially##feature#models##combined#with# •  #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.## Search#for#valid#products:## 9#state#of#the#art#theorem# provers##[Pohl,#ASE’11]# # Bad## scalability# And#these#were# “simple”#models# 12#
  • 13. Diving##deeper#•  Much#prior#work#explored## Nny#objecNve#spaces## –  Two#or#three#objecNves# –  Or,#higher#(but#only#for#small#models)# # # #•  So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)# The# usual# 2.  That#offers#most#features# suspects# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 13# 5.  Using#features#with#least#known#defects#
  • 14. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 15. MOEA=#MulN4objecNve### evoluNonary#algorithms#######################•  Repeat#Nll#happy#or#exhausted# –  SelecNon#(cull#the#herd)# –  Cross4over#(the#rude#bit)# –  MutaNon#(stochasNc#jiggle)# 15#
  • 16. Some#MOEA#ApplicaNon#Domain Application TypesControl gas pipeline, pole balancing, missile evasion, pursuitDesign semiconductor layout, aircraft design, keyboard configuration, communication networksScheduling manufacturing, facility scheduling, resource allocationRobotics trajectory planningMachine Learning designing neural networks, improving classification algorithms, classifier systemsSignal Processing filter designGame Playing poker, checkers, prisoner’s dilemmaCombinatorial set covering, travelling salesman, routing, bin packing,Optimization graph colouring and partitioning
  • 17. MOEA#for#Search4based#SE#TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams#Requirements## #Bagnall,#Mansouri,#Zhang#Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd##Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe#Heap#allocaNon #Cohen,#Kooi,#Srisa4an##Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer##SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani##Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS#Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,## ## # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,# ## # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins#Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7#Model#checking #Alba,#Chicano,#Godefroid#Probing # # #Cohen,#Elbaum##UIOs# # # #Derderian,#Guo,#Hierons# So#study#FODA4#Comprehension #Gold,#Li,#Mahdavi# to#learn#how#to#Protocols# # #Alba,#Clark,#Jacob,#Troya# improve#these#Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo# tasks.#Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis# 17#
  • 18. Much#increased#interest## in#Search4based#SE# 18#
  • 19. The#Pareto#FronNer#•  Mutants#=#<D,O>#=#<decisions,#objecNves># –  E.g.#car# •  Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels# •  ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)# –  E.g.#learning#formula# •  Decisions:#what#variables#and#constants#to#use# •  ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may# complete)#•  Pareto#fronNer:#select#the#non4dominated#mutants# –  X#dominates#Y## •  if#for#all#objecNves,##X#is#never#any#worse##than#Y# •  If#for#one#objecNve,#X#beSer#than#Y# 19#
  • 20. Once#you#know#fronNer# •#Select#from#here#Issues:# •#Ignore#here#1)  Spread#2)  Hypervolume#3)  ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)# 20#
  • 21. The#usual#suspects:# ##=#NSGA4II### #=#SPEA2# In#this#case,#NSGA4II# gets#more#spread# Combines#N#objecNves## to#one##with#some## weighNng#scheme# 21#
  • 22. Some#details#on#the#usual#suspects#NSGA(II,, SPEA2,,•  Is#a#geneNc#algorithm# •  Is#a#geneNc#algorithm# # # •  Changes#the#definiNon#of#•  Non4dominated#sort# “dominaNon”# –  HeurisNc#way#to#fast#group# –  SPEA#(version#1)#scored#mutants#by# mutants#into#bands# how#many#others#they#dominated# # –  Got#confused#by#overlaps#in#the# dominaNon#sets#•  Crowd#pruning##via#approximate# # hypercube#around#each#mutant:# •  SPEA#(version#2):# O(Onlogn)# –  Adds#a#“local#density#factor”#to#the# dominaNon#weight# 3# –  Mutants#in#dense#areas#valued## 2# more# ## 1# •  SPEA2#beSer#than#SPEA1# 2# 22#
  • 23. Any#number#of#opNmizaNons## to#tradiNonal#GAs#•  The#history#of#MOEAs#in#the#last# •  DifferenNal#evoluNon#(Storn#1996)# 15#years#is# –  Mutate#by#interpolaNons#between# –  OpNmize#via#hybrid#GA#+#other# exisNng#mutants# search#method# # –  For#x#in#mutants## ####y#=#any1#+#extrapolate(any3#–#any2)#•  Local#search:## ####if#y#dominates#x#then#x#=#y# –  before#select,#do#a#liSle#simulated# # annealing#on#X%#of#the#populaNon# # •  Cellular#automata,#•  ScaSer#search#(Glover’s#next# •  #Ant#colony#opNmizaNon,#### generaNon#tabu#search)# –  Includes#a#liSle#local#search# •  Bayesian#staNsNcs#to#bias#the# # mutaNon,##•  ParNcle#swam#opNmizaNon# •  Etc## –  May#do#as#well#as#scaSer#search##(Yin# •  etc# and#Glover#2007)# 23#
  • 24. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 25. Three#groups#of#Algorithms#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 25#
  • 26. IBEA#•  Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences#•  I(x1,x2):# –  Least#adjust#objecNve#scores#such#that#x1#dominates#x2#•  Repeat#Nll#just#a#few#le7# –  Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else# # K=# # 0.05# # # # # –  Sort#all#instances#by#F# –  Delete#worst#•  Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors# ## 26#
  • 27. Case#studies#Data#from#hSp://www.splot4research.org/#Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/## Cross4tree# constraints# 27#
  • 28. 4#studies:# Bi,#tri,#quad,#five4#objecNves#So7ware#engineering#=#navigaNng#compeNng#concerns# 1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤# 100%)# 2.  That#offers#most#features# 3.  Build#“stuff”#In#least#Nme# 4.  That#we#have#used#most#before# 5.  Using#features#with#least## known#defects# # # Binary#objecNves#=#1,2# Tri4objecNve#########=#1,2,3# Quad4objecNve####=#1,2,3,4# Five4objecNve######=#1,2,3,4,5# 28#
  • 29. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied# 29#
  • 30. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,1:,all#about#the#same#for#the#24objecNve#problem# 30#
  • 31. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,2:,E4shop#is#a#nasty#problem:#needs#50M#evals# 31#
  • 32. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,3:,IBEA#has#no#spread#operators,#but#gets#best#spread#32#
  • 33. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,4:,IBEA#has#no#HV#operators,#but#usually#gets#best#HV#33#
  • 34. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,5:,All#the#non4IBEA#algorithms#are#very#similar# 34#
  • 35. #HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#Comment,6:,IBEA#does#much,#much##beSer#on#constraints# 35#
  • 36. Why#is#this#interesNng?#Other,MOEAs, IBEA,•  The#usual#suspects#are#widely,# •  Rather#stupid#on#those# uncriNcally#used#in#many#MOEA# applicaNons# internal#tricks# –  E.g.#especially#NSGA4II#and#SPEA2# –  Just#does#a#ye#olde#crossover#•  Focused#on#internal#algorithmic# mutate#GA# tricks# –  Plus:#aggressive#exploraNon# –  Techniques#for# of#the#preference#space# •  #improving#spread## •  Improving#HV# •  And#the#net#effect#of#all# •  Avoid#overlaps#in#cross4over#of# dominated#space# those#differences# •  etc# –  BeSer#spreads#•  And#the#net#effect#of#all#those# –  BeSer#HV# differences?# –  Not#much# –  Fewer#constraint#violaNons# Conclusion:# preference#is#power# 36#
  • 37. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 38. What#about#non4MOEA#soluNons?#DominaNon# DominaNon#Is#a#binary# PSO# Is#a#conNnuous#concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 38#
  • 39. Ethan’s#complaint#•  So7ware#engineer## designs#are#o7en## nested#hierarchical## constraints#•  Ethan#Jackson,#Microso7,## advocate#for#the#Z3#SMT#solver:# –  Why#mutate#at#random,#then#check#for# constraint?# –  BeSer#to#drive#the#mutaNons#by#the# constraints?# 39#
  • 40. Dump#MOEAs?## Move#to#more#logical#forms?#Pro:, Con:,move,to,,say,,SMT,solvers, stay,with,MOEA,•  Next#generaNon#of# •  ImplementaNon#complexity# algorithm# less# –  The#next#big#thing# •  More#tools#•  BeSer#to#drive#the# •  Easier#modificaNon#and# mutaNons#by#the# experimentaNon# constraints?# •  Models#can#be#expressed# anyway#you#like# 40#
  • 41. Tree#mutaNon#•  Work#in#progress#•  Simple#adaptaNon#of#current#MOEAs#for#systems# of#hierarchical#constraints# #if#rand(0,1)<#mutaNon_probability:############Dont#mutate#if#youre#violaNng#one#of#the#rules:# # #1)#if#deselecNng#root#feature# # #2)#if#selecNng#feature#whose############################parent#is#not#selected# # #3)#if#deselecNng#feature#that##########################another#selected#feature#requires# # #4)#if#group#cardinality#violaNon# #else:# # #flip#this#bit# # # ##### # # #if#selecNng#(turning#on)#a#feature#then# # # # #turn#on#children### IBEA# # # #else#if#deselecNng#(turning#off)# stabilizes#70########################################feature#then:# Nmes#faster# # # ######### #########turn#off#all#children# 41#
  • 42. Tree#mutaNon#preserves## domain#constraints# So#what## case#for# SMT?# 42#
  • 43. Roadmap# ①  Feature(based,SE, ②  Algorithms, ③  IBEA, ④  Tree,muta<on, ⑤  Conclusion, #
  • 44. Sound#bites#•  The#new#age#of#the#app# –  In,this,new,worlde:,,use,FODA, (feature(oriented,domain,analysis),•  Stop#Nnkering#with#small#stuff## –  Many,MOEAs,have,strikingly,, similar,performance,•  Enough#with#the#usual#suspects:## –  NSGA4II,#SPEA2,#etc# –  Too,much,uncri<cal,applica<on,of,these,algorithms,•  If#preferences#maSer# –  Then#the##best#opNmizer#understands##preferences#the#best# –  IBEA:,aggressive,preference,explora<on, –  Tree,muta<on:,respect,your,domain, 44#