Sayyad slides ase13_v4
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Sayyad slides ase13_v4

on

  • 308 views

Scalable Product Line Configuration: ...

Scalable Product Line Configuration:
A Straw to Break the Camel’s Back

Abdel Salam Sayyad
Joseph Ingram
Tim Menzies
Hany Ammar

IEEE Automated SE,
Palo Alto, CA
Nov 2013

Statistics

Views

Total Views
308
Views on SlideShare
308
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Sayyad slides ase13_v4 Presentation Transcript

  • 1. Scalable  Product  Line  Configura4on:   A  Straw  to  Break  the  Camel’s  Back   Abdel  Salam  Sayyad   Joseph  Ingram   Tim  Menzies   Hany  Ammar  
  • 2. Sound  bites   Real  SoGware  Product  Lines…   …  are  big,  complex   …  representa4ve  feature  models  are  now  available  for  research   …  scale  up  or  die  !   What  does  it  take?   Mul4-­‐objec4ve  problem  formula4on   An  op4mizer  that  relies  heavily  on  user  preferences   Respect  domain  constraints,  and...   …  A  Straw  to  Break  the  Camel’s  Back   2  
  • 3. Roadmap   ①  State  of  the  Art   ②  Experiment  &  Results   ③  Conclusion    
  • 4. Roadmap   ①  State  of  the  Art   ②  Experiment  &  Results   ③  Conclusion    
  • 5. Feature  Models   •  Feature  models  =  a   lightweight  method  for   defining  a  space  of    op4ons   •  De  facto  standard  for   modeling  variability       Cross-­‐Tree  Constraints   Size  ?  10  Features,  8  Rules   Cross-­‐Tree  Constraints   5  
  • 6. Feature  Model  Repository   Size  ?  290  Features,  421  Rules     cardinal  =  2.26x1049  [Pohl  ‘11]   6  
  • 7. Feature  Models  of  Real  SoGware  Projects   Size  ?  6888  Features;  344,000  Rules   cardinal?  Forgeddabou4t!   7  
  • 8. Size  isn’t  all…   Property   Size   Constraints   Feature  Groups   Leaves   SPLOT   Linux   Significantly  smaller   Significantly  larger   Significantly  less   Significantly  more   High  ra4o   Low  ra4o   Deeper   Shallower   T.  Berger,  S.  She,  R.  Lotufo,  A.  Wasowski,  and  K.  Czarnecki,  "A  Study  of  Variability   Models  and  Languages  in  the  Systems  SoGware  Domain,"  IEEE  Tran  So<  Eng.  2013   8  
  • 9. A  Mul4-­‐Objec4ve  Formula4on   Mul4-­‐objec4ve  op4miza4on  =  naviga4ng  compe4ng  concerns   –  Success  criteria  =  choose  features  that  achieve  the  user  preferences!   Suppose  each  feature  had  the  following  metrics:     1.  Boolean    USED_BEFORE?   2.  Integer  DEFECTS   3.  Real    COST   Show  me  the  space  of  “best  op4ons”  according  to  the  objec4ves:   1.  2.  3.  4.  5.  That  sa4sfies  most  domain  constraints  (0  ≤    #viola4ons  ≤  100%)   That  offers  most  features   That  we  have  used  most  before   Using  features  with  least  known  defects   Using  features  with  least  cost     9  
  • 10. No  single  “op4mum”  solu4on   The  Pareto  Front   Higher-­‐level   Decision  Making   The  Chosen  Solu4on   10  
  • 11. Features   Single-­‐obj   Mul4-­‐obj   Linux  (LVAT)   6888   State  of  the  Art   544   SPLOT   290   9   Objec4ves   11  
  • 12. Features   6888   State  of  the  Art   Mul4-­‐obj   Linux  (LVAT)   Single-­‐obj   544   SPLOT   290   Single-­‐obj,  CSP   9   Up  to  25  features   Exponen[al  [me   Benavides   ‘05   Objec4ves   12  
  • 13. Features   State  of  the  Art   6888   Mul4-­‐obj   Linux  (LVAT)   Single-­‐obj   544   SPLOT   290   9   Pohl  ‘11   Benavides   ‘05   BDD,  SAT,  CSP,  90  models   “All  products”  not  a`empted   if  cardinal  >  3x106   Objec4ves   13  
  • 14. Features   State  of  the  Art   6888   Mul4-­‐obj   Linux  (LVAT)   Single-­‐obj   544   SPLOT   290   9   Pohl  ‘11   Benavides   ‘05   Lopez-­‐ Herrejon   ‘11   Fixing  model  inconsistencies   27  min.  for  94  features   Objec4ves   14  
  • 15. Features   State  of  the  Art   6888   Mul4-­‐obj   Linux  (LVAT)   Single-­‐obj   544   SPLOT   290   9   Pohl  ‘11   Benavides   ‘05   Lopez-­‐ Herrejon   ‘11   Sayyad   ’13a   Mul[-­‐obj  configura[on,  Z3   >15  days  for  E-­‐Shop   Objec4ves   15  
  • 16. State  of  the  Art   Features   6888   Mul4-­‐obj   Linux  (LVAT)   Single-­‐obj   544   White  ‘07,  ‘08,  09a,  09b,     Shi  ‘10,  Guo  ‘11   SPLOT   290   9   Pohl  ‘11   Benavides   ‘05   Lopez-­‐ Herrejon   ‘11   Scale  up  with  randomly-­‐ generated  feature  models   Sayyad   ’13a   Velazco   ‘13   Objec4ves   16  
  • 17. State  of  the  Art   Features   Linux  (LVAT)   6888   544   SPLOT   290   9   Single-­‐obj   Mul4-­‐obj   Johansen   ‘11   Test  covering  arrays,  [med  out   on  all  ops  for  Linux  6888   features  (and  some  ops  on   smaller  FMs)   Pohl  ‘11   Benavides   ‘05   Lopez-­‐ Herrejon   ‘11   Sayyad   ’13a   Velazco   ‘13   Objec4ves   17  
  • 18. State  of  the  Art   Features   Linux  (LVAT)   6888   Single-­‐obj   Johansen   ‘11   Mul4-­‐obj   Henard   ‘12   Test  covering  arrays   20  hours  for  Linux  6888  features   6-­‐wise  coverage   544   SPLOT   290   9   Pohl  ‘11   Benavides   ‘05   Lopez-­‐ Herrejon   ‘11   Sayyad   ’13a   Velazco   ‘13   Objec4ves   18  
  • 19. State  of  the  Art   Features   Linux  (LVAT)   6888   Single-­‐obj   Johansen   ‘11   Mul4-­‐obj   Henard   ‘12   Sayyad  ’13b   Mul[-­‐obj  configura[on,  IBEA   30  configs  in  30  minutes  for   Linux  6888  features   544   SPLOT   290   9   Pohl  ‘11   Benavides   ‘05   Lopez-­‐ Herrejon   ‘11   Sayyad   ’13a   Velazco   ‘13   Objec4ves   19  
  • 20. Roadmap   ①  State  of  the  Art   ②  Experiment  &  Results   ③  Conclusion    
  • 21. Two  Algorithms     1)  NSGA-­‐II  [Deb  et  al.  ‘02]   Non-­‐dominated  Sor4ng   Gene4c  Algorithm   Binary  Dominance   2)      IBEA  [Zitzler  and  Kunzli  ‘04]   Indicator-­‐Based  Evolu4onary   Algorithm   Con4nuous  Dominance   21  
  • 22. 7  Feature  Models   22  
  • 23. Feature  Fixing   •  Look  for  mandatory  or  dead  features.   •  Fix  those  feature  in  the  evolu4on.   •  Skip  rules  with  only  those  features.     23  
  • 24. Experiment   Low  Crossover  &  Muta[on  rates   Stopping  Criteria   24  
  • 25. HV                          =  Hypervolume  of  dominated  region   %Correct    =  %  of  solu4ons  that  are  fully  correct  (out  of  300  solu4ons)   25  
  • 26. Comparing  %Correct   i.e.  %  of  solu4ons  that  are  fully  correct  (out  of  300  solu4ons)   NSGA-­‐II  No  FF   100%   NSGA-­‐II  with  FF   80%   IBEA  No  FF   IBEA  with  FF   60%   40%   20%   0%   ToyBox   axTLS   eCos   FreeBSD   Fiasco   uClinux   Linux  X86   26  
  • 27. How  to  crack  the  nut?   •  What  if  we  told  IBEA  what  good  solu4ons  look   like?   •  Good  solu4on?  …  A  correct  solu4on,  with:   •  Minimal  skeleton?   •  Maximum  features?   •  Then  what?   •  Seed  the  ini4al  popula4on  with  a  pre-­‐calculated   “good”  solu4on.   27  
  • 28. •  How  to  generate  good  (correct)  solu4ons?   •  Z3  SMT  (Sa4sfiability  Module  Theory)  Solver   •  Two-­‐objec4ve  IBEA,  min  viola4ons,  max  features   •  Z3:  super  fast,  low  on  selected  features   •  2-­‐obj  IBEA:  slow,  but  feature-­‐rich   28  
  • 29. Correct  solu4ons  aGer  30  minutes  for   Linux  Kernel  feature  model   29  
  • 30. Correct  solu4ons  aGer  30  minutes  for   Linux  Kernel  feature  model   …  A  Straw  to  Break  the  Camel’s  Back   30  
  • 31. More  recent  result   •  added  the  “PULL”  trick  (give  more  objec4ve  weight   to  constraint  viola4ons)   31  
  • 32. Why  is  this  interes4ng?   •  20  correct  solu4ons  to  choose  from  within  2  minutes   32  
  • 33. Do  I  have  the  user’s  aven4on?   •  2  seconds…   •  to  stay  focused     •  10  seconds…     •  to  stay  on  task   •  few  minutes?     •  To  provide     4mely  input   at  a  mee4ng   33  
  • 34. Roadmap   ①  State  of  the  Art   ②  Experiment  &  Results   ③  Conclusion    
  • 35. Sound  bites   Scalability   Method  innova[on  is  key   Problem  Formula4on     Give  the  best  (comprehensive  yet   concise)  picture  to  the  user.   IBEA  with  Feature  Fixing     Con[nuous  dominance   Respect  domain  constraints   Enter  Z3?     Acknowledgment This  research  work  was   funded  by  the  Qatar   Na[onal  Research  Fund   (QNRF)  under  the  Na[onal   Priori[es  Research  Program   (NPRP)     Grant  No.:  09-­‐1205-­‐2-­‐470.   One  day  we  will  build  that  bridge!   35