Effec%ve	
  Test	
  Execu%on	
  for	
  SPLs	
  
Sabrina	
  Souto	
  	
  
(Supervisor:	
  Marcelo	
  d’Amorim)	
  
	
  
Fede...
General	
  Assump%on	
  
•  We	
  assume	
  that	
  tests	
  are	
  not	
  designed	
  for	
  
one	
  specific	
  product	
...
General	
  Problem	
  
•  Which	
  products	
  to	
  select	
  for	
  running	
  a	
  
par%cular	
  SPL	
  test	
  against...
Previous	
  selec%on	
  strategies	
  
•  Black-­‐box	
  tes6ng	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
 ...
Hypothesis	
  
Few	
  valid	
  configura%ons	
  are	
  
reachable	
  from	
  individual	
  tests	
  
Substan%ated	
  from	
...
Ques%ons	
  (revisited	
  in	
  the	
  end)	
  
•  Is	
  efficiency	
  important	
  in	
  this	
  context?	
  
•  Is	
  comb...
Ques%ons	
  (revisited	
  in	
  the	
  end)	
  
•  Is	
  efficiency	
  important	
  in	
  this	
  context?	
  
•  Is	
  comb...
Ques%ons	
  (revisited	
  in	
  the	
  end)	
  
•  Is	
  efficiency	
  important	
  in	
  this	
  context?	
  
•  Is	
  comb...
Ques%ons	
  (revisited	
  in	
  the	
  end)	
  
•  Is	
  efficiency	
  important	
  in	
  this	
  context?	
  
•  Is	
  comb...
Symbolic	
  Execu%on	
  of	
  Features	
  –	
  SEF	
  	
  
SEF	
  
Test	
  
Program	
  
schemata	
  
Feature	
  
Model	
  ...
Program	
  Schemata	
  
#IFDEF FEATURE_1
x = 10;
#ELIF
x = 20;
#ENDIF
static boolean feature1;
if(feature1){x = 10;} else ...
Symbolic	
  Execu%on	
  of	
  Features	
  –	
  SEF	
  	
  
SEF	
  
Test	
  
Program	
  
schemata	
  
Feature	
  
Model	
  ...
Symbolic	
  Execu%on	
  of	
  Features	
  –	
  SEF	
  	
  
[<o1, c1 >, ...]
feature
model
Driver
Test
SAT
solver
test sche...
Example	
  
Valid	
  paths:	
  
[2	
  3	
  8	
  6	
  7	
  5]	
  
[2	
  3	
  8	
  -­‐6	
  7	
  5]	
  
Invalid	
  paths:	
  ...
Example	
  
Sudoku	
  Feature	
  Model	
   Test1	
  
Dagstuhl	
  Seminar	
  13091,	
  Sabrina	
  Souto,	
  
sfs@cin.ufpe.b...
Example	
  
Valid	
  paths:	
  
[2	
  3	
  8	
  6	
  7	
  5]	
  
[2	
  3	
  8	
  -­‐6	
  7	
  5]	
  
Invalid	
  paths:	
  ...
Example	
  
Valid	
  paths:	
  
[2	
  3	
  8	
  6	
  7	
  5]	
  
[2	
  3	
  8	
  -­‐6	
  7	
  5]	
  
Invalid	
  paths:	
  ...
public class SudokuGenerator{ //…!
public Board generate() {!
if(GENERATOR)){!
Board board = new Board(); !
fillBoard(boar...
Example	
  
Valid	
  paths:	
  
[2	
  3	
  8	
  6	
  7	
  5]	
  
[2	
  3	
  8	
  -­‐6	
  7	
  5]	
  
Invalid	
  paths:	
  ...
Par%%on	
  of	
  configura%on	
  space	
  
…	
  
P1	
  	
  	
  	
  	
  	
  P2	
  
	
  	
  	
  	
  	
  P3 	
  	
  	
  	
  P4...
Research	
  Ques%ons	
  
•  RQ1:	
  How	
  SEF	
  compares	
  to	
  an	
  ideal	
  sampler?	
  
•  RQ2:	
  What	
  is	
  t...
Subjects	
  of	
  Analysis	
  	
  
Dagstuhl	
  Seminar	
  13091,	
  Sabrina	
  Souto,	
  
sfs@cin.ufpe.br	
  
RQ1	
  
How	
  SEF	
  compares	
  to	
  an	
  ideal	
  sampler?	
  
101.1.1
101.2.1
101.3.1
101.4.1
101.5.1
101.6.1
101.7....
gpl.1.1
gpl.1.2
gpl.1.3
gpl.1.4
gpl.1.5
gpl.1.6
gpl.1.7
gpl.1.8
gpl.1.9
jto.1.3
jto.1.4
jto.1.5
jto.1.6
jto.1.7
jto.1.8
jt...
Observa%ons	
  
•  We	
   considered	
   9	
   previously-­‐used	
   SPLs	
   with	
  
exis%ng	
  test	
  cases	
  (We	
  ...
RQ2	
  
What	
  is	
  the	
  impact	
  of	
  further	
  constraining	
  the	
  
model	
  on	
  SEF?	
  
Can	
  one	
  iden...
Possible	
  Future	
  Work	
  	
  
(I	
  need	
  your	
  help	
  here)	
  
•  Bener	
  understand	
  scalability	
  for	
 ...
Possible	
  Future	
  Work	
  	
  
(I	
  need	
  your	
  help	
  here)	
  
•  Improve	
  automa%on	
  
– Program	
  schema...
Ques%ons	
  Revisited	
  
•  Is	
  efficiency	
  important?	
  
•  Is	
  combinatorial	
  explosion	
  (scalability)	
  
rel...
Upcoming SlideShare
Loading in …5
×

Effective Test Execution for SPLs

243 views
205 views

Published on

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
243
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Effective Test Execution for SPLs

  1. 1. Effec%ve  Test  Execu%on  for  SPLs   Sabrina  Souto     (Supervisor:  Marcelo  d’Amorim)     Federal  University  of  Pernambuco  -­‐  UFPE  
  2. 2. General  Assump%on   •  We  assume  that  tests  are  not  designed  for   one  specific  product   – A  test  can  run  against  several  products!   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  3. 3. General  Problem   •  Which  products  to  select  for  running  a   par%cular  SPL  test  against?   Soundness   Efficiency   – One  can  miss  fault-­‐revealing   products  in  this  set     – One  can  add  irrelevant  products            in  this  set   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  4. 4. Previous  selec%on  strategies   •  Black-­‐box  tes6ng                                                                                                     (e.g.,  CIT,  sampling,  exhaus%ve…)                                                 [Cohen  et.  al.,  ROSETEA  2006];  [Yilmaz  et.  al.,  TSE  2006]  ;   [Perrouin  et.  al.,  ICST  2010];  [Cabral  et.  al.,  SPLC  2010]  …   •  White-­‐box  tes6ng                                                                                                                   [Reisner  et.  al.,  ICSE  2010],  [Kim  et.  al.,  ISSRE  2012];                           [Kim  et.  al.,  AOSD  2011]  …   •  Gray-­‐box  tes6ng                                                                                                                         [Garvin  et.  al.,  ISSRE  2011];  [Shi  et.  al.,  FASE  2012];                               [Song  et.  al.,  ICSE  2012]  …   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  5. 5. Hypothesis   Few  valid  configura%ons  are   reachable  from  individual  tests   Substan%ated  from  results  in  literature:   [Reisner  et.  al.,  ICSE  2010]  and  [Kim  et.  al.,  AOSD  2011]   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  6. 6. Ques%ons  (revisited  in  the  end)   •  Is  efficiency  important  in  this  context?   •  Is  combinatorial  explosion  (scalability)   rela%vely  less  important  for  tes%ng?   •  Is  soundness  important  for  tes%ng?     Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  7. 7. Ques%ons  (revisited  in  the  end)   •  Is  efficiency  important  in  this  context?   •  Is  combinatorial  explosion  (scalability)   rela%vely  less  important  for  tes%ng?   •  Is  soundness  important  for  tes%ng?     Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  8. 8. Ques%ons  (revisited  in  the  end)   •  Is  efficiency  important  in  this  context?   •  Is  combinatorial  explosion  (scalability)   rela%vely  less  important  for  tes%ng?   •  Is  soundness  important  for  tes%ng?     Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  9. 9. Ques%ons  (revisited  in  the  end)   •  Is  efficiency  important  in  this  context?   •  Is  combinatorial  explosion  (scalability)   rela%vely  less  important  for  tes%ng?   •  Is  soundness  important  for  tes%ng?   Our  answer  is  YES  to  all  ques%ons!   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  10. 10. Symbolic  Execu%on  of  Features  –  SEF     SEF   Test   Program   schemata   Feature   Model   Configura6ons   Symbolic  Execu%on  has  been   proposed  in  the  70’s   Reisner  et  al.  proposed  SEF  in   2010  (ICSE)   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  11. 11. Program  Schemata   #IFDEF FEATURE_1 x = 10; #ELIF x = 20; #ENDIF static boolean feature1; if(feature1){x = 10;} else { x = 20;} Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  12. 12. Symbolic  Execu%on  of  Features  –  SEF     SEF   Test   Program   schemata   Feature   Model   Configura6ons   Symbolic  Execu%on  has  been   proposed  in  the  70’s   Reisner  et  al.  proposed  SEF  in   2010  (ICSE)   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  13. 13. Symbolic  Execu%on  of  Features  –  SEF     [<o1, c1 >, ...] feature model Driver Test SAT solver test schemata program Our  goal  is  to  apply  SEF  to   reach  sound  and  efficient   tes%ng  for  SPLs   Symbolic  Execu%on  has  been   proposed  in  the  70’s   Reisner  et  al.  proposed  SEF  in   2010  (ICSE)   Insight  for  efficient   implementa%on  is  to   par%%on  state  in  two:  one   concrete  and  one  symbolic.   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  14. 14. Example   Valid  paths:   [2  3  8  6  7  5]   [2  3  8  -­‐6  7  5]   Invalid  paths:   [-­‐2]        [2  3  8  -­‐6  -­‐7]   [2  -­‐3  8]        [2  3  8  6  -­‐7]   [2  -­‐3  -­‐8]        [2  3  8  -­‐6  7  -­‐5]   [2    3  -­‐8]      [2  3  8    6  7  -­‐5]       Sudoku  Feature  Model   Test1   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br   Base(2)   Generator(8)   States(3)   Generator(8)   Color(6)   Solver(7)   Solver(7)   Undo(8)   Undo(8)   X   X   X   X   X   X   X   X  ✓ ✓ -­‐2   2   -­‐3   3   8  -­‐8   8   -­‐8   -­‐6   6   -­‐7   -­‐7  7   7   5   5  -­‐5  -­‐5  
  15. 15. Example   Sudoku  Feature  Model   Test1   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br   Base(2)   Generator(8)   States(3)   Generator(8)   Color(6)   Solver(7)   Solver(7)   Undo(8)   Undo(8)   X   X   X   X   X   X   X   X  ✓ ✓ -­‐2   2   -­‐3   3   8  -­‐8   8   -­‐8   -­‐6   6   -­‐7   -­‐7  7   7   5   5  -­‐5  -­‐5   Valid  paths:   [2  3  8  6  7  5]   [2  3  8  -­‐6  7  5]   Invalid  paths:   [-­‐2]        [2  3  8  -­‐6  -­‐7]   [2  -­‐3  8]        [2  3  8  6  -­‐7]   [2  -­‐3  -­‐8]        [2  3  8  -­‐6  7  -­‐5]   [2    3  -­‐8]      [2  3  8    6  7  -­‐5]      
  16. 16. Example   Valid  paths:   [2  3  8  6  7  5]   [2  3  8  -­‐6  7  5]   Invalid  paths:   [-­‐2]        [2  3  8  -­‐6  -­‐7]   [2  -­‐3  8]        [2  3  8  6  -­‐7]   [2  -­‐3  -­‐8]        [2  3  8  -­‐6  7  -­‐5]   [2    3  -­‐8]      [2  3  8    6  7  -­‐5]       Sudoku  Feature  Model   Test1   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br   Feature   Model   Base(2)   Generator(8)   States(3)   Generator(8)   Color(6)   Solver(7)   Solver(7)   Undo(8)   Undo(8)   X   X   X   X   X   X   X   X  ✓ ✓ -­‐2   2   -­‐3   3   8  -­‐8   8   -­‐8   -­‐6   6   -­‐7   -­‐7  7   7   5   5  -­‐5  -­‐5  
  17. 17. Example   Valid  paths:   [2  3  8  6  7  5]   [2  3  8  -­‐6  7  5]   Invalid  paths:   [-­‐2]        [2  3  8  -­‐6  -­‐7]   [2  -­‐3  8]        [2  3  8  6  -­‐7]   [2  -­‐3  -­‐8]        [2  3  8  -­‐6  7  -­‐5]   [2    3  -­‐8]      [2  3  8    6  7  -­‐5]       Sudoku  Feature  Model   Test1   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br   Feature   Model   Test   Base(2)   Generator(8)   States(3)   Generator(8)   Color(6)   Solver(7)   Solver(7)   Undo(8)   Undo(8)   X   X   X   X   X   X   X   X  ✓ ✓ -­‐2   2   -­‐3   3   8  -­‐8   8   -­‐8   -­‐6   6   -­‐7   -­‐7  7   7   5   5  -­‐5  -­‐5  
  18. 18. public class SudokuGenerator{ //…! public Board generate() {! if(GENERATOR)){! Board board = new Board(); ! fillBoard(board); ! ! makeSolvable(board, 50) ! ! ! return board;! }! }! //…! private void fillBoard(Board board){! if(GENERATOR)){ ! BoardManager bm = new BoardManager(); ! ! //…! ! bm.undo(); !! //…! }! }! //…! }! Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br   public class Test{ //…! BoardManager bm;! SudokuGenerator sGen;! public static void test1(){! if(BASE){! this.bm = new BoardManager();! if(GENERATOR){! this.sGen = new SudokuGenerator();! } ! }! } //…! }! public class BoardManager{ //…! public BoardManager(){! if(BASE){// …! if(STATES){/*…*/} ! !} ! }! //…! public void undo(){! if(UNDO){/*…*/}! }! //…! }! public class Board { //…! public Board () {! if(BASE){ //…! this.board[i] = new Field();! } ! }//…! }! public class Field{ //…! public Field() {! if(BASE){ //…! if(COLOR){! ! !//…! ! } ! ! if(SOLVER){/*…*/} ! } ! }//…! }! 1   2   3   4   5   6  
  19. 19. Example   Valid  paths:   [2  3  8  6  7  5]   [2  3  8  -­‐6  7  5]   Invalid  paths:   [-­‐2]        [2  3  8  -­‐6  -­‐7]   [2  -­‐3  8]        [2  3  8  6  -­‐7]   [2  -­‐3  -­‐8]        [2  3  8  -­‐6  7  -­‐5]   [2    3  -­‐8]      [2  3  8    6  7  -­‐5]       Sudoku  Feature  Model   Test1   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br   Feature   Model   Test   Base(2)   Generator(8)   States(3)   Generator(8)   Color(6)   Solver(7)   Solver(7)   Undo(8)   Undo(8)   X   X   X   X   X   X   X   X  ✓ ✓ -­‐2   2   -­‐3   3   8  -­‐8   8   -­‐8   -­‐6   6   -­‐7   -­‐7  7   7   5   5  -­‐5  -­‐5  
  20. 20. Par%%on  of  configura%on  space   …   P1            P2            P3        P4   P5   P1                P2                P3          ...Pm   P1    P2       P1    P2      P3                  …Pn   P4   P1      P2      P3          ...Pm   2n   There  is  no  dis%nc%on  amongst  the  configura%ons  from  the  same  par%%on  with   respect  to  the  states  they  elaborate  during  the  test  execu%on.   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  21. 21. Research  Ques%ons   •  RQ1:  How  SEF  compares  to  an  ideal  sampler?   •  RQ2:  What  is  the  impact  of  further  constraining   the  model  for  test  execu%on?   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  22. 22. Subjects  of  Analysis     Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  23. 23. RQ1   How  SEF  compares  to  an  ideal  sampler?   101.1.1 101.2.1 101.3.1 101.4.1 101.5.1 101.6.1 101.7.1 101.8.1 101.9.1 01.10.1 01.11.1 01.12.1 01.13.1 bdb.5.1 jto.1.1 jto.1.2 jto.2.1 jto.3.1 jto.4.1 npd.1.1 npd.2.1 npd.3.1 prv.2.1 prv.3.1 prv.4.1 sdk.1.1 sdk.2.1 sdk.3.1 sdk.4.1 xst.1.1 xst.1.2 xst.1.3 xst.1.4 xst.1.5 xst.1.6 xst.1.7 xst.1.8 xst.1.9 zip.1.1 zip.2.1 time(inseconds) 0.00.20.40.6 running feasible paths running infeasible paths solving constraints <  1  second  
  24. 24. gpl.1.1 gpl.1.2 gpl.1.3 gpl.1.4 gpl.1.5 gpl.1.6 gpl.1.7 gpl.1.8 gpl.1.9 jto.1.3 jto.1.4 jto.1.5 jto.1.6 jto.1.7 jto.1.8 jto.1.9 prv.1.1 time(inseconds) 0102030 running feasible paths running infeasible paths solving constraints RQ1   How  SEF  compares  to  an  ideal  sampler?   Average  overhead  of   SEF  compared  to  an   ideal  sampler  is  of  7%     >  1  second   <  1  minute  
  25. 25. Observa%ons   •  We   considered   9   previously-­‐used   SPLs   with   exis%ng  test  cases  (We  did  not  generate  new)   •  For  most  SEF  runs  it  took  less  than  1  second   •  BerkeleyDB  took  much  longer  to  run!   – System  tests  covering  most  features  in  each  path     Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  26. 26. RQ2   What  is  the  impact  of  further  constraining  the   model  on  SEF?   Can  one  iden%fy   groups  of  related   features  from   changes  and  one   test?  
  27. 27. Possible  Future  Work     (I  need  your  help  here)   •  Bener  understand  scalability  for  SPL  tes%ng   – Evaluate  the  importance  of  scalability  for  tes%ng  on   a  much  larger  set  of  SPLs   •  Address  scalability  (if  indeed  a  big  problem)   – Possible  Solu%ons:   •  Use  composi%onal  symbolic  execu%on   •  Collapse  paths  with  similar  states   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  28. 28. Possible  Future  Work     (I  need  your  help  here)   •  Improve  automa%on   – Program  schemata  genera%on  can  be  imprac%cal   – Possible  Solu%on:   •  Generate   products   on   demand   and   use   incremental   compila%on   •  Use  Concolic  Execu%on  of  Features  (CEF)   •  We  can  also  contribute  to  find  type-­‐errors   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  
  29. 29. Ques%ons  Revisited   •  Is  efficiency  important?   •  Is  combinatorial  explosion  (scalability)   rela%vely  less  important  for  tes%ng?   •  Is  soundness  important  for  tes%ng?   Our  answer  was  YES  to  all!   What  is  your  opinion?   Dagstuhl  Seminar  13091,  Sabrina  Souto,   sfs@cin.ufpe.br  

×