Henry Muccini, profile picture
Uploaded byHenry Muccini
PDF, PPTX1,337 views

TAROT2013 Testing School - Paolo Tonella presentation

The document details the Fittest project funded by the European Commission, focusing on search-based test case generation methods. It discusses various search algorithms, such as hill climbing, simulated annealing, tabu search, genetic algorithms, and particle swarm optimization, emphasizing their mechanisms, advantages, and trade-offs between exploration and exploitation. Additionally, it provides insights into automated test case generation and presents a running example of a triangle class in programming.

Embed presentation

Download as PDF, PPTX
1 / 151
2 / 151
3 / 151
4 / 151
5 / 151
6 / 151
7 / 151
8 / 151
9 / 151
10 / 151
11 / 151
12 / 151
13 / 151
14 / 151
15 / 151
16 / 151
17 / 151
18 / 151
19 / 151
20 / 151
21 / 151
22 / 151
23 / 151
24 / 151
25 / 151
26 / 151
27 / 151
28 / 151
29 / 151
30 / 151
31 / 151
32 / 151
33 / 151
34 / 151
35 / 151
36 / 151
37 / 151
38 / 151
39 / 151
40 / 151
41 / 151
42 / 151
43 / 151
44 / 151
45 / 151
46 / 151
47 / 151
48 / 151
49 / 151
50 / 151
51 / 151
52 / 151
53 / 151
54 / 151
55 / 151
56 / 151
57 / 151
58 / 151
59 / 151
60 / 151
61 / 151
62 / 151
63 / 151
64 / 151
65 / 151
66 / 151
67 / 151
68 / 151
69 / 151
70 / 151
71 / 151
72 / 151
73 / 151
74 / 151
75 / 151
76 / 151
77 / 151
78 / 151
79 / 151
80 / 151
81 / 151
82 / 151
83 / 151
84 / 151
85 / 151
86 / 151
87 / 151
88 / 151
89 / 151
90 / 151
91 / 151
92 / 151
93 / 151
94 / 151
95 / 151
96 / 151
97 / 151
98 / 151
99 / 151
100 / 151
101 / 151
102 / 151
103 / 151
104 / 151
105 / 151
106 / 151
107 / 151
108 / 151
109 / 151
110 / 151
111 / 151
112 / 151
113 / 151
114 / 151
115 / 151
116 / 151
117 / 151
118 / 151
119 / 151
120 / 151
121 / 151
122 / 151
123 / 151
124 / 151
125 / 151
126 / 151
127 / 151
128 / 151
129 / 151
130 / 151
131 / 151
132 / 151
133 / 151
134 / 151
135 / 151
136 / 151
137 / 151
138 / 151
139 / 151
140 / 151
141 / 151
142 / 151
143 / 151
144 / 151
145 / 151
146 / 151
147 / 151
148 / 151
149 / 151
150 / 151
151 / 151
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Paolo Tonella Fondazione Bruno Kessler Trento, Italy http://se.fbk.eu/tonella Search Based Test Case Generation 1
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Outline   •  Search  based  algorithms   •  Automated  test  case  genera4on   •  Object  oriented  tes4ng   •  Future  internet  tes4ng   •  Dynamic  symbolic  execu4on   2
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Search  based  algorithms   3
The FITTEST project is funded by the European Commission (FP7-ICT-257574) The  search  problem   Global  op4mum  Local   op4mum   4
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Search  algorithms   •  Exhaus4ve  search   •  Random  search   •  Metaheuris4c  search   Metaheuris6c  algorithm:  itera4ve  op4miza4on  of  candidate   solu4ons  for  arbitrary  problem  instances.  It  is  not  ensured  to   find  a  global  op4mum,  it  finds  a  so-­‐called  near-­‐op(mal  solu(on. 5
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Explora6on  vs.  exploita6on   Since  the  search  budget  is   finite,  metaheuris4c   algorithms  aim  for  a  balance   between:     •  local  search:  exploita4on   (intensifica4on);   •  global  search:  explora4on   (diversifica4on).   Search  budget:  max  number  of  fitness  evalua4ons  (number  of   samples  x  taken)  compa4ble  with  the  max  algorithm  execu4on   4me. 6
The FITTEST project is funded by the European Commission (FP7-ICT-257574) No  free  lunch  theorems   David  H.  Wolpert,  William  G.  Macready:  No  Free  Lunch  Theorems  for   Op(miza(on,  IEEE  Transac4ons  on  Evolu4onary  Computa4on,  vol.  1,   n.  1,  April  1997. NFL  theorem:  given  a  search  budget  m,  the  average  probability  of   obtaining  the  near-­‐op4mal  value  f*  using  algorithm  A  is  the  same   as  the  probability  of  obtaining  the  same  near-­‐op4mal  value  using   another,  arbitrarily  chosen  algorithm  B. Hence,  if  metaheuris4c  A  performs  beVer  than  random  search  on   problem  instance  f1,  there  will  be  another  problem  instance  f2  on   which  random  performs  beVer  than  A. 7
The FITTEST project is funded by the European Commission (FP7-ICT-257574) f1*   Decep6ve  fitness  func6on   X   f1   X   f2   P(f1*|f1,  m,  HC)  =  1   P(f1*|f1,  m,  RND)  =  |m|  /  |X|   ε f2*   P(f2*|f2,  m,  HC)  =  |ε|  /  |X|   P(f2*|f2,  m,  RND)  =  |m|  /  |X|   HC  =  hill  climbing  (steepest  ascent)   RND  =  random  search   8
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Metaheuris6c  algorithms   •  Hill  climbing   •  Simulated  annealing   •  Tabu  search   •  Gene4c  algorithms   •  Ant  colony  op4miza4on   •  Par4cle  swarm  op4miza4on   9
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Hill  climbing   1.  x_opt = x0 // random 2.  max = f(x_opt) 3.  LOOP // with search budget m 4.  improved = FALSE 5.  FOR x IN Neighbors(x_opt) 6.  IF f(x) > max 7.  max = f(x) 8.  x_opt = x 9.  improved = TRUE 10.  END IF 11.  END FOR 12.  IF NOT improved 13.  RETURN x_opt 14.  END IF 15.  END LOOP 16.  RETURN x_opt X   E.g.:  find  input  that   crashes       myFunc(int a, int b, int c){…} E.g.:  (a±1,  b±1,  c±1) Equivalent  to  greedy  or  steepest  ascent:  exploita4on,  no  explora4on.   10
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Hill  climbing  variants   •  Stochas6c  hill  climbing:  selects  randomly  among  the   improving  solu4ons  in  the  neighborhood.   •  Random-­‐restart  (shotgun)  hill  climbing:  randomly   restarts  when  no  improving  solu4on  is  found  (or  the   improvement  is  small).   They  aVempt  to  add  more  explora4on,  when  local  search  is   ineffec4ve.   11
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Simulated  annealing   1.  x_opt = x0 // random 2.  k = 1 // time 3.  T = T_max // temperature 4.  x = x_opt 5.  LOOP // with search budget m 6.  x_new = RNDNeighbor(x) 7.  Df = f(x_new) - f(x) 8.  IF Df > 0 9.  x_opt = x_new 10.  END IF 11.  IF Df > 0 OR 12.  P(Df, T) > rand(0, 1) 13.  x = x_new 14.  END IF 15.  T = cool(T, k) 16.  k = k + 1 17.  END LOOP 18.  RETURN x_opt Parameters:   E.g.:   P(Df,  T)  =  exp(Df  /  T)     if  Df  <  0;  1  otherwise     cool(T,  k)  =  α  T   if  k  %  K  =  0;  T  otherwise   +   -­‐   -­‐   +   Df  >  0  means   lower  energy   state     12
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Simulated  annealing   •  Ini6al  high  temperature:  ini4ally,  explora4on  is  privileged   over  exploita4on:  worsening  solu4ons  are  ini4ally   accepted  with  high  probability   •  Low  temperature  at  the  end:  only  improving  solu4ons  are   accepted  at  the  end  of  the  search  (similar  to  hill  climbing).   Cooling  schedule:   T’  =  α  T      if  k  %  K  =  0   T’  =  T              otherwise     with  0  ≤  α  ≤  1   k   T   K   2K   3K   4K   5K   P(Df,  T)  =  exp(Df  /  T)     13
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Tabu  search   1.  x_opt = x0 // random 2.  add(tabuFIFOList, x_opt) 3.  LOOP // with search budget m 4.  x_new = RNDMutate(x_opt) 5.  IF x_new NOT IN tabuFIFOList 6.  add(tabuFIFOList, x_new) 7.  IF f(x_new) > f(x_opt) 8.  x_opt = x_new 9.  END IF 10.  END IF 11.  END LOOP 12.  RETURN x_opt Local  search  (exploita4on),   op4mized  to  avoid   revisi4ng  previously   examined  solu4ons   +   -­‐   -­‐   +   14
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Tabu  search  variants   •  Neighborhood  tabu:  the  en4re  neighborhood  of  an   already  visited  solu4on  is  declared  tabu;  muta4ons  jump   beyond  such  neighborhood.   •  Property  based  tabu  list:  tabu  solu4ons  are  characterized   by  their  proper4es,  such  that  any  (even  non  visited)   solu4on  with  the  proper4es  of  a  tabu  solu4on  is   considered  tabu  as  well.   •  Reac6ve  tabu  search:  when  the  same  (tabu)  solu4ons   reappear  frequently  during  the  search,  a  random  walk  is   made  to  explore  a  different  por4on  of  the  search  space.   15
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Gene6c  algorithms   1.  Pop = {x0, …, xN} // random 2.  x_opt = NULL 3.  LOOP // with search budget m 4.  assignFitness(Pop) 5.  x_opt = best(Pop, x_opt) 6.  Pop = select(Pop) 7.  Pop = reproduce(Pop) 8.  END LOOP 9.  RETURN x_opt Key  decisions:   1.  Representa4on   2.  Fitness  func4on   3.  Selec4on   4.  Reproduc4on  (crossover   and  muta4on)   Initialization   Fitness   assignment   Selection  Reproduction   Evolu6onary  loop   16
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Representa6on   Fixed  length  chromosome:   Variable  length  chromosome:  k  varies  across  individuals   y1   y2   y3   y4   y5   y6   …   yk   yi  may  be  a  single  bit,  a   number,  a  character,  etc.   E.g.:  find  input  that  crashes    myFunc(int a, int b, int c){…} Chromosomes  =  (2,  0,  3),  (0,  0,  -­‐1)   Chromosomes  =  (‘a’,  ‘b’,  ‘c’,  ‘0’),  (‘z’,  ‘0’)   Intron:  por4on  of  the  chromosome  that  does  not  contribute  to   the  phenotype  (i.e.,  the  fitness  value)   E.g.:  find  input  that  crashes    myFunc(char* s){…} Chromosomes  =  (‘a’,  ‘b’,  ‘c’,  ‘0’,  ‘t’),  (‘z’,  ‘0’,  ‘a’,  ‘g’,  ‘h’)   17
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Selec6on   Pop  =  select(Pop)   •  Trunca6on  selec6on:  the  best  M  individuals  are  selected;  to   obtain  the  number  of  individuals  required  to  fill-­‐in  Pop,  the   selected  individuals  are  replicated  mul4ple  4mes.   •  RouleQe  wheel  selec6on:  the  probability  of  selec4ng  an   individual  is  propor4onal  to  its  fitness;  individuals  can  be   selected  mul4ple  4mes.   •  Tournament  selec6on:  T  individuals  (e.g.,  2)  are  randomly   selected  and  compared  (according  to  their  fitness);  the  winner   is  inserted  into  the  new  popula4on.   •  Ordered  selec6on:  the  probability  of  selec4ng  an  individual  is   propor4onal  to  its  posi4on  in  the  list  of  all  individuals  ranked  by   increasing  fitness.   •  Eli6sm:  at  least  one  copy  of  the  best  individual(s)  is  propagated   to  the  next  genera4on.   18
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Reproduc6on   Pop  =  reproduce(Pop)   Muta6on:  y’i  =  µμ(yi)   y1   y2   y3   y4   y5   y6   …   yk   y1   y2   y3   y4   y’5   y6   …   yk   Crossover:  (y’i,  y’j)  =  yi  ⊗  yj   y1   y2   y3   y4   y5   y6   …   yk   z1   z2   z3   z4   z5   z6   …   zk   Single  point  crossover   y1   y2   y3   z4   z5   z6   …   zk   z1   z2   z3   y4   y5   y6   …   yk   E.g.,  P  =  80%   E.g.,  P  =  10%   19
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Gene6c  algorithms   The  parameters  of  these  algorithms  (popula4on  size,  kind  of   selec4on,  crossover  and  muta4on  rates,  etc.)  and  the  design   of  the  muta4on/crossover  operators  determine  the  trade-­‐off   between  explora4on  and  exploita4on.  E.g.:   •  Neighborhood  muta4on  /  two-­‐parent  crossover  /  low   muta4on  and  crossover  rate  /  eli4sm  =>  local  search.   •  Disrup4ve  muta4on  /  mul4-­‐parent  crossover  /  high   muta4on  and  crossover  rate  /  ordered  selec4on  =>  global   search.   20
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Ant  colony  op6miza6on   1.  Pop = {x0, …, xN} // random 2.  x_opt = NULL 3.  LOOP // with search budget m 4.  FOR x IN Pop 5.  FOR y IN Neighbors(x) 6.  computeProb(P(x, y)) 7.  END FOR 8.  x = move(x, P(x, *)) 9.  END FOR 10.  FOR EACH move (x, y) 11.  depositPheromone(x, y) 12.  END FOR 13.  updatePheromone() 14.  x_opt = best(Pop, x_opt) 15.  END LOOP 16.  RETURN x_opt S6gmery  principle:  coordina4on  between  agents   is  achieved  through  traces  lem  in  the  environment.   21
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Pheromone  calcula6on   computeProb(P(x,  y))   depositPheromone(x,  y)   updatePheromone()   τxy  :  amount  of  pheromone  on   (x,  y)   ηxy  :  desirability  of  move  (x,  y)   (e.g.,  f(y))   ρ  :  pheromone  evapora4on   coefficient   22
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Ant  colony  op6miza6on   The  decay  of  the  pheromone  (evapora4on)  determines  the   tradeoff  between  exploita4on  and  explora4on:   •  Pheromone  decays  slowly:  ants  remain  longer  in  the  same   region  (local  search).   •  Pheromone  decays  quickly:  ants  move  randomly  to  other   regions  (global  search)  amer  a  limited  amount  of  local   search.   23
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Par6cle  swarm  op6miza6on   1.  Pop = {x0, …, xN} // random 2.  x_opt = NULL 3.  LOOP // with search budget m 4.  FOR x IN Pop 5.  IF f(x) > f(p[x]) 6.  p[x] = x 7.  END FOR 8.  IF f(x) > f(x_opt) 9.  x_opt = x 10.  END FOR 11.  v[x] = updateVelocity(x) 12.  x = updatePosition(x) 13.  END FOR 14.  END LOOP 15.  RETURN x_opt x_opt   p[x]   x   Par4cle  x  tends  to:   1.  proceed  along  current   direc4on;   2.  be  aVracted  by  the  best   posi4on  in  its  trajectory   3.  be  aVracted  by  the   globally  best  posi4on  p[x]:  best  posi4on  of  par4cle  x   v[x]:  speed  of  par4cle  x     x:  current  posi4on  of  par4cle  x   24
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Par6cle  dynamics   updateVelocity(x)   updatePosi6on(x)   25
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Par6cle  Swarm  Op6miza6on   Variants   •  Neighborhood  communica6on:  if  any  posi4on  in  the   neighborhood  of  a  par4cle  has  been  visited  previously  by   other  par4cles,  the  best  posi4on  in  the  neighborhood   determines  a  fourth  direc4on  to  follow.   26
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Par6cle  swarm  op6miza6on   The  parameters  of  the  algorithm  determine  the  tradeoff   between  exploita4on  and  explora4on:   ω:  contributes  to  keeping  current  speed  and  direc4on,   possibly  moving  towards  unexplored  regions.   ϕp:  directs  the  search  toward  the  best  solu4on  found  by  the   current  par4cle,  usually  increasing  the  amount  of  local   search.   ϕg:  directs  the  search  toward  the  globally  best  solu4on,   which  may  increase  the  local  search  in  the  region  around   xopt.   Neighborhood  communica4on  increases  the  amount  of  local   search.   27
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Automated  test  case  genera4on   Phil  McMinn,  Search-­‐based  soDware  test  data  genera(on:  a  survey.   Journal  of  Somware  Tes4ng,  Verifica4on  and  Reliability,  vol.  14,  n.  2,   pp.  105-­‐156,  June  2004.   28
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle(){…} void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) { int a = Integer.parseInt(args[0]); int b = Integer.parseInt(args[1]); int c = Integer.parseInt(args[2]); Triangle t = new Triangle(a, b, c); if (t.isTriangle()) t.computeTriangleType(); System.out.println(typeToString(t.type)); } } 29
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() {…} boolean isTriangle() { if (a <= 0 || b <= 0 || c <= 0) return false; if (a + b <= c || a + c <= b || b + c <= a) return false; return true; } public static void main(String args[]) {…} } 30
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() { if (a == b) if (b == c) type = EQUILATERAL; else type = ISOSCELE; else if (a == c) type = ISOSCELE; else if (b == c) type = ISOSCELE; else checkRightAngle(); } boolean isTriangle() {…} public static void main(String args[]) {…} } 31
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() { if (a*a + b*b == c*c) type = RIGHT_ANGLE; else if (b*b + c*c == a*a) type = RIGHT_ANGLE; else if (a*a + c*c == b*b) type = RIGHT_ANGLE; else type = SCALENE; } void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) {…} } 32
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Coverage  tes6ng   void f(int a, int b) { 1 if (a < 0) 2 print(“a is negative”); 3 if (b < 0) 4 print(“b is negative”); 5 return; } 1   2   3   4   5   Statement  coverage   Branch  coverage   Path  coverage   Coverage  targets   {<1,  3,  5>,  <1,  2,  3,  5>,   <1,  3,  4,  5>,  <1,  2,  3,  4,  5>}   {<1,  2>,  <2,  3>,  <1,  3>,   <3,  4>,  <4,  5>,  <3,  5>}   {1,  2,  3,  4,  5}   Test  cases   f(-­‐1,  -­‐1),  f(1,  -­‐1)   f(-­‐1,  1),  f(1,  1)   f(-­‐1,  -­‐1),  f(1,  1)   f(1,  1)   33
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle(){…} void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]){…} } Goal:   Automa4c   genera4on  of  test   cases  using  gene4c   algorithms  so  as  to   achieve  statement   coverage.   Initialization   Fitness   assignment   Selection  Reproduction   Key  decisions:   1.  Representa4on   2.  Fitness  func4on   3.  Selec4on   4.  Reproduc4on  (crossover   and  muta4on)   34
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Representa6on   Fixed  length  chromosome   a   b   c   The  chromosome  used  for  test  case  genera4on  is  the  input  vector,   (sequence  of  input  values  used  by  the  test  case  running  the   program)  which  may  be  fixed  length  or  variable  length.   Example  of  (randomly  generated)  ini4al  popula4on:   Pop  =  {(2,  2,  -­‐2),  (2,  1,  1),  (2,  1,  2),  (3,  4,  2),  (4,  3,  3),  (4,  -­‐5,  6),  (3,  5,   2),  (-­‐3,  0,  -­‐2)}   E.g.,  in  our  running  example:   35
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  func6on   For  statement  and  branch  coverage,  given  a  specific  coverage   target  t,  a  widely  used  fitness  func4on  (to  be  minimized)  is:   f(x)  =  approach_level(P(x),  t)  +  branch_distance(P(x),  t)   approach_level(P(x),  t)  :   Given  the  execu4on  trace  obtained  by  running  program  P  with   input  vector  x,  the  approach  level  is  the  minimum  number  of   control  nodes  between  an  executed  statement  and  the  coverage   target  t.   branch_distance(P(x),  t)   Given  the  first  control  node  where  the  execu4on  diverges  from  the   target  t,  the  predicate  at  such  node  is  converted  to  a  distance   (from  taking  the  desired  branch),  normalized  between  0  and  1.   36
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Approach  level   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } 1   2   3   4   5   6   7   8   9   10   1   2   3   4   5   6   7   8   9   Control  flow   graph   Control   dependency   graph   37
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Approach  level   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } 1   2   3   4   5   6   7   8   9   10   1   2   3   4   5   6   7   8   9   Control  flow   graph   Control   dependency   graph   Ch1  =  (2,  2,  2)    P(Ch1)  =  <1,  2,  3,  10>    AL  =  2   Ch2  =  (2,  3,  4)    P(Ch2)  =  <1,  5,  7,  9,  10>  AL  =  0   38
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Branch  distance   The  predicate  of  the  control  node  which  is  closest  to  the  target  is   converted  to  a  distance,  which  measures  how  far  the  test  case  is   from  taking  the  desired  branch.  For  boolean/numeric  variables  a,  b:   Condi6on  c  =  atomic  predicate   Distance  BD(c)  =  d  /  (d  +  1)   a   d  =  {0  if  a  ==  true;  K  otherwise}   !a   d  =  {K  if  a  ==  true;  0  otherwise}   a  ==  b   d  =  {0  if  a  ==  b;  abs(a  -­‐  b)  +  K   otherwise}   a  !=  b   d  =  {0  if  a  !=  b;  K  otherwise}   a  <  b   d  =  {0  if  a  <  b;  a  -­‐  b  +  K  otherwise}   a  <=  b   d  =  {0  if  a  <=  b;  a  -­‐  b  +  K  otherwise}   a  >  b   d  =  {0  if  a  >  b;  b  -­‐  a  +  K  otherwise}   a  >=  b   d  =  {0  if  a  >=  b;  b  -­‐  a  +  K  otherwise}   39
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Branch  distance   For  string  variables  a,  b:   Condi6on  c  =  atomic  predicate   Distance  BD(c)  =  d  /  (d  +  1)   a  ==  b   d  =  {0  if  a  ==  b;  edit_dist(a,  b)  +  K   otherwise}   a  !=  b   d  =  {0  if  a  !=  b;  K  otherwise}   a  <  b   d  =  {0  if  a  <  b;  a[j]  -­‐  b[j]  +  K  otherwise}   a  <=  b   d  =  {0  if  a  <=  b;  a[j]  -­‐  b[j]  +  K  otherwise}   a  >  b   d  =  {0  if  a  >  b;  b[j]  -­‐  a[j]  +  K  otherwise}   a  >=  b   d  =  {0  if  a  >=  b;  b[j]  -­‐  a[j]  +  K  otherwise}   where  j  is  the  posi4on  of  the  first  different  character:  a[j]  !=  b[j],   while  a[i]  ==  b[i]  for  i  <  j  (a[j]  –b[j]  is  set  to  zero  if  a  ==  b).   Example  of  edit  distance:  edit_dist(“strqqvt”,  “trwwv”)  =  6   40
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Branch  distance   Condi6on  c  =  composite  predicate   Distance  BD(c)  =  d  /  (d  +  1)   !p   Nega4on  is  propagated  inside  p   p  &  q   d  =  d(p)  +  d(q)   p  |  q   d  =  min(d(p),  d(q))   p  XOR  q  =  p  &  !q  |  !p  &  q   d  =  min(d(p)+d(!q),  d(!p)+d(q))   41
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Branch  distance   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Ch1  =  (2,  2,  2)    P(Ch1)  =  <1,  2,  3,  10>    AL  =  2  f  =  2.5   Ch2  =  (2,  3,  4)    P(Ch2)  =  <1,  5,  7,  9,  10>  AL  =  0   1   2   3   4   5   6   7   8   9   d(a  !=  b)  =  K  =  1   BD(a  !=  b)  =  1  /  (1  +  1)  =  0.5   f(Ch1)  =  2  +  0.5  =  2.5   a  ==  b  a  !=  b   42
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Branch  distance   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Ch1  =  (2,  2,  2)    P(Ch1)  =  <1,  2,  3,  10>    AL  =  2  f  =  2.5   Ch2  =  (2,  3,  4)    P(Ch2)  =  <1,  5,  7,  9,  10>  AL  =  0  f  =  0.66   1   2   3   4   5   6   7   8   9   d(b  ==  c)  =  abs(b  -­‐  c)  +  K  =  2   BD(b  ==  c)  =  2  /  (2  +  1)  =  0.66   f(Ch2)  =  0  +  0.66  =  0.66   b  ==  c   b  !=  c   43
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Selec6on   Ch1  =  (2,  2,  2)  f  =  2.5   Ch2  =  (2,  3,  4)  f  =  0.66   Ch3  =  (-­‐2,  3,  6)  f  =  ∞   Ch4  =  (2,  3,  7)  f  =  ∞   Ch5  =  (2,  2,  3)  f  =  2.5   Ch6  =  (3,  4,  5)  f  =  0.66   Ch7  =  (3,  5,  7)  f  =  0.75   Ch8  =  (6,  8,  4)  f  =  0.83   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } 44
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Selec6on   Ch6  =  (3,  4,  5)  f  =  0.66   Ch2  =  (2,  3,  4)  f  =  0.66   Ch7  =  (3,  5,  7)  f  =  0.75   Ch8  =  (6,  8,  4)  f  =  0.83   Ch1  =  (2,  2,  2)  f  =  2.5   Ch5  =  (2,  2,  3)  f  =  2.5   Ch3  =  (-­‐2,  3,  6)  f  =  ∞   Ch4  =  (2,  3,  7)  f  =  ∞   Trunca6on  selec6on   Ch6  =  (3,  4,  5)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch8  =  (6,  8,  4)     Ch6  =  (3,  4,  5)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch8  =  (6,  8,  4)     45
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Selec6on   Ch6  =  (3,  4,  5)  P  ≈  1/f  =  0.23   Ch2  =  (2,  3,  4)  P  ≈  1/f  =  0.23   Ch7  =  (3,  5,  7)  P  ≈  1/f  =  0.20   Ch8  =  (6,  8,  4)  P  ≈  1/f  =  0.18   Ch1  =  (2,  2,  2)  P  ≈  1/f  =  0.06   Ch5  =  (2,  2,  3)  P  ≈  1/f  =  0.06   Ch3  =  (-­‐2,  3,  6)  P  ≈  1/f  =  0   Ch4  =  (2,  3,  7)  P  ≈  1/f  =  0   RouleQe  wheel  selec6on   Ch6  =  (3,  4,  5)     Ch2  =  (2,  3,  4)     Ch6  =  (3,  4,  5)     Ch1  =  (2,  2,  2)     Ch8  =  (6,  8,  4)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch6  =  (3,  4,  5)     46
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Selec6on   Tournament  selec6on   Ch2  =  (2,  3,  4)       Ch5  =  (2,  2,  3)       Ch8  =  (6,  8,  4)   Ch2  =  (2,  3,  4)       Ch6  =  (3,  4,  5)   Ch6  =  (3,  4,  5)   Ch8  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   Ch1  =  (2,  2,  2)  f  =  2.5   Ch2  =  (2,  3,  4)  f  =  0.66   Ch3  =  (-­‐2,  3,  6)  f  =  ∞   Ch4  =  (2,  3,  7)  f  =  ∞   Ch5  =  (2,  2,  3)  f  =  2.5   Ch6  =  (3,  4,  5)  f  =  0.66   Ch7  =  (3,  5,  7)  f  =  0.75   Ch8  =  (6,  8,  4)  f  =  0.83   <Ch2,  Ch7>   <Ch1,  Ch5>   <Ch3,  Ch8>   <Ch3,  Ch2>   <Ch6,  Ch5>   <Ch6,  Ch4>   <Ch8,  Ch3>   <Ch8,  Ch1>   47
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Selec6on   1  Ch4  =  (2,  3,  7)  f  =  ∞    P  =  1/33  =  0.03     1  Ch3  =  (-­‐2,  3,  6)  f  =  ∞    P  =  1/33  =  0.03   3  Ch5  =  (2,  2,  3)  f  =  2.5    P  =  3/33  =  0.09   3  Ch1  =  (2,  2,  2)  f  =  2.5    P  =  3/33  =  0.09   5  Ch8  =  (6,  8,  4)  f  =  0.83  P  =  5/33  =  0.15   6  Ch7  =  (3,  5,  7)  f  =  0.75  P  =  6/33  =  0.18   7  Ch6  =  (3,  4,  5)  f  =  0.66  P  =  7/33  =  0.21   7  Ch2  =  (2,  3,  4)  f  =  0.66  P  =  7/33  =  0.21   Ordered  selec6on   Ch2  =  (2,  3,  4)     Ch2  =  (2,  3,  4)     Ch6  =  (3,  4,  5)     Ch4  =  (2,  3,  7)     Ch8  =  (6,  8,  4)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch6  =  (3,  4,  5)     Rank  sum  =  33     48
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Reproduc6on:  crossover   Ch1  =  (2,  3,  4)       Ch2  =  (2,  2,  3)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  4)       Ch5  =  (3,  4,  5)   Ch6  =  (3,  4,  5)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   One-­‐point  crossover,  with  P  =  0.8   -­‐   <Ch2,  Ch5>   <Ch3,  Ch8>   <Ch4,  Ch6>   SEL   SEL   -­‐   SEL   Ch1  =  (2,  3,  4)       Ch2  =  (2,  4,  5)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  5)       Ch5  =  (3,  2,  3)   Ch6  =  (3,  4,  4)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   49
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Reproduc6on:  muta6on   Muta4on  probability:  P  =  0.2   Muta4on  operators:  increment/decrement  (P  =  50%)   Ch1  =  (2,  3,  4)       Ch2  =  (2,  4,  5)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  5)       Ch5  =  (3,  2,  3)   Ch6  =  (3,  4,  4)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   Ch1  =  (2,  3,  4)       Ch2  =  (2,  5,  5)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  5)       Ch5  =  (2,  2,  3)   Ch6  =  (3,  4,  4)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   50
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Itera6ng   Initialization   Fitness   assignment   Selection  Reproduction   Un6l  search  budget  (number   of  fitness  evalua4ons)  is  over   or  target  is  covered.   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Then,  select  the  next  target  (yet  to   cover  statement)  and  repeat  the   test  case  genera4on  cycle.   51
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Collateral  coverage   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } When  a  target  is  covered   accidentally  (e.g.,  (2,  2,  3))  in  a  test   case  genera4on  cycle,  it  is   removed  from  the  list  of  yet  to   cover  targets  and  the  related  test   case  is  stored,  to  be  included  in  the   final  test  suite.   Andrea  Arcuri,  Muhammad  Zohaib  Z.  Iqbal,  Lionel  C.  Briand:  Formal  analysis  of  the   effec(veness  and  predictability  of  random  tes(ng.  ISSTA,  pp.  219-­‐230,  2010.   Collateral  (serendipitous)  coverage:  if  test  cases  involved  in  collateral   coverage  are  not  detected  and  stored,  it  can  be  shown  that  random   tes4ng  performs  asympto4cally  beVer  than  search  based  tes4ng.   52
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Final  test  suite   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Ch1  =  (2,  2,  3)   Ch2  =  (2,  5,  5)   Ch3  =  (2,  2,  2)   Ch4  =  (4,  3,  4)   Ch5  =  (3,  4,  5)   The  final  test  suite  consists  of  all   chromosomes  that  have  been   found  to  cover  (even   accidentally)  one  or  more  yet  to   cover  statements.   The  final  test  suite  might  contain  redundancies  that  can  be   eliminated  in  a  post-­‐processing  (by  keeping  only  the  test  cases  that   cover  at  least  one  target  uniquely).   53
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Object  oriented  tes4ng   Paolo  Tonella,  Evolu(onary  tes(ng  of  classes.  Proc.  of  the  Interna4onal  Symposium   on  Somware  Tes4ng  and  Analysis  (ISSTA),  pp.  119-­‐128,  Boston,  USA,  July  2004.   Gordon  Fraser,  Andrea  Arcuri,  Whole  test  suite  genera(on.  IEEE  Transac4ons  on   Somware  Engineering,  vol.  38,  n.  2,  pp.  276-­‐291,  2013.   54
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Unit  tes6ng  of  classes   1.  An  object  of  the  class  under  test  is  created  using  one  of   the  available  constructors.   2.  A  sequence  of  zero  or  more  methods  is  invoked  on  it.   3.  The  method  currently  under  test  is  executed.   4.  The  final  state  of  the  object  is  examined  to  produce  the   pass/fail  result.   Drivers/stubs  are  created  whenever  necessary.   Steps  1,  2  are  repeated  for  each  parameter  of  object  type.   55
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  of  test  case   1 A a = new A(); 2 B b = new B(); 3 b.f(2); 4 a.m(5, b); •  Object  b  is  created  because  it  is  required  to  call  m. •  Its  state  is  changed  by  calling  f  on  it.   •  Sequence  of  required  input  values:  <2,  5>.   Let  us  assume  that  m  is  the  method  under  test.   56
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Features  of  object-­‐oriented     unit  tes6ng   •  The  number  and  order  of  method  invoca4ons  is  variable.   •  The  number  of  input  values  is  also  variable.   •  Some  parameters  in  method  calls  are  objects   themselves,  thus  requiring  further  object  construc4ons.   •  The  state  of  the  object  under  test  and  of  the  object   parameters  affects  the  result.   Chromosomes  are  not  just  sequences  of  input  values   57
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Chromosomes   Procedural  code:   Object-­‐Oriented  code:   (v1,  v2,  …,  vN)   $x0=A():  $x0.f():  $x0.g()  @  v1,  v2,  …,  vN   class TestA extends TestCase { public void testCase1() { A a = new A(-1); B b = new B(); b.f(2); a.m(5, b); } } $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   chromosome   variables   input   values   58
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Random  chromosome   construc6on   1.  A  constructor  for  the  object  under  test  is  randomly  selected:   $a=A(int)@-1 2.  The  invoca4on  of  the  method  under  test  is  appended:   $a=A(int):$a.m(int,$b)@-1,5 3.  All  required  object  construc4ons  are  inserted:  $a=A(int): $b=B():$a.m(int,$b)@-1,5 4.  Method  invoca4ons  to  change  the  state  of  the  created  objects   are  randomly  inserted:  $a=A(int):$b=B():$b.f(int): $a.m(int,$b)@-1,2,5 Steps  3  and  4  are  repeated  un4l  all  chromosome  variables   used  as  method  or  constructor  parameters  are  properly   ini4alized  (well-­‐formedness  of  the  resul4ng  chromosome).   59
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Random  input  data  genera6on   A.m(int) Default  integer  generator:  uniform  selec4on  in  [0,  100]   A.m(int[-2;2]) Parameterized  integer  generator:  uniform  selec4on  in  [-­‐2,  2]   A.m(int[MyIntGenerator]) Customized  integer  generator:  method  newIntValue()   from  class  MyIntGenerator  is  called  to  obtain  the  value   A.m(boolean) Default  boolean  generator:  true  and  false  are  equally  likely   A.m(String) Default  string  generator:  characters  are  uniformly  chosen  from   [a-­‐zA-­‐Z0-­‐9],  with  the  string  length  decaying  exponen4ally   A.m(String[DateGenerator]) Customized  string  generator:  only  strings   represen4ng  legal  dates  are  produced  (e.g.,   “3/3/2003”)   Default,  parameterized,  pooled  and  customized  input  generators:   A.m(int<pool>) Randomly  selected  from  the  program’s  pool  of  integer  constants   A.m(String<pool>) Randomly  selected  from  the  program’s  pool  of  string  constants   60
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Muta6on  operators   Change  input  value:   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  4,  5   61
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Muta6on  operators   Change  constructor:   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   $a=A():$b=B():$b.f(int):$a.m(int,  $b)  @  2,  5   62
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Muta6on  operators   Insert/remove  method  call:   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   $a=A(int):$b=B():$a.m(int,  $b)  @  -­‐1,  5   63
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Crossover   $a=A():$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2   $a=A(int):$b=B():$b.g():$b.f(int):$a.m(int,  $b)  @  0,  -­‐1,  2   $a=A(int):$b=B():$b.g():$a.m(int,  $b)  @  0,  -­‐3   $a=A():$b=B():$a.m(int,  $b)  @  -­‐3   $a=A(int,  int):$b=B():$a.m(int,  $b)  @  0,  3,  6   $a=A():$b=B(int):$a.m(int,  $b)  @  1,  6   $a=A():$b=B(int):$c=C(int):$b.h($c):$b.f():$a.m(int,  $b)  @  1,  4,  5   $a=A(int,  int):$b=B():$c=C():$b.h($c):$b.f():$a.m(int,  $b)  @  0,  3,  5   Well-­‐formedness   must  be  maintained   The  alterna4ve   would  be  to  use  null   64
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness   Initialization   Fitness   assignment   Selection  Reproduction   f(x)  =  approach_level(P(x),  t)  +  branch_distance(P(x),  t)   •  Trunca6on  selec6on   •  RouleQe  wheel  selec6on   •  Tournament  selec6on   •  Ordered  selec6on   •  Eli6sm   65
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Test  case  generator   $x0=BinaryTree():  $x1=Integer(int):  $x2=BinaryTreeNode($x1):   $x0.insert($x2):  $x3=Integer(int):  $x0.search($x3)  @  5,  5   class TestBinaryTree extends TestCase { … public void testCase2() { BinaryTree x0 = new BinaryTree(); Integer x1 = new Integer(5); BinaryTreeNode x2 = new BinaryTreeNode(x1); x0.insert(x2); Integer x3 = new Integer(5); assertTrue(x0.search(x3)); } } Asser6on  added  manually   66
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Asser6ons   Asser4ons  are  inserted  manually  as  follows:   –  The  value  returned  from  each  method  call  is  compared  with  the   expected  one.   –  If  a  method  call  can  raise  an  excep6on,  an  asser4on  on  its   actual  occurrence  is  added.   –  At  the  end  of  each  test  case,  the  final  state  of  the  object  under   test  is  compared  with  the  expected  one.   67
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Asser6ons   Asser4ons  are  added  at  method  calls  to  check  the  actually   returned  values  vs.  the  expected  ones.   class TestLinkedList extends TestCase { public void testCase1() { LinkedList x0 = new LinkedList(); … Integer x1 = null; x0.addFirst(x1); Integer x2 = null; assertTrue(x0.remove(x2)); … } } 68
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Asser6ons   Asser4ons  are  added  at  method  calls  possibly  raising   excep6ons.   class TestLinkedList extends TestCase { public void testCase2() { LinkedList x0 = new LinkedList(); … try { x0.listIterator(78); fail(); } catch(IndexOutOfBoundException e) {} } 69
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Asser6ons   class TestLinkedList extends TestCase { public void testCase3() { LinkedList x0 = new LinkedList(); … Integer x1 = new Integer(2); x0.addLast(x1); assertTrue(x0.toString().equals(“[2]”)); } } Asser4ons  are  added  at  to  check  the  final  object  state.     70
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Whole  test  suite  genera6on   •  Some  coverage  targets  may  be  infeasible.   •  Some  coverage  targets  may  be  very  difficult  to  achieve.   Since  a  limited  search  budget  is  available  for  test  case  genera4on:   •  Infeasible  targets  may  use  the  en4re  search  budget  without   achieving  any  target.   •  Difficult  targets  may  use  most  of  the  search  budget,  leaving  lots   of  easier  coverage  targets  uncovered.   •  The  order  in  which  targets  are  considered  affects  the  final   result.   Whole  test  suite  genera4on  addresses  all  these  problems:   •  A  set  of  test  cases  is  generated  to  cover  all  targets  at  the  same  4me.   •  The  result  is  independent  of  the  order  of  the  targets  and  the  presence   of  infeasible  or  difficult  targets.   71
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example   class Stack { int[] values = new int[INIT_SIZE]; int size = 0; void push(int x) { if (size >= values.length) resize(); if (size < values.length) values[size++] = x; } int pop() {…} private void resize() {…} } Gordon  Fraser,  Andrea  Arcuri,  Evolu(onary  genera(on  of  whole  test  suites.  Proc.  of  the  Int.   Conference  on  the  Quality  of  Somware  (QSIC),  pp.  31-­‐40,  Madrid,  Spain,  July  2011.   Difficult  to  cover   Else  branch  is   infeasible   Whole  test  suite  genera4on   72
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Chromosome   An  individual  is  an  en4re  test  suite,  i.e.  a  set  of  test  cases:   Ch1  =  {TC1,  TC2,  …}  =      {<$a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5>,      <$a=A(int):$b=B():$a.m(int,  $b)  @  3,  7>,  …}   Ch1:    cov  =  30   Ch2:    cov  =  15   Ch3:    cov  =  17   Ch4:    cov  =  34   Ch5:    cov  =  21   Ch6:    cov  =  11   Ch7:    cov  =  13   Ch8:    cov  =  45   Intui4vely,  individuals  (test   suites)  with  higher  coverage  are   beVer,  so  fitness  must  be  based   on  coverage.   73
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness   Ties  are  resolved  by  rewarding  the  chromosomes  represen4ng   smaller  test  suites  (i.e.,  test  suites  with  less  statements).   74
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Muta6on   In  addi4on  to  the  muta4on  operators  for  the  individual  test  cases:   •   Remove  method/constructor  call;   •   Insert  method/constructor  call;   •   Change  input  value.   Muta4on  operators  that  work  on  whole  test  suites  include:   •   Add  test  case:  Ch  =  {TC1,  …}  →  Ch’  =  {TC_new,  TC1,  …}   •   Remove  test  case:  Ch  =  {TC_k,  TC1,  …}  →  Ch’  =  {TC1,  …}  if  TC_k   becomes  empty  amer  (repeatedly)  applying  remove  method/ constructor  call  to  it.   75
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Crossover   Each  test  suite  is  par44oned  into  two  non  empty  subsets,  which  are   exchanged  between  the  two  individuals.   Ch1  =  {TC1,  TC2,  TC3,  TC4}    Ch2  =  {TC5,  TC6,  TC7}     Ch1’  =  {TC1,  TC2,  TC7}    Ch2’  =  {TC5,  TC6,  TC3,  TC4}   76
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Tool   EvoSuite:  hVp://www.evosuite.org/   by  Gordon  Fraser  &  Andrea  Arcuri.   •  Implements  the  gene4c  algorithm  for  whole  test  suite   genera4on   •  Instruments  the  Java  bytecode  under  test  for  coverage   analysis.   •  Uses  Java  reflec4on  to  execute  test  cases  encoded  as   chromosomes  and  to  determine  the  execu4on  traces.   •  Only  mutated  test  cases  need  to  be  re-­‐executed  from  one   itera4on  to  the  next  one.   •  Test  case  execu4on  is  4med  out  to  prevent  non-­‐termina4on.   •  Produces  a  JUnit  test  class  as  output.   77
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Future  internet  tes4ng   78
The FITTEST project is funded by the European Commission (FP7-ICT-257574) FITTEST EU  FP7  STREP  project  n.  257574   Sep  1,  2010  –  Dec  31,  2013   Par4cipant  organisa4on Country Universidad  Politécnica  de  Valencia  (coordinator) Spain University  College  London United  Kingdom Berner  &  MaVner  Systemtechnik Germany IBM Israel Fondazione  Bruno  Kessler Italy Utrecht  University The  Netherlands SOFTeam France Sulake Finland Goal:  automated,  con4nuous  tes4ng  of  Future   Internet  applica4ons   79
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Future internet •  Highly  interac4ve   •  Communica4ng   asynchronously  with   server   •  Based  on  Web  2.0   technologies  (e.g.,  Ajax,   Flash/Ac4onScript)   •  Composed  and  updated   dynamically   80
The FITTEST project is funded by the European Commission (FP7-ICT-257574) FI testing challenges §  CH1  Self  modifica4on     §  Rich  clients  have  increased  capability  to  dynamically   adapt  the  structure  of  the  Web  pages;  server-­‐side   services  are  replaced  and  recomposed  dynamically  based   on  Service  Level  Agreements  (SLA)  and  newly  discovered   services;  components  are  dynamically  loaded   §  CH2  Autonomic  behaviour   §  FI  applica4ons  are  highly  autonomous;  their  correct   behaviour  cannot  be  specified  precisely  at  design-­‐4me.   §  CH3  Low  observability   §   FI  applica4ons  are  composed  of  an  increasing  number  of   3rd-­‐party  components  and  services,accessed  as  a  black   box,  which  are  hard  to  test.   81
The FITTEST project is funded by the European Commission (FP7-ICT-257574) §  CH4  Asynchronous  interac4ons   §  FI  applica4ons  are  highly  asynchronous  and  hence  hard  to  test.  Each  client   submits  mul4ple  requests  asynchronously;  mul4ple  clients  run  in  parallel;   server-­‐side  computa4on  is  distributed  over  the  network  and  concurrent.   §  CH5  Time  and  load  dependent  behaviour   §  For  FI  applica4ons,  factors  like  4ming  and  load  condi4ons  make  it  hard  to   reproduce  errors  during  debugging.   §  CH6  Huge  feature  configura4on  space   §  FI  applica4ons  are  highly  customisable  and  self-­‐configuring,  and  contain  a   huge  number  of  configurable  features,  such  as  user-­‐,  context-­‐,  and   environment-­‐dependent  parameters.   §  CH7  Ultra-­‐large  scale     §  FI  applica4ons  are  omen  systems  of  systems;  tradi4onal  tes4ng  adequacy   criteria  cannot  be  applied,  since  even  in  good  tes4ng  situa4ons  low   coverage  will  be  achieved.   FI testing challenges 82
The FITTEST project is funded by the European Commission (FP7-ICT-257574) FI testing approach Automated  Con4nous  Evolu4onary  Tes4ng   Evolving testware (models, testcases, oracles) FITTEST FI applications, dynamically composed of 3rd party components and services Self-modifying Changed requirements Autonomic 83
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Global picture 84
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Model-based FI testing 85
The FITTEST project is funded by the European Commission (FP7-ICT-257574) FI example: Cart public class Cart { public Cart() { ... } public void add(int c) { ... } public void rem(int c) { ... } public void pay() { ... } public int n() { ... } } 86
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Inferred model 87
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Architecture Branch  coverage  of  the  FSM  monitor  is  equivalent  to   transi4on  coverage  of  the  model   88
The FITTEST project is funded by the European Commission (FP7-ICT-257574) FSM monitor as a “man in the middle” 89
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Monitor generation through testability transformation void rem(int x) { sut.rem(x); n = sut.n(); switch (s_state) { case 2: if ( n > 1 ) { s_state = 2; } else if ( n == 1 ) { s_state = 1; } else if ( n == 0 ) { s_state = 0; } else { throw new UnderSpecEx(...); } break; case 1: ... } Branch  condi4ons  become   branch  distances  in  the   fitness  func4on  of  the   search  based  TC  generator   90
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Underspecified model 91
The FITTEST project is funded by the European Commission (FP7-ICT-257574) From Branch to Transition Coverage §  Theorem:  given  the  FSM  model,  we  can  apply  a   testability  transforma4on  to  the  model,  which  builds  a   class  FSM_monitor  such  that,  if  the  SUT  is  compliant   with  the  FSM,  then:   All-­‐transi4on  coverage  on  the  FSM  is  sa4sfied   iff   All  (non-­‐error)  branch  coverage  on  the   FSM_monitor  is  sa4sfied   92
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Implementation •  Existing search-based test case generators (e.g., EvoSuite) can be used •  FSM_monitor generator: – java – template based (Velocity) •  highly configurable •  different implementation styles can be supported •  graphical representations can be produced, with proper templates 93
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Test Suite//Test case number: 0 testCart0.add(1); testCart0.add(1); testCart0.add(1); //Test case number: 1 testCart0.add(2); testCart0.pay(); //Test case number: 2 testCart0.add(11); testCart0.rem(11); //Test case number: 3 testCart0.add(1); testCart0.rem(1); //Test case number: 4 testCart0.add(1734); testCart0.rem(1); //Test case number: 5 testCart0.add(2); testCart0.rem(1); //Test case number: 6 testCart0.add(1); testCart0.pay(); 94
The FITTEST project is funded by the European Commission (FP7-ICT-257574) FI testing •  Models  are  inferred  con4nuously   •  Test  case  generators  cover  all  branches  of  the   FSM  monitor  è  all  model  transi4ons   •  Test  suites  are  updated  and  re-­‐executed   incrementally   The  testware  evolves  autonomously  with  the  FI   applica6on  under  test   95
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Dynamic  symbolic  execu4on   Slides are partially based on the ESEC-FSE’05 presentation: http://srl.cs.berkeley.edu/~ksen 96
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   Path  coverage:   1.  Select  a  path  p  to  be  covered.   2.  Determine  the  path  condi4on  (path  constraint),  by   propaga4on  of  symbolic  input  values  along  the  path   (expressions  are  evaluated  symbolically).  The  path  condi4on  is   a  boolean  expression  containing  only  input  variables.   3.  Solve  the  path  condi4on  using  an  SMT  solver.   97
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() { if (a == b) if (b == c) type = EQUILATERAL; // target else type = ISOSCELE; else if (a == c) type = ISOSCELE; else if (b == c) type = ISOSCELE; else checkRightAngle(); } boolean isTriangle() {…} public static void main(String args[]) {…} } 98
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle(){…} void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) { int a = Integer.parseInt(args[0]); int b = Integer.parseInt(args[1]); int c = Integer.parseInt(args[2]); Triangle t = new Triangle(a, b, c); if (t.isTriangle()) t.computeTriangleType(); System.out.println(typeToString(t.type)); } } 99
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() {…} boolean isTriangle() { if (a <= 0 || b <= 0 || c <= 0) return false; if (a + b <= c || a + c <= b || b + c <= a) return false; return true; } public static void main(String args[]) {…} } Path  condi6on  =  !  (a  <=  0  ||  b  <=  0  ||  c  <=  0)  &&  !(a  +  b  <=  c  ||  a   +  c  <=  b  ||  b  +  c  <=  a)  &&  (a  ==  b)  &&  (b  ==  c)   100
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Yices  SMT  solver   Path  condi6on  =  !  (a  <=  0  ||  b  <=  0  ||  c  <=  0)  &&  !(a  +  b  <=  c  ||  a   +  c  <=  b  ||  b  +  c  <=  a)  &&  (a  ==  b)  &&  (b  ==  c)   =  (a  >  0)  &&  (b  >  0)  &&  (c  >  0)  &&  (a  +  b  >  c)  &&  (a  +  c  >  b)  &&  (b  +   c  >  a)  &&  (a  ==  b)  &&  (b  ==  c)   (define a::int) (define b::int) (define c::int) (assert (> a 0)) (assert (> b 0)) (assert (> c 0)) (assert (> (+ a b) c)) (assert (> (+ a c) b)) (assert (> (+ b c) a)) (assert (= a b)) (assert (= b c)) (check) sat (= a 1) (= b 1) (= c 1) 101
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Problems  of  sta6c  symbolic   execu6on   In  prac4ce,  sta6c  symbolic  execu4on  has  several  limita4ons  that   make  it  inapplicable  for  test  case  genera4on  for  non  trivial   programs:   Ø Loops  must  be  unrolled  a  fixed  number  of  4mes.   Ø Sta4c  symbolic  execu4on  may  require  a  lot  of  (SMT-­‐solving)   4me  and  does  not  scale  to  large  programs.   Ø   SMT  solvers  usually  do  not  handle  non  linear  constraints.   Ø   SMT  solvers  cannot  deal  with  black  box  func4ons.   Dynamic  symbolic  execu6on:   Start  from  an  available  (random),  concrete  execu(on  and  use   symbolic  execu(on  to  explore  alterna(ve  paths.   102
The FITTEST project is funded by the European Commission (FP7-ICT-257574) In  a  nutshell   Goal:  to  generate  data  inputs  that  exercise  all  feasible   execu(on  paths  up  to  a  given  length.   1. Execute a random test case. 2. Collect symbolic constraints along the concretely executed path. 3. Negate one branch condition in the path constraint. 4. Use constraint solvers to generate a new test input for the negated constraint. 5. Execute the new test case, covering a new path; iterate from 2. 103
The FITTEST project is funded by the European Commission (FP7-ICT-257574) In  a  nutshell   Problem:  the  constraint  solver  may  not  be  powerful  enough   to  determine  concrete  values  that  sa(sfy  the  negated  path   constraint.   4. Use constraint solvers to generate a new test input for the negated constraint. 4’ If necessary, simplify the path constraint by replacing some of the symbolic values with concrete values. In  par4cular,  simplifica4on  may  be  necessary  whenever  non   linear  constraints  are  involved  or  black-­‐box,  external  func4ons   are  called.   104
The FITTEST project is funded by the European Commission (FP7-ICT-257574) x = y + 2; Symb_exec(“x=y+2”); … if (x > z) { Symb_exec(“x>z”); 1.  Update  symbolic  state   2.  Accumulate  path  condi4on   Implementa6on   Instrumenta6on  can  be  used  to  perform  concrete  and  symbolic   execu4on  at  the  same  4me.   When  a  func4on  is  called  for  which  the  code  is  not  available  for  symbolic   execu4on,  the  outcome  of  its  concrete  execu4on  is  used  as  an   approxima4on  of  its  symbolic  execu4on.   1.  Update  symbolic  state   2.  Accumulate  path  condi4on   105
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  1   int testme(int x) { 1 int y = x + 3; 2 if (y == 13) 3 target(); 4 return 0; } Execu4on  constraints  are  collected  dynamically  and  negated  to   traverse    alterna4ve  paths   1 2 3 4 <x  =  0>  è  <1,  2,  4>  è  (x0  +  3  ≠  13)   (x0  +  3  =  13)  è  <x  =  10>  è  <1,  2,  3>   concrete state symbolic state constraints x  =  0     y  =  3   x  =  x0   y  =  x0  +  3   (x0  +  3  ≠  13)   106
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  ≠  ‘a’)  &  (x2  ≠  ‘d’)  &  (x3  =   ‘!’)  è  <x  =  “goo!”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “good”     n  =  0   x  =  {x0,  x1,  x2,   x3}   n  =  0     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  ≠  ‘d’)  &   (x3  ≠  ‘!’)  &   (0  <  4)   107
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  ≠  ‘a’)  &  (x2  =  ‘d’)  &  (x3  =   ‘!’)  è  <x  =  “god!”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “goo!”     n  =  1   x  =  {x0,  x1,  x2,   x3}   n  =  1     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  ≠  ‘d’)  &   (x3  =  ‘!’)  &   (1  <  4)   108
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  ≠  ‘a’)  &  (x2  =  ‘d’)  &  (x3  ≠   ‘!’)  è  <x  =  “godd”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “god!”     n  =  2   x  =  {x0,  x1,  x2,   x3}   n  =  2     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  =  ‘d’)  &   (x3  =  ‘!’)  &   (2  <  4)   109
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  =  ‘a’)  &  (x2  =  ‘d’)  &  (x3  ≠   ‘!’)  è  <x  =  “gadd”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “godd”     n  =  1   x  =  {x0,  x1,  x2,   x3}   n  =  1     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  =  ‘d’)  &   (x3  ≠  ‘!’)  &   (1  <  4)   110
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  =  ‘b’)  &  (x1  =  ‘a’)  &  (x2  =  ‘d’)  &  (x3  =   ‘!’)  è  <x  =  “bad!”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “badd”     n  =  3   x  =  {x0,  x1,  x2,   x3}   n  =  3     (x0  =  ‘b’)  &   (x1  =  ‘a’)  &   (x2  =  ‘d’)  &   (x3  ≠  ‘!’)  &   (3  <  4)  Target  reached   Amer  16   steps   111
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  randomly   112
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  randomly   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   •  take  then  branch  with  constraint   x0*x0*x0+  3*x0*x0+9  !=  y0   113
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  randomly   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   •  take  then  branch  with  constraint   x0*x0*x0+  3*x0*x0+9  !=  y0   •  solve  x0*x0*x0+  3*x0*x0+9  =  y0  to   take  else  branch   •  Don’t  know  how  to  solve  !!   –  Stuck  ?   114
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   115
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   116
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   •  take  then  branch  with  constraint     –  9  !=  y0   117
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   •  take  then  branch  with  constraint     –  9  !=  y0   •  solve  9  =  y0    to  take  else  branch   •  execute  next  run  with  x  =  -­‐3  and   y=  9   –  Target  reached   118
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   •  take  then  branch  with  constraint     –  9  !=  y0   •  solve  9  =  y0    to  take  else  branch   •  execute  next  run  with  x  =  -­‐3  and   y=  9   –  Target  reached   Replace  symbolic   expression  with  concrete   value  when  symbolic   expression  becomes   unmanageable  (e.  g.  non-­‐ linear)   119
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Black-box functions   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }         void  test_me(int  x,int  y){    z  =  black_box_fun(x);    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }     The  same  approach  is  applied  to  any  called  func4on  which  cannot  be   executed  or  modeled  symbolically.   120
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Black-box functions   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }         void  test_me(int  x,int  y){    z  =  black_box_fun(x);    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }     The  same  approach  is  applied  to  any  called  func4on  which  cannot  be   executed  or  modeled  symbolically.   Use  the  concrete  value   returned  by  the  func4on   whenever  the  symbolic   expression  cannot  be   computed   121
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Proper6es   Ø Incompleteness:  dynamic  symbolic  execu4on  is  complete  only   if  the  constraint  solver  can  successfully  solve  all  constraints   without  having  to  simplify  them.     Ø   Approxima6on:  an  input  sa4sfying  a  simplified  path  condi4on,   where  some  symbolic  values  are  replaced  by  concrete  ones,  is  not   ensured  to  execute  the  path  of  interest  (i.e.,  its  execu4on  may   diverge  from  the  expected  one).   Ø   Soundness:  dynamic  symbolic  execu4on  is  always  sound,  since   generated  test  cases  are  executable,  hence  revealed  faults  are   real  faults.   122
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  of  approxima6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() { if (a*a + b*b == c*c) type = RIGHT_ANGLE; else if (b*b + c*c == a*a) type = RIGHT_ANGLE; else if (a*a + c*c == b*b) type = RIGHT_ANGLE; else type = SCALENE; } void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) {…} } Concrete  input  =  (2,  3,  4)  [SCALENE]   Path  condi6on  =  !  (a  <=  0  ||  b  <=  0  ||   c  <=  0)  &&  !(a  +  b  <=  c  ||  a  +  c  <=  b  ||   b  +  c  <=  a)  &&  !(a*a  +  b*b  ==  c*c)   &&  !(b*b  +  c*c  ==  a*a)  &&  !(a*a  +   c*c  ==  b*b)     123
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Example  of  approxima6on   Concrete  input  =  (2,  3,  4)  [SCALENE]   SMT  solver:     (a  >  0  &&  b  >  0  &&  c  >  0)  &&  (a  +  b  >  c  &&  a  +  c  >  b  &&  b  +  c  >  a)  &&   (a*a  +  b*b  !=  c*c)  &&  (b*b  +  c*c  !=  a*a)  &&     (a*a  +  c*c  ==  b*b)     Error:  feature  not  supported:  non  linear  problem.   Simplified  constraint:     (a  >  0  &&  b  >  0  &&  c  >  0)  &&  (a  +  b  >  c  &&  a  +  c  >  b  &&  b  +  c  >  a)  &&   (2*a  +  3*b  !=  4*c)  &&  (3*b  +  4*c  !=  2*a)  &&     (2*a  +  4*c  ==  3*b)     sat (= a 4) (= b 4) (= c 1) Not  a  right  angle  triangle!   Target  not  covered.   124
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Strengths  and  weaknesses   Dynamic  symbolic  execu6on:   Ø   Incompleteness:  not  all  paths  are  covered  if  concrete  values   are  used  and  the  simplified  constraint  admits  no  solu4on.     Ø   Approxima6on:  solu4ons  are  not  ensured  to  execute  the  path   of  interest  if  concrete  values  are  used.   Ø   Soundness:  generated  test  cases  are  executable,  faults  are  real.     Search  based  test  case  genera6on:   Ø   Incompleteness:  feasible  coverage  targets  might  be  missed.     Ø   Robustness:  non  linear  expressions  and  black  box  func4ons  do   not  represent  any  special  problem  in  fitness  func4on  defini4on.   Ø   Soundness:  generated  test  cases  are  executable,  faults  are  real.   Search  based  algorithms  can  easily  accommodate  approxima4ons,  but     might  miss  exact  solu4ons  which  are  available  via  SMT  solving.   125
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Combining  SBST  &  DSE   §  Kobi  Inkumsah,  Tao  Xie.  Improving  Structural  Tes(ng  of  Object-­‐Oriented   Programs  via  Integra(ng  Evolu(onary  Tes(ng  and  Symbolic  Execu(on.  Proc.   of  Automated  Somware  Engineering  (ASE),  pp.  297-­‐306,  2008.     §  Jan  Malburg,  Gordon  Fraser.  Combining  search-­‐based  and  constraint-­‐based   tes(ng.  Proc.  of  Automated  Somware  Engineering  (ASE),  pp.  436-­‐439,  2011.   §  Tao  Xie,  Nikolai  Tillmann,  Jonathan  de  Halleux,  Wolfram  Schulte.  Fitness-­‐ guided  path  explora(on  in  dynamic  symbolic  execu(on.  Proc.  of  the   Interna4onal  Conference  on  Dependable  Systems  and  Networks  (DSN),  pp.   359-­‐368,  2009.   §  Arthur  I.  Baars,  Mark  Harman,  Youssef  Hassoun,  Kiran  Lakho4a,  Phil  McMinn,   Paolo  Tonella,  Tanja  E.  J.  Vos.  Symbolic  search-­‐based  tes(ng.  Proc.  of   Automated  Somware  Engineering  (ASE),  pp.  53-­‐62,  2011.   126
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Genera6ng  new  individuals  by   means  of  DSE   Jan  Malburg,  Gordon  Fraser.  Combining  search-­‐based  and  constraint-­‐based  tes(ng.  Proc.  of   Automated  Somware  Engineering  (ASE),  pp.  436-­‐439,  2011.   New  individuals  are  generated  using  DSE.   Kobi  Inkumsah,  Tao  Xie.  Improving  Structural  Tes(ng  of  Object-­‐Oriented  Programs  via   Integra(ng  Evolu(onary  Tes(ng  and  Symbolic  Execu(on.  Proc.  of  Automated  Somware   Engineering  (ASE),  pp.  297-­‐306,  2008.     127
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Genera6ng  new  individuals  by   means  of  DSE   Initialization   Fitness   assignment   Selection  Reproduction   •  Muta4on   •  Crossover   •  DSE  based  muta6on   Ch1:   1   5   -­‐1   0   3   pc1:  C1  &&  C2  &&  C3  &&  C4   pc1’:  C1  &&  C2  &&  !C3   Ch1’:   1   0   2   1   1   128
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Primi6ve  values  genera6on   $x0=BankAccount():  $x0.deposit(int):  $x0.withdraw(int)  @  1,  20   Path  condi6on  (withdraw)  =  (z1  >  z0)   Error  handling  branch:   void withdraw(int amount) { if (amount > balance) { printError(); } else { balance = balance – amount; //target } … } Primi6ve  values  are  made  symbolic:  z0,  z1   129
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Primi6ve  values  genera6on   $x0=BankAccount():  $x0.deposit(int):  $x0.withdraw(int)  @  1,  20   Path  condi6on  (withdraw)  =  (z1  >  z0)   Primi6ve  values  are  made  symbolic:  z0,  z1   Negated  path  condi6on  =  (z1  <=  z0)   sat (= z1 1) (= z0 20) $x0=BankAccount():  $x0.deposit(int):  $x0.withdraw(int)  @  20,  1   Added  to  current  popula6on   130 Non-­‐error  branch:   if (amount > balance) { printError(); } else { balance = balance – amount; …
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Primi6ve  values  genera6on   double testMe(int x, int y, double z) { 1 boolean flag = y > 1000; 2 //… 3 if (x + y == 1024) 4 if (flag) 5 if (Math.cos(z) – 0.95 < Math.exp(z)) 6 abort(); 7 return 0.0; } Ch1:   1024   0   0.0   Standard  muta4on  may  take  a  while  to  make  y  >  1000   and  x  +  y  =  1024   pc1:  (x  +  y  =  1024)  &&  (y  ≤  1000)   pc1’:  (x  +  y  =  1024)  &&  (y  >  1000)   Ch1’:   23   1001   0.0   Normal  muta4on  and  crossover  are  s4ll  required  for   condi4ons  such  as  5.   131
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   A  fitness  func6on  is  used  to  select  which  path  to  explore  by   means  of  DSE.   Tao  Xie,  Nikolai  Tillmann,  Jonathan  de  Halleux,  Wolfram  Schulte.  Fitness-­‐guided  path   explora(on  in  dynamic  symbolic  execu(on.  Proc.  of  the  Interna4onal  Conference  on   Dependable  Systems  and  Networks  (DSN),  pp.  359-­‐368,  2009.   132
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } Ø   The  (randomly  selected)  size  of  y  must  be  ≥  20.   Ø   DSE  has  ≥  220  paths  to  explore,  among  which  only  a  small  frac4on   contain  exactly  twenty  3T,  which  is  required  to  make  5  true.   DSE   Ini4al  random  TC:     x  =  0,  y  =  {0}   p  =  <1F,  7>   pc  =  (x  !=  90)   pc’  =  (x  ==  90)   New  TC:   x  =  90,  y  =  {0}   p  =  <1T,  2T,  3F,  2F,  5F,  7>   133
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   d5T  =  {0  if  x  ==  110;  abs(x  -­‐  110)  otherwise}   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } 134
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC0:      TestLoop(0,  {0})      p  =  <1F,  7>   TC1:      TestLoop(90,  {0})      p  =  <1T,  2T,  3F,  2F,  5F,  7>   TC2:      TestLoop(90,  {15})      p  =  <1T,  2T,  3T,  2F,  5F,  7>   TC3:      TestLoop(90,  {15,  0})      p  =  <1T,  2T,  3T,  2T,  3F,  2F,  5F,  7>   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   §  f(TC0)  =  1  +  BD(1T)  =  1.989   §  d(TC1)  =  |90  –  110|  =  20,  BD  =  0.952   §  d(TC2)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC3)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC4)  =  |92  –  110|  =  18,  BD  =  0.947   135
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC0:      TestLoop(0,  {0})      p  =  <1F,  7>   TC1:      TestLoop(90,  {0})      p  =  <1T,  2T,  3F,  2F,  5F,  7>   TC2:      TestLoop(90,  {15})      p  =  <1T,  2T,  3T,  2F,  5F,  7>   TC3:      TestLoop(90,  {15,  0})      p  =  <1T,  2T,  3T,  2T,  3F,  2F,  5F,  7>   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   TC4  is  selected  for   branching  node  flipping   §  f(TC0)  =  1  +  BD(1T)  =  1.989   §  d(TC1)  =  |90  –  110|  =  20,  BD  =  0.952   §  d(TC2)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC3)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC4)  =  |92  –  110|  =  18,  BD  =  0.947   136
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  gain   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>     Which  branch  shall  be  flipped?   Fitness  gain:  how  much  the  fitness  value  has  improved  across   paths  amer  flipping  a  given  branch  in  the  past.     137
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  gain   DSE   TC0:      TestLoop(0,  {0})      p  =  <1F,  7>   TC1:  d  =  20      TestLoop(90,  {0})      p  =  <1T,  2T,  3F,  2F,  5F,  7>   TC2:  d  =  19      TestLoop(90,  {15})      p  =  <1T,  2T,  3T,  2F,  5F,  7>   TC3:  d  =  19      TestLoop(90,  {15,  0})      p  =  <1T,  2T,  3T,  2T,  3F,  2F,  5F,  7>   TC4:  d  =  18      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } §  ∆d(3F)  =  (1  +  1  +  2)  /  3  =  1.33   §  ∆d(3T)  =  (-­‐1  -­‐  1  -­‐  2)  /  3  =  -­‐1.33   §  ∆d(2F)  =  (0  +  1)  /  2  =  0.5   §  ∆d(2T)  =  (0  -­‐  1)  /  2  =  -­‐0.5   138
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  gain   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   Es6mated  distance:  the  new  branch  distance  associated  with   flipping  branch  b  is  es4mated  as:  d’(TC)  =  d(TC)  -­‐  ∆d(b)   §  Flip  2T:  d’(TC4)  =  18  +  0.5  =  18.5   §  Flip  3T:  d’(TC4)  =  18  +  1.33  =  19.33   §  Flip  2F:  d’(TC4)  =  18  -­‐  0.5  =  17.5   139
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC4:  TestLoop(90,  {15,  15}),  p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>,  d  =  18   TC5:  TestLoop(90,  {15,  15,  0}),  p  =  <1T,  2T,  3T,  2T,  3T,  2T,  3F,  2F,  5F,  7>,  d  =  18   TC6:  TestLoop(90,  {15,  15,  15}),  p  =  <1T,  2T,  3T,  2T,  3T,  2T,  3T,  2F,  5F,  7>,  d  =  17   …   TC40:  TestLoop(90,  {15,  15,  …,  15}),  p  =  <1T,  2T,  3T,  2T,  3T,  2T,  3T,  2T,  …,  2F,  5T,  7>,  d   =  0   41  paths  are  explored  to   cover  5T;  full  explora4on   of  all  paths  would  require   ≥  220  test  cases.   140
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   The  fitness  func6on  is  improved  by  means  of  symbolic   execu4on.   Arthur  I.  Baars,  Mark  Harman,  Youssef  Hassoun,  Kiran  Lakho4a,  Phil  McMinn,  Paolo  Tonella,   Tanja  E.  J.  Vos.  Symbolic  search-­‐based  tes(ng.  Proc.  of  ASE,  pp.  53-­‐62,  2011.   141
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   ¢  Concrete execution of SUT ¢  Symbolic execution of each path between critical node and target ¢  Compute cumulative branch distance for each path condition ¢  Return minimal distance ¢  Example: —  abef —  abcdef Z > 0 ∧ X = Z ∧ Z ≤ 0 ∧ Y = 10 a b c d e f Z > 0 ∧ X = Z ∧ Z ≤ 1 ∧ Y = 9 bd>(z,0) + bd=(x,z) + bd<=(z,1) + bd=(y,9) 142
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   c ¢  Path expression: —  ab(cd)*ef ¢  Choose a maximum loop unrolling (e.g., 1) —  abef —  abcdD[cd]ef e Z > 0 ∧ X = Z ∧ Z > 0 ∧ Z ≤ 1 ∧ Y = 9 a b d f 143
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   Z > 0 ∧ X = Z ∧ Z > 0 ∧ Z ≤ 1 ∧ Y = 9 FF1 = bd>(z,0) + bd+(x,z) + bd>(z,0) + 1 + 1 path distance approximation level 144
The FITTEST project is funded by the European Commission (FP7-ICT-257574) int foo(int x, int y, int z) { if(z > 0) if(x == z) { for(int i = 0; i < z; i++) y++ if(y == 10) return 0; } return 1; } FF1 = bd>(z,0) + bd=(x,z) + bd>(z,0) + 1 + 1 Fitness  func6on  improvement   target FF2 FF5 FF3 FF1 Fitness = min(FF1,FF2,FF3,FF5) 145
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Conclusions  and  future  work   146
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Conclusions   •  High  coverage  can  be  achieved  automa4cally   using  search  based  test  case  genera4on.   •  Func4ons  and  classes  under  test  are  exercised  in   a  quite  sophis4cated  way.   •  Resul4ng  test  suites  are  generally  quite  compact   and  their  size  can  be  controlled  as  a  further   op4miza4on  objec4ve.   147
The FITTEST project is funded by the European Commission (FP7-ICT-257574) Future  work   •  Inves4ga4on  of  alterna4ve  adequacy  criteria  and   tes4ng  goals  (e.g.,  non-­‐func4onal  tes4ng).   •  Automa4ng  the  genera4on  of  likely  oracles  (i.e.,   anomaly  detectors),  based  on  invariants  and   temporal  proper4es  of  the  func4ons  and  classes   under  test.   •  Further  empirical  studies,  especially  on  combined   approaches.   148
The FITTEST project is funded by the European Commission (FP7-ICT-257574) References   §  Shaukat  Ali,  Lionel  C.  Briand,  Hadi  Hemma4,  Rajwinder  Kaur  Panesar-­‐Walawege:  A   Systema(c  Review  of  the  Applica(on  and  Empirical  Inves(ga(on  of  Search-­‐Based  Test  Case   Genera(on.  IEEE  Transac4ons  on  Somware  Engineering,  vol.  36  n.  6,  pp.  742-­‐762,  2010.   §  Andrea  Arcuri,  Muhammad  Zohaib  Z.  Iqbal,  Lionel  C.  Briand:  Formal  analysis  of  the   effec(veness  and  predictability  of  random  tes(ng.  ISSTA,  pp.  219-­‐230,  2010.   §  Gordon  Fraser,  Andrea  Arcuri:  It  is  Not  the  Length  That  Ma`ers,  It  is  How  You  Control  It.   Proc.  of  the  Interna4onal  Conference  on  Somware  Tes4ng  (ICST),  pp.  150-­‐159,  2011.   §  Gordon  Fraser,  Andrea  Arcuri,  Evolu(onary  genera(on  of  whole  test  suites.  Proc.  of  the   Interna4onal  Conf.  on  the  Quality  of  Somware  (QSIC),  pp.  31-­‐40,  Madrid,  Spain,  July  2011.   §  Gordon  Fraser,  Andrea  Arcuri,  Whole  test  suite  genera(on.  IEEE  Transac4ons  on  Somware   Engineering,  vol.  38,  n.  2,  pp.  276-­‐291,  2013.   §  Arthur  I.  Baars,  Mark  Harman,  Youssef  Hassoun,  Kiran  Lakho4a,  Phil  McMinn,  Paolo  Tonella,   Tanja  E.  J.  Vos.  Symbolic  search-­‐based  tes(ng.  Proc.  of  Automated  Somware  Engineering   (ASE),  pp.  53-­‐62,  2011.   §  Patrice  Godefroid,  Nils  Klarlund,  and  Koushik  Sen,  DART:  Directed  Automated  Random   Tes4ng,  PLDI’05.   §  Mark  Harman  and  Phil  McMinn.  A  Theore(cal  and  Empirical  Study  of  Search  Based  Tes(ng:   Local,  Global  and  Hybrid  Search.  IEEE  Transac4ons  on  Somware  Engineering,  vol.  36,  n.  2,  pp.   226-­‐247,  2010.   149
The FITTEST project is funded by the European Commission (FP7-ICT-257574) References   §  Mark  Harman,  Lin  Hu,  Rob  Hierons,  Joachim  Wegener,  Harmen  Sthamer,  Andre  Baresel  and   Marc  Roper.  Testability  Transforma(on.  IEEE  Transac4ons  on  Somware  Engineering,  vol.  30,   n.  1,  pp.  3-­‐16,  2004.   §  Kobi  Inkumsah,  Tao  Xie.  Evacon:  a  framework  for  integra(ng  evolu(onary  and  concolic   tes(ng  for  object-­‐oriented  programs.  Proc.  of  Automated  Somware  Engineering  (ASE),  pp.   425-­‐428,  2007.   §  Kobi  Inkumsah,  Tao  Xie.  Improving  Structural  Tes(ng  of  Object-­‐Oriented  Programs  via   Integra(ng  Evolu(onary  Tes(ng  and  Symbolic  Execu(on.  Proc.  of  Automated  Somware   Engineering  (ASE),  pp.  297-­‐306,  2008.     §  Bogdan  Korel:  Automated  SoDware  Test  Data  Genera(on,  IEEE  Transac4ons  on  Somware   Engineering,  vol.  16,  n.  8,  pp.  870-­‐879,  1990.   §  Jan  Malburg,  Gordon  Fraser.  Combining  search-­‐based  and  constraint-­‐based  tes(ng.  Proc.  of   Automated  Somware  Engineering  (ASE),  pp.  436-­‐439,  2011.   §  Phil  McMinn,  Search-­‐based  soDware  test  data  genera(on:  a  survey.  Journal  of  Somware   Tes4ng,  Verifica4on  and  Reliability,  vol.  14,  n.  2,  pp.  105-­‐156,  June  2004.   §  Roy  P.  Pargas,  Mary  Jean  Harrold  and  Robert  R.  Peck:  Test-­‐data  genera(on  using  gene(c   algorithms,  Somware  Tes4ng,  Verifica4on  and  Reliability,  vol.  9,  n.  4,  pp.  263-­‐282,  1999.   §  Koushik  Sen,  Darko  Marinov,  Cul  Agha,  CUTE:  A  Concolic  Unit  Tes4ng  Engine  for  C,  ESEC-­‐ FSE’05.   150
The FITTEST project is funded by the European Commission (FP7-ICT-257574) References   §  Paolo  Tonella,  Evolu(onary  tes(ng  of  classes.  Proc.  of  the  Interna4onal  Symposium  on   Somware  Tes4ng  and  Analysis  (ISSTA),  pp.  119-­‐128,  Boston,  USA,  July  2004.   §  David  H.  Wolpert,  William  G.  Macready:  No  Free  Lunch  Theorems  for  Op(miza(on,  IEEE   Transac4ons  on  Evolu4onary  Computa4on,  vol.  1,  n.  1,  April  1997.   §  Tao  Xie,  Nikolai  Tillmann,  Jonathan  de  Halleux,  Wolfram  Schulte.  Fitness-­‐guided  path   explora(on  in  dynamic  symbolic  execu(on.  Proc.  of  the  Interna4onal  Conference  on   Dependable  Systems  and  Networks  (DSN),  pp.  359-­‐368,  2009.   Slides  on  dynamic  symbolic  execu4on  are  par4ally  based  on  the  ESEC-­‐FSE’05  presenta4on:  hVp:// srl.cs.berkeley.edu/~ksen   151

More Related Content

PDF
Pythran: Static compiler for high performance by Mehdi Amini PyData SV 2014
byPyData
 
PPTX
IFLA trends report - Segametsi Molawa
byHELIGLIASA
 
PDF
sarah resume timeline 11 with dansco 4
bySarah Miller
 
PPT
1) in what ways does the media prezi
bybrookiebuzzin
 
PPTX
VPInstruments VPVision
bymenno_verbeek
 
PPTX
LieDM atviras pazangos forumas 2015
byAirina Volungeviciene
 
PDF
30 Measurable Use Cases for IoT by M Capone
byProf. Dr. Michael J. Capone
 
PPT
hardware
byfenris97
 
Pythran: Static compiler for high performance by Mehdi Amini PyData SV 2014
byPyData
 
IFLA trends report - Segametsi Molawa
byHELIGLIASA
 
sarah resume timeline 11 with dansco 4
bySarah Miller
 
1) in what ways does the media prezi
bybrookiebuzzin
 
VPInstruments VPVision
bymenno_verbeek
 
LieDM atviras pazangos forumas 2015
byAirina Volungeviciene
 
30 Measurable Use Cases for IoT by M Capone
byProf. Dr. Michael J. Capone
 
hardware
byfenris97
 

Viewers also liked

PPTX
Treating Apps as Startups
byDean Johnson
 
PDF
Tech Talk: Achieve a Customer-Centric View in an Omni-Channel World  
byCA Technologies
 
PDF
Reality of the Consumer Experience
byiQmetrixCorp
 
PPSX
Crazy Hats: Royal Ascot 2016
bymaditabalnco
 
PPTX
The Practice of Extreme Listening
byCaravan Studios, a division of TechSoup
 
PDF
Cotizacion cx350 b
byICA PROYECTOS Y CONSTRUCCIONES SRL
 
PPT
Top 40 Fan Pages on Facebook Revealed
bytheKbuzz
 
PPT
2011 ford mustang
byn156250
 
PDF
Culture Code @ Arbunize
byArbunize
 
Treating Apps as Startups
byDean Johnson
 
Tech Talk: Achieve a Customer-Centric View in an Omni-Channel World  
byCA Technologies
 
Reality of the Consumer Experience
byiQmetrixCorp
 
Crazy Hats: Royal Ascot 2016
bymaditabalnco
 
The Practice of Extreme Listening
byCaravan Studios, a division of TechSoup
 
Cotizacion cx350 b
byICA PROYECTOS Y CONSTRUCCIONES SRL
 
Top 40 Fan Pages on Facebook Revealed
bytheKbuzz
 
2011 ford mustang
byn156250
 
Culture Code @ Arbunize
byArbunize
 

Similar to TAROT2013 Testing School - Paolo Tonella presentation

PPTX
module 3_updated ai machine learning.pptx
byadityasrivastav8769
 
PPT
Genetic Algorithms-1.ppt
byDrSanjeevPunia
 
PPT
BeyondClassicalSearch.ppt
byjpradha86
 
PPT
BeyondClassicalSearch.ppt
byGauravWani20
 
PDF
Tutorial rpo
bymosi2005
 
PDF
Self-configuring Classical Logic Gate Circuits using Genetic Programming in J...
byAritra Sarkar
 
PPTX
Artificial Intelligence - Informed search algorithms
bylynangelina1
 
PPT
Informed Search on machine learning .ppt
bynamia10
 
PPTX
Artificial Intelligence_Anjali_Kumari_26900122059.pptx
byCCBProduction
 
PPTX
Genetic algorithms
byswapnac12
 
PPTX
3.informed search
byKONGU ENGINEERING COLLEGE
 
PPT
Searchadditional2
bychandsek666
 
PPT
Heuristic Search Techniques Unit -II.ppt
bykarthikaparthasarath
 
PPT
hillclimb algorithm for heuristic search
byRaghavendraPrasad179187
 
PPT
Heuristic Search Techniques Unit -II.ppt
bykarthikaparthasarath
 
PPTX
Thesis_Presentation
byMaruf Alam
 
PDF
Genetic Algorithms: Programming by the Seat of Your Genes
byJeremy Fisher
 
PDF
Lec 7 genetic algorithms
byEyob Seyfu
 
PDF
hillclimbing in Artificial intelligence.pdf
byratnababum
 
PPTX
Genetic Algorithm
byFatemeh Karimi
 
module 3_updated ai machine learning.pptx
byadityasrivastav8769
 
Genetic Algorithms-1.ppt
byDrSanjeevPunia
 
BeyondClassicalSearch.ppt
byjpradha86
 
BeyondClassicalSearch.ppt
byGauravWani20
 
Tutorial rpo
bymosi2005
 
Self-configuring Classical Logic Gate Circuits using Genetic Programming in J...
byAritra Sarkar
 
Artificial Intelligence - Informed search algorithms
bylynangelina1
 
Informed Search on machine learning .ppt
bynamia10
 
Artificial Intelligence_Anjali_Kumari_26900122059.pptx
byCCBProduction
 
Genetic algorithms
byswapnac12
 
3.informed search
byKONGU ENGINEERING COLLEGE
 
Searchadditional2
bychandsek666
 
Heuristic Search Techniques Unit -II.ppt
bykarthikaparthasarath
 
hillclimb algorithm for heuristic search
byRaghavendraPrasad179187
 
Heuristic Search Techniques Unit -II.ppt
bykarthikaparthasarath
 
Thesis_Presentation
byMaruf Alam
 
Genetic Algorithms: Programming by the Seat of Your Genes
byJeremy Fisher
 
Lec 7 genetic algorithms
byEyob Seyfu
 
hillclimbing in Artificial intelligence.pdf
byratnababum
 
Genetic Algorithm
byFatemeh Karimi
 

More from Henry Muccini

PPTX
Human Behaviour Centred Design
byHenry Muccini
 
PDF
How cultural heritage, cyber-physical spaces, and software engineering can wo...
byHenry Muccini
 
PDF
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle Segreterie
byHenry Muccini
 
PDF
Turismo 4.0: l'ICT a supporto del turismo sostenibile
byHenry Muccini
 
PDF
Sustainable Tourism - IoT and crowd management
byHenry Muccini
 
PDF
Software Engineering at the age of the Internet of Things
byHenry Muccini
 
PDF
The influence of Group Decision Making on Architecture Design Decisions
byHenry Muccini
 
PDF
An IoT Software Architecture for an Evacuable Building Architecture
byHenry Muccini
 
PDF
Web Engineering L8: User-centered Design (8/8)
byHenry Muccini
 
PDF
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)
byHenry Muccini
 
PDF
Web Engineering L6: Software Architecture for the Web (6/8)
byHenry Muccini
 
PDF
Web Engineering L5: Content Model (5/8)
byHenry Muccini
 
PDF
Web Engineering L3: Project Planning (3/8)
byHenry Muccini
 
PDF
Web Engineering L2: Requirements Elicitation for the Web (2/8)
byHenry Muccini
 
PDF
Web Engineering L1: introduction to Web Engineering (1/8)
byHenry Muccini
 
PDF
Web Engineering L4: Requirements and Planning in concrete (4/8)
byHenry Muccini
 
PDF
Collaborative aspects of Decision Making and its impact on Sustainability
byHenry Muccini
 
PDF
Engineering Cyber Physical Spaces
byHenry Muccini
 
PDF
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPIS
byHenry Muccini
 
PDF
Exploring the Temporal Aspects of Software Architecture
byHenry Muccini
 
Human Behaviour Centred Design
byHenry Muccini
 
How cultural heritage, cyber-physical spaces, and software engineering can wo...
byHenry Muccini
 
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle Segreterie
byHenry Muccini
 
Turismo 4.0: l'ICT a supporto del turismo sostenibile
byHenry Muccini
 
Sustainable Tourism - IoT and crowd management
byHenry Muccini
 
Software Engineering at the age of the Internet of Things
byHenry Muccini
 
The influence of Group Decision Making on Architecture Design Decisions
byHenry Muccini
 
An IoT Software Architecture for an Evacuable Building Architecture
byHenry Muccini
 
Web Engineering L8: User-centered Design (8/8)
byHenry Muccini
 
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)
byHenry Muccini
 
Web Engineering L6: Software Architecture for the Web (6/8)
byHenry Muccini
 
Web Engineering L5: Content Model (5/8)
byHenry Muccini
 
Web Engineering L3: Project Planning (3/8)
byHenry Muccini
 
Web Engineering L2: Requirements Elicitation for the Web (2/8)
byHenry Muccini
 
Web Engineering L1: introduction to Web Engineering (1/8)
byHenry Muccini
 
Web Engineering L4: Requirements and Planning in concrete (4/8)
byHenry Muccini
 
Collaborative aspects of Decision Making and its impact on Sustainability
byHenry Muccini
 
Engineering Cyber Physical Spaces
byHenry Muccini
 
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPIS
byHenry Muccini
 
Exploring the Temporal Aspects of Software Architecture
byHenry Muccini
 

Recently uploaded

PDF
Computer-Based Training (CBT) The Backbone of Modern Technical & Defence Trai...
bycomputerbasedtrainin1
 
PDF
Taxonomy Alignment for SDG Tagging - ASHA Case Study
byEnterprise Knowledge
 
PDF
AI assisted Software Development - Agentic and Vibe Coding
byAndrei Oros
 
PDF
Bettersize | BeSEC Series Product Brochure
byBettersize Instruments
 
PDF
"When every team does things "right", but together it turns into chaos", Yozh...
byFwdays
 
PDF
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
PDF
ICT500 - CRITICAL AND CREATIVE THINKING FOR INFORMATION TECHNOLOGY SOLUTIONS:...
by2024432452
 
PPTX
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
PDF
Agentic-Document-Extraction-2026-Presentation.pdf
byTamanna
 
PDF
Real-Time AI at Scale Masterclass: Feature Store at Scale
byScyllaDB
 
PPTX
CRM for the Insurance Industry Lessons for Insurance CIOs and Digital Leaders...
byKerry Millar
 
PDF
Certified AI-Empowered SAFe 6 Scrum Master.pdf
byMarc Entsminger SPC6, RTE, SSM, SA, SDP
 
PDF
2026_01_28 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
PDF
React Mastery: Visual Mental Models to Understand React Deeply
byThomas Gaye
 
PPTX
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
PDF
TrustArc Webinar - From Zero to Privacy Hero: Launching Your Program Right an...
byTrustArc
 
PDF
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
PDF
What Thema can do: Leveraging metadata to support the discoverability of Firs...
byBookNet Canada
 
PDF
Fortinet Certified Fundamentals in Cybersecurity
byVICTOR MAESTRE RAMIREZ
 
PDF
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 
Computer-Based Training (CBT) The Backbone of Modern Technical & Defence Trai...
bycomputerbasedtrainin1
 
Taxonomy Alignment for SDG Tagging - ASHA Case Study
byEnterprise Knowledge
 
AI assisted Software Development - Agentic and Vibe Coding
byAndrei Oros
 
Bettersize | BeSEC Series Product Brochure
byBettersize Instruments
 
"When every team does things "right", but together it turns into chaos", Yozh...
byFwdays
 
How the EU Ecolabel's Record Growth Creates ESG Opportunities for CRE
byWastify AI
 
ICT500 - CRITICAL AND CREATIVE THINKING FOR INFORMATION TECHNOLOGY SOLUTIONS:...
by2024432452
 
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
Agentic-Document-Extraction-2026-Presentation.pdf
byTamanna
 
Real-Time AI at Scale Masterclass: Feature Store at Scale
byScyllaDB
 
CRM for the Insurance Industry Lessons for Insurance CIOs and Digital Leaders...
byKerry Millar
 
Certified AI-Empowered SAFe 6 Scrum Master.pdf
byMarc Entsminger SPC6, RTE, SSM, SA, SDP
 
2026_01_28 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
React Mastery: Visual Mental Models to Understand React Deeply
byThomas Gaye
 
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
TrustArc Webinar - From Zero to Privacy Hero: Launching Your Program Right an...
byTrustArc
 
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
What Thema can do: Leveraging metadata to support the discoverability of Firs...
byBookNet Canada
 
Fortinet Certified Fundamentals in Cybersecurity
byVICTOR MAESTRE RAMIREZ
 
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 

TAROT2013 Testing School - Paolo Tonella presentation

  • 1.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Paolo Tonella Fondazione Bruno Kessler Trento, Italy http://se.fbk.eu/tonella Search Based Test Case Generation 1
  • 2.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Outline   •  Search  based  algorithms   •  Automated  test  case  genera4on   •  Object  oriented  tes4ng   •  Future  internet  tes4ng   •  Dynamic  symbolic  execu4on   2
  • 3.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Search  based  algorithms   3
  • 4.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) The  search  problem   Global  op4mum  Local   op4mum   4
  • 5.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Search  algorithms   •  Exhaus4ve  search   •  Random  search   •  Metaheuris4c  search   Metaheuris6c  algorithm:  itera4ve  op4miza4on  of  candidate   solu4ons  for  arbitrary  problem  instances.  It  is  not  ensured  to   find  a  global  op4mum,  it  finds  a  so-­‐called  near-­‐op(mal  solu(on. 5
  • 6.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Explora6on  vs.  exploita6on   Since  the  search  budget  is   finite,  metaheuris4c   algorithms  aim  for  a  balance   between:     •  local  search:  exploita4on   (intensifica4on);   •  global  search:  explora4on   (diversifica4on).   Search  budget:  max  number  of  fitness  evalua4ons  (number  of   samples  x  taken)  compa4ble  with  the  max  algorithm  execu4on   4me. 6
  • 7.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) No  free  lunch  theorems   David  H.  Wolpert,  William  G.  Macready:  No  Free  Lunch  Theorems  for   Op(miza(on,  IEEE  Transac4ons  on  Evolu4onary  Computa4on,  vol.  1,   n.  1,  April  1997. NFL  theorem:  given  a  search  budget  m,  the  average  probability  of   obtaining  the  near-­‐op4mal  value  f*  using  algorithm  A  is  the  same   as  the  probability  of  obtaining  the  same  near-­‐op4mal  value  using   another,  arbitrarily  chosen  algorithm  B. Hence,  if  metaheuris4c  A  performs  beVer  than  random  search  on   problem  instance  f1,  there  will  be  another  problem  instance  f2  on   which  random  performs  beVer  than  A. 7
  • 8.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) f1*   Decep6ve  fitness  func6on   X   f1   X   f2   P(f1*|f1,  m,  HC)  =  1   P(f1*|f1,  m,  RND)  =  |m|  /  |X|   ε f2*   P(f2*|f2,  m,  HC)  =  |ε|  /  |X|   P(f2*|f2,  m,  RND)  =  |m|  /  |X|   HC  =  hill  climbing  (steepest  ascent)   RND  =  random  search   8
  • 9.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Metaheuris6c  algorithms   •  Hill  climbing   •  Simulated  annealing   •  Tabu  search   •  Gene4c  algorithms   •  Ant  colony  op4miza4on   •  Par4cle  swarm  op4miza4on   9
  • 10.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Hill  climbing   1.  x_opt = x0 // random 2.  max = f(x_opt) 3.  LOOP // with search budget m 4.  improved = FALSE 5.  FOR x IN Neighbors(x_opt) 6.  IF f(x) > max 7.  max = f(x) 8.  x_opt = x 9.  improved = TRUE 10.  END IF 11.  END FOR 12.  IF NOT improved 13.  RETURN x_opt 14.  END IF 15.  END LOOP 16.  RETURN x_opt X   E.g.:  find  input  that   crashes       myFunc(int a, int b, int c){…} E.g.:  (a±1,  b±1,  c±1) Equivalent  to  greedy  or  steepest  ascent:  exploita4on,  no  explora4on.   10
  • 11.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Hill  climbing  variants   •  Stochas6c  hill  climbing:  selects  randomly  among  the   improving  solu4ons  in  the  neighborhood.   •  Random-­‐restart  (shotgun)  hill  climbing:  randomly   restarts  when  no  improving  solu4on  is  found  (or  the   improvement  is  small).   They  aVempt  to  add  more  explora4on,  when  local  search  is   ineffec4ve.   11
  • 12.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Simulated  annealing   1.  x_opt = x0 // random 2.  k = 1 // time 3.  T = T_max // temperature 4.  x = x_opt 5.  LOOP // with search budget m 6.  x_new = RNDNeighbor(x) 7.  Df = f(x_new) - f(x) 8.  IF Df > 0 9.  x_opt = x_new 10.  END IF 11.  IF Df > 0 OR 12.  P(Df, T) > rand(0, 1) 13.  x = x_new 14.  END IF 15.  T = cool(T, k) 16.  k = k + 1 17.  END LOOP 18.  RETURN x_opt Parameters:   E.g.:   P(Df,  T)  =  exp(Df  /  T)     if  Df  <  0;  1  otherwise     cool(T,  k)  =  α  T   if  k  %  K  =  0;  T  otherwise   +   -­‐   -­‐   +   Df  >  0  means   lower  energy   state     12
  • 13.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Simulated  annealing   •  Ini6al  high  temperature:  ini4ally,  explora4on  is  privileged   over  exploita4on:  worsening  solu4ons  are  ini4ally   accepted  with  high  probability   •  Low  temperature  at  the  end:  only  improving  solu4ons  are   accepted  at  the  end  of  the  search  (similar  to  hill  climbing).   Cooling  schedule:   T’  =  α  T      if  k  %  K  =  0   T’  =  T              otherwise     with  0  ≤  α  ≤  1   k   T   K   2K   3K   4K   5K   P(Df,  T)  =  exp(Df  /  T)     13
  • 14.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Tabu  search   1.  x_opt = x0 // random 2.  add(tabuFIFOList, x_opt) 3.  LOOP // with search budget m 4.  x_new = RNDMutate(x_opt) 5.  IF x_new NOT IN tabuFIFOList 6.  add(tabuFIFOList, x_new) 7.  IF f(x_new) > f(x_opt) 8.  x_opt = x_new 9.  END IF 10.  END IF 11.  END LOOP 12.  RETURN x_opt Local  search  (exploita4on),   op4mized  to  avoid   revisi4ng  previously   examined  solu4ons   +   -­‐   -­‐   +   14
  • 15.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Tabu  search  variants   •  Neighborhood  tabu:  the  en4re  neighborhood  of  an   already  visited  solu4on  is  declared  tabu;  muta4ons  jump   beyond  such  neighborhood.   •  Property  based  tabu  list:  tabu  solu4ons  are  characterized   by  their  proper4es,  such  that  any  (even  non  visited)   solu4on  with  the  proper4es  of  a  tabu  solu4on  is   considered  tabu  as  well.   •  Reac6ve  tabu  search:  when  the  same  (tabu)  solu4ons   reappear  frequently  during  the  search,  a  random  walk  is   made  to  explore  a  different  por4on  of  the  search  space.   15
  • 16.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Gene6c  algorithms   1.  Pop = {x0, …, xN} // random 2.  x_opt = NULL 3.  LOOP // with search budget m 4.  assignFitness(Pop) 5.  x_opt = best(Pop, x_opt) 6.  Pop = select(Pop) 7.  Pop = reproduce(Pop) 8.  END LOOP 9.  RETURN x_opt Key  decisions:   1.  Representa4on   2.  Fitness  func4on   3.  Selec4on   4.  Reproduc4on  (crossover   and  muta4on)   Initialization   Fitness   assignment   Selection  Reproduction   Evolu6onary  loop   16
  • 17.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Representa6on   Fixed  length  chromosome:   Variable  length  chromosome:  k  varies  across  individuals   y1   y2   y3   y4   y5   y6   …   yk   yi  may  be  a  single  bit,  a   number,  a  character,  etc.   E.g.:  find  input  that  crashes    myFunc(int a, int b, int c){…} Chromosomes  =  (2,  0,  3),  (0,  0,  -­‐1)   Chromosomes  =  (‘a’,  ‘b’,  ‘c’,  ‘0’),  (‘z’,  ‘0’)   Intron:  por4on  of  the  chromosome  that  does  not  contribute  to   the  phenotype  (i.e.,  the  fitness  value)   E.g.:  find  input  that  crashes    myFunc(char* s){…} Chromosomes  =  (‘a’,  ‘b’,  ‘c’,  ‘0’,  ‘t’),  (‘z’,  ‘0’,  ‘a’,  ‘g’,  ‘h’)   17
  • 18.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Selec6on   Pop  =  select(Pop)   •  Trunca6on  selec6on:  the  best  M  individuals  are  selected;  to   obtain  the  number  of  individuals  required  to  fill-­‐in  Pop,  the   selected  individuals  are  replicated  mul4ple  4mes.   •  RouleQe  wheel  selec6on:  the  probability  of  selec4ng  an   individual  is  propor4onal  to  its  fitness;  individuals  can  be   selected  mul4ple  4mes.   •  Tournament  selec6on:  T  individuals  (e.g.,  2)  are  randomly   selected  and  compared  (according  to  their  fitness);  the  winner   is  inserted  into  the  new  popula4on.   •  Ordered  selec6on:  the  probability  of  selec4ng  an  individual  is   propor4onal  to  its  posi4on  in  the  list  of  all  individuals  ranked  by   increasing  fitness.   •  Eli6sm:  at  least  one  copy  of  the  best  individual(s)  is  propagated   to  the  next  genera4on.   18
  • 19.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Reproduc6on   Pop  =  reproduce(Pop)   Muta6on:  y’i  =  µμ(yi)   y1   y2   y3   y4   y5   y6   …   yk   y1   y2   y3   y4   y’5   y6   …   yk   Crossover:  (y’i,  y’j)  =  yi  ⊗  yj   y1   y2   y3   y4   y5   y6   …   yk   z1   z2   z3   z4   z5   z6   …   zk   Single  point  crossover   y1   y2   y3   z4   z5   z6   …   zk   z1   z2   z3   y4   y5   y6   …   yk   E.g.,  P  =  80%   E.g.,  P  =  10%   19
  • 20.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Gene6c  algorithms   The  parameters  of  these  algorithms  (popula4on  size,  kind  of   selec4on,  crossover  and  muta4on  rates,  etc.)  and  the  design   of  the  muta4on/crossover  operators  determine  the  trade-­‐off   between  explora4on  and  exploita4on.  E.g.:   •  Neighborhood  muta4on  /  two-­‐parent  crossover  /  low   muta4on  and  crossover  rate  /  eli4sm  =>  local  search.   •  Disrup4ve  muta4on  /  mul4-­‐parent  crossover  /  high   muta4on  and  crossover  rate  /  ordered  selec4on  =>  global   search.   20
  • 21.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Ant  colony  op6miza6on   1.  Pop = {x0, …, xN} // random 2.  x_opt = NULL 3.  LOOP // with search budget m 4.  FOR x IN Pop 5.  FOR y IN Neighbors(x) 6.  computeProb(P(x, y)) 7.  END FOR 8.  x = move(x, P(x, *)) 9.  END FOR 10.  FOR EACH move (x, y) 11.  depositPheromone(x, y) 12.  END FOR 13.  updatePheromone() 14.  x_opt = best(Pop, x_opt) 15.  END LOOP 16.  RETURN x_opt S6gmery  principle:  coordina4on  between  agents   is  achieved  through  traces  lem  in  the  environment.   21
  • 22.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Pheromone  calcula6on   computeProb(P(x,  y))   depositPheromone(x,  y)   updatePheromone()   τxy  :  amount  of  pheromone  on   (x,  y)   ηxy  :  desirability  of  move  (x,  y)   (e.g.,  f(y))   ρ  :  pheromone  evapora4on   coefficient   22
  • 23.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Ant  colony  op6miza6on   The  decay  of  the  pheromone  (evapora4on)  determines  the   tradeoff  between  exploita4on  and  explora4on:   •  Pheromone  decays  slowly:  ants  remain  longer  in  the  same   region  (local  search).   •  Pheromone  decays  quickly:  ants  move  randomly  to  other   regions  (global  search)  amer  a  limited  amount  of  local   search.   23
  • 24.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Par6cle  swarm  op6miza6on   1.  Pop = {x0, …, xN} // random 2.  x_opt = NULL 3.  LOOP // with search budget m 4.  FOR x IN Pop 5.  IF f(x) > f(p[x]) 6.  p[x] = x 7.  END FOR 8.  IF f(x) > f(x_opt) 9.  x_opt = x 10.  END FOR 11.  v[x] = updateVelocity(x) 12.  x = updatePosition(x) 13.  END FOR 14.  END LOOP 15.  RETURN x_opt x_opt   p[x]   x   Par4cle  x  tends  to:   1.  proceed  along  current   direc4on;   2.  be  aVracted  by  the  best   posi4on  in  its  trajectory   3.  be  aVracted  by  the   globally  best  posi4on  p[x]:  best  posi4on  of  par4cle  x   v[x]:  speed  of  par4cle  x     x:  current  posi4on  of  par4cle  x   24
  • 25.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Par6cle  dynamics   updateVelocity(x)   updatePosi6on(x)   25
  • 26.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Par6cle  Swarm  Op6miza6on   Variants   •  Neighborhood  communica6on:  if  any  posi4on  in  the   neighborhood  of  a  par4cle  has  been  visited  previously  by   other  par4cles,  the  best  posi4on  in  the  neighborhood   determines  a  fourth  direc4on  to  follow.   26
  • 27.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Par6cle  swarm  op6miza6on   The  parameters  of  the  algorithm  determine  the  tradeoff   between  exploita4on  and  explora4on:   ω:  contributes  to  keeping  current  speed  and  direc4on,   possibly  moving  towards  unexplored  regions.   ϕp:  directs  the  search  toward  the  best  solu4on  found  by  the   current  par4cle,  usually  increasing  the  amount  of  local   search.   ϕg:  directs  the  search  toward  the  globally  best  solu4on,   which  may  increase  the  local  search  in  the  region  around   xopt.   Neighborhood  communica4on  increases  the  amount  of  local   search.   27
  • 28.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Automated  test  case  genera4on   Phil  McMinn,  Search-­‐based  soDware  test  data  genera(on:  a  survey.   Journal  of  Somware  Tes4ng,  Verifica4on  and  Reliability,  vol.  14,  n.  2,   pp.  105-­‐156,  June  2004.   28
  • 29.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle(){…} void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) { int a = Integer.parseInt(args[0]); int b = Integer.parseInt(args[1]); int c = Integer.parseInt(args[2]); Triangle t = new Triangle(a, b, c); if (t.isTriangle()) t.computeTriangleType(); System.out.println(typeToString(t.type)); } } 29
  • 30.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() {…} boolean isTriangle() { if (a <= 0 || b <= 0 || c <= 0) return false; if (a + b <= c || a + c <= b || b + c <= a) return false; return true; } public static void main(String args[]) {…} } 30
  • 31.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() { if (a == b) if (b == c) type = EQUILATERAL; else type = ISOSCELE; else if (a == c) type = ISOSCELE; else if (b == c) type = ISOSCELE; else checkRightAngle(); } boolean isTriangle() {…} public static void main(String args[]) {…} } 31
  • 32.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() { if (a*a + b*b == c*c) type = RIGHT_ANGLE; else if (b*b + c*c == a*a) type = RIGHT_ANGLE; else if (a*a + c*c == b*b) type = RIGHT_ANGLE; else type = SCALENE; } void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) {…} } 32
  • 33.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Coverage  tes6ng   void f(int a, int b) { 1 if (a < 0) 2 print(“a is negative”); 3 if (b < 0) 4 print(“b is negative”); 5 return; } 1   2   3   4   5   Statement  coverage   Branch  coverage   Path  coverage   Coverage  targets   {<1,  3,  5>,  <1,  2,  3,  5>,   <1,  3,  4,  5>,  <1,  2,  3,  4,  5>}   {<1,  2>,  <2,  3>,  <1,  3>,   <3,  4>,  <4,  5>,  <3,  5>}   {1,  2,  3,  4,  5}   Test  cases   f(-­‐1,  -­‐1),  f(1,  -­‐1)   f(-­‐1,  1),  f(1,  1)   f(-­‐1,  -­‐1),  f(1,  1)   f(1,  1)   33
  • 34.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Running  example   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle(){…} void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]){…} } Goal:   Automa4c   genera4on  of  test   cases  using  gene4c   algorithms  so  as  to   achieve  statement   coverage.   Initialization   Fitness   assignment   Selection  Reproduction   Key  decisions:   1.  Representa4on   2.  Fitness  func4on   3.  Selec4on   4.  Reproduc4on  (crossover   and  muta4on)   34
  • 35.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Representa6on   Fixed  length  chromosome   a   b   c   The  chromosome  used  for  test  case  genera4on  is  the  input  vector,   (sequence  of  input  values  used  by  the  test  case  running  the   program)  which  may  be  fixed  length  or  variable  length.   Example  of  (randomly  generated)  ini4al  popula4on:   Pop  =  {(2,  2,  -­‐2),  (2,  1,  1),  (2,  1,  2),  (3,  4,  2),  (4,  3,  3),  (4,  -­‐5,  6),  (3,  5,   2),  (-­‐3,  0,  -­‐2)}   E.g.,  in  our  running  example:   35
  • 36.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  func6on   For  statement  and  branch  coverage,  given  a  specific  coverage   target  t,  a  widely  used  fitness  func4on  (to  be  minimized)  is:   f(x)  =  approach_level(P(x),  t)  +  branch_distance(P(x),  t)   approach_level(P(x),  t)  :   Given  the  execu4on  trace  obtained  by  running  program  P  with   input  vector  x,  the  approach  level  is  the  minimum  number  of   control  nodes  between  an  executed  statement  and  the  coverage   target  t.   branch_distance(P(x),  t)   Given  the  first  control  node  where  the  execu4on  diverges  from  the   target  t,  the  predicate  at  such  node  is  converted  to  a  distance   (from  taking  the  desired  branch),  normalized  between  0  and  1.   36
  • 37.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Approach  level   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } 1   2   3   4   5   6   7   8   9   10   1   2   3   4   5   6   7   8   9   Control  flow   graph   Control   dependency   graph   37
  • 38.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Approach  level   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } 1   2   3   4   5   6   7   8   9   10   1   2   3   4   5   6   7   8   9   Control  flow   graph   Control   dependency   graph   Ch1  =  (2,  2,  2)    P(Ch1)  =  <1,  2,  3,  10>    AL  =  2   Ch2  =  (2,  3,  4)    P(Ch2)  =  <1,  5,  7,  9,  10>  AL  =  0   38
  • 39.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Branch  distance   The  predicate  of  the  control  node  which  is  closest  to  the  target  is   converted  to  a  distance,  which  measures  how  far  the  test  case  is   from  taking  the  desired  branch.  For  boolean/numeric  variables  a,  b:   Condi6on  c  =  atomic  predicate   Distance  BD(c)  =  d  /  (d  +  1)   a   d  =  {0  if  a  ==  true;  K  otherwise}   !a   d  =  {K  if  a  ==  true;  0  otherwise}   a  ==  b   d  =  {0  if  a  ==  b;  abs(a  -­‐  b)  +  K   otherwise}   a  !=  b   d  =  {0  if  a  !=  b;  K  otherwise}   a  <  b   d  =  {0  if  a  <  b;  a  -­‐  b  +  K  otherwise}   a  <=  b   d  =  {0  if  a  <=  b;  a  -­‐  b  +  K  otherwise}   a  >  b   d  =  {0  if  a  >  b;  b  -­‐  a  +  K  otherwise}   a  >=  b   d  =  {0  if  a  >=  b;  b  -­‐  a  +  K  otherwise}   39
  • 40.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Branch  distance   For  string  variables  a,  b:   Condi6on  c  =  atomic  predicate   Distance  BD(c)  =  d  /  (d  +  1)   a  ==  b   d  =  {0  if  a  ==  b;  edit_dist(a,  b)  +  K   otherwise}   a  !=  b   d  =  {0  if  a  !=  b;  K  otherwise}   a  <  b   d  =  {0  if  a  <  b;  a[j]  -­‐  b[j]  +  K  otherwise}   a  <=  b   d  =  {0  if  a  <=  b;  a[j]  -­‐  b[j]  +  K  otherwise}   a  >  b   d  =  {0  if  a  >  b;  b[j]  -­‐  a[j]  +  K  otherwise}   a  >=  b   d  =  {0  if  a  >=  b;  b[j]  -­‐  a[j]  +  K  otherwise}   where  j  is  the  posi4on  of  the  first  different  character:  a[j]  !=  b[j],   while  a[i]  ==  b[i]  for  i  <  j  (a[j]  –b[j]  is  set  to  zero  if  a  ==  b).   Example  of  edit  distance:  edit_dist(“strqqvt”,  “trwwv”)  =  6   40
  • 41.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Branch  distance   Condi6on  c  =  composite  predicate   Distance  BD(c)  =  d  /  (d  +  1)   !p   Nega4on  is  propagated  inside  p   p  &  q   d  =  d(p)  +  d(q)   p  |  q   d  =  min(d(p),  d(q))   p  XOR  q  =  p  &  !q  |  !p  &  q   d  =  min(d(p)+d(!q),  d(!p)+d(q))   41
  • 42.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Branch  distance   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Ch1  =  (2,  2,  2)    P(Ch1)  =  <1,  2,  3,  10>    AL  =  2  f  =  2.5   Ch2  =  (2,  3,  4)    P(Ch2)  =  <1,  5,  7,  9,  10>  AL  =  0   1   2   3   4   5   6   7   8   9   d(a  !=  b)  =  K  =  1   BD(a  !=  b)  =  1  /  (1  +  1)  =  0.5   f(Ch1)  =  2  +  0.5  =  2.5   a  ==  b  a  !=  b   42
  • 43.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Branch  distance   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Ch1  =  (2,  2,  2)    P(Ch1)  =  <1,  2,  3,  10>    AL  =  2  f  =  2.5   Ch2  =  (2,  3,  4)    P(Ch2)  =  <1,  5,  7,  9,  10>  AL  =  0  f  =  0.66   1   2   3   4   5   6   7   8   9   d(b  ==  c)  =  abs(b  -­‐  c)  +  K  =  2   BD(b  ==  c)  =  2  /  (2  +  1)  =  0.66   f(Ch2)  =  0  +  0.66  =  0.66   b  ==  c   b  !=  c   43
  • 44.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Selec6on   Ch1  =  (2,  2,  2)  f  =  2.5   Ch2  =  (2,  3,  4)  f  =  0.66   Ch3  =  (-­‐2,  3,  6)  f  =  ∞   Ch4  =  (2,  3,  7)  f  =  ∞   Ch5  =  (2,  2,  3)  f  =  2.5   Ch6  =  (3,  4,  5)  f  =  0.66   Ch7  =  (3,  5,  7)  f  =  0.75   Ch8  =  (6,  8,  4)  f  =  0.83   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } 44
  • 45.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Selec6on   Ch6  =  (3,  4,  5)  f  =  0.66   Ch2  =  (2,  3,  4)  f  =  0.66   Ch7  =  (3,  5,  7)  f  =  0.75   Ch8  =  (6,  8,  4)  f  =  0.83   Ch1  =  (2,  2,  2)  f  =  2.5   Ch5  =  (2,  2,  3)  f  =  2.5   Ch3  =  (-­‐2,  3,  6)  f  =  ∞   Ch4  =  (2,  3,  7)  f  =  ∞   Trunca6on  selec6on   Ch6  =  (3,  4,  5)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch8  =  (6,  8,  4)     Ch6  =  (3,  4,  5)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch8  =  (6,  8,  4)     45
  • 46.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Selec6on   Ch6  =  (3,  4,  5)  P  ≈  1/f  =  0.23   Ch2  =  (2,  3,  4)  P  ≈  1/f  =  0.23   Ch7  =  (3,  5,  7)  P  ≈  1/f  =  0.20   Ch8  =  (6,  8,  4)  P  ≈  1/f  =  0.18   Ch1  =  (2,  2,  2)  P  ≈  1/f  =  0.06   Ch5  =  (2,  2,  3)  P  ≈  1/f  =  0.06   Ch3  =  (-­‐2,  3,  6)  P  ≈  1/f  =  0   Ch4  =  (2,  3,  7)  P  ≈  1/f  =  0   RouleQe  wheel  selec6on   Ch6  =  (3,  4,  5)     Ch2  =  (2,  3,  4)     Ch6  =  (3,  4,  5)     Ch1  =  (2,  2,  2)     Ch8  =  (6,  8,  4)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch6  =  (3,  4,  5)     46
  • 47.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Selec6on   Tournament  selec6on   Ch2  =  (2,  3,  4)       Ch5  =  (2,  2,  3)       Ch8  =  (6,  8,  4)   Ch2  =  (2,  3,  4)       Ch6  =  (3,  4,  5)   Ch6  =  (3,  4,  5)   Ch8  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   Ch1  =  (2,  2,  2)  f  =  2.5   Ch2  =  (2,  3,  4)  f  =  0.66   Ch3  =  (-­‐2,  3,  6)  f  =  ∞   Ch4  =  (2,  3,  7)  f  =  ∞   Ch5  =  (2,  2,  3)  f  =  2.5   Ch6  =  (3,  4,  5)  f  =  0.66   Ch7  =  (3,  5,  7)  f  =  0.75   Ch8  =  (6,  8,  4)  f  =  0.83   <Ch2,  Ch7>   <Ch1,  Ch5>   <Ch3,  Ch8>   <Ch3,  Ch2>   <Ch6,  Ch5>   <Ch6,  Ch4>   <Ch8,  Ch3>   <Ch8,  Ch1>   47
  • 48.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Selec6on   1  Ch4  =  (2,  3,  7)  f  =  ∞    P  =  1/33  =  0.03     1  Ch3  =  (-­‐2,  3,  6)  f  =  ∞    P  =  1/33  =  0.03   3  Ch5  =  (2,  2,  3)  f  =  2.5    P  =  3/33  =  0.09   3  Ch1  =  (2,  2,  2)  f  =  2.5    P  =  3/33  =  0.09   5  Ch8  =  (6,  8,  4)  f  =  0.83  P  =  5/33  =  0.15   6  Ch7  =  (3,  5,  7)  f  =  0.75  P  =  6/33  =  0.18   7  Ch6  =  (3,  4,  5)  f  =  0.66  P  =  7/33  =  0.21   7  Ch2  =  (2,  3,  4)  f  =  0.66  P  =  7/33  =  0.21   Ordered  selec6on   Ch2  =  (2,  3,  4)     Ch2  =  (2,  3,  4)     Ch6  =  (3,  4,  5)     Ch4  =  (2,  3,  7)     Ch8  =  (6,  8,  4)     Ch2  =  (2,  3,  4)     Ch7  =  (3,  5,  7)     Ch6  =  (3,  4,  5)     Rank  sum  =  33     48
  • 49.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Reproduc6on:  crossover   Ch1  =  (2,  3,  4)       Ch2  =  (2,  2,  3)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  4)       Ch5  =  (3,  4,  5)   Ch6  =  (3,  4,  5)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   One-­‐point  crossover,  with  P  =  0.8   -­‐   <Ch2,  Ch5>   <Ch3,  Ch8>   <Ch4,  Ch6>   SEL   SEL   -­‐   SEL   Ch1  =  (2,  3,  4)       Ch2  =  (2,  4,  5)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  5)       Ch5  =  (3,  2,  3)   Ch6  =  (3,  4,  4)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   49
  • 50.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Reproduc6on:  muta6on   Muta4on  probability:  P  =  0.2   Muta4on  operators:  increment/decrement  (P  =  50%)   Ch1  =  (2,  3,  4)       Ch2  =  (2,  4,  5)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  5)       Ch5  =  (3,  2,  3)   Ch6  =  (3,  4,  4)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   Ch1  =  (2,  3,  4)       Ch2  =  (2,  5,  5)       Ch3  =  (6,  8,  4)   Ch4  =  (2,  3,  5)       Ch5  =  (2,  2,  3)   Ch6  =  (3,  4,  4)   Ch7  =  (6,  8,  4)       Ch8  =  (6,  8,  4)   50
  • 51.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Itera6ng   Initialization   Fitness   assignment   Selection  Reproduction   Un6l  search  budget  (number   of  fitness  evalua4ons)  is  over   or  target  is  covered.   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Then,  select  the  next  target  (yet  to   cover  statement)  and  repeat  the   test  case  genera4on  cycle.   51
  • 52.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Collateral  coverage   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } When  a  target  is  covered   accidentally  (e.g.,  (2,  2,  3))  in  a  test   case  genera4on  cycle,  it  is   removed  from  the  list  of  yet  to   cover  targets  and  the  related  test   case  is  stored,  to  be  included  in  the   final  test  suite.   Andrea  Arcuri,  Muhammad  Zohaib  Z.  Iqbal,  Lionel  C.  Briand:  Formal  analysis  of  the   effec(veness  and  predictability  of  random  tes(ng.  ISSTA,  pp.  219-­‐230,  2010.   Collateral  (serendipitous)  coverage:  if  test  cases  involved  in  collateral   coverage  are  not  detected  and  stored,  it  can  be  shown  that  random   tes4ng  performs  asympto4cally  beVer  than  search  based  tes4ng.   52
  • 53.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Final  test  suite   class Triangle { void computeTriangleType() { 1 if (a == b) 2 if (b == c) 3 type = EQUILATERAL; 4 else type = ISOSCELE; 5 else if (a == c) 6 type = ISOSCELE; 7 else if (b == c) 8 type = ISOSCELE; 9 else checkRightAngle(); 10 return; } } Ch1  =  (2,  2,  3)   Ch2  =  (2,  5,  5)   Ch3  =  (2,  2,  2)   Ch4  =  (4,  3,  4)   Ch5  =  (3,  4,  5)   The  final  test  suite  consists  of  all   chromosomes  that  have  been   found  to  cover  (even   accidentally)  one  or  more  yet  to   cover  statements.   The  final  test  suite  might  contain  redundancies  that  can  be   eliminated  in  a  post-­‐processing  (by  keeping  only  the  test  cases  that   cover  at  least  one  target  uniquely).   53
  • 54.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Object  oriented  tes4ng   Paolo  Tonella,  Evolu(onary  tes(ng  of  classes.  Proc.  of  the  Interna4onal  Symposium   on  Somware  Tes4ng  and  Analysis  (ISSTA),  pp.  119-­‐128,  Boston,  USA,  July  2004.   Gordon  Fraser,  Andrea  Arcuri,  Whole  test  suite  genera(on.  IEEE  Transac4ons  on   Somware  Engineering,  vol.  38,  n.  2,  pp.  276-­‐291,  2013.   54
  • 55.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Unit  tes6ng  of  classes   1.  An  object  of  the  class  under  test  is  created  using  one  of   the  available  constructors.   2.  A  sequence  of  zero  or  more  methods  is  invoked  on  it.   3.  The  method  currently  under  test  is  executed.   4.  The  final  state  of  the  object  is  examined  to  produce  the   pass/fail  result.   Drivers/stubs  are  created  whenever  necessary.   Steps  1,  2  are  repeated  for  each  parameter  of  object  type.   55
  • 56.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  of  test  case   1 A a = new A(); 2 B b = new B(); 3 b.f(2); 4 a.m(5, b); •  Object  b  is  created  because  it  is  required  to  call  m. •  Its  state  is  changed  by  calling  f  on  it.   •  Sequence  of  required  input  values:  <2,  5>.   Let  us  assume  that  m  is  the  method  under  test.   56
  • 57.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Features  of  object-­‐oriented     unit  tes6ng   •  The  number  and  order  of  method  invoca4ons  is  variable.   •  The  number  of  input  values  is  also  variable.   •  Some  parameters  in  method  calls  are  objects   themselves,  thus  requiring  further  object  construc4ons.   •  The  state  of  the  object  under  test  and  of  the  object   parameters  affects  the  result.   Chromosomes  are  not  just  sequences  of  input  values   57
  • 58.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Chromosomes   Procedural  code:   Object-­‐Oriented  code:   (v1,  v2,  …,  vN)   $x0=A():  $x0.f():  $x0.g()  @  v1,  v2,  …,  vN   class TestA extends TestCase { public void testCase1() { A a = new A(-1); B b = new B(); b.f(2); a.m(5, b); } } $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   chromosome   variables   input   values   58
  • 59.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Random  chromosome   construc6on   1.  A  constructor  for  the  object  under  test  is  randomly  selected:   $a=A(int)@-1 2.  The  invoca4on  of  the  method  under  test  is  appended:   $a=A(int):$a.m(int,$b)@-1,5 3.  All  required  object  construc4ons  are  inserted:  $a=A(int): $b=B():$a.m(int,$b)@-1,5 4.  Method  invoca4ons  to  change  the  state  of  the  created  objects   are  randomly  inserted:  $a=A(int):$b=B():$b.f(int): $a.m(int,$b)@-1,2,5 Steps  3  and  4  are  repeated  un4l  all  chromosome  variables   used  as  method  or  constructor  parameters  are  properly   ini4alized  (well-­‐formedness  of  the  resul4ng  chromosome).   59
  • 60.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Random  input  data  genera6on   A.m(int) Default  integer  generator:  uniform  selec4on  in  [0,  100]   A.m(int[-2;2]) Parameterized  integer  generator:  uniform  selec4on  in  [-­‐2,  2]   A.m(int[MyIntGenerator]) Customized  integer  generator:  method  newIntValue()   from  class  MyIntGenerator  is  called  to  obtain  the  value   A.m(boolean) Default  boolean  generator:  true  and  false  are  equally  likely   A.m(String) Default  string  generator:  characters  are  uniformly  chosen  from   [a-­‐zA-­‐Z0-­‐9],  with  the  string  length  decaying  exponen4ally   A.m(String[DateGenerator]) Customized  string  generator:  only  strings   represen4ng  legal  dates  are  produced  (e.g.,   “3/3/2003”)   Default,  parameterized,  pooled  and  customized  input  generators:   A.m(int<pool>) Randomly  selected  from  the  program’s  pool  of  integer  constants   A.m(String<pool>) Randomly  selected  from  the  program’s  pool  of  string  constants   60
  • 61.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Muta6on  operators   Change  input  value:   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  4,  5   61
  • 62.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Muta6on  operators   Change  constructor:   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   $a=A():$b=B():$b.f(int):$a.m(int,  $b)  @  2,  5   62
  • 63.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Muta6on  operators   Insert/remove  method  call:   $a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5   $a=A(int):$b=B():$a.m(int,  $b)  @  -­‐1,  5   63
  • 64.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Crossover   $a=A():$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2   $a=A(int):$b=B():$b.g():$b.f(int):$a.m(int,  $b)  @  0,  -­‐1,  2   $a=A(int):$b=B():$b.g():$a.m(int,  $b)  @  0,  -­‐3   $a=A():$b=B():$a.m(int,  $b)  @  -­‐3   $a=A(int,  int):$b=B():$a.m(int,  $b)  @  0,  3,  6   $a=A():$b=B(int):$a.m(int,  $b)  @  1,  6   $a=A():$b=B(int):$c=C(int):$b.h($c):$b.f():$a.m(int,  $b)  @  1,  4,  5   $a=A(int,  int):$b=B():$c=C():$b.h($c):$b.f():$a.m(int,  $b)  @  0,  3,  5   Well-­‐formedness   must  be  maintained   The  alterna4ve   would  be  to  use  null   64
  • 65.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness   Initialization   Fitness   assignment   Selection  Reproduction   f(x)  =  approach_level(P(x),  t)  +  branch_distance(P(x),  t)   •  Trunca6on  selec6on   •  RouleQe  wheel  selec6on   •  Tournament  selec6on   •  Ordered  selec6on   •  Eli6sm   65
  • 66.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Test  case  generator   $x0=BinaryTree():  $x1=Integer(int):  $x2=BinaryTreeNode($x1):   $x0.insert($x2):  $x3=Integer(int):  $x0.search($x3)  @  5,  5   class TestBinaryTree extends TestCase { … public void testCase2() { BinaryTree x0 = new BinaryTree(); Integer x1 = new Integer(5); BinaryTreeNode x2 = new BinaryTreeNode(x1); x0.insert(x2); Integer x3 = new Integer(5); assertTrue(x0.search(x3)); } } Asser6on  added  manually   66
  • 67.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Asser6ons   Asser4ons  are  inserted  manually  as  follows:   –  The  value  returned  from  each  method  call  is  compared  with  the   expected  one.   –  If  a  method  call  can  raise  an  excep6on,  an  asser4on  on  its   actual  occurrence  is  added.   –  At  the  end  of  each  test  case,  the  final  state  of  the  object  under   test  is  compared  with  the  expected  one.   67
  • 68.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Asser6ons   Asser4ons  are  added  at  method  calls  to  check  the  actually   returned  values  vs.  the  expected  ones.   class TestLinkedList extends TestCase { public void testCase1() { LinkedList x0 = new LinkedList(); … Integer x1 = null; x0.addFirst(x1); Integer x2 = null; assertTrue(x0.remove(x2)); … } } 68
  • 69.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Asser6ons   Asser4ons  are  added  at  method  calls  possibly  raising   excep6ons.   class TestLinkedList extends TestCase { public void testCase2() { LinkedList x0 = new LinkedList(); … try { x0.listIterator(78); fail(); } catch(IndexOutOfBoundException e) {} } 69
  • 70.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Asser6ons   class TestLinkedList extends TestCase { public void testCase3() { LinkedList x0 = new LinkedList(); … Integer x1 = new Integer(2); x0.addLast(x1); assertTrue(x0.toString().equals(“[2]”)); } } Asser4ons  are  added  at  to  check  the  final  object  state.     70
  • 71.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Whole  test  suite  genera6on   •  Some  coverage  targets  may  be  infeasible.   •  Some  coverage  targets  may  be  very  difficult  to  achieve.   Since  a  limited  search  budget  is  available  for  test  case  genera4on:   •  Infeasible  targets  may  use  the  en4re  search  budget  without   achieving  any  target.   •  Difficult  targets  may  use  most  of  the  search  budget,  leaving  lots   of  easier  coverage  targets  uncovered.   •  The  order  in  which  targets  are  considered  affects  the  final   result.   Whole  test  suite  genera4on  addresses  all  these  problems:   •  A  set  of  test  cases  is  generated  to  cover  all  targets  at  the  same  4me.   •  The  result  is  independent  of  the  order  of  the  targets  and  the  presence   of  infeasible  or  difficult  targets.   71
  • 72.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example   class Stack { int[] values = new int[INIT_SIZE]; int size = 0; void push(int x) { if (size >= values.length) resize(); if (size < values.length) values[size++] = x; } int pop() {…} private void resize() {…} } Gordon  Fraser,  Andrea  Arcuri,  Evolu(onary  genera(on  of  whole  test  suites.  Proc.  of  the  Int.   Conference  on  the  Quality  of  Somware  (QSIC),  pp.  31-­‐40,  Madrid,  Spain,  July  2011.   Difficult  to  cover   Else  branch  is   infeasible   Whole  test  suite  genera4on   72
  • 73.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Chromosome   An  individual  is  an  en4re  test  suite,  i.e.  a  set  of  test  cases:   Ch1  =  {TC1,  TC2,  …}  =      {<$a=A(int):$b=B():$b.f(int):$a.m(int,  $b)  @  -­‐1,  2,  5>,      <$a=A(int):$b=B():$a.m(int,  $b)  @  3,  7>,  …}   Ch1:    cov  =  30   Ch2:    cov  =  15   Ch3:    cov  =  17   Ch4:    cov  =  34   Ch5:    cov  =  21   Ch6:    cov  =  11   Ch7:    cov  =  13   Ch8:    cov  =  45   Intui4vely,  individuals  (test   suites)  with  higher  coverage  are   beVer,  so  fitness  must  be  based   on  coverage.   73
  • 74.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness   Ties  are  resolved  by  rewarding  the  chromosomes  represen4ng   smaller  test  suites  (i.e.,  test  suites  with  less  statements).   74
  • 75.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Muta6on   In  addi4on  to  the  muta4on  operators  for  the  individual  test  cases:   •   Remove  method/constructor  call;   •   Insert  method/constructor  call;   •   Change  input  value.   Muta4on  operators  that  work  on  whole  test  suites  include:   •   Add  test  case:  Ch  =  {TC1,  …}  →  Ch’  =  {TC_new,  TC1,  …}   •   Remove  test  case:  Ch  =  {TC_k,  TC1,  …}  →  Ch’  =  {TC1,  …}  if  TC_k   becomes  empty  amer  (repeatedly)  applying  remove  method/ constructor  call  to  it.   75
  • 76.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Crossover   Each  test  suite  is  par44oned  into  two  non  empty  subsets,  which  are   exchanged  between  the  two  individuals.   Ch1  =  {TC1,  TC2,  TC3,  TC4}    Ch2  =  {TC5,  TC6,  TC7}     Ch1’  =  {TC1,  TC2,  TC7}    Ch2’  =  {TC5,  TC6,  TC3,  TC4}   76
  • 77.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Tool   EvoSuite:  hVp://www.evosuite.org/   by  Gordon  Fraser  &  Andrea  Arcuri.   •  Implements  the  gene4c  algorithm  for  whole  test  suite   genera4on   •  Instruments  the  Java  bytecode  under  test  for  coverage   analysis.   •  Uses  Java  reflec4on  to  execute  test  cases  encoded  as   chromosomes  and  to  determine  the  execu4on  traces.   •  Only  mutated  test  cases  need  to  be  re-­‐executed  from  one   itera4on  to  the  next  one.   •  Test  case  execu4on  is  4med  out  to  prevent  non-­‐termina4on.   •  Produces  a  JUnit  test  class  as  output.   77
  • 78.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Future  internet  tes4ng   78
  • 79.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) FITTEST EU  FP7  STREP  project  n.  257574   Sep  1,  2010  –  Dec  31,  2013   Par4cipant  organisa4on Country Universidad  Politécnica  de  Valencia  (coordinator) Spain University  College  London United  Kingdom Berner  &  MaVner  Systemtechnik Germany IBM Israel Fondazione  Bruno  Kessler Italy Utrecht  University The  Netherlands SOFTeam France Sulake Finland Goal:  automated,  con4nuous  tes4ng  of  Future   Internet  applica4ons   79
  • 80.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Future internet •  Highly  interac4ve   •  Communica4ng   asynchronously  with   server   •  Based  on  Web  2.0   technologies  (e.g.,  Ajax,   Flash/Ac4onScript)   •  Composed  and  updated   dynamically   80
  • 81.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) FI testing challenges §  CH1  Self  modifica4on     §  Rich  clients  have  increased  capability  to  dynamically   adapt  the  structure  of  the  Web  pages;  server-­‐side   services  are  replaced  and  recomposed  dynamically  based   on  Service  Level  Agreements  (SLA)  and  newly  discovered   services;  components  are  dynamically  loaded   §  CH2  Autonomic  behaviour   §  FI  applica4ons  are  highly  autonomous;  their  correct   behaviour  cannot  be  specified  precisely  at  design-­‐4me.   §  CH3  Low  observability   §   FI  applica4ons  are  composed  of  an  increasing  number  of   3rd-­‐party  components  and  services,accessed  as  a  black   box,  which  are  hard  to  test.   81
  • 82.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) §  CH4  Asynchronous  interac4ons   §  FI  applica4ons  are  highly  asynchronous  and  hence  hard  to  test.  Each  client   submits  mul4ple  requests  asynchronously;  mul4ple  clients  run  in  parallel;   server-­‐side  computa4on  is  distributed  over  the  network  and  concurrent.   §  CH5  Time  and  load  dependent  behaviour   §  For  FI  applica4ons,  factors  like  4ming  and  load  condi4ons  make  it  hard  to   reproduce  errors  during  debugging.   §  CH6  Huge  feature  configura4on  space   §  FI  applica4ons  are  highly  customisable  and  self-­‐configuring,  and  contain  a   huge  number  of  configurable  features,  such  as  user-­‐,  context-­‐,  and   environment-­‐dependent  parameters.   §  CH7  Ultra-­‐large  scale     §  FI  applica4ons  are  omen  systems  of  systems;  tradi4onal  tes4ng  adequacy   criteria  cannot  be  applied,  since  even  in  good  tes4ng  situa4ons  low   coverage  will  be  achieved.   FI testing challenges 82
  • 83.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) FI testing approach Automated  Con4nous  Evolu4onary  Tes4ng   Evolving testware (models, testcases, oracles) FITTEST FI applications, dynamically composed of 3rd party components and services Self-modifying Changed requirements Autonomic 83
  • 84.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Global picture 84
  • 85.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Model-based FI testing 85
  • 86.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) FI example: Cart public class Cart { public Cart() { ... } public void add(int c) { ... } public void rem(int c) { ... } public void pay() { ... } public int n() { ... } } 86
  • 87.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Inferred model 87
  • 88.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Architecture Branch  coverage  of  the  FSM  monitor  is  equivalent  to   transi4on  coverage  of  the  model   88
  • 89.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) FSM monitor as a “man in the middle” 89
  • 90.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Monitor generation through testability transformation void rem(int x) { sut.rem(x); n = sut.n(); switch (s_state) { case 2: if ( n > 1 ) { s_state = 2; } else if ( n == 1 ) { s_state = 1; } else if ( n == 0 ) { s_state = 0; } else { throw new UnderSpecEx(...); } break; case 1: ... } Branch  condi4ons  become   branch  distances  in  the   fitness  func4on  of  the   search  based  TC  generator   90
  • 91.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Underspecified model 91
  • 92.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) From Branch to Transition Coverage §  Theorem:  given  the  FSM  model,  we  can  apply  a   testability  transforma4on  to  the  model,  which  builds  a   class  FSM_monitor  such  that,  if  the  SUT  is  compliant   with  the  FSM,  then:   All-­‐transi4on  coverage  on  the  FSM  is  sa4sfied   iff   All  (non-­‐error)  branch  coverage  on  the   FSM_monitor  is  sa4sfied   92
  • 93.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Implementation •  Existing search-based test case generators (e.g., EvoSuite) can be used •  FSM_monitor generator: – java – template based (Velocity) •  highly configurable •  different implementation styles can be supported •  graphical representations can be produced, with proper templates 93
  • 94.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Test Suite//Test case number: 0 testCart0.add(1); testCart0.add(1); testCart0.add(1); //Test case number: 1 testCart0.add(2); testCart0.pay(); //Test case number: 2 testCart0.add(11); testCart0.rem(11); //Test case number: 3 testCart0.add(1); testCart0.rem(1); //Test case number: 4 testCart0.add(1734); testCart0.rem(1); //Test case number: 5 testCart0.add(2); testCart0.rem(1); //Test case number: 6 testCart0.add(1); testCart0.pay(); 94
  • 95.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) FI testing •  Models  are  inferred  con4nuously   •  Test  case  generators  cover  all  branches  of  the   FSM  monitor  è  all  model  transi4ons   •  Test  suites  are  updated  and  re-­‐executed   incrementally   The  testware  evolves  autonomously  with  the  FI   applica6on  under  test   95
  • 96.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Dynamic  symbolic  execu4on   Slides are partially based on the ESEC-FSE’05 presentation: http://srl.cs.berkeley.edu/~ksen 96
  • 97.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   Path  coverage:   1.  Select  a  path  p  to  be  covered.   2.  Determine  the  path  condi4on  (path  constraint),  by   propaga4on  of  symbolic  input  values  along  the  path   (expressions  are  evaluated  symbolically).  The  path  condi4on  is   a  boolean  expression  containing  only  input  variables.   3.  Solve  the  path  condi4on  using  an  SMT  solver.   97
  • 98.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() { if (a == b) if (b == c) type = EQUILATERAL; // target else type = ISOSCELE; else if (a == c) type = ISOSCELE; else if (b == c) type = ISOSCELE; else checkRightAngle(); } boolean isTriangle() {…} public static void main(String args[]) {…} } 98
  • 99.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle(){…} void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) { int a = Integer.parseInt(args[0]); int b = Integer.parseInt(args[1]); int c = Integer.parseInt(args[2]); Triangle t = new Triangle(a, b, c); if (t.isTriangle()) t.computeTriangleType(); System.out.println(typeToString(t.type)); } } 99
  • 100.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Sta6c  symbolic  execu6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() {…} void computeTriangleType() {…} boolean isTriangle() { if (a <= 0 || b <= 0 || c <= 0) return false; if (a + b <= c || a + c <= b || b + c <= a) return false; return true; } public static void main(String args[]) {…} } Path  condi6on  =  !  (a  <=  0  ||  b  <=  0  ||  c  <=  0)  &&  !(a  +  b  <=  c  ||  a   +  c  <=  b  ||  b  +  c  <=  a)  &&  (a  ==  b)  &&  (b  ==  c)   100
  • 101.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Yices  SMT  solver   Path  condi6on  =  !  (a  <=  0  ||  b  <=  0  ||  c  <=  0)  &&  !(a  +  b  <=  c  ||  a   +  c  <=  b  ||  b  +  c  <=  a)  &&  (a  ==  b)  &&  (b  ==  c)   =  (a  >  0)  &&  (b  >  0)  &&  (c  >  0)  &&  (a  +  b  >  c)  &&  (a  +  c  >  b)  &&  (b  +   c  >  a)  &&  (a  ==  b)  &&  (b  ==  c)   (define a::int) (define b::int) (define c::int) (assert (> a 0)) (assert (> b 0)) (assert (> c 0)) (assert (> (+ a b) c)) (assert (> (+ a c) b)) (assert (> (+ b c) a)) (assert (= a b)) (assert (= b c)) (check) sat (= a 1) (= b 1) (= c 1) 101
  • 102.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Problems  of  sta6c  symbolic   execu6on   In  prac4ce,  sta6c  symbolic  execu4on  has  several  limita4ons  that   make  it  inapplicable  for  test  case  genera4on  for  non  trivial   programs:   Ø Loops  must  be  unrolled  a  fixed  number  of  4mes.   Ø Sta4c  symbolic  execu4on  may  require  a  lot  of  (SMT-­‐solving)   4me  and  does  not  scale  to  large  programs.   Ø   SMT  solvers  usually  do  not  handle  non  linear  constraints.   Ø   SMT  solvers  cannot  deal  with  black  box  func4ons.   Dynamic  symbolic  execu6on:   Start  from  an  available  (random),  concrete  execu(on  and  use   symbolic  execu(on  to  explore  alterna(ve  paths.   102
  • 103.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) In  a  nutshell   Goal:  to  generate  data  inputs  that  exercise  all  feasible   execu(on  paths  up  to  a  given  length.   1. Execute a random test case. 2. Collect symbolic constraints along the concretely executed path. 3. Negate one branch condition in the path constraint. 4. Use constraint solvers to generate a new test input for the negated constraint. 5. Execute the new test case, covering a new path; iterate from 2. 103
  • 104.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) In  a  nutshell   Problem:  the  constraint  solver  may  not  be  powerful  enough   to  determine  concrete  values  that  sa(sfy  the  negated  path   constraint.   4. Use constraint solvers to generate a new test input for the negated constraint. 4’ If necessary, simplify the path constraint by replacing some of the symbolic values with concrete values. In  par4cular,  simplifica4on  may  be  necessary  whenever  non   linear  constraints  are  involved  or  black-­‐box,  external  func4ons   are  called.   104
  • 105.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) x = y + 2; Symb_exec(“x=y+2”); … if (x > z) { Symb_exec(“x>z”); 1.  Update  symbolic  state   2.  Accumulate  path  condi4on   Implementa6on   Instrumenta6on  can  be  used  to  perform  concrete  and  symbolic   execu4on  at  the  same  4me.   When  a  func4on  is  called  for  which  the  code  is  not  available  for  symbolic   execu4on,  the  outcome  of  its  concrete  execu4on  is  used  as  an   approxima4on  of  its  symbolic  execu4on.   1.  Update  symbolic  state   2.  Accumulate  path  condi4on   105
  • 106.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  1   int testme(int x) { 1 int y = x + 3; 2 if (y == 13) 3 target(); 4 return 0; } Execu4on  constraints  are  collected  dynamically  and  negated  to   traverse    alterna4ve  paths   1 2 3 4 <x  =  0>  è  <1,  2,  4>  è  (x0  +  3  ≠  13)   (x0  +  3  =  13)  è  <x  =  10>  è  <1,  2,  3>   concrete state symbolic state constraints x  =  0     y  =  3   x  =  x0   y  =  x0  +  3   (x0  +  3  ≠  13)   106
  • 107.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  ≠  ‘a’)  &  (x2  ≠  ‘d’)  &  (x3  =   ‘!’)  è  <x  =  “goo!”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “good”     n  =  0   x  =  {x0,  x1,  x2,   x3}   n  =  0     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  ≠  ‘d’)  &   (x3  ≠  ‘!’)  &   (0  <  4)   107
  • 108.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  ≠  ‘a’)  &  (x2  =  ‘d’)  &  (x3  =   ‘!’)  è  <x  =  “god!”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “goo!”     n  =  1   x  =  {x0,  x1,  x2,   x3}   n  =  1     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  ≠  ‘d’)  &   (x3  =  ‘!’)  &   (1  <  4)   108
  • 109.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  ≠  ‘a’)  &  (x2  =  ‘d’)  &  (x3  ≠   ‘!’)  è  <x  =  “godd”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “god!”     n  =  2   x  =  {x0,  x1,  x2,   x3}   n  =  2     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  =  ‘d’)  &   (x3  =  ‘!’)  &   (2  <  4)   109
  • 110.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  ≠  ‘b’)  &  (x1  =  ‘a’)  &  (x2  =  ‘d’)  &  (x3  ≠   ‘!’)  è  <x  =  “gadd”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “godd”     n  =  1   x  =  {x0,  x1,  x2,   x3}   n  =  1     (x0  ≠  ‘b’)  &   (x1  ≠  ‘a’)  &   (x2  =  ‘d’)  &   (x3  ≠  ‘!’)  &   (1  <  4)   110
  • 111.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  2   void testme(char x[4]) { 1 int n = 0; 2 if (x[0] == ‘b’) 3 n++; 4 if (x[1] == ‘a’) 5 n++; 6 if (x[0] == ‘d’) 7 n++; 8 if (x[0] == ‘!’) 9 n++; 10 if (n >= 4) 11 target(); } 1 2 3 4 5 6 7 8 9 10   11   (x0  =  ‘b’)  &  (x1  =  ‘a’)  &  (x2  =  ‘d’)  &  (x3  =   ‘!’)  è  <x  =  “bad!”>   Concrete   Execu4on   Symbolic   Execu4on   concrete state symbolic state constraints x  =  “badd”     n  =  3   x  =  {x0,  x1,  x2,   x3}   n  =  3     (x0  =  ‘b’)  &   (x1  =  ‘a’)  &   (x2  =  ‘d’)  &   (x3  ≠  ‘!’)  &   (3  <  4)  Target  reached   Amer  16   steps   111
  • 112.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  randomly   112
  • 113.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  randomly   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   •  take  then  branch  with  constraint   x0*x0*x0+  3*x0*x0+9  !=  y0   113
  • 114.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  randomly   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   •  take  then  branch  with  constraint   x0*x0*x0+  3*x0*x0+9  !=  y0   •  solve  x0*x0*x0+  3*x0*x0+9  =  y0  to   take  else  branch   •  Don’t  know  how  to  solve  !!   –  Stuck  ?   114
  • 115.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   115
  • 116.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   116
  • 117.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   •  take  then  branch  with  constraint     –  9  !=  y0   117
  • 118.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      target();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   •  take  then  branch  with  constraint     –  9  !=  y0   •  solve  9  =  y0    to  take  else  branch   •  execute  next  run  with  x  =  -­‐3  and   y=  9   –  Target  reached   118
  • 119.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Non linear constraints   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }         •  Let  ini4ally  x  =  -­‐3  and  y  =  7   generated  by  random  test-­‐driver   •  concrete  z  =  9   •  symbolic  z  =  x0*x0*x0  +  3*x0*x0+9   –  cannot  handle  symbolic  value   of  z   –  make  symbolic  z  =  9  and   proceed   •  take  then  branch  with  constraint     –  9  !=  y0   •  solve  9  =  y0    to  take  else  branch   •  execute  next  run  with  x  =  -­‐3  and   y=  9   –  Target  reached   Replace  symbolic   expression  with  concrete   value  when  symbolic   expression  becomes   unmanageable  (e.  g.  non-­‐ linear)   119
  • 120.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Black-box functions   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }         void  test_me(int  x,int  y){    z  =  black_box_fun(x);    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }     The  same  approach  is  applied  to  any  called  func4on  which  cannot  be   executed  or  modeled  symbolically.   120
  • 121.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Black-box functions   void  test_me(int  x,int  y){    z  =  x*x*x  +  3*x*x  +  9;    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }         void  test_me(int  x,int  y){    z  =  black_box_fun(x);    if(z  !=  y){      prin•(“Good  branch”);    }  else  {      prin•(“Bad  branch”);      abort();    }   }     The  same  approach  is  applied  to  any  called  func4on  which  cannot  be   executed  or  modeled  symbolically.   Use  the  concrete  value   returned  by  the  func4on   whenever  the  symbolic   expression  cannot  be   computed   121
  • 122.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Proper6es   Ø Incompleteness:  dynamic  symbolic  execu4on  is  complete  only   if  the  constraint  solver  can  successfully  solve  all  constraints   without  having  to  simplify  them.     Ø   Approxima6on:  an  input  sa4sfying  a  simplified  path  condi4on,   where  some  symbolic  values  are  replaced  by  concrete  ones,  is  not   ensured  to  execute  the  path  of  interest  (i.e.,  its  execu4on  may   diverge  from  the  expected  one).   Ø   Soundness:  dynamic  symbolic  execu4on  is  always  sound,  since   generated  test  cases  are  executable,  hence  revealed  faults  are   real  faults.   122
  • 123.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  of  approxima6on   class Triangle { int a, b, c; // sides int type = NOT_A_TRIANGLE; Triangle(int a, int b, int c) {…} void checkRightAngle() { if (a*a + b*b == c*c) type = RIGHT_ANGLE; else if (b*b + c*c == a*a) type = RIGHT_ANGLE; else if (a*a + c*c == b*b) type = RIGHT_ANGLE; else type = SCALENE; } void computeTriangleType() {…} boolean isTriangle() {…} public static void main(String args[]) {…} } Concrete  input  =  (2,  3,  4)  [SCALENE]   Path  condi6on  =  !  (a  <=  0  ||  b  <=  0  ||   c  <=  0)  &&  !(a  +  b  <=  c  ||  a  +  c  <=  b  ||   b  +  c  <=  a)  &&  !(a*a  +  b*b  ==  c*c)   &&  !(b*b  +  c*c  ==  a*a)  &&  !(a*a  +   c*c  ==  b*b)     123
  • 124.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Example  of  approxima6on   Concrete  input  =  (2,  3,  4)  [SCALENE]   SMT  solver:     (a  >  0  &&  b  >  0  &&  c  >  0)  &&  (a  +  b  >  c  &&  a  +  c  >  b  &&  b  +  c  >  a)  &&   (a*a  +  b*b  !=  c*c)  &&  (b*b  +  c*c  !=  a*a)  &&     (a*a  +  c*c  ==  b*b)     Error:  feature  not  supported:  non  linear  problem.   Simplified  constraint:     (a  >  0  &&  b  >  0  &&  c  >  0)  &&  (a  +  b  >  c  &&  a  +  c  >  b  &&  b  +  c  >  a)  &&   (2*a  +  3*b  !=  4*c)  &&  (3*b  +  4*c  !=  2*a)  &&     (2*a  +  4*c  ==  3*b)     sat (= a 4) (= b 4) (= c 1) Not  a  right  angle  triangle!   Target  not  covered.   124
  • 125.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Strengths  and  weaknesses   Dynamic  symbolic  execu6on:   Ø   Incompleteness:  not  all  paths  are  covered  if  concrete  values   are  used  and  the  simplified  constraint  admits  no  solu4on.     Ø   Approxima6on:  solu4ons  are  not  ensured  to  execute  the  path   of  interest  if  concrete  values  are  used.   Ø   Soundness:  generated  test  cases  are  executable,  faults  are  real.     Search  based  test  case  genera6on:   Ø   Incompleteness:  feasible  coverage  targets  might  be  missed.     Ø   Robustness:  non  linear  expressions  and  black  box  func4ons  do   not  represent  any  special  problem  in  fitness  func4on  defini4on.   Ø   Soundness:  generated  test  cases  are  executable,  faults  are  real.   Search  based  algorithms  can  easily  accommodate  approxima4ons,  but     might  miss  exact  solu4ons  which  are  available  via  SMT  solving.   125
  • 126.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Combining  SBST  &  DSE   §  Kobi  Inkumsah,  Tao  Xie.  Improving  Structural  Tes(ng  of  Object-­‐Oriented   Programs  via  Integra(ng  Evolu(onary  Tes(ng  and  Symbolic  Execu(on.  Proc.   of  Automated  Somware  Engineering  (ASE),  pp.  297-­‐306,  2008.     §  Jan  Malburg,  Gordon  Fraser.  Combining  search-­‐based  and  constraint-­‐based   tes(ng.  Proc.  of  Automated  Somware  Engineering  (ASE),  pp.  436-­‐439,  2011.   §  Tao  Xie,  Nikolai  Tillmann,  Jonathan  de  Halleux,  Wolfram  Schulte.  Fitness-­‐ guided  path  explora(on  in  dynamic  symbolic  execu(on.  Proc.  of  the   Interna4onal  Conference  on  Dependable  Systems  and  Networks  (DSN),  pp.   359-­‐368,  2009.   §  Arthur  I.  Baars,  Mark  Harman,  Youssef  Hassoun,  Kiran  Lakho4a,  Phil  McMinn,   Paolo  Tonella,  Tanja  E.  J.  Vos.  Symbolic  search-­‐based  tes(ng.  Proc.  of   Automated  Somware  Engineering  (ASE),  pp.  53-­‐62,  2011.   126
  • 127.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Genera6ng  new  individuals  by   means  of  DSE   Jan  Malburg,  Gordon  Fraser.  Combining  search-­‐based  and  constraint-­‐based  tes(ng.  Proc.  of   Automated  Somware  Engineering  (ASE),  pp.  436-­‐439,  2011.   New  individuals  are  generated  using  DSE.   Kobi  Inkumsah,  Tao  Xie.  Improving  Structural  Tes(ng  of  Object-­‐Oriented  Programs  via   Integra(ng  Evolu(onary  Tes(ng  and  Symbolic  Execu(on.  Proc.  of  Automated  Somware   Engineering  (ASE),  pp.  297-­‐306,  2008.     127
  • 128.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Genera6ng  new  individuals  by   means  of  DSE   Initialization   Fitness   assignment   Selection  Reproduction   •  Muta4on   •  Crossover   •  DSE  based  muta6on   Ch1:   1   5   -­‐1   0   3   pc1:  C1  &&  C2  &&  C3  &&  C4   pc1’:  C1  &&  C2  &&  !C3   Ch1’:   1   0   2   1   1   128
  • 129.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Primi6ve  values  genera6on   $x0=BankAccount():  $x0.deposit(int):  $x0.withdraw(int)  @  1,  20   Path  condi6on  (withdraw)  =  (z1  >  z0)   Error  handling  branch:   void withdraw(int amount) { if (amount > balance) { printError(); } else { balance = balance – amount; //target } … } Primi6ve  values  are  made  symbolic:  z0,  z1   129
  • 130.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Primi6ve  values  genera6on   $x0=BankAccount():  $x0.deposit(int):  $x0.withdraw(int)  @  1,  20   Path  condi6on  (withdraw)  =  (z1  >  z0)   Primi6ve  values  are  made  symbolic:  z0,  z1   Negated  path  condi6on  =  (z1  <=  z0)   sat (= z1 1) (= z0 20) $x0=BankAccount():  $x0.deposit(int):  $x0.withdraw(int)  @  20,  1   Added  to  current  popula6on   130 Non-­‐error  branch:   if (amount > balance) { printError(); } else { balance = balance – amount; …
  • 131.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Primi6ve  values  genera6on   double testMe(int x, int y, double z) { 1 boolean flag = y > 1000; 2 //… 3 if (x + y == 1024) 4 if (flag) 5 if (Math.cos(z) – 0.95 < Math.exp(z)) 6 abort(); 7 return 0.0; } Ch1:   1024   0   0.0   Standard  muta4on  may  take  a  while  to  make  y  >  1000   and  x  +  y  =  1024   pc1:  (x  +  y  =  1024)  &&  (y  ≤  1000)   pc1’:  (x  +  y  =  1024)  &&  (y  >  1000)   Ch1’:   23   1001   0.0   Normal  muta4on  and  crossover  are  s4ll  required  for   condi4ons  such  as  5.   131
  • 132.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   A  fitness  func6on  is  used  to  select  which  path  to  explore  by   means  of  DSE.   Tao  Xie,  Nikolai  Tillmann,  Jonathan  de  Halleux,  Wolfram  Schulte.  Fitness-­‐guided  path   explora(on  in  dynamic  symbolic  execu(on.  Proc.  of  the  Interna4onal  Conference  on   Dependable  Systems  and  Networks  (DSN),  pp.  359-­‐368,  2009.   132
  • 133.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } Ø   The  (randomly  selected)  size  of  y  must  be  ≥  20.   Ø   DSE  has  ≥  220  paths  to  explore,  among  which  only  a  small  frac4on   contain  exactly  twenty  3T,  which  is  required  to  make  5  true.   DSE   Ini4al  random  TC:     x  =  0,  y  =  {0}   p  =  <1F,  7>   pc  =  (x  !=  90)   pc’  =  (x  ==  90)   New  TC:   x  =  90,  y  =  {0}   p  =  <1T,  2T,  3F,  2F,  5F,  7>   133
  • 134.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   d5T  =  {0  if  x  ==  110;  abs(x  -­‐  110)  otherwise}   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } 134
  • 135.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC0:      TestLoop(0,  {0})      p  =  <1F,  7>   TC1:      TestLoop(90,  {0})      p  =  <1T,  2T,  3F,  2F,  5F,  7>   TC2:      TestLoop(90,  {15})      p  =  <1T,  2T,  3T,  2F,  5F,  7>   TC3:      TestLoop(90,  {15,  0})      p  =  <1T,  2T,  3T,  2T,  3F,  2F,  5F,  7>   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   §  f(TC0)  =  1  +  BD(1T)  =  1.989   §  d(TC1)  =  |90  –  110|  =  20,  BD  =  0.952   §  d(TC2)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC3)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC4)  =  |92  –  110|  =  18,  BD  =  0.947   135
  • 136.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC0:      TestLoop(0,  {0})      p  =  <1F,  7>   TC1:      TestLoop(90,  {0})      p  =  <1T,  2T,  3F,  2F,  5F,  7>   TC2:      TestLoop(90,  {15})      p  =  <1T,  2T,  3T,  2F,  5F,  7>   TC3:      TestLoop(90,  {15,  0})      p  =  <1T,  2T,  3T,  2T,  3F,  2F,  5F,  7>   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   TC4  is  selected  for   branching  node  flipping   §  f(TC0)  =  1  +  BD(1T)  =  1.989   §  d(TC1)  =  |90  –  110|  =  20,  BD  =  0.952   §  d(TC2)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC3)  =  |91  –  110|  =  19,  BD  =  0.95   §  d(TC4)  =  |92  –  110|  =  18,  BD  =  0.947   136
  • 137.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  gain   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>     Which  branch  shall  be  flipped?   Fitness  gain:  how  much  the  fitness  value  has  improved  across   paths  amer  flipping  a  given  branch  in  the  past.     137
  • 138.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  gain   DSE   TC0:      TestLoop(0,  {0})      p  =  <1F,  7>   TC1:  d  =  20      TestLoop(90,  {0})      p  =  <1T,  2T,  3F,  2F,  5F,  7>   TC2:  d  =  19      TestLoop(90,  {15})      p  =  <1T,  2T,  3T,  2F,  5F,  7>   TC3:  d  =  19      TestLoop(90,  {15,  0})      p  =  <1T,  2T,  3T,  2T,  3F,  2F,  5F,  7>   TC4:  d  =  18      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } §  ∆d(3F)  =  (1  +  1  +  2)  /  3  =  1.33   §  ∆d(3T)  =  (-­‐1  -­‐  1  -­‐  2)  /  3  =  -­‐1.33   §  ∆d(2F)  =  (0  +  1)  /  2  =  0.5   §  ∆d(2T)  =  (0  -­‐  1)  /  2  =  -­‐0.5   138
  • 139.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  gain   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC4:      TestLoop(90,  {15,  15})      p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>   Es6mated  distance:  the  new  branch  distance  associated  with   flipping  branch  b  is  es4mated  as:  d’(TC)  =  d(TC)  -­‐  ∆d(b)   §  Flip  2T:  d’(TC4)  =  18  +  0.5  =  18.5   §  Flip  3T:  d’(TC4)  =  18  +  1.33  =  19.33   §  Flip  2F:  d’(TC4)  =  18  -­‐  0.5  =  17.5   139
  • 140.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  guided  path  explora6on   bool testLoop(int x, int[] y) { 1 if (x == 90) { 2 for (int i = 0 ; i < y.length ; i++) 3 if (y[i] == 15) 4 x++; 5 if (x == 110) 6 abort(); } 7 return false; } DSE   TC4:  TestLoop(90,  {15,  15}),  p  =  <1T,  2T,  3T,  2T,  3T,  2F,  5F,  7>,  d  =  18   TC5:  TestLoop(90,  {15,  15,  0}),  p  =  <1T,  2T,  3T,  2T,  3T,  2T,  3F,  2F,  5F,  7>,  d  =  18   TC6:  TestLoop(90,  {15,  15,  15}),  p  =  <1T,  2T,  3T,  2T,  3T,  2T,  3T,  2F,  5F,  7>,  d  =  17   …   TC40:  TestLoop(90,  {15,  15,  …,  15}),  p  =  <1T,  2T,  3T,  2T,  3T,  2T,  3T,  2T,  …,  2F,  5T,  7>,  d   =  0   41  paths  are  explored  to   cover  5T;  full  explora4on   of  all  paths  would  require   ≥  220  test  cases.   140
  • 141.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   The  fitness  func6on  is  improved  by  means  of  symbolic   execu4on.   Arthur  I.  Baars,  Mark  Harman,  Youssef  Hassoun,  Kiran  Lakho4a,  Phil  McMinn,  Paolo  Tonella,   Tanja  E.  J.  Vos.  Symbolic  search-­‐based  tes(ng.  Proc.  of  ASE,  pp.  53-­‐62,  2011.   141
  • 142.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   ¢  Concrete execution of SUT ¢  Symbolic execution of each path between critical node and target ¢  Compute cumulative branch distance for each path condition ¢  Return minimal distance ¢  Example: —  abef —  abcdef Z > 0 ∧ X = Z ∧ Z ≤ 0 ∧ Y = 10 a b c d e f Z > 0 ∧ X = Z ∧ Z ≤ 1 ∧ Y = 9 bd>(z,0) + bd=(x,z) + bd<=(z,1) + bd=(y,9) 142
  • 143.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   c ¢  Path expression: —  ab(cd)*ef ¢  Choose a maximum loop unrolling (e.g., 1) —  abef —  abcdD[cd]ef e Z > 0 ∧ X = Z ∧ Z > 0 ∧ Z ≤ 1 ∧ Y = 9 a b d f 143
  • 144.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Fitness  func6on  improvement   Z > 0 ∧ X = Z ∧ Z > 0 ∧ Z ≤ 1 ∧ Y = 9 FF1 = bd>(z,0) + bd+(x,z) + bd>(z,0) + 1 + 1 path distance approximation level 144
  • 145.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) int foo(int x, int y, int z) { if(z > 0) if(x == z) { for(int i = 0; i < z; i++) y++ if(y == 10) return 0; } return 1; } FF1 = bd>(z,0) + bd=(x,z) + bd>(z,0) + 1 + 1 Fitness  func6on  improvement   target FF2 FF5 FF3 FF1 Fitness = min(FF1,FF2,FF3,FF5) 145
  • 146.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Conclusions  and  future  work   146
  • 147.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Conclusions   •  High  coverage  can  be  achieved  automa4cally   using  search  based  test  case  genera4on.   •  Func4ons  and  classes  under  test  are  exercised  in   a  quite  sophis4cated  way.   •  Resul4ng  test  suites  are  generally  quite  compact   and  their  size  can  be  controlled  as  a  further   op4miza4on  objec4ve.   147
  • 148.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) Future  work   •  Inves4ga4on  of  alterna4ve  adequacy  criteria  and   tes4ng  goals  (e.g.,  non-­‐func4onal  tes4ng).   •  Automa4ng  the  genera4on  of  likely  oracles  (i.e.,   anomaly  detectors),  based  on  invariants  and   temporal  proper4es  of  the  func4ons  and  classes   under  test.   •  Further  empirical  studies,  especially  on  combined   approaches.   148
  • 149.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) References   §  Shaukat  Ali,  Lionel  C.  Briand,  Hadi  Hemma4,  Rajwinder  Kaur  Panesar-­‐Walawege:  A   Systema(c  Review  of  the  Applica(on  and  Empirical  Inves(ga(on  of  Search-­‐Based  Test  Case   Genera(on.  IEEE  Transac4ons  on  Somware  Engineering,  vol.  36  n.  6,  pp.  742-­‐762,  2010.   §  Andrea  Arcuri,  Muhammad  Zohaib  Z.  Iqbal,  Lionel  C.  Briand:  Formal  analysis  of  the   effec(veness  and  predictability  of  random  tes(ng.  ISSTA,  pp.  219-­‐230,  2010.   §  Gordon  Fraser,  Andrea  Arcuri:  It  is  Not  the  Length  That  Ma`ers,  It  is  How  You  Control  It.   Proc.  of  the  Interna4onal  Conference  on  Somware  Tes4ng  (ICST),  pp.  150-­‐159,  2011.   §  Gordon  Fraser,  Andrea  Arcuri,  Evolu(onary  genera(on  of  whole  test  suites.  Proc.  of  the   Interna4onal  Conf.  on  the  Quality  of  Somware  (QSIC),  pp.  31-­‐40,  Madrid,  Spain,  July  2011.   §  Gordon  Fraser,  Andrea  Arcuri,  Whole  test  suite  genera(on.  IEEE  Transac4ons  on  Somware   Engineering,  vol.  38,  n.  2,  pp.  276-­‐291,  2013.   §  Arthur  I.  Baars,  Mark  Harman,  Youssef  Hassoun,  Kiran  Lakho4a,  Phil  McMinn,  Paolo  Tonella,   Tanja  E.  J.  Vos.  Symbolic  search-­‐based  tes(ng.  Proc.  of  Automated  Somware  Engineering   (ASE),  pp.  53-­‐62,  2011.   §  Patrice  Godefroid,  Nils  Klarlund,  and  Koushik  Sen,  DART:  Directed  Automated  Random   Tes4ng,  PLDI’05.   §  Mark  Harman  and  Phil  McMinn.  A  Theore(cal  and  Empirical  Study  of  Search  Based  Tes(ng:   Local,  Global  and  Hybrid  Search.  IEEE  Transac4ons  on  Somware  Engineering,  vol.  36,  n.  2,  pp.   226-­‐247,  2010.   149
  • 150.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) References   §  Mark  Harman,  Lin  Hu,  Rob  Hierons,  Joachim  Wegener,  Harmen  Sthamer,  Andre  Baresel  and   Marc  Roper.  Testability  Transforma(on.  IEEE  Transac4ons  on  Somware  Engineering,  vol.  30,   n.  1,  pp.  3-­‐16,  2004.   §  Kobi  Inkumsah,  Tao  Xie.  Evacon:  a  framework  for  integra(ng  evolu(onary  and  concolic   tes(ng  for  object-­‐oriented  programs.  Proc.  of  Automated  Somware  Engineering  (ASE),  pp.   425-­‐428,  2007.   §  Kobi  Inkumsah,  Tao  Xie.  Improving  Structural  Tes(ng  of  Object-­‐Oriented  Programs  via   Integra(ng  Evolu(onary  Tes(ng  and  Symbolic  Execu(on.  Proc.  of  Automated  Somware   Engineering  (ASE),  pp.  297-­‐306,  2008.     §  Bogdan  Korel:  Automated  SoDware  Test  Data  Genera(on,  IEEE  Transac4ons  on  Somware   Engineering,  vol.  16,  n.  8,  pp.  870-­‐879,  1990.   §  Jan  Malburg,  Gordon  Fraser.  Combining  search-­‐based  and  constraint-­‐based  tes(ng.  Proc.  of   Automated  Somware  Engineering  (ASE),  pp.  436-­‐439,  2011.   §  Phil  McMinn,  Search-­‐based  soDware  test  data  genera(on:  a  survey.  Journal  of  Somware   Tes4ng,  Verifica4on  and  Reliability,  vol.  14,  n.  2,  pp.  105-­‐156,  June  2004.   §  Roy  P.  Pargas,  Mary  Jean  Harrold  and  Robert  R.  Peck:  Test-­‐data  genera(on  using  gene(c   algorithms,  Somware  Tes4ng,  Verifica4on  and  Reliability,  vol.  9,  n.  4,  pp.  263-­‐282,  1999.   §  Koushik  Sen,  Darko  Marinov,  Cul  Agha,  CUTE:  A  Concolic  Unit  Tes4ng  Engine  for  C,  ESEC-­‐ FSE’05.   150
  • 151.
    The FITTEST projectis funded by the European Commission (FP7-ICT-257574) References   §  Paolo  Tonella,  Evolu(onary  tes(ng  of  classes.  Proc.  of  the  Interna4onal  Symposium  on   Somware  Tes4ng  and  Analysis  (ISSTA),  pp.  119-­‐128,  Boston,  USA,  July  2004.   §  David  H.  Wolpert,  William  G.  Macready:  No  Free  Lunch  Theorems  for  Op(miza(on,  IEEE   Transac4ons  on  Evolu4onary  Computa4on,  vol.  1,  n.  1,  April  1997.   §  Tao  Xie,  Nikolai  Tillmann,  Jonathan  de  Halleux,  Wolfram  Schulte.  Fitness-­‐guided  path   explora(on  in  dynamic  symbolic  execu(on.  Proc.  of  the  Interna4onal  Conference  on   Dependable  Systems  and  Networks  (DSN),  pp.  359-­‐368,  2009.   Slides  on  dynamic  symbolic  execu4on  are  par4ally  based  on  the  ESEC-­‐FSE’05  presenta4on:  hVp:// srl.cs.berkeley.edu/~ksen   151