Data Mutation Testing --  A Method for Automated Generation of Structurally Complex Test Cases Hong Zhu Dept. of Computing...
Outline <ul><li>Motivation  </li></ul><ul><ul><li>Overview of existing work on software test case generation </li></ul></u...
Motivation  <ul><li>Test case generation  </li></ul><ul><ul><li>Need to meet multiple goals </li></ul></ul><ul><ul><ul><li...
Existing Work <ul><li>Program-based test case generation </li></ul><ul><ul><li>Static : analysis of code without execution...
<ul><li>Specification-based test case generation </li></ul><ul><ul><li>Derive from either formal or semi-formal specificat...
<ul><li>Random testing </li></ul><ul><ul><li>Through random sampling over input domain based on probabilistic models of th...
<ul><li>Domain-specific techniques </li></ul><ul><ul><li>Database applications   </li></ul></ul><ul><ul><ul><li>Zhang, J.,...
The Challenge <ul><li>How to generate  adequate  test cases of high  reality  for programs that process  structurally comp...
Basic Ideas of Data Mutation Testing <ul><li>Preparing the seeds, i.e. a small set of test cases </li></ul><ul><ul><li>Con...
Illustrative Example <ul><li>Triangle classification </li></ul><ul><ul><li>Input:  x ,  y ,  z : Natural Numbers; </li></u...
<ul><li>Mutation operators </li></ul><ul><ul><li>IVP: Increase the value of a parameter by 1; </li></ul></ul><ul><ul><li>D...
<ul><li>Generation of mutant test cases  </li></ul><ul><ul><li>For example, by applying the mutation operator IVP to test ...
<ul><li>Execution of program and classification of mutants </li></ul><ul><ul><li>A mutant is classified as  dead , if the ...
<ul><li>Analyse test effectiveness   </li></ul><ul><ul><li>Reasons why a mutant can remain alive: </li></ul></ul><ul><ul><...
Measurements of Data Mutation  <ul><li>Equivalent mutant score  EMS :   </li></ul><ul><li>A high equivalent mutant score E...
Process of Data Mutation Testing
Analysis of Program Correctness  <ul><li>Can data mutation testing be helpful to the analysis of program correctness?  </l...
Case Study <ul><li>The subject </li></ul><ul><ul><li>CAMLE: Caste-centric Agent-oriented Modelling Language and Environmen...
Complexity of the Input Data <ul><li>Input: models in CAMLE language </li></ul><ul><ul><li>Multiple views:  </li></ul></ul...
The Function to Be Tested <ul><li>Consistency checker  </li></ul><ul><ul><li>Consistency constraints are formally defined ...
Types of Data Mutation Operators Delete an existing env node in a sub-collaboration diagram Delete env node 12 Generate a ...
Change the Start or End node of an existing edge to an env node Change edge end to env 24 Delete the Action List of an exi...
The Seed Test Cases <ul><li>Models developed in previous case studies of agent-oriented software development methodology  ...
The Seed Test Cases and Their Mutants  7808 1082 3260 3466 Number of Mutants 257 38 99 110 #Edges 352 54 140 158 #Nodes 34...
The Results: Fault Detecting Ability 5 114 (97%) 61 (52%) 118 Total 0 19 (100%) 12 (63%) 19 Transposition of statements 1 ...
Detecting Design Errors <ul><li>In the case study, we found that a large number of mutants remain alive  </li></ul>Table ....
Statistics on Amalthaea test suite   Some typed mutation score is very low Design of consistency checker has errors! Espec...
Results: Detecting Design Errors <ul><li>Hypothesis </li></ul><ul><ul><li>Design of the tool is weak in detecting certain ...
Test Adequacy <ul><li>Our experiments show that high test adequacy can be achieved through data mutation.  </li></ul><ul><...
Coverage of scenario diagram variants 40 0 0 40 17 11 0 1 10 16 10 0 2 8 14 10 0 2 8 10 24 0 4 20 9 24 0 4 20 8 24 0 4 20 ...
Coverage of Program Structure and Functions The test data achieved 100% coverage of the functions of the consistency check...
Test Cost The seeds were readily available from previous case studies of the tool.  Table. Summary of the test cost spent ...
Analysis Program’s Correctness <ul><li>The experiment took the black-box approach  </li></ul><ul><ul><li>The output on a t...
Experiments <ul><li>The experiments </li></ul><ul><ul><li>Mutants are selected at random  </li></ul></ul><ul><ul><li>The p...
The Experiment Data <ul><li>Results: </li></ul><ul><ul><li>Checking correctness on dead mutants: 3 minute/per mutant </li>...
Related Works <ul><li>Mutation testing </li></ul><ul><ul><li>Program or specification is modified </li></ul></ul><ul><ul><...
Future Work <ul><li>More case studies with potential applications </li></ul><ul><ul><li>Security control software: Role-Ba...
Perspectives and Future Work <ul><li>Integration of data mutation testing, metamorphic testing and algebraic testing metho...
Example <ul><li>Consider the Triangle Classification program  P </li></ul><ul><ul><li>The following is a metamorphic relat...
Integration with Algebraic Testing <ul><li>In algebraic software testing, axioms are written in the form of </li></ul><ul>...
Screen Snapshot of Algebraic Testing Tool CASCAT
References <ul><li>Lijun Shan and Hong Zhu,  Generating Structurally Complex Test Cases by Data Mutation: A Case Study of ...
Upcoming SlideShare
Loading in …5
×

Aug./Sept. 2009 Data Mutation Software Testing

798 views

Published on

  • Be the first to comment

  • Be the first to like this

Aug./Sept. 2009 Data Mutation Software Testing

  1. 1. Data Mutation Testing -- A Method for Automated Generation of Structurally Complex Test Cases Hong Zhu Dept. of Computing and Electronics, Oxford Brookes Univ., Oxford, OX33 1HX, UK Email: [email_address]
  2. 2. Outline <ul><li>Motivation </li></ul><ul><ul><li>Overview of existing work on software test case generation </li></ul></ul><ul><ul><li>The challenges to software testing </li></ul></ul><ul><li>The Data Mutation Testing Method </li></ul><ul><ul><li>Basic ideas </li></ul></ul><ul><ul><li>Process </li></ul></ul><ul><ul><li>Measurements </li></ul></ul><ul><li>A Case study </li></ul><ul><ul><li>Subject software under test </li></ul></ul><ul><ul><li>The mutation operators </li></ul></ul><ul><ul><li>Experiment process </li></ul></ul><ul><ul><li>Main results </li></ul></ul><ul><li>Perspectives and future works </li></ul><ul><ul><li>Potential applications </li></ul></ul><ul><ul><li>Integration with other black box testing methods </li></ul></ul>
  3. 3. Motivation <ul><li>Test case generation </li></ul><ul><ul><li>Need to meet multiple goals </li></ul></ul><ul><ul><ul><li>Reality : to represent real operation of the system </li></ul></ul></ul><ul><ul><ul><li>Coverage : functions, program code, input/output data space, and their combinations </li></ul></ul></ul><ul><ul><ul><li>Efficiency : not to overkill, easy to execute, etc. </li></ul></ul></ul><ul><ul><ul><li>Effective : capable of detecting faults, which implies easy to check the correctness of program’s output </li></ul></ul></ul><ul><ul><ul><li>Externally useful : help with debugging, reliability estimation, etc. </li></ul></ul></ul><ul><ul><li>Huge impact on test effectiveness and efficiency </li></ul></ul><ul><ul><li>One of the most labour intensive tasks in practices </li></ul></ul>
  4. 4. Existing Work <ul><li>Program-based test case generation </li></ul><ul><ul><li>Static : analysis of code without execution, e.g. symbolic execution </li></ul></ul><ul><ul><ul><li>Path oriented </li></ul></ul></ul><ul><ul><ul><ul><li>Howden, W. E. (1975, 1977, 1978); Ramamoorthy, C., Ho, S. and Chen, W. (1976) ; King, J. (1975) ; Clarke, L. (1976) ; Xie T., Marinov, D., and Notkin, D. (2004); J. Zhang. (2004), Xu, Z. and Zhang, J. (2006) </li></ul></ul></ul></ul><ul><ul><ul><li>Goal oriented </li></ul></ul></ul><ul><ul><ul><ul><li>DeMillo, R. A., Guindi, D. S., McCracken, W. M., Offutt, A. J. and King, K. N. (1988) ; Pargas, R. P., Harrold, M. J. and Peck, R. R. (1999); Gupta, N., Mathur, A. P. and Soffa, M. L. (2000); </li></ul></ul></ul></ul><ul><ul><li>Dynamic : through execution of the program </li></ul></ul><ul><ul><ul><ul><li>Korel, B. (1990) , Beydeda, S. and Gruhn, V. (2003) </li></ul></ul></ul></ul><ul><ul><li>Hybrid : combination of dynamic execution with symbolic execution, e.g. concolic techniques </li></ul></ul><ul><ul><ul><ul><li>Godefroid, P., Klarlund, N., and Sen, K.. (2005); </li></ul></ul></ul></ul><ul><ul><li>Techniques : </li></ul></ul><ul><ul><ul><li>Constraint solver, Heuristic search , e.g. genetic algorithms: </li></ul></ul></ul><ul><ul><ul><ul><li>McMinn, P. and Holcombe, M. (2003) , Survey: McMinn, P. (2004) </li></ul></ul></ul></ul>
  5. 5. <ul><li>Specification-based test case generation </li></ul><ul><ul><li>Derive from either formal or semi-formal specifications of the required functions and/or the designs </li></ul></ul><ul><ul><li>Formal specification-based : </li></ul></ul><ul><ul><ul><li>First order logic, Z spec and Logic programs : : Tai, K.-C. (1993); Stocks, P. A. and Carrington, D. A. (1993) ; Ammann, P. and Offutt, J. (1994) ; Denney, R. (1991) </li></ul></ul></ul><ul><ul><ul><li>Algebraic specification : Bouge, L., Choquet, N., Fribourg, L. and Gaudel, M.-C. (1986) ; Doong, R. K. and Frankl, P. G. (1994) ; Chen, H. Y., Tse, T. H. and Chen, T. Y. (2001) ; Zhu (2007); </li></ul></ul></ul><ul><ul><ul><li>Finite state machines : Fujiwara, S., et al.. (1991) ; Lee, D. and Yannakakis, M. (1996) ; Hierons, R. M. (2001) ; Zhu, H., Jin, L. & Diaper, D. (1999) ; </li></ul></ul></ul><ul><ul><ul><li>Petri nets : Morasca, S. and Pezze, M. (eds). (1990) ; Zhu, H. and He, X. (2002) </li></ul></ul></ul><ul><ul><li>Model-based : derive from semi-formal graphic models </li></ul></ul><ul><ul><ul><li>SSADM models : Zhu, H., Jin, L. and Diaper, D. (1999, 2001); </li></ul></ul></ul><ul><ul><ul><li>UML models : Offutt, J. and Abdurazik, A. (2000) ; Tahat, L. H., et al. (2001); Hartman, A. and Nagin, K. (2004); Li, S., Wang, J. and Qi, Z.-C. (2004) ; </li></ul></ul></ul><ul><ul><li>Techniques: </li></ul></ul><ul><ul><ul><li>Constraint solving ; Theorem prover ; Model-checker </li></ul></ul></ul>
  6. 6. <ul><li>Random testing </li></ul><ul><ul><li>Through random sampling over input domain based on probabilistic models of the operation of the software under test. </li></ul></ul><ul><ul><li>Profile-based : Sampling over an existing operation profile at random </li></ul></ul><ul><ul><li>Stochastic model based : Use a probabilistic model of software usages </li></ul></ul><ul><ul><ul><li>Markov chain : </li></ul></ul></ul><ul><ul><ul><ul><li>Avritzer, A. Larson, B. (1993) ; Avritzer, A. Weyuker, E. J. (1994) ; Whittaker, J. A. and Poore, J. H. (1993) ; Guen, H. L., Marie, R. and Thelin, T. (2004) ; Prowell, S. J. (2005) </li></ul></ul></ul></ul><ul><ul><ul><li>Stochastic automata networks : </li></ul></ul></ul><ul><ul><ul><ul><li>Farina, A. G., Fernandes, P. and Oliveira, F. M. (2002, 2004) ; </li></ul></ul></ul></ul><ul><ul><ul><li>Bayesian networks : </li></ul></ul></ul><ul><ul><ul><ul><li>Fine, S. and Ziv, A. (2003) </li></ul></ul></ul></ul><ul><ul><li>Adaptive random testing : Even spread of randomly test cases ( Chen, T. Y., Leung, H. and Mak, I. K. (2004) ) </li></ul></ul><ul><ul><ul><li>Variants : Mirror , Restricted , and Probabilistic ART </li></ul></ul></ul>
  7. 7. <ul><li>Domain-specific techniques </li></ul><ul><ul><li>Database applications </li></ul></ul><ul><ul><ul><li>Zhang, J., Xu, C. and Cheung, S. C. (2001) </li></ul></ul></ul><ul><ul><li>Spreadsheets : </li></ul></ul><ul><ul><ul><li>Fisher, M., Cao, M., Rothermel, G., Cook, C. and Burnett, M. (2002) </li></ul></ul></ul><ul><ul><ul><li>Erwig, M., Abraham, R., Cooperstein, I., and Kollmansberger S. (2005) </li></ul></ul></ul><ul><ul><li>XML Scheme : </li></ul></ul><ul><ul><ul><li>Lee, S. C. and Offutt, J. (2001) ; Li, J. B. and Miller, J. (2005) </li></ul></ul></ul><ul><ul><li>Compiler : </li></ul></ul><ul><ul><ul><li>See Boujarwah, A. S. and Saleh, K. (1997) for a survey. </li></ul></ul></ul>
  8. 8. The Challenge <ul><li>How to generate adequate test cases of high reality for programs that process structurally complex inputs ? </li></ul><ul><ul><li>Structural complexity: </li></ul></ul><ul><ul><ul><li>A large number of elements </li></ul></ul></ul><ul><ul><ul><li>A large number of possible explicitly represented relationships between the elements </li></ul></ul></ul><ul><ul><ul><li>A large number of constraints imposed on the relationships </li></ul></ul></ul><ul><ul><ul><li>Meaning of the data depends on not only the values of the elements, but also the relationships and thus their processing </li></ul></ul></ul><ul><ul><li>Reality: </li></ul></ul><ul><ul><ul><li>Likely or close to be a correct real input in the operation of the system </li></ul></ul></ul><ul><ul><ul><li>Likely or close to be an input that contains errors that a user inputs to the system in operation </li></ul></ul></ul><ul><ul><li>Examples : </li></ul></ul><ul><ul><li>CAD, Word processor, Web browser, Spreadsheets, Powerpoint, Software modelling tools, Language processor, Theorem provers, Model-checkers, Speech recognition, Hand writing recognition, Search engine,… </li></ul></ul>
  9. 9. Basic Ideas of Data Mutation Testing <ul><li>Preparing the seeds, i.e. a small set of test cases </li></ul><ul><ul><li>Contain various types of elements and relationships between them </li></ul></ul><ul><ul><li>Highly close to the real input data </li></ul></ul><ul><ul><li>Easy to check their correctness </li></ul></ul><ul><li>Generating mutant test cases by modifying the seeds slightly </li></ul><ul><ul><li>Preserve the validity of the input </li></ul></ul><ul><ul><li>Change at one place a time unless imposed by the constraints (but may use second order even higher order mutants) </li></ul></ul><ul><ul><li>Make as many different mutants as possible </li></ul></ul><ul><li>Executing the software under test on both seeds and their mutants </li></ul><ul><ul><li>What to observe: </li></ul></ul><ul><ul><ul><li>program’s correctness on both seeds and mutants </li></ul></ul></ul><ul><ul><ul><li>the differences of the program’s behaviours on seed and their mutants </li></ul></ul></ul><ul><ul><li>Uses of metrics and measurements </li></ul></ul><ul><ul><ul><li>seeds are sufficient </li></ul></ul></ul><ul><ul><ul><li>mutations are effective and/or sufficient </li></ul></ul></ul><ul><ul><ul><li>Feedback to step 1 and 2 if necessary, or to improve the observation. </li></ul></ul></ul>
  10. 10. Illustrative Example <ul><li>Triangle classification </li></ul><ul><ul><li>Input: x , y , z : Natural Numbers; </li></ul></ul><ul><ul><li>Output: </li></ul></ul><ul><ul><li>{ equilateral, isosceles, scalene, non-triangle } </li></ul></ul><ul><li>Seeds: </li></ul>The lengths of the sides The type of triangles Non-triangle ( x =3, y =5, z =9) t 4 Scalene ( x =5, y =7, z =9) t 3 Isosceles ( x =5, y =5, z =7) t 2 Equilateral ( x =5, y =5, z =5) t 1 Expected output Input ID
  11. 11. <ul><li>Mutation operators </li></ul><ul><ul><li>IVP: Increase the value of a parameter by 1; </li></ul></ul><ul><ul><li>DVP: Decrease the value of a parameter by 1; </li></ul></ul><ul><ul><li>SPL: Set the value of a parameter to a very large number, say 1000000; </li></ul></ul><ul><ul><li>SPZ: Set the value of a parameter to 0; </li></ul></ul><ul><ul><li>SPN: Set the value of a parameter to a negative number, say -2; </li></ul></ul><ul><ul><li>WXY: Swap the values of parameters x and y; </li></ul></ul><ul><ul><li>WXZ: Swap the values of parameters x and z; </li></ul></ul><ul><ul><li>WYZ: Swap the values of parameters y and z; </li></ul></ul><ul><ul><li>RPL: Rotate the values of parameters towards left; </li></ul></ul><ul><ul><li>RPR: Rotate the values of parameters towards right. </li></ul></ul>
  12. 12. <ul><li>Generation of mutant test cases </li></ul><ul><ul><li>For example, by applying the mutation operator IVP to test case t 1 on parameter x , we can obtain the following test case t 5 . </li></ul></ul><ul><ul><li>IVP ( t 1 , x ) = t 5 = Input: ( x =6, y =5, z =5). </li></ul></ul><ul><ul><li>Total number of mutants: </li></ul></ul><ul><ul><ul><li>(5*3 +5)*4 = 80 </li></ul></ul></ul><ul><ul><ul><li>Covering all sorts of combinations of data elements </li></ul></ul></ul><ul><ul><ul><li>Systematically produced from the four seeds </li></ul></ul></ul>
  13. 13. <ul><li>Execution of program and classification of mutants </li></ul><ul><ul><li>A mutant is classified as dead , if the execution of the software under test on the mutant is different from the execution on the seed test case. Otherwise, the mutant is classified as alive . </li></ul></ul><ul><ul><li>For example </li></ul></ul><ul><ul><ul><li>For a correctly implemented Triangle Classification program, the execution on the mutant test case t 5 will output isosceles while the execution on its seed t 1 will output equilateral . </li></ul></ul></ul><ul><ul><ul><li>TrC(t 5 )  TrC(t 1 )  t 5 is dead </li></ul></ul></ul>It depends on how you observe the behaviour!
  14. 14. <ul><li>Analyse test effectiveness </li></ul><ul><ul><li>Reasons why a mutant can remain alive: </li></ul></ul><ul><ul><ul><li>The mutant is equivalent to the original with respect to the functionality or property of the software under test. </li></ul></ul></ul><ul><ul><ul><li>RPL(t 1 )= t 1 </li></ul></ul></ul><ul><ul><ul><li>The observation on the behaviour and output of the software under test is not sufficient to detect the difference </li></ul></ul></ul><ul><ul><ul><li>RPL(t 2 )= t 6 = Input: ( x =5, y =7, z =5). </li></ul></ul></ul><ul><ul><ul><li>The software is incorrectly designed and/or implemented so that it is unable to differentiate the mutants from the original. </li></ul></ul></ul>Same output, but different execution paths for a correct program.
  15. 15. Measurements of Data Mutation <ul><li>Equivalent mutant score EMS : </li></ul><ul><li>A high equivalent mutant score EMS indicates that the mutation operators have not been well designed to achieve variety in the test cases. </li></ul><ul><li>Live mutant score LMS : </li></ul><ul><li>A high LMS indicates that the observation on the behaviour and output of the software under test is insufficient. </li></ul><ul><li>Typed live mutant score LMS  , </li></ul><ul><li>where  is a type of mutation operators </li></ul><ul><li>A high LMS  reveals that the program is not sensitive to the type of mutation probably because a fault in design or implementation. </li></ul>Number of equivalent mutants Total number of mutants Number of life mutants
  16. 16. Process of Data Mutation Testing
  17. 17. Analysis of Program Correctness <ul><li>Can data mutation testing be helpful to the analysis of program correctness? </li></ul><ul><li>Consider the examples in Triangle Classification: </li></ul><ul><ul><li>IVP or DVP to test case t 1 , we can expect the output to be isosceles. </li></ul></ul><ul><ul><li>For the RPL, RPR, WXY, WYZ, and WYZ mutation operators, we can expect that the program should output the same classification on a seed and its mutant test cases. </li></ul></ul><ul><ul><li>If the software’s behaviour on a mutant is not as expected, an error in the software under test can be detected. </li></ul></ul>
  18. 18. Case Study <ul><li>The subject </li></ul><ul><ul><li>CAMLE: Caste-centric Agent-oriented Modelling Language and Environment </li></ul></ul><ul><ul><ul><li>Automated modelling tool for agent-oriented methodology </li></ul></ul></ul><ul><ul><ul><li>Developed at NUDT of China </li></ul></ul></ul><ul><li>Potential threats to the validity of the case study </li></ul><ul><ul><li>Subject is developed by the tester </li></ul></ul><ul><ul><li>The developer is not professional software developer </li></ul></ul><ul><li>Validation of the case study against the potential threats </li></ul><ul><ul><li>The test method is black box testing . The knowledge of the code and program structure affect the outcomes. </li></ul></ul><ul><ul><li>The subject was developed before the case study and no change at all was made during the course to enable the case study to be carried out. </li></ul></ul><ul><ul><li>In software testing practice, systems are often tested by the developers. </li></ul></ul><ul><ul><li>The developer is a capable master degree student with sufficient training at least equivalent to an average programmer. </li></ul></ul><ul><ul><li>The correctness of the program’s output can be judges objectively. </li></ul></ul>
  19. 19. Complexity of the Input Data <ul><li>Input: models in CAMLE language </li></ul><ul><ul><li>Multiple views: </li></ul></ul><ul><ul><ul><li>a caste diagram that describes the static structure of a multi-agent system, </li></ul></ul></ul><ul><ul><ul><li>a set of collaboration diagrams that describe how agents collaborate with each other, </li></ul></ul></ul><ul><ul><ul><li>a set of scenario diagrams that describe typical scenarios namely situations in the operation of the system, and </li></ul></ul></ul><ul><ul><ul><li>a set of behaviour diagrams that define the behaviour rules of the agents in the context of various scenarios. </li></ul></ul></ul><ul><ul><li>Well-formedness constraints </li></ul></ul><ul><ul><ul><li>Each diagram has a number of different types of nodes and arcs, etc. </li></ul></ul></ul><ul><ul><ul><li>Each diagram and the whole model must satisfy a set of well-formedness conditions to be considered as a valid input (e.g. the types of nodes and arcs must match with each other) </li></ul></ul></ul>
  20. 20. The Function to Be Tested <ul><li>Consistency checker </li></ul><ul><ul><li>Consistency constraints are formally defined in first order logic </li></ul></ul><ul><li>Potential threat to the validity </li></ul><ul><ul><li>The program is not representative. </li></ul></ul><ul><li>Validation of the case study </li></ul><ul><ul><li>The program’s input is structurally complex </li></ul></ul><ul><ul><li>The program is non-trivial </li></ul></ul>4 1 4 Inter-model  8 8 Inter-diagram   10 Intra-diagram Intra-model Global Local Vertical Consistency Horizontal Consistency Table 1. Summary of CAMLE’s Consistency Constraints
  21. 21. Types of Data Mutation Operators Delete an existing env node in a sub-collaboration diagram Delete env node 12 Generate a sub-collaboration diagram for an existing node Add sub diagram 11 Replace an existing node with a new node of another type Change node type 10 Rename an existing node in a diagram Rename node 9 Delete an existing node in a diagram Delete node 8 Replicate an existing node in a diagram Replicate node 7 Add an edge of some type to a diagram Add edge 6 Add a node and link it to an existing node Add node with edge 5 Add a node of some type to a diagram Add node 4 Change the title of an existing diagram Rename diagram 3 Delete an existing diagram Delete diagram 2 Add a collaboration or behaviour or scenario diagram Add diagram 1 Description Operator type No.
  22. 22. Change the Start or End node of an existing edge to an env node Change edge end to env 24 Delete the Action List of an existing interaction edge Delete edge annotation 23 Change the Action List annotated to an existing interaction edge Change edge annotation 22 Replicate an existing interaction edge with Action List Replicate interaction 21 Replicate an existing interaction edge without Action List Replicate interaction edge 20 Replace an existing edge in a diagram with a new edge of another type Change edge type 19 Reverse the direction of an existing edge Change edge direction 18 Change the Start or End node of an existing edge Change edge association 17 Delete an existing edge in a diagram Delete edge 16 Replicate an existing non-interaction edge Replicate edge 15 Remove an annotation on an existing node Delete node annotation 14 Rename an existing environment node in a sub-collaboration diagram Rename env node 13
  23. 23. The Seed Test Cases <ul><li>Models developed in previous case studies of agent-oriented software development methodology </li></ul><ul><ul><li>The evolutionary multi-agent Internet information retrieval system Amalthaea (originally developed at MIT media lab); </li></ul></ul><ul><ul><li>Online auction web service; </li></ul></ul><ul><ul><li>The agent-oriented model of the United Nation’s Security Council on the organisational structure and the work procedure to pass resolutions at UNSC. </li></ul></ul><ul><li>All seeds passed consistency check before the case study started </li></ul><ul><li>No change was made to these seeds in this case study </li></ul>
  24. 24. The Seed Test Cases and Their Mutants 7808 1082 3260 3466 Number of Mutants 257 38 99 110 #Edges 352 54 140 158 #Nodes 34 7 13 14 #Diagrams Total 11 0 1 10 #Edges 26 0 4 22 #Nodes 3 0 1 2 #Diagrams Scenario Diagram 170 28 75 67 #Edges 270 43 115 112 #Nodes 16 2 6 8 #Diagrams Behaviour Diagram 49 6 17 26 #Edges 37 8 14 15 #Nodes 12 4 5 3 #Diagrams Collaboration Diagram 17 4 6 7 # Edges 19 3 7 9 #Nodes 3 1 1 1 #Diagrams Caste Diagram Total UNSC Auction Amalthaea
  25. 25. The Results: Fault Detecting Ability 5 114 (97%) 61 (52%) 118 Total 0 19 (100%) 12 (63%) 19 Transposition of statements 1 14 (93%) 9 (60%) 15 Incorrect expression 0 31 (100%) 13 (42%) 31 Omission of statements 0 21 (88%) 14 (58%) 24 Incorrect variable Computation 2 17 (100%) 8 (47%) 17 Path selection 2 12 (100%) 5 (42%) 12 Missing path Domain Indigenous Inserted By mutants By seeds No. of Detected Faults No. of Inserted Faults Fault Type
  26. 26. Detecting Design Errors <ul><li>In the case study, we found that a large number of mutants remain alive </li></ul>Table . The numbers of alive and dead mutants <ul><li>Review: Three possible reasons: </li></ul><ul><li>improper design of data mutation operators, </li></ul><ul><li>insufficient observation on the behaviour and output </li></ul><ul><li>defects in the software under test. </li></ul>16.47% 6522 1286 7808 Total 15.43% 915 167 1082 UNSC 12.94% 2838 422 3260 Auction 20.11% 2769 697 3466 Amalthaea %Dead #Alive #Dead #Mutant Seed
  27. 27. Statistics on Amalthaea test suite Some typed mutation score is very low Design of consistency checker has errors! Especially, the consistency constraints are weak. … … … … … 0% 22 0 22 Replicate edge 0% 39 0 39 Delete annotation on node 100% 0 4 4 Rename environment node 100% 0 4 4 Delete environment node 100% 0 8 8 Add sub diagram 39% 37 24 61 Change node type 63% 46 77 123 Rename node 25% 110 37 147 Delete node 0% 130 0 130 Replicate node 13% 1205 173 1378 Add edge 64% 22 39 61 Combine node 16% 74 14 88 Add node 100% 0 9 9 Rename diagram 22% 7 2 9 Delete diagram 67% 1 2 3 Add diagram %Dead #Live #Dead #Total Operator type
  28. 28. Results: Detecting Design Errors <ul><li>Hypothesis </li></ul><ul><ul><li>Design of the tool is weak in detecting certain types of inconsistency or incompleteness </li></ul></ul><ul><li>Validation of the hypothesis </li></ul><ul><ul><li>Strengthening the well-formedness constraints </li></ul></ul><ul><ul><li>Strengthening the consistency constraints: 3 constraints modified </li></ul></ul><ul><ul><li>Introducing new completeness constraints: 13 new constraints introduced </li></ul></ul><ul><ul><li>Test again using the same seeds and the same mutation operators </li></ul></ul><ul><ul><li>A significant change in the statistic data is observed. </li></ul></ul>85.18% 1060 6092 7152 Total 82.76% 171 821 992 UNSC 83.33% 516 2579 3095 Auction 87.83% 373 2692 3065 Amalthaea %Dead #Alive #Dead #Mutant Seed Table. The statistics of alive and dead mutants after modification
  29. 29. Test Adequacy <ul><li>Our experiments show that high test adequacy can be achieved through data mutation. </li></ul><ul><ul><li>Coverage of input data space </li></ul></ul><ul><ul><ul><li>Measured by the coverage of various kinds of mutants </li></ul></ul></ul><ul><ul><li>Coverage of program structure </li></ul></ul><ul><ul><ul><li>Measured by code coverage (equivalent to the branches covered) </li></ul></ul></ul><ul><ul><li>Coverage of the functions of the requirements </li></ul></ul><ul><ul><ul><li>Measured by the consistency constraints used in checking </li></ul></ul></ul><ul><li>Two factors the determines the test adequacy: </li></ul><ul><ul><li>the seeds </li></ul></ul><ul><ul><li>the mutation operators </li></ul></ul>
  30. 30. Coverage of scenario diagram variants 40 0 0 40 17 11 0 1 10 16 10 0 2 8 14 10 0 2 8 10 24 0 4 20 9 24 0 4 20 8 24 0 4 20 7 24 0 0 24 6 17 0 3 14 5 21 0 7 14 4 3 0 1 2 3 3 0 1 2 2 3 1 1 1 1 Total UNSC Auction Amalthaea Mutation operator type
  31. 31. Coverage of Program Structure and Functions The test data achieved 100% coverage of the functions of the consistency checker and 100% of the branches in the code.
  32. 32. Test Cost The seeds were readily available from previous case studies of the tool. Table. Summary of the test cost spent in the case study 2 man-month (estimated) Analysis of program correctness on each test case 0 man-month Development of seed test cases 1.5 man-month Design and implementation of data mutation operators Amount in case study Source of cost
  33. 33. Analysis Program’s Correctness <ul><li>The experiment took the black-box approach </li></ul><ul><ul><li>The output on a test case consists of </li></ul></ul><ul><ul><ul><li>Whether the input (a model) is consistent and complete </li></ul></ul></ul><ul><ul><ul><li>The error message(s) and/or warning message(s), if any </li></ul></ul></ul><ul><ul><li>The expected output on a mutant is specified </li></ul></ul>14, (Interaction edges in the main collaboration diagram) E016 5 6, (Caste nodes in the main collaboration diagram) E004 2 5, (Agent nodes in the main collaboration diagram) E003 1 Add a new Collaboration diagram / Top of model 1 #Messages, Message Content Message ID Violated Constraint Expected Output Operator /Location Mutant No.
  34. 34. Experiments <ul><li>The experiments </li></ul><ul><ul><li>Mutants are selected at random </li></ul></ul><ul><ul><li>The program’s correctness on each mutant is checked manually </li></ul></ul><ul><ul><li>Time is measured for how long it needs to check the correctness of the program on each test case </li></ul></ul><ul><ul><li>Two experiments were conducted </li></ul></ul><ul><li>Experiment 1 </li></ul><ul><ul><li>1 mutant selected at random from each set of the mutants generated by one type of mutation operator (24 mutants in total) </li></ul></ul><ul><ul><li>Detected 2 faults in the checker and 1 fault in other parts of the tool </li></ul></ul><ul><li>Experiment 2 </li></ul><ul><ul><li>22 live mutants from the Amalthaea suite selected at random </li></ul></ul><ul><ul><li>Detected 2 faults in the other parts of the tool </li></ul></ul>
  35. 35. The Experiment Data <ul><li>Results: </li></ul><ul><ul><li>Checking correctness on dead mutants: 3 minute/per mutant </li></ul></ul><ul><ul><li>Checking correctness on live mutants: 1 minute/per mutant </li></ul></ul>2 11 Dead Non-equivalent 1 1 Alive Non-equivalent 0 0 Dead Equivalent 2 34 Alive Equivalent #Detected Faults #Mutants Aliveness Type of Mutant
  36. 36. Related Works <ul><li>Mutation testing </li></ul><ul><ul><li>Program or specification is modified </li></ul></ul><ul><ul><li>Used as a criteria to measure test adequacy </li></ul></ul><ul><ul><li>Data mutation testing adopted the idea of mutation operators, but applied to test cases to generate test case, rather than to measure adequacy. </li></ul></ul><ul><li>Meek and Siu (1989) </li></ul><ul><ul><li>Randomisation in error seeding into programs to test compiler </li></ul></ul><ul><li>Adaptive Random Testing (Chen, et al. 2003, 2004) </li></ul><ul><ul><li>Random test cases as far apart as possible </li></ul></ul><ul><ul><li>Not yet applied to structurally complex input space </li></ul></ul><ul><li>Data perturbation testing (Offutt, 2001) </li></ul><ul><ul><li>Test XML message for web services </li></ul></ul><ul><ul><li>As a application specific technique and applicable to XML files </li></ul></ul><ul><li>Metamorphic testing (Chen, Tse, et al. 2003) </li></ul><ul><ul><li>As a test oracle automation technique and focus on the metamorphic relations rather than to generate test cases </li></ul></ul><ul><ul><li>Could be integrated with data mutation method </li></ul></ul>
  37. 37. Future Work <ul><li>More case studies with potential applications </li></ul><ul><ul><li>Security control software: Role-Base Access Control </li></ul></ul><ul><ul><ul><li>Input: Role model, User assignments </li></ul></ul></ul><ul><ul><ul><li>< Roles , Resources , Permissions : Role  Resources , </li></ul></ul></ul><ul><ul><ul><li>Constraints  Roles X Resources X Permissions > </li></ul></ul></ul><ul><ul><ul><li>User assignments : Users  P (Roles ) </li></ul></ul></ul><ul><ul><li>Virus detection </li></ul></ul><ul><ul><ul><li>Input: files infected by virus </li></ul></ul></ul><ul><ul><ul><ul><li>Virus are programs in assembly/binary code format </li></ul></ul></ul></ul><ul><ul><ul><ul><li>One virus may have many variants obtained by equivalent transformation of the code. </li></ul></ul></ul></ul><ul><ul><li>Spreadsheet processing software and spreadsheets applications </li></ul></ul><ul><ul><ul><li>Input: spreadsheets <data cells, program cells> </li></ul></ul></ul>
  38. 38. Perspectives and Future Work <ul><li>Integration of data mutation testing, metamorphic testing and algebraic testing methods </li></ul>Let be the program under test Data mutation testing generates test cases using a set of data mutation operators Metamorphic testing used a set of metamorphic relations to check output correctness We can use  i to define metamorphic relations as follows:
  39. 39. Example <ul><li>Consider the Triangle Classification program P </li></ul><ul><ul><li>The following is a metamorphic relation </li></ul></ul><ul><ul><li>P ( t )= equilateral  P ( IPV ( t )) = isosceles </li></ul></ul><ul><ul><li>For each of the data mutation operators  = WXY, WXZ, WYZ, RPL, or RPR, the following is a metamorphic relation </li></ul></ul><ul><ul><li>P (  ( t )) =P ( t ) </li></ul></ul>We observed in case study that data mutation operators are very helpful to find metamorphic relations.
  40. 40. Integration with Algebraic Testing <ul><li>In algebraic software testing, axioms are written in the form of </li></ul><ul><ul><li>T 1 = T’ 1 ^ T 2 = T’ 2 ^ … ^ T n = T’ n => T=T’, </li></ul></ul><ul><ul><li>Where T i , T’ i are terms constructed from variables and function/procedure/methods of the program under test. </li></ul></ul><ul><li>The integration of data mutation testing, metamorphic testing and algebraic testing by developing </li></ul><ul><ul><li>A black box software testing specification language </li></ul></ul><ul><ul><li>An automated tool to check metamorphic relations </li></ul></ul><ul><ul><li>Using observation context to check if a relation is true </li></ul></ul><ul><ul><li>To allow user defined data mutation operators to be invoked </li></ul></ul><ul><ul><li>To allow metamorphic relations to be specified </li></ul></ul>
  41. 41. Screen Snapshot of Algebraic Testing Tool CASCAT
  42. 42. References <ul><li>Lijun Shan and Hong Zhu, Generating Structurally Complex Test Cases by Data Mutation: A Case Study of Testing an Automated Modelling Tool , Special Issue on Automation of Software Test, the Computer Journal, (In press). </li></ul><ul><li>Shan, L. and Zhu, H., Testing Software Modelling Tools Using Data Mutation , Proc. of AST’06, ACM Press, 2006, pp43-49. </li></ul><ul><li>Zhu, H. and Shan, L., Caste-Centric Modelling of Multi-Agent Systems: The CAMLE Modelling Language and Automated Tools , in Beydeda, S. and Gruhn, V. (eds) Model-driven Software Development, Research and Practice in Software Engineering, Vol. II, Springer, 2005, pp57-89. </li></ul><ul><li>Liang Kong, Hong Zhu and Bin Zhou, Automated Testing EJB Components Based on Algebraic Specifications , Proc. of TEST’07, IEEE CS Press, 2007. </li></ul>

×