Software Logic Mutation Testing Presented by


Published on

  • Be the first to comment

  • Be the first to like this

Software Logic Mutation Testing Presented by

  1. 1. Software Logic Mutation Testing Presented by Gary Kaminski
  2. 2. Overview <ul><li>Introduction and Motivation </li></ul><ul><li>Logic Faults and Detection </li></ul><ul><li>Case Study Comparison </li></ul><ul><li>Context </li></ul><ul><li>Conclusion </li></ul>
  3. 3. Logic Mutation Testing Process <ul><li>Systematically create program variations (mutants) each containing a single typical logic fault </li></ul><ul><li>if (a && b)  if (a || b) </li></ul><ul><li>For each mutant, find an input that distinguishes the original program from the mutant program (killing) </li></ul><ul><li>- a && b and a || b evaluate to different truth values </li></ul><ul><li>Focus is on step 1) </li></ul>
  4. 4. Logic Mutation Testing <ul><li>Logic mutants are created by applying mutation operators to Boolean predicates </li></ul><ul><li>Boolean predicates may come from: </li></ul><ul><li>UML Diagrams </li></ul><ul><li>2) Security Requirements </li></ul><ul><li>3) Looping constructs (while / do-while / for) </li></ul><ul><li>4) Preconditions for methods (if - then) </li></ul>
  5. 5. Motivation <ul><li>Current mutation systems: </li></ul><ul><ul><li>generate excess logic mutants </li></ul></ul><ul><ul><li>do not detect certain classes of logic faults </li></ul></ul><ul><li>Goal: </li></ul><ul><li>- generate fewer logic mutants </li></ul><ul><li>- detect more logic faults </li></ul><ul><li>Assumption: </li></ul><ul><li>- formally restrict attention to minimal Disjunctive Normal Form (DNF) Boolean predicates </li></ul>
  6. 6. Minimal DNF <ul><li>Terms separated by OR, literals by AND </li></ul><ul><li>ab + a!c vs. a(b + !c) </li></ul><ul><li>Make each term true and other terms false </li></ul><ul><li>ab + ac vs. ab + abc </li></ul><ul><li>Impossible to remove a literal or term without changing the predicate </li></ul><ul><li>ab vs. abc + ab!c </li></ul>
  7. 7. Excess Logic Mutants - Example <ul><li>Original: ab + b!c + !bc </li></ul><ul><li>1) mutant generated multiple times: ab + _ + !bc </li></ul><ul><li>2) equivalent mutants: a c + b!c + !bc </li></ul><ul><li>3) mutants guaranteed to be killed by an input that kills some other mutant: </li></ul><ul><li>!a b + b!c + !bc and c b + b!c + !bc </li></ul><ul><li>4) highly effective mutation operators not used: ab c + b!c + !bc </li></ul>
  8. 8. Minimal DNF Logic Faults <ul><li>Original: ab + b!c + !bc </li></ul><ul><li>Expression Negation Fault: ! (ab + b!c + !bc) </li></ul><ul><li>Expression Stuck At Fault 0: FALSE </li></ul><ul><li>Expression Stuck At Fault 1: TRUE </li></ul><ul><li>Term Negation Fault: ! (ab) + b!c + !bc </li></ul><ul><li>Term Omission Fault: ab + _ + !bc </li></ul><ul><li>Scalar Variable Replacement TOF: ab + c !c + !bc </li></ul><ul><li>TOF generates same mutant twice: ab + _ + !bc </li></ul>
  9. 9. Minimal DNF Logic Faults <ul><li>Original: ab + b!c + !bc </li></ul><ul><li>Operator Reference Fault+: a bb !c + !bc </li></ul><ul><li>Operator Reference Fault .: a + b + b!c + !bc </li></ul><ul><li>Operator Reference Fault+0: FALSE + !bc </li></ul><ul><li>Operator Reference Fault+1: TRUE + !bc </li></ul><ul><li>Operator Reference Fault.0: FALSE + b!c + !bc </li></ul><ul><li>Operator Reference Fault.1: TRUE + b!c + !bc </li></ul>
  10. 10. Minimal DNF Logic Faults <ul><li>Original: ab + b!c + !bc </li></ul><ul><li>Literal Negation Fault: !a b + b!c + !bc </li></ul><ul><li>Literal Stuck At Fault 0: FALSE b + b!c + !bc </li></ul><ul><li>Literal Stuck At Fault 1: TRUE b + b!c + !bc </li></ul><ul><li>Literal Omission Fault: _ b + b!c + !bc </li></ul><ul><li>SVR LOF: b b + b!c + !bc </li></ul><ul><li>LOF generates same mutant twice </li></ul><ul><li>original: abc  _ bc, a _ c, a _ c, ab _ </li></ul>
  11. 11. Minimal DNF Logic Faults <ul><li>Original: ab + b!c + !bc </li></ul><ul><li>Literal Reference Fault: a c + b!c + !bc </li></ul><ul><li>Literal Reference Fault: a !c + b!c + !bc </li></ul><ul><li>Literal Insertion Fault: ab c + b!c + !bc </li></ul><ul><li>Literal Insertion Fault: ab !c + b!c + !bc </li></ul><ul><li>LRF and LIF generate equivalent mutants </li></ul>
  12. 12. Logic Fault Minimal DNF Detection Green Arrow – Detection may not hold Black Arrow – Detection always holds Red Box – Mutation Operator does not exist Blue Box – Mutation Operator partially exists Black Box – Mutation Operator exists LOF ORF. LRF LNF TNF LIF TOF LSTF0 ORF.0 ESTF0 ORF+ ORF+0 ENF LSTF1 ORF.1 ORF+1 ESTF1 SVRTOF SVRLOF
  13. 13. Logic Mutation Comparison <ul><li>Original: ab + b!c + !bc </li></ul><ul><li>61 mutants vs. 4 LIFs, 3 LOFs, 3 LRFs </li></ul><ul><li>Equivalent LIF: ab c + b!c + !bc </li></ul><ul><li>LRF: c b + b!c + !bc & a c + b!c + !bc </li></ul><ul><li>LOF: a _ + b!c + !bc </li></ul><ul><li>Equivalent LIF: ab + b!c !a + !bc </li></ul><ul><li>LRFs: ab + !a !c + !bc & ab + b !a + !bc </li></ul><ul><li>No Equivalent LIFs for term !bc </li></ul><ul><li>LOFs: ab + b!c + _ c & ab + b!c + !b _ </li></ul>
  14. 14. Mutation Efficiency Measures <ul><li>Mutation Efficiency Score = Faults Detected - Mutants Generated </li></ul><ul><li>Mutation Efficiency Ratio = Faults Detected / Mutants Generated </li></ul><ul><li>FD assumes an input is found to kill the mutant </li></ul><ul><li>FD includes equivalent mutants detected automatically </li></ul><ul><li>ab + b!c + !bc </li></ul><ul><li>Current Mutation Systems: </li></ul><ul><li>MES: 60 FD - 61 MG = -1 </li></ul><ul><li>MER: 60 FD / 61 MG = .98 </li></ul><ul><li>LIF-LRF-LOF </li></ul><ul><li>MES: 73 FD – 10 MG = 63 </li></ul><ul><li>MER: 73 FD / 10 MG = 7.30 </li></ul>
  15. 15. Case Study <ul><li>Analyzed 19 Boolean predicates in an avionics software system </li></ul><ul><li>Number of unique literals range: 5 to 13 </li></ul><ul><li>Compared MES and MER for current mutation systems vs. LIF-LRF-LOF </li></ul><ul><li>Examined how MES and MER vary based on number of unique literals </li></ul>
  16. 16. Case Study Results <ul><li>LIF-LRF-LOF generated 18% of the mutants current mutation systems would (150 vs. 819) </li></ul><ul><li>Current mutation systems guarantee detecting 75% of faults LIF-LRF-LOF does (819 vs. 1082) </li></ul><ul><li>Number of literals matters for MES but not for MER </li></ul><ul><li>MES = FD – MG and MER = FD / MG </li></ul>7.22 1.00 MER 932 0 MES LIF-LRF-LOF Current
  17. 17. Mutation Efficiency Score (FD – MG) vs. Number of Unique Literals
  18. 18. Mutation Efficiency Ratio (FD/MG) vs. Number of Unique Literals
  19. 19. Context: Internal Variable Problem <ul><li>What input values kill a mutant? </li></ul><ul><li>Mutation may be buried deep in the code </li></ul><ul><li>Must reach predicate and have mutant predicate and original predicate evaluate to different truth values </li></ul><ul><li>Partial solutions using constraints exist </li></ul>
  20. 20. Context: Minimal DNF in Practice <ul><li>95% of 20,256 Boolean predicates in avionics software were in minimal DNF* </li></ul><ul><li>2) Detected > 99% of corresponding faults in non-minimal DNF Boolean predicates* </li></ul><ul><li>*Source: Y.T Yu and M.F. Lau. Comparing Several Coverage Criteria for Detecting Faults in Logical Decisions. In Proceedings QSIC 2004: 4th International Conference on Quality Software, Pages 14-21. </li></ul>
  21. 21. Context: Industry Audience <ul><li>LIF-LRF-LOF for software with Boolean predicates having >= 5 unique literals </li></ul><ul><li>Exhaustive coverage for < 5 unique literals </li></ul><ul><li>ab + b!c + bc (8 tests vs. 10 mutants) </li></ul><ul><li>Avionics software* </li></ul><ul><li>*Source: J.J Chilenski and S.P. Miller. Applicability of modified condition/decision coverage to software testing. IEE/BCS Software Engineering Journal , 9(5): 193-200, September 1994. </li></ul>
  22. 22. Related Work <ul><li>J.J. Chilenski. An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion. Final Technical Report, DOT/FAA/AR-01/18, April 2001. </li></ul><ul><li>Non-Minimal DNF based logic coverage criteria </li></ul><ul><li>2) T.Y. Chen, M.F. Lau, and Y.T. Yu. MUMCUT: A Fault-Based Strategy for Testing Boolean Specifications. Software Engineering Conference, 1999. (APSEC ’99) Proceedings. Sixth Asia Pacific. Pages 606-613. </li></ul><ul><li>Minimal DNF based logic coverage criteria and Double Faults </li></ul><ul><li>3) D. Richard Kuhn. Fault Classes and Error Detection Capability of Specification Based Testing. ACM Transactions on Software Engineering and Methodology, 8(4): 411-424, October 1999. </li></ul><ul><li>Seminal work in logic fault hierarchy </li></ul>
  23. 23. Conclusion <ul><li>Current logic mutation testing is inefficient </li></ul><ul><li>Excess mutants generated and faults are missed </li></ul><ul><li>Fault hierarchy approach based on minimal DNF: detect more faults with less mutants </li></ul><ul><li>Applications in avionics software </li></ul>
  24. 24. Other Research and Publications <ul><li>Logic Coverage Criteria: </li></ul><ul><li>Reduce test set size and maintain fault detection </li></ul><ul><li>Use feasibility of smaller, less expensive component criteria </li></ul><ul><li>Internal variable problem </li></ul><ul><li>CoJava and constraint programming </li></ul><ul><li>Publications: </li></ul><ul><li>G. Kaminski, G. Williams, and P. Ammann. Reconciling Perspectives </li></ul><ul><li>of Logic Testing for Software. Submitted to Journal of Software </li></ul><ul><li>Testing, Verification, and Reliability, January 2008. Online. </li></ul>