SlideShare a Scribd company logo
1 of 37
Download to read offline
Fastest
Test Case Generation from Z Specifications

                 Maximiliano Cristi´
                                   a

                         CIFASIS
             Universidad Nacional de Rosario
                   Rosario – Argentina
          cristia@cifasis-conicet.gov.ar

         Alpen-Adria Univesit¨t – Klagenfurt – March, 2012
                             a
Software Testing Basics



  Program Correctness
  A program is correct if it behaves according to its specification

  Edsger Dijkstra
  “Program testing can be a very effective way to show the presence
  of bugs, but it is hopelessly inadequate for showing their absence.”

  Donald Knuth
  “Beware of bugs in the above code; I have only proved it correct,
  not tried it.”




                                                                         2 / 34
Model-Based Testing (MBT)



    MBT analyses a model of the           MBT Process
    program to derive test cases which     requeriments
    later are provided as input.
    At the beginning, the model is used       model                       error?
    to generate test cases.                                           verify
                                                  generate
    At end, the model becomes the
                                             abstract                    abstract
    oracle.
                                              tests                       results
    If a model is provided, the process
                                                                    abstract
    is almost automatic.                          refine
                                                          execute
                                              tests                       results




                                                                                    3 / 34
The Test Template Framework and Fastest


  The Test Template Framework (TTF)
      A particular method of MBT which uses Z specifications.
      Unit testing.
      Stocks and Carrington, A Framework for Specification-Based
      Testing, IEEE TOSE 1996.

  Fastest
       It is the only existing implementation of the TTF.
      Available at www.fceia.unr.edu.ar/~mcristia.
      Cristi´ and Rodr´
            a         ıguez Monetti, Implementing and Applying
      the Stocks-Carrington Framework for MBT, ICFEM 2009.



                                                                  4 / 34
TTF Automation


The TTF in 9 Steps                                    MBT Process
  1   Partition the IS by applying testing tactics.    requeriments

  2   Generate a testing tree
  3   Prune the testing tree.                             model                       error?
                                                                                  verify
  4   Generate abstract test cases.                           generate

           (Optionally) Translate test cases into        abstract                    abstract
           natural language.                              tests                       results
  5   Refine abstract test cases.
                                                              refine             abstract
  6   Execute test cases.                                             execute
                                                          tests                       results
  7   Get the program output.
  8   Abstract the program output.
  9   Check for errors.


                                                                                                5 / 34
An Example




  The Requirements
      A bank has a certain number of savings accounts.
      Customers can deposit and withdraw money.
      Each account is identified with an account number.
      An amount of money is considered to be a natural number.
      Only the deposit operation is going to be shown.




                                                                 6 / 34
The Z Specification
Types and State Space




         [ACCNUM]         A type for account numbers

         BALANCE == N A type for balances
         AMOUNT == N      A type for amounts of money



           Bank
           sa : ACCNUM → BALANCE




                                                        7 / 34
The Z Specification
The Deposit Operation


           DepositOk
           ∆Bank
           n? : ACCNUM
           a? : AMOUNT
           n? ∈ dom sa
           a? > 0
           sa = sa ⊕ {n? → sa n? + a?}


        DepositE 1 == [ΞBank; n? : ACCNUM | n? ∈ dom sa]
                                               /
        DepositE 2 == [ΞBank; a? : Z | a? ≤ 0]
        Deposit == DepositOk ∨ DepositE 1 ∨ DepositE 2

    loadspec bank.tex
    selop Deposit
                                                           8 / 34
The TTF in 9 Steps                                    MBT Process
  1   Partition the IS by applying testing tactics.    requeriments

  2   Generate a testing tree.
  3   Prune the testing tree.                             model                       error?
                                                                                  verify
  4   Generate abstract test cases.                           generate

           (Optionally) Translate test cases into        abstract                    abstract
           natural language.                              tests                       results
  5   Refine abstract test cases.
                                                              refine             abstract
  6   Execute test cases.                                             execute
                                                          tests                       results
  7   Get the program output.
  8   Abstract the program output.
  9   Check for errors.
Domain Partition

  Given a Z operation, take its input space and get a partition of it.


      input      testing       input           testing         input   testing
      space       tactic       space            tactic        space     tactic


                           P             Q               P P1    Q Q1
                           atomic predicates



  Each member of a partition is given by a Z predicate (i.e.,
  first-order logic and Zermelo-Fraenkel set theory).

      Partitioning can be represented as a tree.
      Test cases are derived only from the “leaves”
      Leaves are called test conditions.
                                                                             10 / 34
Testing Tactics
  The TTF performs domain partition by applying testing tactics.
  Disjunctive Normal Form (DNF)
  Write the operation predicate in DNF and take the precondition of
  each disjunct.

  Standard Partition (SP) for R ⊕ G
  R   = {}, G   = {}
  R   = {}, G   = {}
  R   = {}, G   = {}
  R   = {}, G   = {}, dom R = dom G
  R   = {}, G   = {}, dom G ⊂ dom R
  R   = {}, G   = {}, (dom R ∩ dom G ) = {}
  R   = {}, G   = {}, dom R ⊂ dom G
  R   = {}, G   = {}, (dom R ∩ dom G ) = {},
                        ¬ (dom G ⊆ dom R), ¬ (dom R ⊆ dom G )

                                                                      11 / 34
Applying DNF and SP to Deposit
  First DNF
         DNF == [Deposit VIS | n? ∈ dom sa ∧ a? > 0]
  Deposit1
         DNF == [Deposit VIS | n? ∈ dom sa]
  Deposit2                        /
         DNF == [Deposit VIS | a? ≤ 0]
  Deposit3

                                DNF
  Second SP, but only to Deposit1
         SP
  Deposit1               DNF
              == [Deposit1        | sa = {} ∧ {n? → (sa n? + a?)} = {}]
         SP
  Deposit2               DNF
              == [Deposit1        | sa = {} ∧ {n? → (sa n? + a?)} = {}]
         SP
  Deposit3               DNF
              == [Deposit1        | sa = {} ∧ {n? → (sa n? + a?)} = {}]
         SP
  Deposit4               DNF
              == [Deposit1        | sa = {} ∧ {n? → (sa n? + a?)} = {}
                                  ∧ dom sa = dom{n? → (sa n? + a?)}]
         SP           DNF
  Deposit5 == [Deposit1           | sa = {} ∧ {n? → (sa n? + a?)} = {}
                                  ∧ dom{n? → (sa n? + a?)} ⊂ dom sa]
  (three more test conditions not shown here)

  genalltt
  addtactic Deposit_DNF_1 SP oplus sa oplus {(n? , sa~n? + a?)}
                                                                          12 / 34
The TTF in 9 Steps                                    MBT Process
  1   Partition the IS by applying testing tactics.    requeriments

  2   Generate a testing tree
  3   Prune the testing tree.                             model                       error?
                                                                                  verify
  4   Generate abstract test cases.                           generate

           (Optionally) Translate test cases into        abstract                    abstract
           natural language.                              tests                       results
  5   Refine abstract test cases.
                                                              refine             abstract
  6   Execute test cases.                                             execute
                                                          tests                       results
  7   Get the program output.
  8   Abstract the program output.
  9   Check for errors.
Pruning the Testing Tree

  Pruning involves eliminating unsatisfiable test conditions.

 Before pruning                       After pruning
   Deposit VIS                           Deposit VIS
      Deposit DNF 1                         Deposit DNF 1
         Deposit SP     1                      Deposit SP 4
         Deposit SP     2                      Deposit SP 5
         Deposit SP     3                   Deposit DNF 2
         Deposit SP     4                   Deposit DNF 3
         Deposit SP     5
         Deposit SP     6
                                       prunett
         Deposit SP     7
         Deposit SP     8
      Deposit DNF 2
      Deposit DNF 3

                                                               14 / 34
Pruning in Detail


  The problem – I
      Many test conditions are unsatisfiable.
      Most of them are mathematical contradictions.
      Then, no test cases can be derived from them.
      Therefore, unsatisfiable test conditions should be pruned from
      testing trees.
      In general, this problem is undecidable.

  The problem – II
  Z includes many mathematical theories (sets, relations, partial
  functions, sequences, etc.), making the problem harder.



                                                                      15 / 34
Detecting Contradictions


  Detecting logical contradictions
  Since every test condition is a conjunction of atomic predicates,
  then the only possible logical contradiction is of the form:

      ··· ∧ p ∧ ··· ∧ ¬ p ∧ ...

  This is the easy part.

  Detecting mathematical contradictions
      Most of the contradictions are of a mathematical nature.
      Some depend on the particular semantics of the Z
      mathematical theories.



                                                                      16 / 34
A Library of Mathematical Contradictions

      The core of our method is a library of elimination theorems.
      Each elimination theorem should represent a contradiction.

  ETheorem ETheoremName [parameters]
      atomicPred
      ...............
      atomicPred

      parameters is roughly any Z declaration.
      atomicPred is any Z atomic predicate and some keywords.
      Subtyping and equivalence rules.
                       A
      User extensible (LTEX).
      No deduction, only pattern-matching.

                                                                     17 / 34
Some examples

     Update SP 3              ETheorem T1 [x : X ]
     st : SYM → VAL
     s? : SYM                      setExtension(x) = {}
     v ? : VAL
                              setExtension(x) matches any set
     st = {}
                              extension containing x
     {s? → v ?} = {}



     Update SP 7              ETheorem T2
     st : SYM → VAL           [R : X ↔ Y ; x : X ; y : Y ]
     s? : SYM
     v ? : VAL                     R = {}
     st = {}
                                   dom R ⊂ dom{x → y }
     {s? → v ?} = {}
     dom st ⊂ dom{s? → v ?}   subtyping is applied to st




                                                                18 / 34
Some examples

     Update SP 3              ETheorem T1 [x : X ]
     st : SYM → VAL
     s? : SYM                      setExtension(x) = {}
     v ? : VAL
                              setExtension(x) matches any set
     st = {}
                              extension containing x
     {s? → v ?} = {}



     Update SP 7              ETheorem T2
     st : SYM → VAL           [R : X ↔ Y ; x : X ; y : Y ]
     s? : SYM
     v ? : VAL                     R = {}
     st = {}
                                   dom R ⊂ dom{x → y }
     {s? → v ?} = {}
     dom st ⊂ dom{s? → v ?}   subtyping is applied to st




                                                                18 / 34
Some examples

     Update SP 3              ETheorem T1 [x : X ]
     st : SYM → VAL
     s? : SYM                      setExtension(x) = {}
     v ? : VAL
                              setExtension(x) matches any set
     st = {}
                              extension containing x
     {s? → v ?} = {}



     Update SP 7              ETheorem T2
     st : SYM → VAL           [R : X ↔ Y ; x : X ; y : Y ]
     s? : SYM
     v ? : VAL                     R = {}
     st = {}
                                   dom R ⊂ dom{x → y }
     {s? → v ?} = {}
     dom st ⊂ dom{s? → v ?}   subtyping is applied to st




                                                                18 / 34
Some examples

     Update SP 3              ETheorem T1 [x : X ]
     st : SYM → VAL
     s? : SYM                      setExtension(x) = {}
     v ? : VAL
                              setExtension(x) matches any set
     st = {}
                              extension containing x
     {s? → v ?} = {}



     Update SP 7              ETheorem T2
     st : SYM → VAL           [R : X ↔ Y ; x : X ; y : Y ]
     s? : SYM
     v ? : VAL                     R = {}
     st = {}
                                   dom R ⊂ dom{x → y }
     {s? → v ?} = {}
     dom st ⊂ dom{s? → v ?}   subtyping is applied to st




                                                                18 / 34
The TTF in 9 Steps                                    MBT Process
  1   Partition the IS by applying testing tactics.    requeriments

  2   Generate a testing tree
  3   Prune the testing tree.                             model                       error?
                                                                                  verify
  4   Generate abstract test cases.                           generate

           (Optionally) Translate test cases into        abstract                    abstract
           natural language.                              tests                       results
  5   Refine abstract test cases.
                                                              refine             abstract
  6   Execute test cases.                                             execute
                                                          tests                       results
  7   Get the program output.
  8   Abstract the program output.
  9   Check for errors.
Generating Abstract Test Cases


  Find constants satisfying test conditions.
         TC
  Deposit1 ==
            SP
   [Deposit4 | n? = accnum2 ∧ sa = {(accnum2 , 1)} ∧ a? = 1]
         TC
  Deposit2 ==
            SP
   [Deposit5 |
     n? = accnum2 ∧ sa = {(accnum2 , 1), (accnum1 , 1)} ∧ a? = 1]
         TC           DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0]
  Deposit3 == [Deposit2                2
         TC           DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0]
  Deposit4 == [Deposit3                2



  genalltca




                                                                    20 / 34
Test Case Generation in Detail
  Currently, Fastest implements a rough satisfiability algorithm:
    1   Finite sets are bound to variables and types in a test template.

                 ACCNUM, n? → {accnum0 , accnum1 , accnum2 }
   AMOUNT , BALANCE , a? → {0, 1, 2}
  ACCNUM → BALANCE , sa → {∅,
                                        {accnum2 → 1},
                                        {accnum2 → 1, accnum1 → 1},
                                        ...}

    2   The predicate is evaluated on some elements of the Cartesian
        product of those finite sets.
            ZLive (from CZT) is used as evaluator.
            Some simple heuristics are applied.
            The mathematical structure of the predicate is not considered.

                                                                             21 / 34
Is it good enough?


  Fastest has been able to find test cases for an average of 80 % of
  the satisfiable test conditions from eleven case studies.


  For the remaining 20 %, Fastest provides a command to help it to
  find a test case.

  setfinitemodel Deposit_SP_4 -fm "sa=={{accnum_2 mapsto 1}}"




  Nevertheless, Fastest’s algorithm needs to be improved or replaced
  by more sophisticated ones.



                                                                       22 / 34
Difficulties with Test Case Generation in the TTF



       The ideas of test templates and of automatic partition of
       Z schema have been proposed several years ago. Their
       actual application to systems of industrial size was
       problematic due to the lack of efficient and adequate
       constraint solvers and theorem provers. Given the
       progresses recently achieved for these kinds of tools,
       these techniques are worth revisiting . . .
  Ana Cavalcanti and Marie-Claude Gaudel (2011): Testing for refinement in
  Circus. Acta Informatica, 48(2).




                                                                            23 / 34
SMT Solvers for the TTF

  Satisfiability Modulo Theory (SMT) solvers
  SMT solvers are tools that solve the problem of satisfiability for a
  range of mathematical and logical theories.

      No SMT solver supports the Zermelo-Fraenkel set theory.
      Must define a shallow embedding.
      Many questions:
           Is the language of SMT solvers expressive enough?
           Is there just one shallow embedding?
           Will the chosen embedding solve all the satisfiable test
           specifications appearing in the TTF and real Z specifications?
           Which SMT solver and which embedding will be the best in
           satisfying more test specifications in less time?
           Will it be better than Fastest in finding test cases?
           Or should the SMT solver complement Fastest in this task?

                                                                          24 / 34
First Results




  Two shallow embeddings
  We have defined a shallow embedding of the Zermelo-Franekel set
  theory for two mainstream SMT solvers: Yices and CVC3.

  Initial empirical assessment
  We manually translated 69 satisfiable test templates, for which
  Fastest cannot find a test case, into these shallow embeddings and
  ran the SMT solvers over them.




                                                                      25 / 34
Empirical assessment

                             Main embeddings       Variant embeddings
       Case study           Yices      CVC3         Yices      CVC3
                          Sat Unk Sat Unk         Sat Unk Sat Unk
   Savings accounts (3)     8          8            8          8
   Savings accounts (1)           2           2           2           2
   Launching vehicle              8    8                  8    8
   Plavis                        29          29          29         29
   SWPDC                         13          13          13         13
   Scheduler                      4           4           4           4
   Security class                 4           4           4           4
   Pool of sensors          1          1            1          1
   Totals                  9     60   17     52     9    60   17    52

  Time spent in processing all the 69 test templates:
      Yices: 3 s
      CVC3: 420 s
      Fastest: 390 s (but no test case is discovered)


                                                                          26 / 34
The TTF in 9 Steps                                    MBT Process
  1   Partition the IS by applying testing tactics.    requeriments

  2   Generate a testing tree
  3   Prune the testing tree.                             model                       error?
                                                                                  verify
  4   Generate abstract test cases.                           generate

           (Optionally) Translate test cases into        abstract                    abstract
           natural language.                              tests                       results
  5   Refine abstract test cases.
                                                              refine             abstract
  6   Execute test cases.                                             execute
                                                          tests                       results
  7   Get the program output.
  8   Abstract the program output.
  9   Check for errors.
Refining Abstract Test Cases

  The implementation (C programming language)
  Assume sa is implemented as follows:
       struct sanode
            {int an;
              float bal;
              struct sanode *n;} *sa;
  where sa is a global variable. Besides, Deposit is implemented by
  the following function:
       int deposit(int n, float a)

  Refinement rules
      We need to link each specification variable to their
      implementation counterparts.
      This process is called abstract test case refinement.
      Write refinement rules and then run the refine command.
                                                                      28 / 34
Refining Abstract Test Cases (II)


  A refinement rule for Deposit
  @RRULE deposit
  @PREAMBLE
  #include <bank.h>
  @LAWS
  l1:n? ==> n
  l2:a? ==> a
  l3:sa ==> sa AS LIST[SLL,n] WITH[sa.@dom ==> sa.an,
                                   sa.@ran ==> sa.bal]
  @UUT deposit(n, a)


  loadrefrule deposit.txt
  refine Deposit to test /bank/src implemented in C with deposit



                                                                   29 / 34
Concrete Test Cases
                      TC
  Refinement of Deposit2 .

  #include <bank.h>
  int main() {
   int n = 345; n? = accnum2
   float a = 1; a? = 1

  sa = {(accnum2, 1), (accnum1, 1)}
   struct sanode sanode0 = {876,1,NULL};
   struct sanode sanode1 = {345,1,NULL};
   sa = &sanode0;
   sanode0.n = &sanode1;

      deposit(n,a);

      return 1;
  }
                                           30 / 34
The Remaining Steps

      Before executing a concrete test case it is necessary to insert
      code to get the program output.
           Abstraction rules, similar to refinement rules.
      Once a test case has been executed its output must be
      abstracted.
           Same abstraction rules, different command.
      The final verification is performed with another command.
      (Optional) Translating abstract test cases into natural
      language involves writing translation rules.
           Quite different from refinement rules.

  Work in progress
  Fastest is still a prototype: many of these features are roughly
  implemented and there can be bugs.


                                                                        31 / 34
Our Experience with Fastest

  We have applied Fastest in the following projects (only up to
  abstract test case generation):
  On-board flight control software of a satellite launcher vehicle
      Along with Instituto de Aeron´utica e Espa¸o (IAE), Brazil.
                                   a            c
      139 lines of Z.
      One operation, 36 abstract test cases.

  Satellite communication protocols
       Two protocols, along with Instituto Nacional de Pesquisas
       Espaciais (INPE), Brazil.
      Two specifications comprising a total of 1,800 lines of Z.
      26 operations, 195 abstract test cases.


                                                                    32 / 34
Our Experience with Fastest (II)

  The ECSS-E-70-41A aerospace standard
      Along with INVAP, Argentina.
      A single specification of 2,011 lines of Z.
      Abstract test case translation to English.
      256 test conditions.
      Many challenges to solve.

  Tokeneer
      First specification not written by us, 2,057 lines of Z.
      139 schemas selected as operations.
      More than 25,000 test conditions, only DNF was applied.
      We had to modify the DNF algorithm due to “term
      explosion”.

                                                                33 / 34
Current and Future Work



     Continue with the SMT solvers.
         Z3.
         More satisfiable test specifications.
         A decision procedure for the Z notation?
     Test case refinement: greatly improve the implementation.
     Z specifications as program annotations.
     Implement the remaining steps of the TTF.
     Validate with more case studies.




                                                                34 / 34

More Related Content

What's hot

Unit testingandcontinousintegrationfreenest1dot4
Unit testingandcontinousintegrationfreenest1dot4Unit testingandcontinousintegrationfreenest1dot4
Unit testingandcontinousintegrationfreenest1dot4JAMK
 
Ekeko Technology Showdown at SoTeSoLa 2012
Ekeko Technology Showdown at SoTeSoLa 2012Ekeko Technology Showdown at SoTeSoLa 2012
Ekeko Technology Showdown at SoTeSoLa 2012Coen De Roover
 
Формальная верификация как средство тестирования (в Java)
Формальная верификация как средство тестирования (в Java)Формальная верификация как средство тестирования (в Java)
Формальная верификация как средство тестирования (в Java)SQALab
 
The SOUL Tool Suite for Querying Programs in Symbiosis with Eclipse
The SOUL Tool Suite for Querying Programs in Symbiosis with EclipseThe SOUL Tool Suite for Querying Programs in Symbiosis with Eclipse
The SOUL Tool Suite for Querying Programs in Symbiosis with EclipseCoen De Roover
 
Dart function - Recursive functions
Dart function - Recursive functionsDart function - Recursive functions
Dart function - Recursive functionsKoAungThuOo1
 
Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...
Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...
Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...Radoslaw Smilgin
 
Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...
Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...
Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...Radoslaw Smilgin
 
Handling Exceptions In C &amp; C++[Part A]
Handling Exceptions In C &amp; C++[Part A]Handling Exceptions In C &amp; C++[Part A]
Handling Exceptions In C &amp; C++[Part A]ppd1961
 
Www javatpoint-com-corejava-interview-questions-2
Www javatpoint-com-corejava-interview-questions-2Www javatpoint-com-corejava-interview-questions-2
Www javatpoint-com-corejava-interview-questions-2ssuserd2d58b
 
Unit Testing Standards - Recommended Best Practices
Unit Testing Standards - Recommended Best PracticesUnit Testing Standards - Recommended Best Practices
Unit Testing Standards - Recommended Best PracticesVitaliy Kulikov
 
Session 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfacesSession 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfacesNirav Desai
 
System Verilog Functional Coverage
System Verilog Functional CoverageSystem Verilog Functional Coverage
System Verilog Functional Coveragerraimi
 
Std 12 Computer Chapter 7 Java Basics (Part 2)
Std 12 Computer Chapter 7 Java Basics (Part 2)Std 12 Computer Chapter 7 Java Basics (Part 2)
Std 12 Computer Chapter 7 Java Basics (Part 2)Nuzhat Memon
 
Session 7 code_functional_coverage
Session 7 code_functional_coverageSession 7 code_functional_coverage
Session 7 code_functional_coverageNirav Desai
 
Test and refactoring
Test and refactoringTest and refactoring
Test and refactoringKenneth Ceyer
 

What's hot (20)

Unit testingandcontinousintegrationfreenest1dot4
Unit testingandcontinousintegrationfreenest1dot4Unit testingandcontinousintegrationfreenest1dot4
Unit testingandcontinousintegrationfreenest1dot4
 
Ekeko Technology Showdown at SoTeSoLa 2012
Ekeko Technology Showdown at SoTeSoLa 2012Ekeko Technology Showdown at SoTeSoLa 2012
Ekeko Technology Showdown at SoTeSoLa 2012
 
Формальная верификация как средство тестирования (в Java)
Формальная верификация как средство тестирования (в Java)Формальная верификация как средство тестирования (в Java)
Формальная верификация как средство тестирования (в Java)
 
The SOUL Tool Suite for Querying Programs in Symbiosis with Eclipse
The SOUL Tool Suite for Querying Programs in Symbiosis with EclipseThe SOUL Tool Suite for Querying Programs in Symbiosis with Eclipse
The SOUL Tool Suite for Querying Programs in Symbiosis with Eclipse
 
Doulos coverage-tips-tricks
Doulos coverage-tips-tricksDoulos coverage-tips-tricks
Doulos coverage-tips-tricks
 
Dart function - Recursive functions
Dart function - Recursive functionsDart function - Recursive functions
Dart function - Recursive functions
 
Dacj 1-3 b
Dacj 1-3 bDacj 1-3 b
Dacj 1-3 b
 
Dacj 1-3 a
Dacj 1-3 aDacj 1-3 a
Dacj 1-3 a
 
Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...
Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...
Decision Testing and Decision Coverage. ISTQB Whitebox techniques with TestCo...
 
Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...
Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...
Statement Testing and Statement Coverage. ISTQB whitebox techniques with Test...
 
Handling Exceptions In C &amp; C++[Part A]
Handling Exceptions In C &amp; C++[Part A]Handling Exceptions In C &amp; C++[Part A]
Handling Exceptions In C &amp; C++[Part A]
 
Www javatpoint-com-corejava-interview-questions-2
Www javatpoint-com-corejava-interview-questions-2Www javatpoint-com-corejava-interview-questions-2
Www javatpoint-com-corejava-interview-questions-2
 
Coverage and Introduction to UVM
Coverage and Introduction to UVMCoverage and Introduction to UVM
Coverage and Introduction to UVM
 
Unit Testing Standards - Recommended Best Practices
Unit Testing Standards - Recommended Best PracticesUnit Testing Standards - Recommended Best Practices
Unit Testing Standards - Recommended Best Practices
 
Session 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfacesSession 8 assertion_based_verification_and_interfaces
Session 8 assertion_based_verification_and_interfaces
 
System Verilog Functional Coverage
System Verilog Functional CoverageSystem Verilog Functional Coverage
System Verilog Functional Coverage
 
Std 12 Computer Chapter 7 Java Basics (Part 2)
Std 12 Computer Chapter 7 Java Basics (Part 2)Std 12 Computer Chapter 7 Java Basics (Part 2)
Std 12 Computer Chapter 7 Java Basics (Part 2)
 
Paper-review: A Parallel Test Pattern Generation Algorithm to Meet Multiple Q...
Paper-review: A Parallel Test Pattern Generation Algorithm to Meet Multiple Q...Paper-review: A Parallel Test Pattern Generation Algorithm to Meet Multiple Q...
Paper-review: A Parallel Test Pattern Generation Algorithm to Meet Multiple Q...
 
Session 7 code_functional_coverage
Session 7 code_functional_coverageSession 7 code_functional_coverage
Session 7 code_functional_coverage
 
Test and refactoring
Test and refactoringTest and refactoring
Test and refactoring
 

Similar to Fastest Test Case Generation

Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010Clemens Reijnen
 
Software Testing - Invited Lecture at UNSW Sydney
Software Testing - Invited Lecture at UNSW SydneySoftware Testing - Invited Lecture at UNSW Sydney
Software Testing - Invited Lecture at UNSW Sydneyjulien.ponge
 
Software Testing Tecniques
Software Testing TecniquesSoftware Testing Tecniques
Software Testing Tecniquesersanbilik
 
Junit Recipes - Intro
Junit Recipes - IntroJunit Recipes - Intro
Junit Recipes - IntroWill Shen
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Developmentguy_davis
 
Test driven development
Test driven developmentTest driven development
Test driven developmentJohn Walsh
 
Tdd pecha kucha_v2
Tdd pecha kucha_v2Tdd pecha kucha_v2
Tdd pecha kucha_v2Paul Boos
 
SE2_Lec 21_ TDD and Junit
SE2_Lec 21_ TDD and JunitSE2_Lec 21_ TDD and Junit
SE2_Lec 21_ TDD and JunitAmr E. Mohamed
 
Software testing ari force institute of tech.
Software testing ari force institute of tech.Software testing ari force institute of tech.
Software testing ari force institute of tech.Sanjith Ml
 
Behaviour Driven Development and Thinking About Testing
Behaviour Driven Development and Thinking About TestingBehaviour Driven Development and Thinking About Testing
Behaviour Driven Development and Thinking About Testingdn
 
Bdd and-testing
Bdd and-testingBdd and-testing
Bdd and-testingmalcolmt
 
What's software testing
What's software testingWhat's software testing
What's software testingLi-Wei Cheng
 
2016 10-04: tdd++: tdd made easier
2016 10-04: tdd++: tdd made easier2016 10-04: tdd++: tdd made easier
2016 10-04: tdd++: tdd made easierChristian Hujer
 
Fu agile#2 unit_testing
Fu agile#2 unit_testingFu agile#2 unit_testing
Fu agile#2 unit_testingNguyen Anh
 

Similar to Fastest Test Case Generation (20)

Repair dagstuhl jan2017
Repair dagstuhl jan2017Repair dagstuhl jan2017
Repair dagstuhl jan2017
 
Writing tests
Writing testsWriting tests
Writing tests
 
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
 
TDD in Javascript
TDD in JavascriptTDD in Javascript
TDD in Javascript
 
Why do a designed experiment
Why do a designed experimentWhy do a designed experiment
Why do a designed experiment
 
Software Testing - Invited Lecture at UNSW Sydney
Software Testing - Invited Lecture at UNSW SydneySoftware Testing - Invited Lecture at UNSW Sydney
Software Testing - Invited Lecture at UNSW Sydney
 
Computer Engineer Master Project
Computer Engineer Master ProjectComputer Engineer Master Project
Computer Engineer Master Project
 
Software Testing Tecniques
Software Testing TecniquesSoftware Testing Tecniques
Software Testing Tecniques
 
Junit Recipes - Intro
Junit Recipes - IntroJunit Recipes - Intro
Junit Recipes - Intro
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Tdd pecha kucha_v2
Tdd pecha kucha_v2Tdd pecha kucha_v2
Tdd pecha kucha_v2
 
SE2_Lec 21_ TDD and Junit
SE2_Lec 21_ TDD and JunitSE2_Lec 21_ TDD and Junit
SE2_Lec 21_ TDD and Junit
 
Software testing ari force institute of tech.
Software testing ari force institute of tech.Software testing ari force institute of tech.
Software testing ari force institute of tech.
 
10 software testing_technique
10 software testing_technique10 software testing_technique
10 software testing_technique
 
Behaviour Driven Development and Thinking About Testing
Behaviour Driven Development and Thinking About TestingBehaviour Driven Development and Thinking About Testing
Behaviour Driven Development and Thinking About Testing
 
Bdd and-testing
Bdd and-testingBdd and-testing
Bdd and-testing
 
What's software testing
What's software testingWhat's software testing
What's software testing
 
2016 10-04: tdd++: tdd made easier
2016 10-04: tdd++: tdd made easier2016 10-04: tdd++: tdd made easier
2016 10-04: tdd++: tdd made easier
 
Fu agile#2 unit_testing
Fu agile#2 unit_testingFu agile#2 unit_testing
Fu agile#2 unit_testing
 

More from Förderverein Technische Fakultät

The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...Förderverein Technische Fakultät
 
Engineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdfEngineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdfFörderverein Technische Fakultät
 
The Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdfThe Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdfFörderverein Technische Fakultät
 
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...Förderverein Technische Fakultät
 
East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...Förderverein Technische Fakultät
 
Advances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksAdvances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksFörderverein Technische Fakultät
 
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdfIndustriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdfFörderverein Technische Fakultät
 

More from Förderverein Technische Fakultät (20)

Supervisory control of business processes
Supervisory control of business processesSupervisory control of business processes
Supervisory control of business processes
 
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
The Digital Transformation of Education: A Hyper-Disruptive Era through Block...
 
A Game of Chess is Like a Swordfight.pdf
A Game of Chess is Like a Swordfight.pdfA Game of Chess is Like a Swordfight.pdf
A Game of Chess is Like a Swordfight.pdf
 
From Mind to Meta.pdf
From Mind to Meta.pdfFrom Mind to Meta.pdf
From Mind to Meta.pdf
 
Miniatures Design for Tabletop Games.pdf
Miniatures Design for Tabletop Games.pdfMiniatures Design for Tabletop Games.pdf
Miniatures Design for Tabletop Games.pdf
 
Distributed Systems in the Post-Moore Era.pptx
Distributed Systems in the Post-Moore Era.pptxDistributed Systems in the Post-Moore Era.pptx
Distributed Systems in the Post-Moore Era.pptx
 
Don't Treat the Symptom, Find the Cause!.pptx
Don't Treat the Symptom, Find the Cause!.pptxDon't Treat the Symptom, Find the Cause!.pptx
Don't Treat the Symptom, Find the Cause!.pptx
 
Engineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdfEngineering Serverless Workflow Applications in Federated FaaS.pdf
Engineering Serverless Workflow Applications in Federated FaaS.pdf
 
The Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdfThe Role of Machine Learning in Fluid Network Control and Data Planes.pdf
The Role of Machine Learning in Fluid Network Control and Data Planes.pdf
 
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
Nonequilibrium Network Dynamics_Inference, Fluctuation-Respones & Tipping Poi...
 
Towards a data driven identification of teaching patterns.pdf
Towards a data driven identification of teaching patterns.pdfTowards a data driven identification of teaching patterns.pdf
Towards a data driven identification of teaching patterns.pdf
 
Förderverein Technische Fakultät.pptx
Förderverein Technische Fakultät.pptxFörderverein Technische Fakultät.pptx
Förderverein Technische Fakultät.pptx
 
The Computing Continuum.pdf
The Computing Continuum.pdfThe Computing Continuum.pdf
The Computing Continuum.pdf
 
East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...East-west oriented photovoltaic power systems: model, benefits and technical ...
East-west oriented photovoltaic power systems: model, benefits and technical ...
 
Machine Learning in Finance via Randomization
Machine Learning in Finance via RandomizationMachine Learning in Finance via Randomization
Machine Learning in Finance via Randomization
 
IT does not stop
IT does not stopIT does not stop
IT does not stop
 
Advances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial NetworksAdvances in Visual Quality Restoration with Generative Adversarial Networks
Advances in Visual Quality Restoration with Generative Adversarial Networks
 
Recent Trends in Personalization at Netflix
Recent Trends in Personalization at NetflixRecent Trends in Personalization at Netflix
Recent Trends in Personalization at Netflix
 
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdfIndustriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
Industriepraktikum_ Unterstützung bei Projekten in der Automatisierung.pdf
 
Introduction to 5G from radio perspective
Introduction to 5G from radio perspectiveIntroduction to 5G from radio perspective
Introduction to 5G from radio perspective
 

Recently uploaded

So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...itnewsafrica
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 

Recently uploaded (20)

So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 

Fastest Test Case Generation

  • 1. Fastest Test Case Generation from Z Specifications Maximiliano Cristi´ a CIFASIS Universidad Nacional de Rosario Rosario – Argentina cristia@cifasis-conicet.gov.ar Alpen-Adria Univesit¨t – Klagenfurt – March, 2012 a
  • 2. Software Testing Basics Program Correctness A program is correct if it behaves according to its specification Edsger Dijkstra “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” Donald Knuth “Beware of bugs in the above code; I have only proved it correct, not tried it.” 2 / 34
  • 3. Model-Based Testing (MBT) MBT analyses a model of the MBT Process program to derive test cases which requeriments later are provided as input. At the beginning, the model is used model error? to generate test cases. verify generate At end, the model becomes the abstract abstract oracle. tests results If a model is provided, the process abstract is almost automatic. refine execute tests results 3 / 34
  • 4. The Test Template Framework and Fastest The Test Template Framework (TTF) A particular method of MBT which uses Z specifications. Unit testing. Stocks and Carrington, A Framework for Specification-Based Testing, IEEE TOSE 1996. Fastest It is the only existing implementation of the TTF. Available at www.fceia.unr.edu.ar/~mcristia. Cristi´ and Rodr´ a ıguez Monetti, Implementing and Applying the Stocks-Carrington Framework for MBT, ICFEM 2009. 4 / 34
  • 5. TTF Automation The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors. 5 / 34
  • 6. An Example The Requirements A bank has a certain number of savings accounts. Customers can deposit and withdraw money. Each account is identified with an account number. An amount of money is considered to be a natural number. Only the deposit operation is going to be shown. 6 / 34
  • 7. The Z Specification Types and State Space [ACCNUM] A type for account numbers BALANCE == N A type for balances AMOUNT == N A type for amounts of money Bank sa : ACCNUM → BALANCE 7 / 34
  • 8. The Z Specification The Deposit Operation DepositOk ∆Bank n? : ACCNUM a? : AMOUNT n? ∈ dom sa a? > 0 sa = sa ⊕ {n? → sa n? + a?} DepositE 1 == [ΞBank; n? : ACCNUM | n? ∈ dom sa] / DepositE 2 == [ΞBank; a? : Z | a? ≤ 0] Deposit == DepositOk ∨ DepositE 1 ∨ DepositE 2 loadspec bank.tex selop Deposit 8 / 34
  • 9. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree. 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  • 10. Domain Partition Given a Z operation, take its input space and get a partition of it. input testing input testing input testing space tactic space tactic space tactic P Q P P1 Q Q1 atomic predicates Each member of a partition is given by a Z predicate (i.e., first-order logic and Zermelo-Fraenkel set theory). Partitioning can be represented as a tree. Test cases are derived only from the “leaves” Leaves are called test conditions. 10 / 34
  • 11. Testing Tactics The TTF performs domain partition by applying testing tactics. Disjunctive Normal Form (DNF) Write the operation predicate in DNF and take the precondition of each disjunct. Standard Partition (SP) for R ⊕ G R = {}, G = {} R = {}, G = {} R = {}, G = {} R = {}, G = {}, dom R = dom G R = {}, G = {}, dom G ⊂ dom R R = {}, G = {}, (dom R ∩ dom G ) = {} R = {}, G = {}, dom R ⊂ dom G R = {}, G = {}, (dom R ∩ dom G ) = {}, ¬ (dom G ⊆ dom R), ¬ (dom R ⊆ dom G ) 11 / 34
  • 12. Applying DNF and SP to Deposit First DNF DNF == [Deposit VIS | n? ∈ dom sa ∧ a? > 0] Deposit1 DNF == [Deposit VIS | n? ∈ dom sa] Deposit2 / DNF == [Deposit VIS | a? ≤ 0] Deposit3 DNF Second SP, but only to Deposit1 SP Deposit1 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit2 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit3 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {}] SP Deposit4 DNF == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {} ∧ dom sa = dom{n? → (sa n? + a?)}] SP DNF Deposit5 == [Deposit1 | sa = {} ∧ {n? → (sa n? + a?)} = {} ∧ dom{n? → (sa n? + a?)} ⊂ dom sa] (three more test conditions not shown here) genalltt addtactic Deposit_DNF_1 SP oplus sa oplus {(n? , sa~n? + a?)} 12 / 34
  • 13. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  • 14. Pruning the Testing Tree Pruning involves eliminating unsatisfiable test conditions. Before pruning After pruning Deposit VIS Deposit VIS Deposit DNF 1 Deposit DNF 1 Deposit SP 1 Deposit SP 4 Deposit SP 2 Deposit SP 5 Deposit SP 3 Deposit DNF 2 Deposit SP 4 Deposit DNF 3 Deposit SP 5 Deposit SP 6 prunett Deposit SP 7 Deposit SP 8 Deposit DNF 2 Deposit DNF 3 14 / 34
  • 15. Pruning in Detail The problem – I Many test conditions are unsatisfiable. Most of them are mathematical contradictions. Then, no test cases can be derived from them. Therefore, unsatisfiable test conditions should be pruned from testing trees. In general, this problem is undecidable. The problem – II Z includes many mathematical theories (sets, relations, partial functions, sequences, etc.), making the problem harder. 15 / 34
  • 16. Detecting Contradictions Detecting logical contradictions Since every test condition is a conjunction of atomic predicates, then the only possible logical contradiction is of the form: ··· ∧ p ∧ ··· ∧ ¬ p ∧ ... This is the easy part. Detecting mathematical contradictions Most of the contradictions are of a mathematical nature. Some depend on the particular semantics of the Z mathematical theories. 16 / 34
  • 17. A Library of Mathematical Contradictions The core of our method is a library of elimination theorems. Each elimination theorem should represent a contradiction. ETheorem ETheoremName [parameters] atomicPred ............... atomicPred parameters is roughly any Z declaration. atomicPred is any Z atomic predicate and some keywords. Subtyping and equivalence rules. A User extensible (LTEX). No deduction, only pattern-matching. 17 / 34
  • 18. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  • 19. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  • 20. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  • 21. Some examples Update SP 3 ETheorem T1 [x : X ] st : SYM → VAL s? : SYM setExtension(x) = {} v ? : VAL setExtension(x) matches any set st = {} extension containing x {s? → v ?} = {} Update SP 7 ETheorem T2 st : SYM → VAL [R : X ↔ Y ; x : X ; y : Y ] s? : SYM v ? : VAL R = {} st = {} dom R ⊂ dom{x → y } {s? → v ?} = {} dom st ⊂ dom{s? → v ?} subtyping is applied to st 18 / 34
  • 22. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  • 23. Generating Abstract Test Cases Find constants satisfying test conditions. TC Deposit1 == SP [Deposit4 | n? = accnum2 ∧ sa = {(accnum2 , 1)} ∧ a? = 1] TC Deposit2 == SP [Deposit5 | n? = accnum2 ∧ sa = {(accnum2 , 1), (accnum1 , 1)} ∧ a? = 1] TC DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0] Deposit3 == [Deposit2 2 TC DNF | n? = accnum ∧ sa = ∅ ∧ a? = 0] Deposit4 == [Deposit3 2 genalltca 20 / 34
  • 24. Test Case Generation in Detail Currently, Fastest implements a rough satisfiability algorithm: 1 Finite sets are bound to variables and types in a test template. ACCNUM, n? → {accnum0 , accnum1 , accnum2 } AMOUNT , BALANCE , a? → {0, 1, 2} ACCNUM → BALANCE , sa → {∅, {accnum2 → 1}, {accnum2 → 1, accnum1 → 1}, ...} 2 The predicate is evaluated on some elements of the Cartesian product of those finite sets. ZLive (from CZT) is used as evaluator. Some simple heuristics are applied. The mathematical structure of the predicate is not considered. 21 / 34
  • 25. Is it good enough? Fastest has been able to find test cases for an average of 80 % of the satisfiable test conditions from eleven case studies. For the remaining 20 %, Fastest provides a command to help it to find a test case. setfinitemodel Deposit_SP_4 -fm "sa=={{accnum_2 mapsto 1}}" Nevertheless, Fastest’s algorithm needs to be improved or replaced by more sophisticated ones. 22 / 34
  • 26. Difficulties with Test Case Generation in the TTF The ideas of test templates and of automatic partition of Z schema have been proposed several years ago. Their actual application to systems of industrial size was problematic due to the lack of efficient and adequate constraint solvers and theorem provers. Given the progresses recently achieved for these kinds of tools, these techniques are worth revisiting . . . Ana Cavalcanti and Marie-Claude Gaudel (2011): Testing for refinement in Circus. Acta Informatica, 48(2). 23 / 34
  • 27. SMT Solvers for the TTF Satisfiability Modulo Theory (SMT) solvers SMT solvers are tools that solve the problem of satisfiability for a range of mathematical and logical theories. No SMT solver supports the Zermelo-Fraenkel set theory. Must define a shallow embedding. Many questions: Is the language of SMT solvers expressive enough? Is there just one shallow embedding? Will the chosen embedding solve all the satisfiable test specifications appearing in the TTF and real Z specifications? Which SMT solver and which embedding will be the best in satisfying more test specifications in less time? Will it be better than Fastest in finding test cases? Or should the SMT solver complement Fastest in this task? 24 / 34
  • 28. First Results Two shallow embeddings We have defined a shallow embedding of the Zermelo-Franekel set theory for two mainstream SMT solvers: Yices and CVC3. Initial empirical assessment We manually translated 69 satisfiable test templates, for which Fastest cannot find a test case, into these shallow embeddings and ran the SMT solvers over them. 25 / 34
  • 29. Empirical assessment Main embeddings Variant embeddings Case study Yices CVC3 Yices CVC3 Sat Unk Sat Unk Sat Unk Sat Unk Savings accounts (3) 8 8 8 8 Savings accounts (1) 2 2 2 2 Launching vehicle 8 8 8 8 Plavis 29 29 29 29 SWPDC 13 13 13 13 Scheduler 4 4 4 4 Security class 4 4 4 4 Pool of sensors 1 1 1 1 Totals 9 60 17 52 9 60 17 52 Time spent in processing all the 69 test templates: Yices: 3 s CVC3: 420 s Fastest: 390 s (but no test case is discovered) 26 / 34
  • 30. The TTF in 9 Steps MBT Process 1 Partition the IS by applying testing tactics. requeriments 2 Generate a testing tree 3 Prune the testing tree. model error? verify 4 Generate abstract test cases. generate (Optionally) Translate test cases into abstract abstract natural language. tests results 5 Refine abstract test cases. refine abstract 6 Execute test cases. execute tests results 7 Get the program output. 8 Abstract the program output. 9 Check for errors.
  • 31. Refining Abstract Test Cases The implementation (C programming language) Assume sa is implemented as follows: struct sanode {int an; float bal; struct sanode *n;} *sa; where sa is a global variable. Besides, Deposit is implemented by the following function: int deposit(int n, float a) Refinement rules We need to link each specification variable to their implementation counterparts. This process is called abstract test case refinement. Write refinement rules and then run the refine command. 28 / 34
  • 32. Refining Abstract Test Cases (II) A refinement rule for Deposit @RRULE deposit @PREAMBLE #include <bank.h> @LAWS l1:n? ==> n l2:a? ==> a l3:sa ==> sa AS LIST[SLL,n] WITH[sa.@dom ==> sa.an, sa.@ran ==> sa.bal] @UUT deposit(n, a) loadrefrule deposit.txt refine Deposit to test /bank/src implemented in C with deposit 29 / 34
  • 33. Concrete Test Cases TC Refinement of Deposit2 . #include <bank.h> int main() { int n = 345; n? = accnum2 float a = 1; a? = 1 sa = {(accnum2, 1), (accnum1, 1)} struct sanode sanode0 = {876,1,NULL}; struct sanode sanode1 = {345,1,NULL}; sa = &sanode0; sanode0.n = &sanode1; deposit(n,a); return 1; } 30 / 34
  • 34. The Remaining Steps Before executing a concrete test case it is necessary to insert code to get the program output. Abstraction rules, similar to refinement rules. Once a test case has been executed its output must be abstracted. Same abstraction rules, different command. The final verification is performed with another command. (Optional) Translating abstract test cases into natural language involves writing translation rules. Quite different from refinement rules. Work in progress Fastest is still a prototype: many of these features are roughly implemented and there can be bugs. 31 / 34
  • 35. Our Experience with Fastest We have applied Fastest in the following projects (only up to abstract test case generation): On-board flight control software of a satellite launcher vehicle Along with Instituto de Aeron´utica e Espa¸o (IAE), Brazil. a c 139 lines of Z. One operation, 36 abstract test cases. Satellite communication protocols Two protocols, along with Instituto Nacional de Pesquisas Espaciais (INPE), Brazil. Two specifications comprising a total of 1,800 lines of Z. 26 operations, 195 abstract test cases. 32 / 34
  • 36. Our Experience with Fastest (II) The ECSS-E-70-41A aerospace standard Along with INVAP, Argentina. A single specification of 2,011 lines of Z. Abstract test case translation to English. 256 test conditions. Many challenges to solve. Tokeneer First specification not written by us, 2,057 lines of Z. 139 schemas selected as operations. More than 25,000 test conditions, only DNF was applied. We had to modify the DNF algorithm due to “term explosion”. 33 / 34
  • 37. Current and Future Work Continue with the SMT solvers. Z3. More satisfiable test specifications. A decision procedure for the Z notation? Test case refinement: greatly improve the implementation. Z specifications as program annotations. Implement the remaining steps of the TTF. Validate with more case studies. 34 / 34