Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Combinatorial Interaction Testing, an Introduction - 2018

23 views

Published on

Invited talk to the CSE1110 Software Quality and Testing course, Delft University of Technology

Published in: Software
  • Be the first to comment

  • Be the first to like this

Combinatorial Interaction Testing, an Introduction - 2018

  1. 1. Combinatorial Interaction Testing (CIT) An introduction Xavier Devroey - x.d.m.devroey@tudelft.nl
  2. 2. public class Message {
     private long id = -1;     private String author;     private String message;
     public Message(String author, String message) {         this.author = author;         this.message = message;     }     // Gets and Sets ...     /**      * Appends the given text at the end of this' message.      * A space is added before the given text.      *      * @param text The test to append.      */     public void compose(String text) {         this.message = message + " " + text;     } }
  3. 3. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     }
  4. 4. Code coverage Line coverage: percentage of lines covered by tests https://github.com/cobertura/cobertura Achieving line coverage 
 = 
 executing each line at least once
  5. 5. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     } Is the compose method covered by the test case?
  6. 6. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     } Is the compose method covered by the test case? Yes
  7. 7. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     } Is the compose method covered by the test case? Is it enough? Yes
  8. 8. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     } Is the compose method covered by the test case? Is it enough? Yes No
  9. 9. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     } Is the compose method covered by the test case? Is it enough? Yes No There are no assertions on the result/effect of the method
  10. 10. Input/output domain coverage public void compose(String text); (based on pre/post conditions)
  11. 11. Input/output domain coverage public void compose(String text); • Equivalence partitioning (based on pre/post conditions)
  12. 12. Input/output domain coverage public void compose(String text); • Equivalence partitioning Normal case: "Despicable you" (based on pre/post conditions)
  13. 13. Input/output domain coverage public void compose(String text); • Equivalence partitioning Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" (based on pre/post conditions)
  14. 14. Input/output domain coverage public void compose(String text); public void compose(String t1, String t2); • Equivalence partitioning Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" (based on pre/post conditions)
  15. 15. Input/output domain coverage public void compose(String text); public void compose(String t1, String t2); • Equivalence partitioning Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" • t1 • t2 Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" (based on pre/post conditions)
  16. 16. Input/output domain coverage public void compose(String text); public void compose(String t1, String t2); • Equivalence partitioning Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" • t1 • t2 Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" (based on pre/post conditions) public void compose(String t1, String t2, String t3);
  17. 17. Input/output domain coverage public void compose(String text); public void compose(String t1, String t2); • Equivalence partitioning Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" • t1 • t2 Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" ☛ Combinatorial explosion with the number of parameters (based on pre/post conditions) public void compose(String t1, String t2, String t3);
  18. 18. Combinatorial Interaction Testing (CIT) Hyp.: Most of the faults are due to undesired interactions of a limited number (N) of factors (here, equivalent classes of parameter values). D. Kuhn et al, Software fault interactions and implications for software testing. TSE 2004 • In practice: N ≤ 4 • Usually: N = 2 (pairwise)
  19. 19. Combinatorial Interaction Testing (CIT) Hyp.: Most of the faults are due to undesired interactions of a limited number (N) of factors (here, equivalent classes of parameter values). D. Kuhn et al, Software fault interactions and implications for software testing. TSE 2004 • In practice: N ≤ 4 • Usually: N = 2 (pairwise) Idea: Generate test cases such that all combinations of N values are covered at least once.
  20. 20. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values
  21. 21. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  22. 22. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  23. 23. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  24. 24. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  25. 25. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  26. 26. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  27. 27. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  28. 28. http://cse.unl.edu/~citportal/ http://ctweb.abstracta.com.uy/combinatorial.jsp Hyp.: most faults are caused by interactions of at most t factors T-wise testing: cover all t-uples of values
  29. 29. http://cse.unl.edu/~citportal/ http://ctweb.abstracta.com.uy/combinatorial.jsp Hyp.: most faults are caused by interactions of at most t factors T-wise testing: cover all t-uples of values Integration and System testing
  30. 30. app. http://cse.unl.edu/~citportal/ http://ctweb.abstracta.com.uy/combinatorial.jsp Hyp.: most faults are caused by interactions of at most t factors T-wise testing: cover all t-uples of values Integration and System testing
  31. 31. app. http://cse.unl.edu/~citportal/ http://ctweb.abstracta.com.uy/combinatorial.jsp Hyp.: most faults are caused by interactions of at most t factors T-wise testing: cover all t-uples of values Integration and System testing
  32. 32. app. http://cse.unl.edu/~citportal/ http://ctweb.abstracta.com.uy/combinatorial.jsp Hyp.: most faults are caused by interactions of at most t factors T-wise testing: cover all t-uples of values Integration and System testing Which configuration should I use for my tests?
  33. 33. type Android version Screen Processor tablet 4.0.4 XLarge ARM Cortex smartphone 4.1 Large Intel x86 phablet 4.1.1 Normal 4.1.2 Small
  34. 34. type Android version Screen Processor tablet 4.0.4 XLarge ARM Cortex smartphone 4.1 Large Intel x86 phablet 4.1.1 Normal 4.1.2 Small … or using a feature model http://familiar-project.github.io https://featureide.github.io http://www.skalup.com/ http://research.henard.net/SPL/PLEDGE/ http://martinfjohansen.com/models2011/spltool/ http://cse.unl.edu/~citportal/
  35. 35. app. Hyp.: most faults are caused by interactions of at most t factors T-wise testing: cover all t-uples of values Integration and System testing Which configuration should I use for my tests? Under … • tests requirements • $$$ budget • test execution time budget … constraints
  36. 36. https://console.firebase.google.com

×