Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study

303 views

Published on

The role of software testing in the software development process is widely recognized as a key activity for successful projects. This is the reason why in the last decade several automatic unit test generation tools have been proposed, focusing particularly on high code coverage. Despite the effort spent by the research community, there is still a lack of empirical investigation aimed at analyzing the characteristics of the produced test code. Indeed, while some studies inspected the effectiveness and the usability of these tools in practice, it is still unknown whether test code is maintainable. In this paper, we conducted a large scale empirical study in order to analyze the diffusion of bad design solutions, namely test smells, in automatically generated unit test classes. Results of the study show the high diffusion of test smells as well as the frequent co-occurrence of different types of design problems. Finally we found that all test smells have strong positive correlation with structural characteristics of the systems such as size or number of classes.

Published in: Software
  • Login to see the comments

  • Be the first to like this

On the Diffusion of Test Smells in Automatically Generated Test Code: An Empirical Study

  1. 1. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Fabio Palomba*, Dario Di Nucci*,Annibale Panichella+, Rocco Oliveto°,Andrea De Lucia* * University of Salerno, Italy +Delft University ofTechnology,The Netherlands ° University of Molise, Italy SBST 2016 May, 16th 2016 Austin,TX, USA
  2. 2. Automatic Generation ofTest Code method1() method2() … methodN() Production Class test_method1() test_method2() … test_methodN() Test Suite
  3. 3. Automatic Generation ofTest Code [Arcuri and Fraser - SSBSE’14] Effectiveness of Whole Suite Test Case Generation [Panichella et al. - ICSE’16] Impact ofTest Case Summaries on Bug Fixing Performance [Shamshiri et al. - ASE’15] Effectiveness of Generated Test Cases on Effectiveness
  4. 4. Automatic Generation ofTest Code [Rojas et al. - ISSTA’15] Usability of Automatic GenerationTools in Practice
  5. 5. Automatic Generation ofTest Code What about the characteristics of test code produced by such tools?
  6. 6. Test Smells inTest Code [Van Deursen et al. - XP 2001] “Test Smells represent a set of a poor design solutions to write tests ” 11test smells related to the way developers write test fixtures and test cases
  7. 7. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); }
  8. 8. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); } The test method checks the production method isGround()
  9. 9. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); } But also the production method isVariable()
  10. 10. Test Smells inTest Code public void test12 () throws Throwable { JSTerm jSTerm0 = new JSTerm();
 jSTerm0.makeVariable () ; jSTerm0.add((Object) ””); jSTerm0.matches(jSTerm0);
 assertEquals (false, jSTerm0.isGround ()); assertEquals(true, jSTerm0.isVariable()); } This is an EagerTest, namely a test which checks more than one method of the class to be tested, making difficult the comprehension of the actual test target.
  11. 11. Test Smells inTest Code A test case is affected by a Resource Optimism when it makes assumptions about the state or the existence of external resources, providing a non-deterministic result that depend on the state of the resources. An Assertion Roulette comes from having a number of assertions in a test method that have no explanation. If an assertion fails, the identification of the assert that failed can be difficult.
  12. 12. Who cares aboutTest Smells? Test Cases can be re-generated!
  13. 13. Who cares aboutTest Smells? Test Cases can be re-generated! True
  14. 14. Who cares aboutTest Smells? Test Cases can be re-generated! True BUT
  15. 15. Who cares aboutTest Smells? Test Cases can be re-generated! Developers modify and remove test code Developers add tests when automatic tools leave uncovered branches Developers combine generated with manually written tests [Rojas et al. - ISSTA’15] Usability of Automatic GenerationTools in Practice True BUT
  16. 16. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Empirical Study Design
  17. 17. Empirical Study Design 8test smell types “RefactoringTest Code” [Van Deursen et al. - XP 2001]
  18. 18. Empirical Study Design 110software projects 8test smell types [Fraser and Arcuri -TOSEM 2014] “A Large Scale Evaluation of Automated Unit Test Generation using Evosuite” “RefactoringTest Code” [Van Deursen et al. - XP 2001]
  19. 19. Empirical Study Design 110software projects 8test smell types [Fraser and Arcuri -TOSEM 2014] “A Large Scale Evaluation of Automated Unit Test Generation using Evosuite” “RefactoringTest Code” [Van Deursen et al. - XP 2001]
  20. 20. Empirical Study Design 16,603JUnit classes [Fraser and Arcuri -TOSEM 2014] “A Large Scale Evaluation of Automated Unit Test Generation using Evosuite”
  21. 21. Data Extraction test_method1() test_method2() … test_methodN() Test Suite [Bavota et al. - EMSE 2015] “AreTest Smells Harmful? An Empirical Study” Test Smell Detector
  22. 22. Data Extraction test_method1() test_method2() … test_methodN() Test Suite [Bavota et al. - EMSE 2015] “AreTest Smells Harmful? An Empirical Study” Test Smell Detector 75% precision 100% recall Sample size: 378 JUnit classes
  23. 23. Research Questions RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? ?
  24. 24. RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? ? Research Questions
  25. 25. RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? RQ3:WhichTest Smells Co-OccurTogether? ? Research Questions
  26. 26. RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? RQ3:WhichTest Smells Co-OccurTogether? RQ4: IsThere a Relationship Between the Presence ofTest Smells and the Project Characteristics? ? Research Questions
  27. 27. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Analysis of the Results
  28. 28. Results of the Study RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? !13,791smelly JUnit classes
  29. 29. Results of the Study RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? !83%of the JUnit classes analyzed
  30. 30. Results of the Study RQ1:To What ExtentTest Smells are Spread in Automatically GeneratedTest Classes? !RQ1 Test Smells are highly diffused in the automatically generated test suites
  31. 31. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !Assertion Roulette 54%
  32. 32. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !Assertion Roulette 54% Test Code Duplication 33%
  33. 33. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !Assertion Roulette 54% Test Code Duplication EagerTest 33% 29%
  34. 34. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !public void test8 () throws Throwable { Document document0 = new Document();
 assertNotNull(document0); document0.procText.add((Character) ”s”); String string0 = document0.stringify();
 assertEquals (“s”, document0.stringify()); 
 assertNotNull(string0);
 assertEquals(“s”, string0); } Assertion Roulette
  35. 35. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study ! Assertion Roulette What is the behavior under test? Are the generated assertions valid?
  36. 36. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study ! Assertion Roulette [Panichella et al. - ICSE’16] The Impact ofTest Case Summaries on Bug Fixing Performance An Empirical Investigation These problems have a huge impact on developers’ ability to find faults
  37. 37. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study !public void test8 () throws Throwable { GenericProperties generic0 = new GenericProperties();
 boolean boolean0 = generic0.isValidClassname(); … } Test Code Duplication public void test9 () throws Throwable { GenericProperties generic0 = new GenericProperties();
 boolean boolean0 = generic0.isValidClassname(); … }
  38. 38. RQ2:WhichTest Smells Occur More Frequently in Automatically GeneratedTest Classes? Results of the Study ! Test Code Duplication This problem can be avoided by generating test fixtures!
  39. 39. Results of the Study ! RQ3:WhichTest Smells Co-OccurTogether? Assertion Roulette Eager Test Assertion Roulette Sensitive Equality Resource Optimism Mystery Guest
  40. 40. Results of the Study ! RQ3:WhichTest Smells Co-OccurTogether? Assertion Roulette Eager Test Assertion Roulette Sensitive Equality Resource Optimism Mystery Guest Automatic tools have as main goal that of maximize coverage, without considering test code quality
  41. 41. RQ4: IsThere a Relationship Between the Presence ofTest Smells and the Project Characteristics? Results of the Study !Yes! The higher the LOC to be tested, the higher the probability to produce a smelly test case!
  42. 42. RQ4: IsThere a Relationship Between the Presence ofTest Smells and the Project Characteristics? Results of the Study !Yes! The higher the LOC to be tested, the higher the probability to produce a smelly test case! The higher the LOC of the JUnit class, the higher the probability to produce a smelly test case!
  43. 43. Summarizing Lesson 1: Current implementations of search-based algorithms for automatic test case generation do not consider code quality, increasing the probability to introduce smells!
  44. 44. Summarizing Lesson 1: Current implementations of search-based algorithms for automatic test case generation do not consider code quality, increasing the probability to introduce smells! NB: Considering test code quality is important not only to avoid the introduction of smells, but also because the coverage can be increased! [F. Palomba,A. Panichella,A. Zaidman, R. Oliveto,A. De Lucia - ISSTA’16] AutomaticTest Case Generation:What IfTest Code Quality Matters?
  45. 45. Summarizing Lesson 2:Automatic test case generation tools do not produce text fixtures during their computation, and this implies the introduction of several code clones in the resulting JUnit classes. Future research should spend effort in the automatic generation of test fixtures!
  46. 46. From now on… ? Challenge 1: EvaluatingTest Smells inTest Cases automatically generated by other tools
  47. 47. From now on… ? Challenge 1: EvaluatingTest Smells inTest Cases automatically generated by other tools Challenge 2: Defining new algorithms able to solve the design problems analyzed (e.g., test fixtures).
  48. 48. On the Diffusion ofTest Smells in Automatically GeneratedTest Code:An Empirical Study Fabio Palomba*, Dario Di Nucci*,Annibale Panichella+, Rocco Oliveto°,Andrea De Lucia* * University of Salerno, Italy +Delft University ofTechnology,The Netherlands ° University of Molise, Italy SBST 2016 May, 16th 2016 Austin,TX, USA

×