Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing

327 views

Published on

The presentation was given at ISSTA 2019.

  • Login to see the comments

  • Be the first to like this

Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing

  1. 1. Effective and Efficient API Misuse Detection via Exception Propagation and Search-Based Testing Maria Kechagia Univ. College London, London, United Kingdom Xavier Devroey Delft Univ. of Technology Delft, Netherlands Annibale Panichella Delft Univ. of Technology Delft, Netherlands Georgios Gousios Delft Univ. of Technology Delft, Netherlands Arie van Deursen Delft Univ. of Technology Delft, Netherlands
  2. 2. APIs everywhere • An Application Programming Interface (API) is a set of functions and procedures offering services • Applications are built by combining API calls • Example: Maven dependency graph with 186,384 Nodes and 1,229,083 Edges • Usage constraints • Explicit • Implicit • API misuses • Violation of usage constraints !2 Maven dependency graph using Force Atlas 2 https://github.com/ogirardot/meta-deps
  3. 3. !3 Client (joda-time) API (Java JDK) 3 calls to nextToken() without any checks https://github.com/JodaOrg/joda-time/pull/319
  4. 4. https://github.com/JodaOrg/joda-time/pull/319 There is an assumption that the input TimeZone file will always be valid, leading to runtime exceptions with no good error message when the file is invalid. !4
  5. 5. !5 Low Low High High Static analysis Dynamic analysis Running time (machine effort) False positives
 (human effort)
  6. 6. !6 Low Low High High Static analysis Dynamic analysis Static + dynamic analysis Running time (machine effort) False positives
 (human effort)
  7. 7. Catcher !7 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses.
  8. 8. Catcher !8 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. 1. Spotting potential runtime exceptions throws API
  9. 9. Catcher !9 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API
  10. 10. Catcher !10 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch)
  11. 11. Catcher !11 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch) 4. Use the candidate misuses as test objectives for a search- based test case generation algorithm tests
  12. 12. Catcher !12 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch) 4. Use the candidate misuses as test objectives for a search- based test case generation algorithm tests 5. Expose the misuses
  13. 13. Static exception propagation !13 Static exception propagation analysis API Client public void a1(…){ … throw new IllegalStateException(); … } public void a2(…){ … } public void a3(…){ … a1(…) … } public void c1(…){ … a3(…); … } public void c2(…){ … } public void c3(…){ … a1(…); … }
  14. 14. Filtering candidate misuses !14 Static exception propagation analysis API Client public void a1(…){ … throw new IllegalStateException(); … } public void a2(…){ … } public void a3(…){ … a1(…) … } public void c1(…){ … a3(…); … } public void c2(…){ … } public void c3(…){ … a1(…); … } Candidate misuses IllegalStateException at a1() at a3() at c1() IllegalStateException at a1() at c3()
  15. 15. Search-based test case generation !15 Candidate misuses IllegalStateException at a1() at a3() at c1() IllegalStateException at a1() at c3() Focused search-based test generation EvoSuite public void c1(…){ … a3(…); … } public void c2(…){ … } public void c3(…){ … a1(…); … } public void testC1(…){ … } public void testC3(…){ … }
  16. 16. Static exception propagation analysis • Builds the API call graph • Annotated with the list of runtime exceptions • Builds the global call graph • Link the client call graph to the API call graph • Propagate exceptions from the API to the client • Maximal propagation depth (≤4) • Filter exceptions handled by the client • Each propagation is a candidate misuse • Relies on Soot (http://sable.github.io/soot/) !16 a1() a2() a3() a4() a5() c1() c2() c3() c4() API Client
  17. 17. Focused Search-based test generation • Dynamic Many-Objective Sorting Algorithm (DynaMOSA) • Objectives • For each misuse • Reaching propagation point (branch coverage) • Client method input diversity • Client method output diversity • Misuse detected if and only if • Exception is thrown by the test case • Propagation is the same as in the candidate misuse • Relies on EvoSuite (http://www.evosuite.org) !17 Random test cases tests Evolutionary process Test cases exposing misuses tests
  18. 18. Evaluation on the JDK API • RQ1: How do existing unit level coverage-based test generation tools perform in discovering API misuses? • RQ2: Does Catcher improve the performance of existing test coverage-based approaches on detecting API misuses? • RQ3: What types of API misuses does Catcher expose? • Comparison with (plain) EvoSuite with… • DynaMOSA + default coverage criteria • On all classes of the client apps • 25 runs x 3 minutes x each class under test !18 Client project Files (#) LOC (K) bcel-6.2 489 39 commons-cli-1.4 50 7 commons-codec-1.12 124 10 commons-collections-4.2 535 63 commons-compress-1.17 352 43 commons-lang-3.7 323 76 commons-math-3.6.1 1617 209 easymock-3.6 204 14 gson-2.8.5 206 25 hamcrest-core-1.3 152 7 jackson-databind-2.9.6 919 114 javassist-3.23.1 527 82 jcommander-1.71 139 6 jfreechart-1.5.0 990 134 jodat-time-2.10 330 86 jopt-simple-5.0.4 192 9 natty-0.13 27 3 neo4j-java-driver-1.6.2 510 52 shiro-core-1.3.2 653 31 xwiki-commons-job-10.6 67 3 xwiki-commons-text-10.6 3 101
  19. 19. Evaluation results !19 EvoSuite (RQ1) Catcher (RQ2) Effectiveness 165 (mean=123) unique misuses 243 (mean=207) unique misuses Efficiency ~15 days to produce the test cases for all the projects (~17 hours per project) ~2 days to produce the test cases for all the projects (~2 hours per project) Detected misuses overlap: 1632 80 EvoSuite Catcher
  20. 20. Evaluation results (con’t) !20 Type (RQ3) Catcher Type#1: Inconsistent client • API: documented • Client: not handled 199 Type#2: Unaware client • API: not documented • Client: unaware 24 Type#3: Consistent client • API: documented • Client: documented 20
  21. 21. Implications !21 Research New research directions towards focused search-based testing.
 Automated generation of the test oracle. Further examination of the benefits of combining static analysis and testing. Practical Easily deployable in a continuous integration pipelines.
 Improvement of the reliability of APIs leading to fewer client application crashes.
  22. 22. !22 https://jira.xwiki.org/browse/XCOMMONS-1452
  23. 23. !23 https://jira.xwiki.org/browse/XCOMMONS-1452
  24. 24. !24 Catcher !X The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch) 4. Use the candidate misuses as test objectives for a search- based test case generation algorithm tests 5. Expose the misuses Evaluation on the JDK API • RQ1: How do existing unit level coverage-based test generation tools perform in discovering API misuses? • RQ2: Does Catcher improve the performance of existing test coverage-based approaches on detecting API misuses? • RQ3: What types of API misuses does Catcher expose? • Comparison with (plain) EvoSuite • DynaMOSA + default coverage criteria • On all classes of the client apps • 25 runs x 25 minutes x each class under test !X Client project Files (#) LOC (K) bcel-6.2 489 39 commons-cli-1.4 50 7 commons-codec-1.12 124 10 commons-collections-4.2 535 63 commons-compress-1.17 352 43 commons-lang-3.7 323 76 commons-math-3.6.1 1617 209 easymock-3.6 204 14 gson-2.8.5 206 25 hamcrest-core-1.3 152 7 jackson-databind-2.9.6 919 114 javassist-3.23.1 527 82 jcommander-1.71 139 6 jfreechart-1.5.0 990 134 jodat-time-2.10 330 86 jopt-simple-5.0.4 192 9 natty-0.13 27 3 neo4j-java-driver-1.6.2 510 52 shiro-core-1.3.2 653 31 xwiki-commons-job-10.6 67 3 xwiki-commons-text-10.6 3 101 Evaluation results !X EvoSuite (RQ1) Catcher (RQ2) Effectiveness 166 (mean=123) unique misuses 243 (mean=207) unique misuses Efficiency ~15 days to produce the test cases for all the projects (~17 hours per project) ~2 days to produce the test cases for all the projects (~2 hours per project) Detected misuses overlap: 1632 80 EvoSuite Catcher Implications !X Research New research directions towards focused search-based testing.
 Automated generation of the test oracle. Further examination of the benefits of combining static analysis and testing. Practical Easily deployable in a continuous integration pipelines.
 Improvement of the reliability of APIs leading to fewer client application crashes.
  25. 25. !25 Eective and Eicient API Misuse Detection via Exception Propagation and Search-Based Testing Maria Kechagia∗ m.kechagia@ucl.ac.uk University College London London, United Kingdom Xavier Devroey x.d.m.devroey@tudelft.nl Delft University of Technology Delft, Netherlands Annibale Panichella a.panichella@tudelft.nl Delft University of Technology Delft, Netherlands Georgios Gousios g.gousios@tudelft.nl Delft University of Technology Delft, Netherlands Arie van Deursen arie.vandeursen@tudelft.nl Delft University of Technology Delft, Netherlands ABSTRACT Application Programming Interfaces (APIs) typically come with (implicit) usage constraints. The violations of these constraints (API misuses) can lead to software crashes. Even though there are sev- eral tools that can detect API misuses, most of them suer from a very high rate of false positives. We introduce Catcher, a novel API misuse detection approach that combines static exception propa- gation analysis with automatic search-based test case generation to eectively and eciently pinpoint crash-prone API misuses in client applications. We validate Catcher against 21 Java applications, targeting misuses of the Java platform’s API. Our results indicate that Catcher is able to generate test cases that uncover 243 (unique) API misuses that result in crashes. Our empirical evaluation shows that Catcher can detect a large number of misuses (77 cases) that would remain undetected by the traditional coverage-based test case generator EvoSuite. Additionally, on average, Catcher is eight times faster than EvoSuite in generating test cases for the identi- ed misuses. Finally, we nd that the majority of the exceptions ACM Reference Format: Maria Kechagia, Xavier Devroey, Annibale Panichella, Georgios Gousios, and Arie van Deursen. 2019. Eective and Ecient API Misuse Detection via Exception Propagation and Search-Based Testing. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’19), July 15–19, 2019, Beijing, China. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3293882.3330552 1 INTRODUCTION Developers use external libraries to increase the velocity and reduce the production cost of software projects [38]. While increasing pro- ductivity, this form of software reuse comes with several challenges: dependencies need to be kept up to date [15], developers must learn the intricacies of each imported Application Programming Interface (), and resulting client programs should be robust, ecient, and responsive. Correctly using third-party s is not an easy task; many s are millions of lines of code large, interact with various https://github.com/mkechagia/Catcher
  26. 26. Effective and Efficient API Misuse Detection via Exception Propagation and Search-Based Testing Maria Kechagia, Xavier Devroey, Annibale Panichella, Georgios Gousios and Arie van Deursen

×