SlideShare a Scribd company logo
1 of 26
Download to read offline
Effective and Efficient API Misuse Detection via
Exception Propagation and Search-Based Testing
Maria Kechagia
Univ. College London,
London, United Kingdom
Xavier Devroey
Delft Univ. of Technology
Delft, Netherlands
Annibale Panichella
Delft Univ. of Technology
Delft, Netherlands
Georgios Gousios
Delft Univ. of Technology
Delft, Netherlands
Arie van Deursen
Delft Univ. of Technology
Delft, Netherlands
APIs everywhere
• An Application Programming Interface (API)
is a set of functions and procedures offering
services
• Applications are built by combining API
calls
• Example: Maven dependency graph with
186,384 Nodes and 1,229,083 Edges
• Usage constraints
• Explicit
• Implicit
• API misuses
• Violation of usage constraints
!2
Maven dependency graph using Force Atlas 2
https://github.com/ogirardot/meta-deps
!3
Client
(joda-time)
API
(Java JDK)
3 calls to nextToken()
without any checks
https://github.com/JodaOrg/joda-time/pull/319
https://github.com/JodaOrg/joda-time/pull/319
There is an assumption that the
input TimeZone file will always be
valid, leading to runtime
exceptions with no good error
message when the file is invalid.
!4
!5
Low
Low
High
High
Static
analysis
Dynamic
analysis
Running time
(machine effort)
False positives

(human effort)
!6
Low
Low
High
High
Static
analysis
Dynamic
analysis
Static +
dynamic
analysis
Running time
(machine effort)
False positives

(human effort)
Catcher
!7
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Catcher
!8
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
1. Spotting
potential runtime
exceptions throws
API
Catcher
!9
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
Catcher
!10
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
Catcher
!11
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
Catcher
!12
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
5. Expose the misuses
Static exception propagation
!13
Static exception propagation analysis
API Client
public void a1(…){
…
throw new IllegalStateException();
…
}
public void a2(…){
…
}
public void a3(…){
…
a1(…)
…
}
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
Filtering candidate misuses
!14
Static exception propagation analysis
API Client
public void a1(…){
…
throw new IllegalStateException();
…
}
public void a2(…){
…
}
public void a3(…){
…
a1(…)
…
}
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
Candidate misuses
IllegalStateException
at a1()
at a3()
at c1()
IllegalStateException
at a1()
at c3()
Search-based test case generation
!15
Candidate misuses
IllegalStateException
at a1()
at a3()
at c1()
IllegalStateException
at a1()
at c3()
Focused search-based test generation
EvoSuite
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
public void testC1(…){
…
}
public void testC3(…){
…
}
Static exception propagation analysis
• Builds the API call graph
• Annotated with the list of runtime exceptions
• Builds the global call graph
• Link the client call graph to the API call graph
• Propagate exceptions from the API to the client
• Maximal propagation depth (≤4)
• Filter exceptions handled by the client
• Each propagation is a candidate misuse
• Relies on Soot (http://sable.github.io/soot/)
!16
a1() a2() a3()
a4() a5()
c1() c2() c3()
c4()
API
Client
Focused Search-based test generation
• Dynamic Many-Objective Sorting Algorithm (DynaMOSA)
• Objectives
• For each misuse
• Reaching propagation point (branch coverage)
• Client method input diversity
• Client method output diversity
• Misuse detected if and only if
• Exception is thrown by the test case
• Propagation is the same as in the candidate misuse
• Relies on EvoSuite (http://www.evosuite.org)
!17
Random test
cases tests
Evolutionary
process
Test cases
exposing misuses tests
Evaluation on the JDK API
• RQ1: How do existing unit level coverage-based test
generation tools perform in discovering API misuses?
• RQ2: Does Catcher improve the performance of existing
test coverage-based approaches on detecting API
misuses?
• RQ3: What types of API misuses does Catcher expose?
• Comparison with (plain) EvoSuite with…
• DynaMOSA + default coverage criteria
• On all classes of the client apps
• 25 runs x 3 minutes x each class under test
!18
Client project Files
(#)
LOC (K)
bcel-6.2 489 39
commons-cli-1.4 50 7
commons-codec-1.12 124 10
commons-collections-4.2 535 63
commons-compress-1.17 352 43
commons-lang-3.7 323 76
commons-math-3.6.1 1617 209
easymock-3.6 204 14
gson-2.8.5 206 25
hamcrest-core-1.3 152 7
jackson-databind-2.9.6 919 114
javassist-3.23.1 527 82
jcommander-1.71 139 6
jfreechart-1.5.0 990 134
jodat-time-2.10 330 86
jopt-simple-5.0.4 192 9
natty-0.13 27 3
neo4j-java-driver-1.6.2 510 52
shiro-core-1.3.2 653 31
xwiki-commons-job-10.6 67 3
xwiki-commons-text-10.6 3 101
Evaluation results
!19
EvoSuite (RQ1) Catcher (RQ2)
Effectiveness
165 (mean=123)
unique misuses
243 (mean=207) unique
misuses
Efficiency
~15 days to produce
the test cases for all
the projects (~17
hours per project)
~2 days to produce the
test cases for all the
projects (~2 hours per
project)
Detected misuses overlap:
1632 80
EvoSuite Catcher
Evaluation results (con’t)
!20
Type (RQ3) Catcher
Type#1: Inconsistent client
• API: documented
• Client: not handled
199
Type#2: Unaware client
• API: not documented
• Client: unaware
24
Type#3: Consistent client
• API: documented
• Client: documented
20
Implications
!21
Research
New research directions towards focused search-based testing.

Automated generation of the test oracle.

Further examination of the benefits of combining static analysis and testing.
Practical
Easily deployable in a continuous integration pipelines.

Improvement of the reliability of APIs leading to fewer client application crashes.
!22
https://jira.xwiki.org/browse/XCOMMONS-1452
!23
https://jira.xwiki.org/browse/XCOMMONS-1452
!24
Catcher
!X
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
5. Expose the misuses
Evaluation on the JDK API
• RQ1: How do existing unit level coverage-based test
generation tools perform in discovering API misuses?
• RQ2: Does Catcher improve the performance of existing
test coverage-based approaches on detecting API
misuses?
• RQ3: What types of API misuses does Catcher expose?
• Comparison with (plain) EvoSuite
• DynaMOSA + default coverage criteria
• On all classes of the client apps
• 25 runs x 25 minutes x each class under test
!X
Client project Files
(#)
LOC (K)
bcel-6.2 489 39
commons-cli-1.4 50 7
commons-codec-1.12 124 10
commons-collections-4.2 535 63
commons-compress-1.17 352 43
commons-lang-3.7 323 76
commons-math-3.6.1 1617 209
easymock-3.6 204 14
gson-2.8.5 206 25
hamcrest-core-1.3 152 7
jackson-databind-2.9.6 919 114
javassist-3.23.1 527 82
jcommander-1.71 139 6
jfreechart-1.5.0 990 134
jodat-time-2.10 330 86
jopt-simple-5.0.4 192 9
natty-0.13 27 3
neo4j-java-driver-1.6.2 510 52
shiro-core-1.3.2 653 31
xwiki-commons-job-10.6 67 3
xwiki-commons-text-10.6 3 101
Evaluation results
!X
EvoSuite (RQ1) Catcher (RQ2)
Effectiveness
166 (mean=123)
unique misuses
243 (mean=207) unique
misuses
Efficiency
~15 days to produce
the test cases for all
the projects (~17
hours per project)
~2 days to produce the
test cases for all the
projects (~2 hours per
project)
Detected misuses overlap:
1632 80
EvoSuite Catcher
Implications
!X
Research
New research directions towards focused search-based testing.

Automated generation of the test oracle.

Further examination of the benefits of combining static analysis and testing.
Practical
Easily deployable in a continuous integration pipelines.

Improvement of the reliability of APIs leading to fewer client application crashes.
!25
Eective and Eicient API Misuse Detection via Exception
Propagation and Search-Based Testing
Maria Kechagia∗
m.kechagia@ucl.ac.uk
University College London
London, United Kingdom
Xavier Devroey
x.d.m.devroey@tudelft.nl
Delft University of Technology
Delft, Netherlands
Annibale Panichella
a.panichella@tudelft.nl
Delft University of Technology
Delft, Netherlands
Georgios Gousios
g.gousios@tudelft.nl
Delft University of Technology
Delft, Netherlands
Arie van Deursen
arie.vandeursen@tudelft.nl
Delft University of Technology
Delft, Netherlands
ABSTRACT
Application Programming Interfaces (APIs) typically come with
(implicit) usage constraints. The violations of these constraints (API
misuses) can lead to software crashes. Even though there are sev-
eral tools that can detect API misuses, most of them suer from a
very high rate of false positives. We introduce Catcher, a novel API
misuse detection approach that combines static exception propa-
gation analysis with automatic search-based test case generation
to eectively and eciently pinpoint crash-prone API misuses in
client applications. We validate Catcher against 21 Java applications,
targeting misuses of the Java platform’s API. Our results indicate
that Catcher is able to generate test cases that uncover 243 (unique)
API misuses that result in crashes. Our empirical evaluation shows
that Catcher can detect a large number of misuses (77 cases) that
would remain undetected by the traditional coverage-based test
case generator EvoSuite. Additionally, on average, Catcher is eight
times faster than EvoSuite in generating test cases for the identi-
ed misuses. Finally, we nd that the majority of the exceptions
ACM Reference Format:
Maria Kechagia, Xavier Devroey, Annibale Panichella, Georgios Gousios,
and Arie van Deursen. 2019. Eective and Ecient API Misuse Detection via
Exception Propagation and Search-Based Testing. In Proceedings of the 28th
ACM SIGSOFT International Symposium on Software Testing and Analysis
(ISSTA ’19), July 15–19, 2019, Beijing, China. ACM, New York, NY, USA,
12 pages. https://doi.org/10.1145/3293882.3330552
1 INTRODUCTION
Developers use external libraries to increase the velocity and reduce
the production cost of software projects [38]. While increasing pro-
ductivity, this form of software reuse comes with several challenges:
dependencies need to be kept up to date [15], developers must learn
the intricacies of each imported Application Programming Interface
(), and resulting client programs should be robust, ecient, and
responsive. Correctly using third-party s is not an easy task;
many s are millions of lines of code large, interact with various
https://github.com/mkechagia/Catcher
Effective and Efficient API Misuse Detection via
Exception Propagation and Search-Based Testing
Maria Kechagia, Xavier Devroey, Annibale Panichella,
Georgios Gousios and Arie van Deursen

More Related Content

What's hot

What's hot (20)

FaultHunter workshop (SourceMeter for SonarQube plugin module)
FaultHunter workshop (SourceMeter for SonarQube plugin module)FaultHunter workshop (SourceMeter for SonarQube plugin module)
FaultHunter workshop (SourceMeter for SonarQube plugin module)
 
Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...
Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...
Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...
 
Hackfest 2019 Talk
Hackfest 2019 TalkHackfest 2019 Talk
Hackfest 2019 Talk
 
JUnit 5 - Evolution and Innovation - SpringOne Platform 2019
JUnit 5 - Evolution and Innovation - SpringOne Platform 2019JUnit 5 - Evolution and Innovation - SpringOne Platform 2019
JUnit 5 - Evolution and Innovation - SpringOne Platform 2019
 
Introduction of exception in vb.net
Introduction of exception in vb.netIntroduction of exception in vb.net
Introduction of exception in vb.net
 
PVS-Studio and static code analysis technique
PVS-Studio and static code analysis techniquePVS-Studio and static code analysis technique
PVS-Studio and static code analysis technique
 
An Essential Guide to Effective Test Automation Leveraging Open Source
An Essential Guide to Effective Test Automation Leveraging Open SourceAn Essential Guide to Effective Test Automation Leveraging Open Source
An Essential Guide to Effective Test Automation Leveraging Open Source
 
Automation Testing Syllabus - Checklist
Automation Testing Syllabus - ChecklistAutomation Testing Syllabus - Checklist
Automation Testing Syllabus - Checklist
 
Api testing and steps to do it
Api testing and steps to do itApi testing and steps to do it
Api testing and steps to do it
 
Efficient JavaScript Unit Testing, JavaOne China 2013
Efficient JavaScript Unit Testing, JavaOne China 2013Efficient JavaScript Unit Testing, JavaOne China 2013
Efficient JavaScript Unit Testing, JavaOne China 2013
 
Unit testing on embedded target with C++Test
Unit testing on embedded  target with C++TestUnit testing on embedded  target with C++Test
Unit testing on embedded target with C++Test
 
Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...
Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...
Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-b...
 
Resume_Shanthi
Resume_ShanthiResume_Shanthi
Resume_Shanthi
 
DEPLOYMENT OF CALABASH AUTOMATION FRAMEWORK TO ANALYZE THE PERFORMANCE OF AN ...
DEPLOYMENT OF CALABASH AUTOMATION FRAMEWORK TO ANALYZE THE PERFORMANCE OF AN ...DEPLOYMENT OF CALABASH AUTOMATION FRAMEWORK TO ANALYZE THE PERFORMANCE OF AN ...
DEPLOYMENT OF CALABASH AUTOMATION FRAMEWORK TO ANALYZE THE PERFORMANCE OF AN ...
 
150412 38 beamer methods of binary analysis
150412 38 beamer methods of  binary analysis150412 38 beamer methods of  binary analysis
150412 38 beamer methods of binary analysis
 
Testes? Mas isso não aumenta o tempo de projecto? Não quero...
Testes? Mas isso não aumenta o tempo de projecto? Não quero...Testes? Mas isso não aumenta o tempo de projecto? Não quero...
Testes? Mas isso não aumenta o tempo de projecto? Não quero...
 
Introduction to Web Application Penetration Testing
Introduction to Web Application Penetration TestingIntroduction to Web Application Penetration Testing
Introduction to Web Application Penetration Testing
 
Declarative Performance Testing Automation - Automating Performance Testing f...
Declarative Performance Testing Automation - Automating Performance Testing f...Declarative Performance Testing Automation - Automating Performance Testing f...
Declarative Performance Testing Automation - Automating Performance Testing f...
 
Continuous Integration: Live Static Analysis with Puma Scan
Continuous Integration: Live Static Analysis with Puma ScanContinuous Integration: Live Static Analysis with Puma Scan
Continuous Integration: Live Static Analysis with Puma Scan
 
Server Side Template Injection by Mandeep Jadon
Server Side Template Injection by Mandeep JadonServer Side Template Injection by Mandeep Jadon
Server Side Template Injection by Mandeep Jadon
 

Similar to Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing

REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
Sung Kim
 
CodeChecker Overview Nov 2019
CodeChecker Overview Nov 2019CodeChecker Overview Nov 2019
CodeChecker Overview Nov 2019
Olivera Milenkovic
 
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Zhen Huang
 

Similar to Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing (20)

REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
REMI: Defect Prediction for Efficient API Testing (

ESEC/FSE 2015, Industria...
 
Software testing: an introduction - 2017
Software testing: an introduction - 2017Software testing: an introduction - 2017
Software testing: an introduction - 2017
 
ACSAC2016: Code Obfuscation Against Symbolic Execution Attacks
ACSAC2016: Code Obfuscation Against Symbolic Execution AttacksACSAC2016: Code Obfuscation Against Symbolic Execution Attacks
ACSAC2016: Code Obfuscation Against Symbolic Execution Attacks
 
GPCE16: Automatic Non-functional Testing of Code Generators Families
GPCE16: Automatic Non-functional Testing of Code Generators FamiliesGPCE16: Automatic Non-functional Testing of Code Generators Families
GPCE16: Automatic Non-functional Testing of Code Generators Families
 
CodeChecker Overview Nov 2019
CodeChecker Overview Nov 2019CodeChecker Overview Nov 2019
CodeChecker Overview Nov 2019
 
Practices and Tools for Building Better APIs
Practices and Tools for Building Better APIsPractices and Tools for Building Better APIs
Practices and Tools for Building Better APIs
 
API Performance testing with Gatling
API Performance testing with GatlingAPI Performance testing with Gatling
API Performance testing with Gatling
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 
Code Quality - Security
Code Quality - SecurityCode Quality - Security
Code Quality - Security
 
How to Test PowerShell Code Using Pester
How to Test PowerShell Code Using PesterHow to Test PowerShell Code Using Pester
How to Test PowerShell Code Using Pester
 
IEEE Buenaventura cs Chapter March 9 2016 v4
IEEE Buenaventura cs Chapter March 9 2016  v4IEEE Buenaventura cs Chapter March 9 2016  v4
IEEE Buenaventura cs Chapter March 9 2016 v4
 
Parasoft .TEST, Write better C# Code Using Data Flow Analysis
Parasoft .TEST, Write better C# Code Using  Data Flow Analysis Parasoft .TEST, Write better C# Code Using  Data Flow Analysis
Parasoft .TEST, Write better C# Code Using Data Flow Analysis
 
Cloud-native application monitoring powered by Riverbed and Elasticsearch
Cloud-native application monitoring powered by Riverbed and ElasticsearchCloud-native application monitoring powered by Riverbed and Elasticsearch
Cloud-native application monitoring powered by Riverbed and Elasticsearch
 
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...
 
How to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test AutomationHow to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test Automation
 
How do Developers Test Android Applications?
How do Developers Test Android Applications?How do Developers Test Android Applications?
How do Developers Test Android Applications?
 
INTERFACE by apidays 2023 - Production Ready GraphQL, Antoine Carossio & Tri...
INTERFACE by apidays 2023 - Production Ready GraphQL,  Antoine Carossio & Tri...INTERFACE by apidays 2023 - Production Ready GraphQL,  Antoine Carossio & Tri...
INTERFACE by apidays 2023 - Production Ready GraphQL, Antoine Carossio & Tri...
 
Operating a High Velocity Large Organization with Spring Cloud Microservices
Operating a High Velocity Large Organization with Spring Cloud MicroservicesOperating a High Velocity Large Organization with Spring Cloud Microservices
Operating a High Velocity Large Organization with Spring Cloud Microservices
 
Continuous testing - GUERLAIS ARGOT - Air France KLM Sogeti- Soirée du Test L...
Continuous testing - GUERLAIS ARGOT - Air France KLM Sogeti- Soirée du Test L...Continuous testing - GUERLAIS ARGOT - Air France KLM Sogeti- Soirée du Test L...
Continuous testing - GUERLAIS ARGOT - Air France KLM Sogeti- Soirée du Test L...
 
Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)
 

More from XavierDevroey

More from XavierDevroey (10)

Software Variability Management
Software Variability ManagementSoftware Variability Management
Software Variability Management
 
Software variability management - 2019
Software variability management - 2019Software variability management - 2019
Software variability management - 2019
 
Testing Variability-Intensive Systems, tutorial SPLC 2017, part Ii
Testing Variability-Intensive Systems, tutorial SPLC 2017, part IiTesting Variability-Intensive Systems, tutorial SPLC 2017, part Ii
Testing Variability-Intensive Systems, tutorial SPLC 2017, part Ii
 
Testing Variability-Intensive Systems, tutorial SPLC 2017, part I
Testing Variability-Intensive Systems, tutorial SPLC 2017, part ITesting Variability-Intensive Systems, tutorial SPLC 2017, part I
Testing Variability-Intensive Systems, tutorial SPLC 2017, part I
 
Lorentz workshop - 2018
Lorentz workshop - 2018Lorentz workshop - 2018
Lorentz workshop - 2018
 
Combinatorial Interaction Testing, an Introduction - 2018
Combinatorial Interaction Testing, an Introduction - 2018Combinatorial Interaction Testing, an Introduction - 2018
Combinatorial Interaction Testing, an Introduction - 2018
 
Software variability management - 2017
Software variability management - 2017Software variability management - 2017
Software variability management - 2017
 
Software testing: an introduction - 2016
Software testing: an introduction - 2016Software testing: an introduction - 2016
Software testing: an introduction - 2016
 
Software testing: an introduction - 2015
Software testing: an introduction - 2015Software testing: an introduction - 2015
Software testing: an introduction - 2015
 
Research overview Oct. 2018
Research overview Oct. 2018Research overview Oct. 2018
Research overview Oct. 2018
 

Recently uploaded

Recently uploaded (9)

Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdfMicrosoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
Microsoft Fabric Analytics Engineer (DP-600) Exam Dumps 2024.pdf
 
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdfACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
ACM CHT Best Inspection Practices Kinben Innovation MIC Slideshare.pdf
 
Breathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptxBreathing in New Life_ Part 3 05 22 2024.pptx
Breathing in New Life_ Part 3 05 22 2024.pptx
 
Understanding Poverty: A Community Questionnaire
Understanding Poverty: A Community QuestionnaireUnderstanding Poverty: A Community Questionnaire
Understanding Poverty: A Community Questionnaire
 
Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.Deciding The Topic of our Magazine.pptx.
Deciding The Topic of our Magazine.pptx.
 
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptxDAY 0 8 A Revelation 05-19-2024 PPT.pptx
DAY 0 8 A Revelation 05-19-2024 PPT.pptx
 
ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024ServiceNow CIS-Discovery Exam Dumps 2024
ServiceNow CIS-Discovery Exam Dumps 2024
 
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdfOracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
Oracle Database Administration I (1Z0-082) Exam Dumps 2024.pdf
 
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docxThe Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
The Influence and Evolution of Mogul Press in Contemporary Public Relations.docx
 

Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing

  • 1. Effective and Efficient API Misuse Detection via Exception Propagation and Search-Based Testing Maria Kechagia Univ. College London, London, United Kingdom Xavier Devroey Delft Univ. of Technology Delft, Netherlands Annibale Panichella Delft Univ. of Technology Delft, Netherlands Georgios Gousios Delft Univ. of Technology Delft, Netherlands Arie van Deursen Delft Univ. of Technology Delft, Netherlands
  • 2. APIs everywhere • An Application Programming Interface (API) is a set of functions and procedures offering services • Applications are built by combining API calls • Example: Maven dependency graph with 186,384 Nodes and 1,229,083 Edges • Usage constraints • Explicit • Implicit • API misuses • Violation of usage constraints !2 Maven dependency graph using Force Atlas 2 https://github.com/ogirardot/meta-deps
  • 3. !3 Client (joda-time) API (Java JDK) 3 calls to nextToken() without any checks https://github.com/JodaOrg/joda-time/pull/319
  • 4. https://github.com/JodaOrg/joda-time/pull/319 There is an assumption that the input TimeZone file will always be valid, leading to runtime exceptions with no good error message when the file is invalid. !4
  • 7. Catcher !7 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses.
  • 8. Catcher !8 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. 1. Spotting potential runtime exceptions throws API
  • 9. Catcher !9 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API
  • 10. Catcher !10 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch)
  • 11. Catcher !11 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch) 4. Use the candidate misuses as test objectives for a search- based test case generation algorithm tests
  • 12. Catcher !12 The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch) 4. Use the candidate misuses as test objectives for a search- based test case generation algorithm tests 5. Expose the misuses
  • 13. Static exception propagation !13 Static exception propagation analysis API Client public void a1(…){ … throw new IllegalStateException(); … } public void a2(…){ … } public void a3(…){ … a1(…) … } public void c1(…){ … a3(…); … } public void c2(…){ … } public void c3(…){ … a1(…); … }
  • 14. Filtering candidate misuses !14 Static exception propagation analysis API Client public void a1(…){ … throw new IllegalStateException(); … } public void a2(…){ … } public void a3(…){ … a1(…) … } public void c1(…){ … a3(…); … } public void c2(…){ … } public void c3(…){ … a1(…); … } Candidate misuses IllegalStateException at a1() at a3() at c1() IllegalStateException at a1() at c3()
  • 15. Search-based test case generation !15 Candidate misuses IllegalStateException at a1() at a3() at c1() IllegalStateException at a1() at c3() Focused search-based test generation EvoSuite public void c1(…){ … a3(…); … } public void c2(…){ … } public void c3(…){ … a1(…); … } public void testC1(…){ … } public void testC3(…){ … }
  • 16. Static exception propagation analysis • Builds the API call graph • Annotated with the list of runtime exceptions • Builds the global call graph • Link the client call graph to the API call graph • Propagate exceptions from the API to the client • Maximal propagation depth (≤4) • Filter exceptions handled by the client • Each propagation is a candidate misuse • Relies on Soot (http://sable.github.io/soot/) !16 a1() a2() a3() a4() a5() c1() c2() c3() c4() API Client
  • 17. Focused Search-based test generation • Dynamic Many-Objective Sorting Algorithm (DynaMOSA) • Objectives • For each misuse • Reaching propagation point (branch coverage) • Client method input diversity • Client method output diversity • Misuse detected if and only if • Exception is thrown by the test case • Propagation is the same as in the candidate misuse • Relies on EvoSuite (http://www.evosuite.org) !17 Random test cases tests Evolutionary process Test cases exposing misuses tests
  • 18. Evaluation on the JDK API • RQ1: How do existing unit level coverage-based test generation tools perform in discovering API misuses? • RQ2: Does Catcher improve the performance of existing test coverage-based approaches on detecting API misuses? • RQ3: What types of API misuses does Catcher expose? • Comparison with (plain) EvoSuite with… • DynaMOSA + default coverage criteria • On all classes of the client apps • 25 runs x 3 minutes x each class under test !18 Client project Files (#) LOC (K) bcel-6.2 489 39 commons-cli-1.4 50 7 commons-codec-1.12 124 10 commons-collections-4.2 535 63 commons-compress-1.17 352 43 commons-lang-3.7 323 76 commons-math-3.6.1 1617 209 easymock-3.6 204 14 gson-2.8.5 206 25 hamcrest-core-1.3 152 7 jackson-databind-2.9.6 919 114 javassist-3.23.1 527 82 jcommander-1.71 139 6 jfreechart-1.5.0 990 134 jodat-time-2.10 330 86 jopt-simple-5.0.4 192 9 natty-0.13 27 3 neo4j-java-driver-1.6.2 510 52 shiro-core-1.3.2 653 31 xwiki-commons-job-10.6 67 3 xwiki-commons-text-10.6 3 101
  • 19. Evaluation results !19 EvoSuite (RQ1) Catcher (RQ2) Effectiveness 165 (mean=123) unique misuses 243 (mean=207) unique misuses Efficiency ~15 days to produce the test cases for all the projects (~17 hours per project) ~2 days to produce the test cases for all the projects (~2 hours per project) Detected misuses overlap: 1632 80 EvoSuite Catcher
  • 20. Evaluation results (con’t) !20 Type (RQ3) Catcher Type#1: Inconsistent client • API: documented • Client: not handled 199 Type#2: Unaware client • API: not documented • Client: unaware 24 Type#3: Consistent client • API: documented • Client: documented 20
  • 21. Implications !21 Research New research directions towards focused search-based testing.
 Automated generation of the test oracle. Further examination of the benefits of combining static analysis and testing. Practical Easily deployable in a continuous integration pipelines.
 Improvement of the reliability of APIs leading to fewer client application crashes.
  • 24. !24 Catcher !X The combination of static exception propagation analysis and search-based test case generation for the effective and efficient detection of API misuses. Client 1. Spotting potential runtime exceptions throws 2. See which ones of those exceptions can propagate to the client API 3. Filter trivial false positives (e.g. try/ catch) 4. Use the candidate misuses as test objectives for a search- based test case generation algorithm tests 5. Expose the misuses Evaluation on the JDK API • RQ1: How do existing unit level coverage-based test generation tools perform in discovering API misuses? • RQ2: Does Catcher improve the performance of existing test coverage-based approaches on detecting API misuses? • RQ3: What types of API misuses does Catcher expose? • Comparison with (plain) EvoSuite • DynaMOSA + default coverage criteria • On all classes of the client apps • 25 runs x 25 minutes x each class under test !X Client project Files (#) LOC (K) bcel-6.2 489 39 commons-cli-1.4 50 7 commons-codec-1.12 124 10 commons-collections-4.2 535 63 commons-compress-1.17 352 43 commons-lang-3.7 323 76 commons-math-3.6.1 1617 209 easymock-3.6 204 14 gson-2.8.5 206 25 hamcrest-core-1.3 152 7 jackson-databind-2.9.6 919 114 javassist-3.23.1 527 82 jcommander-1.71 139 6 jfreechart-1.5.0 990 134 jodat-time-2.10 330 86 jopt-simple-5.0.4 192 9 natty-0.13 27 3 neo4j-java-driver-1.6.2 510 52 shiro-core-1.3.2 653 31 xwiki-commons-job-10.6 67 3 xwiki-commons-text-10.6 3 101 Evaluation results !X EvoSuite (RQ1) Catcher (RQ2) Effectiveness 166 (mean=123) unique misuses 243 (mean=207) unique misuses Efficiency ~15 days to produce the test cases for all the projects (~17 hours per project) ~2 days to produce the test cases for all the projects (~2 hours per project) Detected misuses overlap: 1632 80 EvoSuite Catcher Implications !X Research New research directions towards focused search-based testing.
 Automated generation of the test oracle. Further examination of the benefits of combining static analysis and testing. Practical Easily deployable in a continuous integration pipelines.
 Improvement of the reliability of APIs leading to fewer client application crashes.
  • 25. !25 Eective and Eicient API Misuse Detection via Exception Propagation and Search-Based Testing Maria Kechagia∗ m.kechagia@ucl.ac.uk University College London London, United Kingdom Xavier Devroey x.d.m.devroey@tudelft.nl Delft University of Technology Delft, Netherlands Annibale Panichella a.panichella@tudelft.nl Delft University of Technology Delft, Netherlands Georgios Gousios g.gousios@tudelft.nl Delft University of Technology Delft, Netherlands Arie van Deursen arie.vandeursen@tudelft.nl Delft University of Technology Delft, Netherlands ABSTRACT Application Programming Interfaces (APIs) typically come with (implicit) usage constraints. The violations of these constraints (API misuses) can lead to software crashes. Even though there are sev- eral tools that can detect API misuses, most of them suer from a very high rate of false positives. We introduce Catcher, a novel API misuse detection approach that combines static exception propa- gation analysis with automatic search-based test case generation to eectively and eciently pinpoint crash-prone API misuses in client applications. We validate Catcher against 21 Java applications, targeting misuses of the Java platform’s API. Our results indicate that Catcher is able to generate test cases that uncover 243 (unique) API misuses that result in crashes. Our empirical evaluation shows that Catcher can detect a large number of misuses (77 cases) that would remain undetected by the traditional coverage-based test case generator EvoSuite. Additionally, on average, Catcher is eight times faster than EvoSuite in generating test cases for the identi- ed misuses. Finally, we nd that the majority of the exceptions ACM Reference Format: Maria Kechagia, Xavier Devroey, Annibale Panichella, Georgios Gousios, and Arie van Deursen. 2019. Eective and Ecient API Misuse Detection via Exception Propagation and Search-Based Testing. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’19), July 15–19, 2019, Beijing, China. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3293882.3330552 1 INTRODUCTION Developers use external libraries to increase the velocity and reduce the production cost of software projects [38]. While increasing pro- ductivity, this form of software reuse comes with several challenges: dependencies need to be kept up to date [15], developers must learn the intricacies of each imported Application Programming Interface (), and resulting client programs should be robust, ecient, and responsive. Correctly using third-party s is not an easy task; many s are millions of lines of code large, interact with various https://github.com/mkechagia/Catcher
  • 26. Effective and Efficient API Misuse Detection via Exception Propagation and Search-Based Testing Maria Kechagia, Xavier Devroey, Annibale Panichella, Georgios Gousios and Arie van Deursen