Join core JUnit 5 committer Sam Brannen to gain insight on the latest new features in JUnit 5 as well as what’s on the horizon.
In this presentation we will look at exciting new features that have been added in JUnit 5 over the past year, including temporary directories, custom display name generators, method ordering, timeouts, the Test Kit, and powerful new extension APIs. If you haven't yet made the switch from JUnit 4 to JUnit 5 you will definitely want to check out this presentation.
In closing, Sam will also provide a few tips on how to use JUnit Jupiter to test Spring and Spring Boot apps.
PVS-Studio and static code analysis techniqueAndrey Karpov
What is «static code analysis»? It is a technique that allows, at the same time with unit-tests, dynamic code analysis, code review and others, to increase code quality, increase its reliability and decrease the development time.
An Essential Guide to Effective Test Automation Leveraging Open SourceRapidValue
Mankind is stepping into a new era completely owned by automation and the advent of mobility and digital
technologies has simplified life by providing easy access to any individual to access technology without any
bias. In the business environment, this pushes companies to deliver high-quality applications in less time with
limited resources. This has ultimately led to a competition to deliver secure and advanced digital solutions
without breaking the requirements and guidelines of the clients. Projects developed under such pressure
have failed to meet objectives and suffer from vital schedule and budget slippage because of a delay in
discovering the defects. This results in an increased percentage of ‘Defect Leakage’ and eventually customer
dissatisfaction due to sub-par products. Organizations have begun to adopt automation to attain better
product quality and develop secure, faster and useful mobile applications. This whitepaper endeavors to
provide insights from our experiences on working with automation and elaborates on its importance with
reference to certain case studies.
Want to know what a leading software testing consultant perform API testing and API Implementation with steps to do it. As API testing focuses more on the testing of data responses, security and performance restrictions its implementation is a must so read thoroughly this blog, to take everything about API testing.
Parasoft delivers a complete framework to create, manage, and extract greater value from unit tests. We help you exercise and test an incomplete system—enabling you to identify problems when they are least difficult, costly, and time-consuming to fix. This reduces the length and cost of downstream processes such as debugging. Moreover, since all tests are written at the unit level, the test suite can be run independent of the complete system. This allows you to isolate code behavior changes, reduces setup complexities, and makes it practical to execute the test suite on a daily basis.
DEPLOYMENT OF CALABASH AUTOMATION FRAMEWORK TO ANALYZE THE PERFORMANCE OF AN ...Journal For Research
The use of mobile phones has grown rapidly and has become a vital part in the present era. The conventional software testing practices are not really feasible for mobile applications. Test Automation does not cover all the necessary features, which makes manual testing also very crucial. The challenge is even greater. The problem is even greater in projects where agile methodologies are deployed. In such projects Test Automation plays a very significant role and is very important during the development cycle. The Automation Framework should be deployable on multiple heterogeneous platforms. The actions that are frequently used in the mobile devices are extracted. They are then mapped into the corresponding functions of Calabash testing framework.The objective and intention of this paper is to bring out significant merits and demerits of different Automation Frameworks and then use the Calabash Automation Framework to develop the Performance Analysis module which can effectively determine the launch time of the mobile application.
Testes? Mas isso não aumenta o tempo de projecto? Não quero...Comunidade NetPonto
Os Testes são cada vez mais uma necessidade nos projectos de desenvolvimento de software... Sejam eles unitários, de carga ou de "User Interface", uma boa framework de testes ajuda a resolver os problemas mais cedo, de forma mais eficaz e mais barata.
No final da sessão vamos perceber não só para que servem, como são feitos e como o Visual Studio 2010 pode ajudar.
Introduction to Web Application Penetration TestingRana Khalil
Intro to web application penetration testing workshop I held in Atlanta as part of the AnitaBorg Cybersecurity Weekend on Aug. 19. The link for the event can be found here: https://community.anitab.org/event/atl-cybersecurity-day-two/
Slides from OWASP AppSec USA 2016.
For over 10 years, Visual Studio has provided basic source code analysis through FxCop and StyleCop. While these code analyzers focus mainly on design conformance, code consistency, and best practices, there is very little support for enforcing secure coding techniques. To address this gap, Microsoft started a project back in 2011 called CAT.NET to help identify secure coding bugs such as XSS, SQL Injection, and XPath Injection. Unfortunately, CAT.NET failed and never made it past the first version. Aside from purchasing expensive commercial static analysis tools, .NET development teams have been left without an easy way to integrate secure code scanning rules into Visual Studio. At least until now...
With the release of Visual Studio 2015, the open-source .NET Compiler Platform (aka “Roslyn”) exposes a set of code analysis APIs capable of querying the source code, identifying a security issue, and reporting it as code is written! In this talk, we will explore the code analysis APIs and show you how to create a live static analysis rule. Come prepared to see demonstrations of Visual Studio static analysis rules in action, and walk away with a static analysis rule pack to run against your organization’s .NET applications.
Join core JUnit 5 committer Sam Brannen to gain insight on the latest new features in JUnit 5 as well as what’s on the horizon.
In this presentation we will look at exciting new features that have been added in JUnit 5 over the past year, including temporary directories, custom display name generators, method ordering, timeouts, the Test Kit, and powerful new extension APIs. If you haven't yet made the switch from JUnit 4 to JUnit 5 you will definitely want to check out this presentation.
In closing, Sam will also provide a few tips on how to use JUnit Jupiter to test Spring and Spring Boot apps.
PVS-Studio and static code analysis techniqueAndrey Karpov
What is «static code analysis»? It is a technique that allows, at the same time with unit-tests, dynamic code analysis, code review and others, to increase code quality, increase its reliability and decrease the development time.
An Essential Guide to Effective Test Automation Leveraging Open SourceRapidValue
Mankind is stepping into a new era completely owned by automation and the advent of mobility and digital
technologies has simplified life by providing easy access to any individual to access technology without any
bias. In the business environment, this pushes companies to deliver high-quality applications in less time with
limited resources. This has ultimately led to a competition to deliver secure and advanced digital solutions
without breaking the requirements and guidelines of the clients. Projects developed under such pressure
have failed to meet objectives and suffer from vital schedule and budget slippage because of a delay in
discovering the defects. This results in an increased percentage of ‘Defect Leakage’ and eventually customer
dissatisfaction due to sub-par products. Organizations have begun to adopt automation to attain better
product quality and develop secure, faster and useful mobile applications. This whitepaper endeavors to
provide insights from our experiences on working with automation and elaborates on its importance with
reference to certain case studies.
Want to know what a leading software testing consultant perform API testing and API Implementation with steps to do it. As API testing focuses more on the testing of data responses, security and performance restrictions its implementation is a must so read thoroughly this blog, to take everything about API testing.
Parasoft delivers a complete framework to create, manage, and extract greater value from unit tests. We help you exercise and test an incomplete system—enabling you to identify problems when they are least difficult, costly, and time-consuming to fix. This reduces the length and cost of downstream processes such as debugging. Moreover, since all tests are written at the unit level, the test suite can be run independent of the complete system. This allows you to isolate code behavior changes, reduces setup complexities, and makes it practical to execute the test suite on a daily basis.
DEPLOYMENT OF CALABASH AUTOMATION FRAMEWORK TO ANALYZE THE PERFORMANCE OF AN ...Journal For Research
The use of mobile phones has grown rapidly and has become a vital part in the present era. The conventional software testing practices are not really feasible for mobile applications. Test Automation does not cover all the necessary features, which makes manual testing also very crucial. The challenge is even greater. The problem is even greater in projects where agile methodologies are deployed. In such projects Test Automation plays a very significant role and is very important during the development cycle. The Automation Framework should be deployable on multiple heterogeneous platforms. The actions that are frequently used in the mobile devices are extracted. They are then mapped into the corresponding functions of Calabash testing framework.The objective and intention of this paper is to bring out significant merits and demerits of different Automation Frameworks and then use the Calabash Automation Framework to develop the Performance Analysis module which can effectively determine the launch time of the mobile application.
Testes? Mas isso não aumenta o tempo de projecto? Não quero...Comunidade NetPonto
Os Testes são cada vez mais uma necessidade nos projectos de desenvolvimento de software... Sejam eles unitários, de carga ou de "User Interface", uma boa framework de testes ajuda a resolver os problemas mais cedo, de forma mais eficaz e mais barata.
No final da sessão vamos perceber não só para que servem, como são feitos e como o Visual Studio 2010 pode ajudar.
Introduction to Web Application Penetration TestingRana Khalil
Intro to web application penetration testing workshop I held in Atlanta as part of the AnitaBorg Cybersecurity Weekend on Aug. 19. The link for the event can be found here: https://community.anitab.org/event/atl-cybersecurity-day-two/
Slides from OWASP AppSec USA 2016.
For over 10 years, Visual Studio has provided basic source code analysis through FxCop and StyleCop. While these code analyzers focus mainly on design conformance, code consistency, and best practices, there is very little support for enforcing secure coding techniques. To address this gap, Microsoft started a project back in 2011 called CAT.NET to help identify secure coding bugs such as XSS, SQL Injection, and XPath Injection. Unfortunately, CAT.NET failed and never made it past the first version. Aside from purchasing expensive commercial static analysis tools, .NET development teams have been left without an easy way to integrate secure code scanning rules into Visual Studio. At least until now...
With the release of Visual Studio 2015, the open-source .NET Compiler Platform (aka “Roslyn”) exposes a set of code analysis APIs capable of querying the source code, identifying a security issue, and reporting it as code is written! In this talk, we will explore the code analysis APIs and show you how to create a live static analysis rule. Come prepared to see demonstrations of Visual Studio static analysis rules in action, and walk away with a static analysis rule pack to run against your organization’s .NET applications.
ACSAC2016: Code Obfuscation Against Symbolic Execution AttacksSebastian Banescu
Slides from the 2016 Annual Computer Security Applications Conference (ACSAC), about the paper entitled "Code Obfuscation Against Symbolic Execution Attacks"
GPCE16: Automatic Non-functional Testing of Code Generators FamiliesMohamed BOUSSAA
The intensive use of generative programming techniques provides an elegant engineering solution to deal with the heterogeneity of platforms and technological stacks. The use of domain-specific languages for example, leads to the creation of numerous code generators that automatically translate high-level system specifications into multi-target executable code. Producing correct and efficient code generator is complex and error-prone. Although software designers provide generally high-level test suites to verify the functional outcome of generated code, it remains challenging and tedious to verify the behavior of produced code in terms of non-functional properties. This paper describes a practical approach based on a runtime monitoring infrastructure to automatically check the potential inefficient code generators. This infrastructure, based on system containers as execution platforms, allows code-generator developers to evaluate the generated code performance. We evaluate our approach by analyzing the performance of Haxe, a popular high-level programming language that involves a set of cross-platform code generators. Experimental results show that our approach is able to detect some performance inconsistencies that reveal real issues in Haxe code generators.
Using static code analysis tools and detecting and fixing identified issues is very important in order to improve the quality and security of the code baseline.
CodeChecker (https://github.com/Ericsson/codechecker ) is an open source analyzer tooling, defect database and viewer extension for the Clang Static Analyzer and Clang Tidy.
It provides a number of additional features:
- Good visualization of problems in the code
- Overview of results for the whole product
- Filtering
- Cross translational unit analysis and statistical checkers support
- Suppression handling
- And many others...
These features simplify the follow up of results and make it more efficient.
In the video, an overview of features and capabilities of CodeChecker is demonstrated as well as a description and recommendation of how to introduce new tools.
Recording of the demo: https://youtu.be/sQ2Qj0kHoRY published in C++ Dublin User group https://www.youtube.com/channel/UCZ4UNE_1IMUFfAhcdq7CMOg/
Useful links:
open source project: https://github.com/Ericsson/codechecker
http://codechecker-demo.eastus.cloudapp.azure.com/login.html#
demo/demo
https://codechecker.readthedocs.io/en/latest/
http://clang-analyzer.llvm.org/available_checks.html
http://clang.llvm.org/extra/clang-tidy/checks/list.html
Other related videos about Clang Static Analyzer and CodeChecker that goes a bit more deeply into how Clang Static Analyzer works:
Clang Static Analysis - Meeting C++ 2016 Gabor Horvath
https://www.youtube.com/watch?v=UcxF6CVueDM
CppCon 2016: Gabor Horvath “Make Friends with the Clang Static Analysis Tools"
https://www.youtube.com/watch?v=AQF6hjLKsnM
Practices and Tools for Building Better APIsPeter Hendriks
The most important part of well-designed Java code is a nice API. A good API helps developers be more productive and write high-quality code quickly. API design matters for any developer, especially in building larger systems with a team. Modern coding tools such as Eclipse and FindBugs contain advanced tooling to help with designing an API and checking for bad usage. This session demonstrates the latest innovations, including new capabilities in Java 8, by presenting realistic examples based on real experiences in large codebases. They show that just a few Java tricks and simple annotations can make all the difference for building a great API.
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2rm4hFD.
Yevgeniy Brikman talks about how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Topics covered include: unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI / CD for infrastructure code. Filmed at qconsf.com.
Yevgeniy Brikman is the co-founder of Gruntwork, a company that provides DevOps as a Service. He is the author of two books published by O'Reilly Media: Hello, Startup and Terraform: Up & Running. Previously, he worked as a software engineer at LinkedIn, TripAdvisor, Cisco Systems, and Thomson Financial.
Quality of software code for a given product shipped effectively translates not only to its functional quality but as well to its non functional aspects say security. Many of the issues in code can be addressed much before they reach SCM.
Advanced Continuous Test Automation presentation given by Marc Hornbeek, Sr. Solutions Architect for Spirent at the IEEE meeting Buenaventura chapter March 9, 2016.
Cloud-native application monitoring powered by Riverbed and ElasticsearchRichard Juknavorian
Learn improved performance testing for Cloud-native applications by integrating Elasticsearch with Riverbed application performance monitoring (APM). The objective was to create realistic performance testing that was representative of real-world usage of the application.
Talos: Neutralizing Vulnerabilities with Security Workarounds for Rapid Respo...Zhen Huang
There is often a considerable delay between the discovery of a vulnerability and the issue of a patch. One mitigation strategy for this window of vulnerability is to use a configuration workaround, which prevents the vulnerable code from being executed at the cost of some lost functionality -- but if one is available. Since application configurations are not specifically designed to mitigate software vulnerabilities, we find that they only cover 25.2% of vulnerabilities.
To minimize patch delay vulnerabilities and address the limitations of configuration workarounds, we propose Security Workarounds for Rapid Response (SWRRs), which are designed to neutralize security vulnerabilities in a timely, secure, and unobtrusive manner. Similar to configuration workarounds, SWRRs neutralize vulnerabilities by preventing vulnerable code from being executed at the cost of some lost functionality. However, the key difference is that SWRRs use existing error-handling code within applications, which enables them to be mechanically inserted with minimal knowledge of the application and minimal developer effort. This allows SWRRs to achieve high coverage while still being fast and easy to deploy.
We designed and implemented Talos, a system that mechanically instrument SWRRs into a given application, and evaluate it on five popular Linux server applications. We run exploits against 11 real-world software vulnerabilities and show that SWRRs neutralize the vulnerabilities in all cases. Quantitative measurements on 320 SWRRs indicate that SWRRs instrumented by Talos can neutralize 75.1% of all potential vulnerabilities and incur a loss of functionality similar to configuration workarounds in 71.3% of those cases. Our overall conclusion is that automatically generated SWRRs can safely mitigate 2.1x times more vulnerabilities, while only incurring a loss of functionality comparable to that of traditional configuration workarounds.
Getting value from your test automation is fundamental for fast feedback, risk reduction, and return on investment from your testing activities. Once developing the test scenarios, teams cannot stop monitoring and ensuring that their tests continuously bring value, are not flaky, and can support the latest functionalities in your web and mobile apps. Teams often “forget” about their tests once they have been developed and integrated into the CI pipeline — regardless of the value they bring. In this session, learn how to optimize your Appium and Selenium test suites so you can get more value from them.
How do Developers Test Android Applications?Kevin Moran
Enabling fully automated testing of mobile applications has recently become an important topic of study for both researchers and practitioners. A plethora of tools and approaches have been proposed to aid mobile developers both by augmenting manual testing practices and by automating various parts of the testing process. However, current approaches for automated testing fall short in convincing developers about their benefits, leading to a majority of mobile testing being performed manually. With the goal of helping researchers and practitioners – who design approaches supporting mobile testing – to understand developer’s needs, we analyzed survey responses from 102 open source contributors to Android projects about their practices when performing testing. The survey focused on questions regarding practices and preferences of developers/testers in-the-wild for (i) designing and generating test cases, (ii) automated testing practices, and (iii) perceptions of quality metrics such as code coverage for determining test quality. Analyzing the information gleaned from this survey, we compile a body of knowledge to help guide researchers and professionals toward tailoring new automated testing approaches to the need of a diverse set of open source developers.
INTERFACE by apidays 2023 - Production Ready GraphQL, Antoine Carossio & Tri...apidays
INTERFACE by apidays 2023
APIs for a “Smart” economy. Embedding AI to deliver Smart APIs and turn into an exponential organization
June 28 & 29, 2023
Production Ready GraphQL
Antoine Carossio, Cofounder CTO at Escape
Tristan Kalos, Cofounder CEO at Escape
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Operating a High Velocity Large Organization with Spring Cloud MicroservicesNoriaki Tatsumi
Noriaki Tatsumi prepares you to build a microservices architecture that's not only reliable, resilient, and scalable but also addresses the challenges large organizations typically face. He dives into the technical details on how Spring Cloud empowers developers to build the patterns and components of microservices foundation quickly.
Sharpen existing tools or get a new toolbox? Contemporary cluster initiatives...Orkestra
UIIN Conference, Madrid, 27-29 May 2024
James Wilson, Orkestra and Deusto Business School
Emily Wise, Lund University
Madeline Smith, The Glasgow School of Art
This presentation by Morris Kleiner (University of Minnesota), was made during the discussion “Competition and Regulation in Professions and Occupations” held at the Working Party No. 2 on Competition and Regulation on 10 June 2024. More papers and presentations on the topic can be found out at oe.cd/crps.
This presentation was uploaded with the author’s consent.
Have you ever wondered how search works while visiting an e-commerce site, internal website, or searching through other types of online resources? Look no further than this informative session on the ways that taxonomies help end-users navigate the internet! Hear from taxonomists and other information professionals who have first-hand experience creating and working with taxonomies that aid in navigation, search, and discovery across a range of disciplines.
0x01 - Newton's Third Law: Static vs. Dynamic AbusersOWASP Beja
f you offer a service on the web, odds are that someone will abuse it. Be it an API, a SaaS, a PaaS, or even a static website, someone somewhere will try to figure out a way to use it to their own needs. In this talk we'll compare measures that are effective against static attackers and how to battle a dynamic attacker who adapts to your counter-measures.
About the Speaker
===============
Diogo Sousa, Engineering Manager @ Canonical
An opinionated individual with an interest in cryptography and its intersection with secure software development.
This presentation, created by Syed Faiz ul Hassan, explores the profound influence of media on public perception and behavior. It delves into the evolution of media from oral traditions to modern digital and social media platforms. Key topics include the role of media in information propagation, socialization, crisis awareness, globalization, and education. The presentation also examines media influence through agenda setting, propaganda, and manipulative techniques used by advertisers and marketers. Furthermore, it highlights the impact of surveillance enabled by media technologies on personal behavior and preferences. Through this comprehensive overview, the presentation aims to shed light on how media shapes collective consciousness and public opinion.
Acorn Recovery: Restore IT infra within minutesIP ServerOne
Introducing Acorn Recovery as a Service, a simple, fast, and secure managed disaster recovery (DRaaS) by IP ServerOne. A DR solution that helps restore your IT infra within minutes.
Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing
1. Effective and Efficient API Misuse Detection via
Exception Propagation and Search-Based Testing
Maria Kechagia
Univ. College London,
London, United Kingdom
Xavier Devroey
Delft Univ. of Technology
Delft, Netherlands
Annibale Panichella
Delft Univ. of Technology
Delft, Netherlands
Georgios Gousios
Delft Univ. of Technology
Delft, Netherlands
Arie van Deursen
Delft Univ. of Technology
Delft, Netherlands
2. APIs everywhere
• An Application Programming Interface (API)
is a set of functions and procedures offering
services
• Applications are built by combining API
calls
• Example: Maven dependency graph with
186,384 Nodes and 1,229,083 Edges
• Usage constraints
• Explicit
• Implicit
• API misuses
• Violation of usage constraints
!2
Maven dependency graph using Force Atlas 2
https://github.com/ogirardot/meta-deps
7. Catcher
!7
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
8. Catcher
!8
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
1. Spotting
potential runtime
exceptions throws
API
9. Catcher
!9
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
10. Catcher
!10
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
11. Catcher
!11
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
12. Catcher
!12
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
5. Expose the misuses
13. Static exception propagation
!13
Static exception propagation analysis
API Client
public void a1(…){
…
throw new IllegalStateException();
…
}
public void a2(…){
…
}
public void a3(…){
…
a1(…)
…
}
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
14. Filtering candidate misuses
!14
Static exception propagation analysis
API Client
public void a1(…){
…
throw new IllegalStateException();
…
}
public void a2(…){
…
}
public void a3(…){
…
a1(…)
…
}
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
Candidate misuses
IllegalStateException
at a1()
at a3()
at c1()
IllegalStateException
at a1()
at c3()
15. Search-based test case generation
!15
Candidate misuses
IllegalStateException
at a1()
at a3()
at c1()
IllegalStateException
at a1()
at c3()
Focused search-based test generation
EvoSuite
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
public void testC1(…){
…
}
public void testC3(…){
…
}
16. Static exception propagation analysis
• Builds the API call graph
• Annotated with the list of runtime exceptions
• Builds the global call graph
• Link the client call graph to the API call graph
• Propagate exceptions from the API to the client
• Maximal propagation depth (≤4)
• Filter exceptions handled by the client
• Each propagation is a candidate misuse
• Relies on Soot (http://sable.github.io/soot/)
!16
a1() a2() a3()
a4() a5()
c1() c2() c3()
c4()
API
Client
17. Focused Search-based test generation
• Dynamic Many-Objective Sorting Algorithm (DynaMOSA)
• Objectives
• For each misuse
• Reaching propagation point (branch coverage)
• Client method input diversity
• Client method output diversity
• Misuse detected if and only if
• Exception is thrown by the test case
• Propagation is the same as in the candidate misuse
• Relies on EvoSuite (http://www.evosuite.org)
!17
Random test
cases tests
Evolutionary
process
Test cases
exposing misuses tests
18. Evaluation on the JDK API
• RQ1: How do existing unit level coverage-based test
generation tools perform in discovering API misuses?
• RQ2: Does Catcher improve the performance of existing
test coverage-based approaches on detecting API
misuses?
• RQ3: What types of API misuses does Catcher expose?
• Comparison with (plain) EvoSuite with…
• DynaMOSA + default coverage criteria
• On all classes of the client apps
• 25 runs x 3 minutes x each class under test
!18
Client project Files
(#)
LOC (K)
bcel-6.2 489 39
commons-cli-1.4 50 7
commons-codec-1.12 124 10
commons-collections-4.2 535 63
commons-compress-1.17 352 43
commons-lang-3.7 323 76
commons-math-3.6.1 1617 209
easymock-3.6 204 14
gson-2.8.5 206 25
hamcrest-core-1.3 152 7
jackson-databind-2.9.6 919 114
javassist-3.23.1 527 82
jcommander-1.71 139 6
jfreechart-1.5.0 990 134
jodat-time-2.10 330 86
jopt-simple-5.0.4 192 9
natty-0.13 27 3
neo4j-java-driver-1.6.2 510 52
shiro-core-1.3.2 653 31
xwiki-commons-job-10.6 67 3
xwiki-commons-text-10.6 3 101
19. Evaluation results
!19
EvoSuite (RQ1) Catcher (RQ2)
Effectiveness
165 (mean=123)
unique misuses
243 (mean=207) unique
misuses
Efficiency
~15 days to produce
the test cases for all
the projects (~17
hours per project)
~2 days to produce the
test cases for all the
projects (~2 hours per
project)
Detected misuses overlap:
1632 80
EvoSuite Catcher
21. Implications
!21
Research
New research directions towards focused search-based testing.
Automated generation of the test oracle.
Further examination of the benefits of combining static analysis and testing.
Practical
Easily deployable in a continuous integration pipelines.
Improvement of the reliability of APIs leading to fewer client application crashes.
24. !24
Catcher
!X
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
5. Expose the misuses
Evaluation on the JDK API
• RQ1: How do existing unit level coverage-based test
generation tools perform in discovering API misuses?
• RQ2: Does Catcher improve the performance of existing
test coverage-based approaches on detecting API
misuses?
• RQ3: What types of API misuses does Catcher expose?
• Comparison with (plain) EvoSuite
• DynaMOSA + default coverage criteria
• On all classes of the client apps
• 25 runs x 25 minutes x each class under test
!X
Client project Files
(#)
LOC (K)
bcel-6.2 489 39
commons-cli-1.4 50 7
commons-codec-1.12 124 10
commons-collections-4.2 535 63
commons-compress-1.17 352 43
commons-lang-3.7 323 76
commons-math-3.6.1 1617 209
easymock-3.6 204 14
gson-2.8.5 206 25
hamcrest-core-1.3 152 7
jackson-databind-2.9.6 919 114
javassist-3.23.1 527 82
jcommander-1.71 139 6
jfreechart-1.5.0 990 134
jodat-time-2.10 330 86
jopt-simple-5.0.4 192 9
natty-0.13 27 3
neo4j-java-driver-1.6.2 510 52
shiro-core-1.3.2 653 31
xwiki-commons-job-10.6 67 3
xwiki-commons-text-10.6 3 101
Evaluation results
!X
EvoSuite (RQ1) Catcher (RQ2)
Effectiveness
166 (mean=123)
unique misuses
243 (mean=207) unique
misuses
Efficiency
~15 days to produce
the test cases for all
the projects (~17
hours per project)
~2 days to produce the
test cases for all the
projects (~2 hours per
project)
Detected misuses overlap:
1632 80
EvoSuite Catcher
Implications
!X
Research
New research directions towards focused search-based testing.
Automated generation of the test oracle.
Further examination of the benefits of combining static analysis and testing.
Practical
Easily deployable in a continuous integration pipelines.
Improvement of the reliability of APIs leading to fewer client application crashes.
25. !25
Eective and Eicient API Misuse Detection via Exception
Propagation and Search-Based Testing
Maria Kechagia∗
m.kechagia@ucl.ac.uk
University College London
London, United Kingdom
Xavier Devroey
x.d.m.devroey@tudelft.nl
Delft University of Technology
Delft, Netherlands
Annibale Panichella
a.panichella@tudelft.nl
Delft University of Technology
Delft, Netherlands
Georgios Gousios
g.gousios@tudelft.nl
Delft University of Technology
Delft, Netherlands
Arie van Deursen
arie.vandeursen@tudelft.nl
Delft University of Technology
Delft, Netherlands
ABSTRACT
Application Programming Interfaces (APIs) typically come with
(implicit) usage constraints. The violations of these constraints (API
misuses) can lead to software crashes. Even though there are sev-
eral tools that can detect API misuses, most of them suer from a
very high rate of false positives. We introduce Catcher, a novel API
misuse detection approach that combines static exception propa-
gation analysis with automatic search-based test case generation
to eectively and eciently pinpoint crash-prone API misuses in
client applications. We validate Catcher against 21 Java applications,
targeting misuses of the Java platform’s API. Our results indicate
that Catcher is able to generate test cases that uncover 243 (unique)
API misuses that result in crashes. Our empirical evaluation shows
that Catcher can detect a large number of misuses (77 cases) that
would remain undetected by the traditional coverage-based test
case generator EvoSuite. Additionally, on average, Catcher is eight
times faster than EvoSuite in generating test cases for the identi-
ed misuses. Finally, we nd that the majority of the exceptions
ACM Reference Format:
Maria Kechagia, Xavier Devroey, Annibale Panichella, Georgios Gousios,
and Arie van Deursen. 2019. Eective and Ecient API Misuse Detection via
Exception Propagation and Search-Based Testing. In Proceedings of the 28th
ACM SIGSOFT International Symposium on Software Testing and Analysis
(ISSTA ’19), July 15–19, 2019, Beijing, China. ACM, New York, NY, USA,
12 pages. https://doi.org/10.1145/3293882.3330552
1 INTRODUCTION
Developers use external libraries to increase the velocity and reduce
the production cost of software projects [38]. While increasing pro-
ductivity, this form of software reuse comes with several challenges:
dependencies need to be kept up to date [15], developers must learn
the intricacies of each imported Application Programming Interface
(), and resulting client programs should be robust, ecient, and
responsive. Correctly using third-party s is not an easy task;
many s are millions of lines of code large, interact with various
https://github.com/mkechagia/Catcher
26. Effective and Efficient API Misuse Detection via
Exception Propagation and Search-Based Testing
Maria Kechagia, Xavier Devroey, Annibale Panichella,
Georgios Gousios and Arie van Deursen
