SlideShare a Scribd company logo

20181106 arie van_deursen_testday2018

Test amplification tools are being developed to generate and improve test cases. These include DSPOT which generates missing assertions, DESCARTES which makes bigger changes to speed up mutation testing, and CAMP which amplifies tests through Docker mutations. Search-based techniques are also being used to automatically reproduce crashes from stack traces. EvoCrash and Botsing employ genetic algorithms to evolve an initial random test suite towards reproducing the crash. Over 200 real-world crashes have been used to evaluate the techniques. Recent work explores using behavioral models or existing test cases to improve the initial test population.

1 of 29
Download to read offline
Test Amplifcation
Arie van Deursen, TU Delft
Dutch Testing Day, November 6, 2018
Pouria Derakhshanfar, Xavier Devroey, Mozhan Soltani, Annibale Panichella, Andy Zaidman
1
2
Test Amplification Tools Under Construction
DSPOT:
Detect and generate missing assertions for Junit test cases
DESCARTES:
Speed up mutation testing by making bigger changes (drop method body)
CAMP:
Environment test amplification through Docker mutations
EvoCrash / Botsing:
Test suite amplification via crash reproduction
https://www.stamp-project.eu/
https://github.com/STAMP-project
3
java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
at org…..SolrDocumentReferenceResolver.resolve(….java:48)
at …
Crash!?!
4
Crash Reproduction
• File an issue
• Discover steps to
reproduce the crash
• Example: XWIKI-13031
Challenges:
• Labor intensive
• Hard to automate
5
Java Stack Trace (Issue XWIKI-13031)
java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
at org…..SolrDocumentReferenceResolver.resolve(….java:48)
at …
Exception
Frames
{Target ➞
6
Search-Based Crash Reproduction
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
Random initial
test suite
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a()
c() e()d()
e()
c() b() e()
a()
c() e()d()
e()
c() b() e()
Evolutionary
search
a() e()d()
Exception:
at x(…)
at y(…)
at e(…)
Exception:
at x(…)
at y(…)
at e(…)
Crash reproducing
test caseStack trace
7
Soltani, Panichella, van Deursen. “Search-Based Crash Reproduction and Its Impact on Debugging.”
IEEE Transactions on Software Engineering, 2018. pure.tudelft.nl
Crash-reproducing Test Case
public void test0() throws Throwable {
…
SolrEntityReferenceResolver solrEntityReferenceResolver0 = new …();
EntityReferenceResolver entityReferenceResolver0 = … mock(…);
solrDocument0.put("wiki", (Object) entityType0);
Injector.inject(solrEntityReferenceResolver0, …);
Injector.validateBean(solrEntityReferenceResolver0, …);
…
// Undeclared exception!
solrEntityReferenceResolver0.resolve(solrDocument0, entityType0, objectArray0);
}
java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
8
EvoSuite
• Search based test generation
• Many random JUnit tests
• Optimized to maximize (e.g.)
branch coverage
• Combines and improves tests
to optimize overall fitness
• http://www.evosuite.org/
9
Initialize population
Evaluate fitness
Next generation
Selection
Crossover
Mutation
Reinsertion
[fitness == 0 or
budget exhausted]
EvoCrash
• Implemented on top of
EvoSuite
• Requires
• Stack trace
• Binaries: .jar files
• Time budget: Set by user
• Produce single test case
• That reproduces the crash
10
Initialize population
Evaluate fitness
Next generation
Selection
Crossover
Mutation
Reinsertion
[fitness == 0 or
budget exhausted]
Initialize
Population
• Guided initialization
• Random method calls
• Guarantees that a target
method call is inserted in
each test at least once
• Direct for public and
protected methods
• Indirect for private methods
11
Initialize population
Evaluate fitness
Next generation
Selection
Crossover
Mutation
Reinsertion
[fitness == 0 or
budget exhausted]
Evaluate Fitness
• Global fitness function to guide
generation process
• Line coverage
• How far are we from the line
where the exception is thrown?
• Exception coverage
• Is the exception thrown?
• Stack trace similarity
• How similar is the stack trace
compared to the original (given)
stack trace?
12
Initialize population
Evaluate fitness
Next generation
Selection
Crossover
Mutation
Reinsertion
[fitness == 0 or
budget exhausted]
Next Generation
• Selection
• Fittest tests according to the
fitness function
• Guided crossover
• Single-point crossover
• Check that call to target method
is preserved
• Guided mutation
• Add/change/drop statements
• Check that call to target method
is preserved
13
Initialize population
Evaluate fitness
Next generation
Selection
Crossover
Mutation
Reinsertion
[fitness == 0 or
budget exhausted]
Does it Work?
The JCrashPack Crash Replication Benchmark
• 200 crashes from various open source projects
• XWiki
• STAMP partner
• From XWiki issue tracking system: 51 crashes
• Defects4J applications
• State of the art fault localization benchmark
• 73 crashes (with fixes)
• Elasticsearch
• Based on popularity
• From Elasticsearch issue tracking system: 76 crashes
• Filtered, verified, cleaned up, right jar versions, …
14 https://github.com/STAMP-project/JCrashPack
ExRunner:
Running Crash Replication
• Python tool to benchmark crash reproduction tools
• Multithreaded execution
• Monitors tool executions
Stack traces JobGenerator
Observer
Thread 1
Thread n
.
.
Job 1
Logs
results
Tool Configs. Job n
Test case
Jar files
.
.
Logs
results
Test case
15
EvoCrash Applied to JCrashPack
Defects4J XWiki Elasticsearch
crashed
ex.thrown
failed
linereached
reproduced
crashed
ex.thrown
failed
linereached
reproduced
crashed
ex.thrown
failed
linereached
reproduced
1
10
100
Numberofframes(log.scale)
16
Identified 12 Key Challenges
• Input data generation
• For complex inputs, generic types, etc.
• Environmental dependencies
• Environment state hard to manage at unit level
• Complex code
• Long methods, with lot of nested predicates
• Abstract classes and methods
• Cannot be instantiated and one concrete implementation is picked randomly
• […]
17
Mozhan Soltani · Pouria Derakhshanfar · Xavier Devroey · Arie van Deursen
An Empirical Evaluation of Search-Based Crash Reproduction. TU Delft, 2018. In preparation.
Work in Progress:
Improved Seeding
Hypothesis
An initial test suite close to
actual usage of the class is more
likely to lead to a crash
reproducing test case than a
random one
18
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
Random initial
test suite
Exception:
at x(…)
at y(…)
at e(…)
Stack trace
Pouria Derakhshanfar, Xavier Devroey, Gilles Perrouin, Andy Zaidman and Arie van Deursen.
Search-based Crash Reproduction using Behavioral Model Seeding. TU Delft, 2018. Submitted
Test seeding
Use the existing tests to
generate the initial test
suite
19
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() c()
c() e()c()
e()
c() a()
Exception:
at x(…)
at y(…)
at e(…)
Stack trace
a() c()
c() c()
b() a() e()
Existing tests
e()
Random initial
test suite
Existing tests
subset
Model seeding
Use a model of
method usage to
generate the initial
test suite
20
Random initial
test suite
Exception:
at x(…)
at y(…)
at e(…)
Stack trace
a()b()
c() e()
d()
Model
Model-driven
initial test suite
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
b() a()
c() e()d()
e()
c() e()
a()
Call Sequence Models
21
a()b()
c() e()
d()
Model
• Model generated from
sequences of method calls
• Coming from
• Source code
• Static analysis
• Test cases
• Dynamic analysis
• Operations logs
• Online analysis
N-gram inference
[b(), a(), e()]
[c(), d(), a(), e()]
[b(), a(), d(), a(), d(), a(), e()]
…
Example Transitions for java.util.LinkedList
size()
add(Object)iterator()
S0
get(int)
remove(int)
S1
S2S3
remove(int)
S4
S5
SX
size()
add(Object)
size()
add(Object)
22
Model example
23
Model Seeding Approach
Guided Genetic Algorithm
app.jar
tests.jar
Pr[clone] Guided
initialization
Fitness
evaluation
[fitness == 0 or
budget exhausted]
Selection
Guided
crossover
Guided
mutation
Reinsertion
Stacktrace
Test
cases
Objects
pool
Test seeding
Bheavioral model seeding
populatesInstrumented
execution
Static analysis
Instrumented
execution
Call
sequ.
Models
inference
populates
Abstract
test cases
selectionModels
Pr[pick init]
Pr[pick mut]
1
2
3 4
5
A
24
0
25
50
75
not started failed line reached ex. thrown reproduced
Numberofframes
Configurations
no s.
test s. 0.2
test s. 0.5
test s. 0.8
test s. 1.0
model s. 0.2
model s. 0.5
model s. 0.8
model s. 1.0
25
Model Seeding: Results
• Behavioral seeding outperforms test seeding / no seeding
• 13 more crashes could be reproduced
• No performance overhead
• Extra crashes are more complex
• Higher frame levels
• Better at ”industrial” cases
• Model seeding outperforms test seeding.
26
Implementation: Botsing
• Open source implementation of EvoCrash approach
• Uses EvoSuite as library for instrumentation
• Extensible, modular, tested
• Used as test bed for new crash reproduction ideas (model seeding)
• https://github.com/STAMP-project/botsing
27
28https://www.stamp-project.eu/
29https://www.stamp-project.eu/

Recommended

Research overview Oct. 2018
Research overview Oct. 2018Research overview Oct. 2018
Research overview Oct. 2018XavierDevroey
 
"Quantum" performance effects
"Quantum" performance effects"Quantum" performance effects
"Quantum" performance effectsSergey Kuksenko
 
"Quantum" Performance Effects
"Quantum" Performance Effects"Quantum" Performance Effects
"Quantum" Performance EffectsSergey Kuksenko
 
Java Performance: Speedup your application with hardware counters
Java Performance: Speedup your application with hardware countersJava Performance: Speedup your application with hardware counters
Java Performance: Speedup your application with hardware countersSergey Kuksenko
 
Rotten Green Tests
Rotten Green TestsRotten Green Tests
Rotten Green TestsESUG
 
Software Testing:
 A Research Travelogue 
(2000–2014)
Software Testing:
 A Research Travelogue 
(2000–2014)Software Testing:
 A Research Travelogue 
(2000–2014)
Software Testing:
 A Research Travelogue 
(2000–2014)Alex Orso
 

More Related Content

Similar to 20181106 arie van_deursen_testday2018

Lorentz workshop - 2018
Lorentz workshop - 2018Lorentz workshop - 2018
Lorentz workshop - 2018XavierDevroey
 
The Power of Probabilistic Thinking (keynote talk at ASE 2016)
The Power of Probabilistic Thinking (keynote talk at ASE 2016)The Power of Probabilistic Thinking (keynote talk at ASE 2016)
The Power of Probabilistic Thinking (keynote talk at ASE 2016)David Rosenblum
 
Genetic programming with clojure.spec and Beyond
Genetic programming with clojure.spec and BeyondGenetic programming with clojure.spec and Beyond
Genetic programming with clojure.spec and BeyondCarin Meier
 
Introduction to Julia
Introduction to JuliaIntroduction to Julia
Introduction to Julia岳華 杜
 
GTAC 2014: What lurks in test suites?
GTAC 2014: What lurks in test suites?GTAC 2014: What lurks in test suites?
GTAC 2014: What lurks in test suites?Patrick Lam
 
Fast Automated Unpacking and Classification of Malware
Fast Automated Unpacking and Classification of MalwareFast Automated Unpacking and Classification of Malware
Fast Automated Unpacking and Classification of MalwareSilvio Cesare
 
Diversity mechanisms for evolutionary populations in Search-Based Software En...
Diversity mechanisms for evolutionary populations in Search-Based Software En...Diversity mechanisms for evolutionary populations in Search-Based Software En...
Diversity mechanisms for evolutionary populations in Search-Based Software En...Annibale Panichella
 
Esa 2014 qiime
Esa 2014 qiimeEsa 2014 qiime
Esa 2014 qiimeZech Xu
 
Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...
Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...
Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...Pluribus One
 
Robustness Metrics for ML Models based on Deep Learning Methods
Robustness Metrics for ML Models based on Deep Learning MethodsRobustness Metrics for ML Models based on Deep Learning Methods
Robustness Metrics for ML Models based on Deep Learning MethodsData Science Milan
 
Madrid JAM limitaciones - dificultades
Madrid JAM limitaciones - dificultadesMadrid JAM limitaciones - dificultades
Madrid JAM limitaciones - dificultadesJavier Delgado Garrido
 
Neal Gafter Java Evolution
Neal Gafter Java EvolutionNeal Gafter Java Evolution
Neal Gafter Java Evolutiondeimos
 
Two methods for optimising cognitive model parameters
Two methods for optimising cognitive model parametersTwo methods for optimising cognitive model parameters
Two methods for optimising cognitive model parametersUniversity of Huddersfield
 
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...University of Antwerp
 
B Chapman - Toolkit for variation comparison and analysis
B Chapman - Toolkit for variation comparison and analysisB Chapman - Toolkit for variation comparison and analysis
B Chapman - Toolkit for variation comparison and analysisJan Aerts
 

Similar to 20181106 arie van_deursen_testday2018 (20)

Lorentz workshop - 2018
Lorentz workshop - 2018Lorentz workshop - 2018
Lorentz workshop - 2018
 
The Power of Probabilistic Thinking (keynote talk at ASE 2016)
The Power of Probabilistic Thinking (keynote talk at ASE 2016)The Power of Probabilistic Thinking (keynote talk at ASE 2016)
The Power of Probabilistic Thinking (keynote talk at ASE 2016)
 
Machine Learning at Geeky Base 2
Machine Learning at Geeky Base 2Machine Learning at Geeky Base 2
Machine Learning at Geeky Base 2
 
Genetic programming with clojure.spec and Beyond
Genetic programming with clojure.spec and BeyondGenetic programming with clojure.spec and Beyond
Genetic programming with clojure.spec and Beyond
 
Introduction to Julia
Introduction to JuliaIntroduction to Julia
Introduction to Julia
 
Good Practices On Test Automation
Good Practices On Test AutomationGood Practices On Test Automation
Good Practices On Test Automation
 
GTAC 2014: What lurks in test suites?
GTAC 2014: What lurks in test suites?GTAC 2014: What lurks in test suites?
GTAC 2014: What lurks in test suites?
 
Fast Automated Unpacking and Classification of Malware
Fast Automated Unpacking and Classification of MalwareFast Automated Unpacking and Classification of Malware
Fast Automated Unpacking and Classification of Malware
 
Diversity mechanisms for evolutionary populations in Search-Based Software En...
Diversity mechanisms for evolutionary populations in Search-Based Software En...Diversity mechanisms for evolutionary populations in Search-Based Software En...
Diversity mechanisms for evolutionary populations in Search-Based Software En...
 
Esa 2014 qiime
Esa 2014 qiimeEsa 2014 qiime
Esa 2014 qiime
 
Mutant Tests Too: The SQL
Mutant Tests Too: The SQLMutant Tests Too: The SQL
Mutant Tests Too: The SQL
 
Need 4 Speed FI
Need 4 Speed FINeed 4 Speed FI
Need 4 Speed FI
 
Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...
Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...
Battista Biggio @ AISec 2013 - Is Data Clustering in Adversarial Settings Sec...
 
Robustness Metrics for ML Models based on Deep Learning Methods
Robustness Metrics for ML Models based on Deep Learning MethodsRobustness Metrics for ML Models based on Deep Learning Methods
Robustness Metrics for ML Models based on Deep Learning Methods
 
Madrid JAM limitaciones - dificultades
Madrid JAM limitaciones - dificultadesMadrid JAM limitaciones - dificultades
Madrid JAM limitaciones - dificultades
 
Neal Gafter Java Evolution
Neal Gafter Java EvolutionNeal Gafter Java Evolution
Neal Gafter Java Evolution
 
Two methods for optimising cognitive model parameters
Two methods for optimising cognitive model parametersTwo methods for optimising cognitive model parameters
Two methods for optimising cognitive model parameters
 
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
 
R Basics
R BasicsR Basics
R Basics
 
B Chapman - Toolkit for variation comparison and analysis
B Chapman - Toolkit for variation comparison and analysisB Chapman - Toolkit for variation comparison and analysis
B Chapman - Toolkit for variation comparison and analysis
 

More from STAMP Project

STAMP Descartes Presentation
STAMP Descartes PresentationSTAMP Descartes Presentation
STAMP Descartes PresentationSTAMP Project
 
Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenSTAMP Project
 
STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project
 
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP Project
 
Mutation Testing at BzhJUG
Mutation Testing at BzhJUGMutation Testing at BzhJUG
Mutation Testing at BzhJUGSTAMP Project
 
1803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum20181803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum2018STAMP Project
 
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...STAMP Project
 
Mutate and Test your Tests
Mutate and Test your TestsMutate and Test your Tests
Mutate and Test your TestsSTAMP Project
 
Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17STAMP Project
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP Project
 

More from STAMP Project (11)

STAMP Descartes Presentation
STAMP Descartes PresentationSTAMP Descartes Presentation
STAMP Descartes Presentation
 
Mutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, SwedenMutation Testing Workshop at Ericsson, Kista, Sweden
Mutation Testing Workshop at Ericsson, Kista, Sweden
 
STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018STAMP Project at Telecom Valley 6 Dec. 2018
STAMP Project at Telecom Valley 6 Dec. 2018
 
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
STAMP ActiveEon Use Case at Telecom Valley Dec. 2018
 
Mutation Testing at BzhJUG
Mutation Testing at BzhJUGMutation Testing at BzhJUG
Mutation Testing at BzhJUG
 
Mutation @ Spotify
Mutation @ Spotify Mutation @ Spotify
Mutation @ Spotify
 
1803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum20181803_STAMP_OpenCloudForum2018
1803_STAMP_OpenCloudForum2018
 
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Sou...
 
Mutate and Test your Tests
Mutate and Test your TestsMutate and Test your Tests
Mutate and Test your Tests
 
Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17Stamp Project presentation at OW2con'17
Stamp Project presentation at OW2con'17
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017
 

Recently uploaded

Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System AdministratorsGlobus
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaSGlobus
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusGlobus
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentThierry Gayet
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration TopicsGlobus
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with GlobusGlobus
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio, Inc.
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesGlobus
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfssuser82c38d
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Fermin Galan
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019VICTOR MAESTRE RAMIREZ
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxMindInventory
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureHironori Washizaki
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleAmir Moghimi
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowNaoki (Neo) SATO
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارsohilww
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for ResearchersGlobus
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCIOWomenMagazine
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesAnisha Agarwal
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Dmitry Zinoviev
 

Recently uploaded (20)

Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaS
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using Globus
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managment
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All Scales
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdf
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227
 
Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptx
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about Architecture
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scale
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flow
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
 
An Introduction to Globus for Researchers
An Introduction to Globus for ResearchersAn Introduction to Globus for Researchers
An Introduction to Globus for Researchers
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdf
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT Services
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)
 

20181106 arie van_deursen_testday2018

  • 1. Test Amplifcation Arie van Deursen, TU Delft Dutch Testing Day, November 6, 2018 Pouria Derakhshanfar, Xavier Devroey, Mozhan Soltani, Annibale Panichella, Andy Zaidman 1
  • 2. 2
  • 3. Test Amplification Tools Under Construction DSPOT: Detect and generate missing assertions for Junit test cases DESCARTES: Speed up mutation testing by making bigger changes (drop method body) CAMP: Environment test amplification through Docker mutations EvoCrash / Botsing: Test suite amplification via crash reproduction https://www.stamp-project.eu/ https://github.com/STAMP-project 3
  • 4. java.lang.ClassCastException: […] at org…..SolrEntityReferenceResolver.getWikiReference(....java:93) at org…..SolrEntityReferenceResolver.getEntityReference(….java:70) at org…..SolrEntityReferenceResolver.resolve(….java:63) at org…..SolrDocumentReferenceResolver.resolve(….java:48) at … Crash!?! 4
  • 5. Crash Reproduction • File an issue • Discover steps to reproduce the crash • Example: XWIKI-13031 Challenges: • Labor intensive • Hard to automate 5
  • 6. Java Stack Trace (Issue XWIKI-13031) java.lang.ClassCastException: […] at org…..SolrEntityReferenceResolver.getWikiReference(....java:93) at org…..SolrEntityReferenceResolver.getEntityReference(….java:70) at org…..SolrEntityReferenceResolver.resolve(….java:63) at org…..SolrDocumentReferenceResolver.resolve(….java:48) at … Exception Frames {Target ➞ 6
  • 7. Search-Based Crash Reproduction a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() Random initial test suite a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() c() e()d() e() c() b() e() a() c() e()d() e() c() b() e() Evolutionary search a() e()d() Exception: at x(…) at y(…) at e(…) Exception: at x(…) at y(…) at e(…) Crash reproducing test caseStack trace 7 Soltani, Panichella, van Deursen. “Search-Based Crash Reproduction and Its Impact on Debugging.” IEEE Transactions on Software Engineering, 2018. pure.tudelft.nl
  • 8. Crash-reproducing Test Case public void test0() throws Throwable { … SolrEntityReferenceResolver solrEntityReferenceResolver0 = new …(); EntityReferenceResolver entityReferenceResolver0 = … mock(…); solrDocument0.put("wiki", (Object) entityType0); Injector.inject(solrEntityReferenceResolver0, …); Injector.validateBean(solrEntityReferenceResolver0, …); … // Undeclared exception! solrEntityReferenceResolver0.resolve(solrDocument0, entityType0, objectArray0); } java.lang.ClassCastException: […] at org…..SolrEntityReferenceResolver.getWikiReference(....java:93) at org…..SolrEntityReferenceResolver.getEntityReference(….java:70) at org…..SolrEntityReferenceResolver.resolve(….java:63) 8
  • 9. EvoSuite • Search based test generation • Many random JUnit tests • Optimized to maximize (e.g.) branch coverage • Combines and improves tests to optimize overall fitness • http://www.evosuite.org/ 9 Initialize population Evaluate fitness Next generation Selection Crossover Mutation Reinsertion [fitness == 0 or budget exhausted]
  • 10. EvoCrash • Implemented on top of EvoSuite • Requires • Stack trace • Binaries: .jar files • Time budget: Set by user • Produce single test case • That reproduces the crash 10 Initialize population Evaluate fitness Next generation Selection Crossover Mutation Reinsertion [fitness == 0 or budget exhausted]
  • 11. Initialize Population • Guided initialization • Random method calls • Guarantees that a target method call is inserted in each test at least once • Direct for public and protected methods • Indirect for private methods 11 Initialize population Evaluate fitness Next generation Selection Crossover Mutation Reinsertion [fitness == 0 or budget exhausted]
  • 12. Evaluate Fitness • Global fitness function to guide generation process • Line coverage • How far are we from the line where the exception is thrown? • Exception coverage • Is the exception thrown? • Stack trace similarity • How similar is the stack trace compared to the original (given) stack trace? 12 Initialize population Evaluate fitness Next generation Selection Crossover Mutation Reinsertion [fitness == 0 or budget exhausted]
  • 13. Next Generation • Selection • Fittest tests according to the fitness function • Guided crossover • Single-point crossover • Check that call to target method is preserved • Guided mutation • Add/change/drop statements • Check that call to target method is preserved 13 Initialize population Evaluate fitness Next generation Selection Crossover Mutation Reinsertion [fitness == 0 or budget exhausted]
  • 14. Does it Work? The JCrashPack Crash Replication Benchmark • 200 crashes from various open source projects • XWiki • STAMP partner • From XWiki issue tracking system: 51 crashes • Defects4J applications • State of the art fault localization benchmark • 73 crashes (with fixes) • Elasticsearch • Based on popularity • From Elasticsearch issue tracking system: 76 crashes • Filtered, verified, cleaned up, right jar versions, … 14 https://github.com/STAMP-project/JCrashPack
  • 15. ExRunner: Running Crash Replication • Python tool to benchmark crash reproduction tools • Multithreaded execution • Monitors tool executions Stack traces JobGenerator Observer Thread 1 Thread n . . Job 1 Logs results Tool Configs. Job n Test case Jar files . . Logs results Test case 15
  • 16. EvoCrash Applied to JCrashPack Defects4J XWiki Elasticsearch crashed ex.thrown failed linereached reproduced crashed ex.thrown failed linereached reproduced crashed ex.thrown failed linereached reproduced 1 10 100 Numberofframes(log.scale) 16
  • 17. Identified 12 Key Challenges • Input data generation • For complex inputs, generic types, etc. • Environmental dependencies • Environment state hard to manage at unit level • Complex code • Long methods, with lot of nested predicates • Abstract classes and methods • Cannot be instantiated and one concrete implementation is picked randomly • […] 17 Mozhan Soltani · Pouria Derakhshanfar · Xavier Devroey · Arie van Deursen An Empirical Evaluation of Search-Based Crash Reproduction. TU Delft, 2018. In preparation.
  • 18. Work in Progress: Improved Seeding Hypothesis An initial test suite close to actual usage of the class is more likely to lead to a crash reproducing test case than a random one 18 a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() Random initial test suite Exception: at x(…) at y(…) at e(…) Stack trace Pouria Derakhshanfar, Xavier Devroey, Gilles Perrouin, Andy Zaidman and Arie van Deursen. Search-based Crash Reproduction using Behavioral Model Seeding. TU Delft, 2018. Submitted
  • 19. Test seeding Use the existing tests to generate the initial test suite 19 a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() c() c() e()c() e() c() a() Exception: at x(…) at y(…) at e(…) Stack trace a() c() c() c() b() a() e() Existing tests e() Random initial test suite Existing tests subset
  • 20. Model seeding Use a model of method usage to generate the initial test suite 20 Random initial test suite Exception: at x(…) at y(…) at e(…) Stack trace a()b() c() e() d() Model Model-driven initial test suite a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() a() b() c() e()d() e() c() e() b() a() c() e()d() e() c() e() a()
  • 21. Call Sequence Models 21 a()b() c() e() d() Model • Model generated from sequences of method calls • Coming from • Source code • Static analysis • Test cases • Dynamic analysis • Operations logs • Online analysis N-gram inference [b(), a(), e()] [c(), d(), a(), e()] [b(), a(), d(), a(), d(), a(), e()] …
  • 22. Example Transitions for java.util.LinkedList size() add(Object)iterator() S0 get(int) remove(int) S1 S2S3 remove(int) S4 S5 SX size() add(Object) size() add(Object) 22
  • 24. Model Seeding Approach Guided Genetic Algorithm app.jar tests.jar Pr[clone] Guided initialization Fitness evaluation [fitness == 0 or budget exhausted] Selection Guided crossover Guided mutation Reinsertion Stacktrace Test cases Objects pool Test seeding Bheavioral model seeding populatesInstrumented execution Static analysis Instrumented execution Call sequ. Models inference populates Abstract test cases selectionModels Pr[pick init] Pr[pick mut] 1 2 3 4 5 A 24
  • 25. 0 25 50 75 not started failed line reached ex. thrown reproduced Numberofframes Configurations no s. test s. 0.2 test s. 0.5 test s. 0.8 test s. 1.0 model s. 0.2 model s. 0.5 model s. 0.8 model s. 1.0 25
  • 26. Model Seeding: Results • Behavioral seeding outperforms test seeding / no seeding • 13 more crashes could be reproduced • No performance overhead • Extra crashes are more complex • Higher frame levels • Better at ”industrial” cases • Model seeding outperforms test seeding. 26
  • 27. Implementation: Botsing • Open source implementation of EvoCrash approach • Uses EvoSuite as library for instrumentation • Extensible, modular, tested • Used as test bed for new crash reproduction ideas (model seeding) • https://github.com/STAMP-project/botsing 27