SlideShare a Scribd company logo
Universiteit Antwerpen
Reproducible Crashes:


Fuzzing Pharo by Mutating the Test
Methods
Mehrdad Abdi — Henrique Rocha — Serge Demeyer


VST 2021 (March 9th)
VST2021 © Serge Demeyer
• Background


- Fuzzing


- Mutation Coverage


- Test Amplification


• Fuzzing by amplification


- Motivating examples


- Proposed approach


- Challenge


• Conclusions
Outline
2
Fuzz


Testing
Mutation


Coverage
Test


Amplification
VST2021 © Serge Demeyer
Testing
4
Program


Under


Test
Valid Input
Expected output
Software Testing is the process of executing a program or system with
the intent of finding errors.


(Myers, Glenford J., The art of software testing. Wiley, 1979
VST2021 © Serge Demeyer
Fuzz Testing
5
Program


Under


Test
Unexpected Input
Crash/Freeze
Fuzzing (or Fuzz Testing) is an automated testing technique to verify the
robustness of a program-under-test.


Valid input is replaced by random values with the goal to force the
program-under-test into unexpected exceptional behavior.
VST2021 © Serge Demeyer
Black Box Fuzzing
6
Access to
execution
Mutating valid
inputs
VST2021 © Serge Demeyer
White Box Fuzzing
7
Access to code,
models, specs, …
Program Analysis
VST2021 © Serge Demeyer
Grey Box Fuzzing
8
Limited Access


(bytecode, trace, …)
Reverse
Engineering
VST2021 © Serge Demeyer
Code Coverage
9
Program


Under


Test
Valid Input Expected output
coverage
VST2021 © Serge Demeyer
Line Coverage (Statement / Branch / Path / …)
10
c++
java
VST2021 © Serge Demeyer
Mutation Testing
11
int compare(int v1,
int v2)


{if (v1 <v2)


return 1;


return -1
;

}
int compare(int v1,
int v2)


{if (v1 >=v2)


return 1;


return -1
;

}
🙂
🙁
VST2021 © Serge Demeyer
Mutation Coverage
12
VST2021 © Serge Demeyer
Test Amplification
13
Program


Under


Test
Valid Input Expected output
coverage
+coverage
Extra Input
Extra Input
+Extra Input +Extra output
VST2021 © Serge Demeyer
Test amplification Definition
14
[Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit
Baudry. 2019. A snowballing literature study on test ampli
fi
ca
ti
on. Journal of Systems and Software.
Test amplification consists of exploiting the knowledge of a large
number of test cases, in which developers embed meaningful
input data and expected properties in the form of oracles, in
order to enhance these manually written tests with respect to


an engineering goal. [Danglot-19]
mutation coverage
VST2021 © Serge Demeyer
Fuzzing by Amplification
15
[Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit
Baudry. 2019. A snowballing literature study on test ampli
fi
ca
ti
on. Journal of Systems and Software.
Test amplification consists of exploiting the knowledge of a large
number of test cases, in which developers embed meaningful
input data and expected properties in the form of oracles, in
order to enhance these manually written tests with respect to


an engineering goal. [Danglot-19]
Detecting unexpected terminations


(crashes, freezes, …)
VST2021 © Serge Demeyer
Fuzzing by Amplification
16
Image from: https://www.enwild.com/snow-peak-colored-titanium-spork.html
Fuzzing?
Amplification?
VST2021 © Serge Demeyer
Motivating example (1/2)
17
VST2021 © Serge Demeyer
Motivating example (2/2)
18
VST2021 © Serge Demeyer
Proposed approach
19
Profile Test Method
VST2021 © Serge Demeyer
Proposed approach
20
(2) Assertion Removal
VST2021 © Serge Demeyer
Proposed approach
21
(2) Input Amplification
Input Amplification Operators


• Literal mutation


• Nullify objects


• Remove statements


• Duplicate statements


• Add new statements


• Change helper methods


• …
VST2021 © Serge Demeyer
Proposed approach
22
(3) Create new tests


+ install in the test suite
VST2021 © Serge Demeyer
Proposed approach
23
(4) Execute
VST2021 © Serge Demeyer
Proposed approach
24
(5) Sandboxing
VST2021 © Serge Demeyer
Proposed approach
25
Observation
VST2021 © Serge Demeyer
Proposed approach
26
(6) Reporting
VST2021 © Serge Demeyer
Challenge
27
If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
VST2021 © Serge Demeyer
Challenge
28
Explicit


Types
Profiling
White Box Fuzzing
VST2021 © Serge Demeyer
Smalltalk Community
29
Cool dude!
Mehrdad Abdi, Henrique Rocha, and Serge Demeyer. Reproducible crashes: Fuzzing pharo
by mutating the test methods. In Proceedings IWST 2020 (International Workshop on
Smalltalk Technologies). ESUG, 2020.
VST2021 © Serge Demeyer
Testing Community
30
Related Work?


• Fuzzing by mutating test code


• Fuzzing / Test Amplification


- dynamically typed languages


(Python, Javascript, …)
feedback and/or suggestions


serge.demeyer@uantwerpen.be

More Related Content

What's hot

Software testing q as collection by ravi
Software testing q as   collection by raviSoftware testing q as   collection by ravi
Software testing q as collection by ravi
Ravindranath Tagore
 
6 article azojete vol 9 51 67
6 article azojete vol 9 51 676 article azojete vol 9 51 67
6 article azojete vol 9 51 67
Oyeniyi Samuel
 
Www.istqb.guru istqb question-paper5
Www.istqb.guru istqb question-paper5Www.istqb.guru istqb question-paper5
Www.istqb.guru istqb question-paper5
Tomas Vileikis
 
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
Chakkrit (Kla) Tantithamthavorn
 
A survey of software testing
A survey of software testingA survey of software testing
A survey of software testing
Tao He
 
Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)
IJERD Editor
 
'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...
'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...
'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...
TEST Huddle
 
Presentation slides: "How to get 100% code coverage"
Presentation slides: "How to get 100% code coverage" Presentation slides: "How to get 100% code coverage"
Presentation slides: "How to get 100% code coverage"
Rapita Systems Ltd
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directions
Tao He
 
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
Chakkrit (Kla) Tantithamthavorn
 
Multi-Objective Cross-Project Defect Prediction
Multi-Objective Cross-Project Defect PredictionMulti-Objective Cross-Project Defect Prediction
Multi-Objective Cross-Project Defect Prediction
Sebastiano Panichella
 
Defect Prediction: Accomplishments and Future Challenges
Defect Prediction: Accomplishments and Future ChallengesDefect Prediction: Accomplishments and Future Challenges
Defect Prediction: Accomplishments and Future Challenges
Yasutaka Kamei
 
Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...
Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...
Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...
Yuki Ueda
 
Mining Software Defects: Should We Consider Affected Releases?
Mining Software Defects: Should We Consider Affected Releases?Mining Software Defects: Should We Consider Affected Releases?
Mining Software Defects: Should We Consider Affected Releases?
Chakkrit (Kla) Tantithamthavorn
 
312 50-demo
312 50-demo312 50-demo
312 50-demo
Tomas Vileikis
 
Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...
Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...
Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...
Yuki Ueda
 
Code coverage
Code coverageCode coverage
Code coverage
Vijayan Reddy
 
Using Developer Information as a Prediction Factor
Using Developer Information as a Prediction FactorUsing Developer Information as a Prediction Factor
Using Developer Information as a Prediction Factor
Tim Menzies
 
A Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
A Study of the Quality-Impacting Practices of Modern Code Review at Sony MobileA Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
A Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
SAIL_QU
 

What's hot (19)

Software testing q as collection by ravi
Software testing q as   collection by raviSoftware testing q as   collection by ravi
Software testing q as collection by ravi
 
6 article azojete vol 9 51 67
6 article azojete vol 9 51 676 article azojete vol 9 51 67
6 article azojete vol 9 51 67
 
Www.istqb.guru istqb question-paper5
Www.istqb.guru istqb question-paper5Www.istqb.guru istqb question-paper5
Www.istqb.guru istqb question-paper5
 
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
Leveraging HPC Resources to Improve the Experimental Design of Software Analy...
 
A survey of software testing
A survey of software testingA survey of software testing
A survey of software testing
 
Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)
 
'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...
'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...
'Acceptance Test Driven Development Using Robot Framework' by Pekka Klarch & ...
 
Presentation slides: "How to get 100% code coverage"
Presentation slides: "How to get 100% code coverage" Presentation slides: "How to get 100% code coverage"
Presentation slides: "How to get 100% code coverage"
 
Testing survey by_directions
Testing survey by_directionsTesting survey by_directions
Testing survey by_directions
 
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
 
Multi-Objective Cross-Project Defect Prediction
Multi-Objective Cross-Project Defect PredictionMulti-Objective Cross-Project Defect Prediction
Multi-Objective Cross-Project Defect Prediction
 
Defect Prediction: Accomplishments and Future Challenges
Defect Prediction: Accomplishments and Future ChallengesDefect Prediction: Accomplishments and Future Challenges
Defect Prediction: Accomplishments and Future Challenges
 
Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...
Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...
Automatically Customizing Static Analysis Tools to Coding Rules Really Follow...
 
Mining Software Defects: Should We Consider Affected Releases?
Mining Software Defects: Should We Consider Affected Releases?Mining Software Defects: Should We Consider Affected Releases?
Mining Software Defects: Should We Consider Affected Releases?
 
312 50-demo
312 50-demo312 50-demo
312 50-demo
 
Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...
Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...
Impact of Coding Style Checker on Code Review -A case study on the OpenStack ...
 
Code coverage
Code coverageCode coverage
Code coverage
 
Using Developer Information as a Prediction Factor
Using Developer Information as a Prediction FactorUsing Developer Information as a Prediction Factor
Using Developer Information as a Prediction Factor
 
A Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
A Study of the Quality-Impacting Practices of Modern Code Review at Sony MobileA Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
A Study of the Quality-Impacting Practices of Modern Code Review at Sony Mobile
 

Similar to Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods

Test Amplification in Python — An Industrial Experience Report
       Test Amplification in Python — An Industrial Experience Report       Test Amplification in Python — An Industrial Experience Report
Test Amplification in Python — An Industrial Experience Report
University of Antwerp
 
Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...
IRJET Journal
 
Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...
IRJET Journal
 
Comparative Study on Analysis of G+10 Regular Residential Building subjected ...
Comparative Study on Analysis of G+10 Regular Residential Building subjected ...Comparative Study on Analysis of G+10 Regular Residential Building subjected ...
Comparative Study on Analysis of G+10 Regular Residential Building subjected ...
IRJET Journal
 
AI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience ReportsAI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience Reports
University of Antwerp
 
Technical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-UpsTechnical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-Ups
University of Antwerp
 
A technique for parallel gui testing of android applications
A technique for parallel gui testing of android applicationsA technique for parallel gui testing of android applications
A technique for parallel gui testing of android applications
Porfirio Tramontana
 
What is the best approach to tdd
What is the best approach to tddWhat is the best approach to tdd
What is the best approach to tdd
Red Hat
 
MTech- Viva_Voce
MTech- Viva_VoceMTech- Viva_Voce
MTech- Viva_Voce
Vijayananda Mohire
 
Csmr13b.ppt
Csmr13b.pptCsmr13b.ppt
STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)
STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)
STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)
pycontw
 
SOLIDWORKS Simulation Standard
SOLIDWORKS Simulation StandardSOLIDWORKS Simulation Standard
SOLIDWORKS Simulation Standard
Delphine Genouvrier
 
Mdd test qa_test2014_bryan_bakker
Mdd test qa_test2014_bryan_bakkerMdd test qa_test2014_bryan_bakker
Mdd test qa_test2014_bryan_bakker
Bryan Bakker
 
Introduction to Genex Assistance
Introduction to  Genex AssistanceIntroduction to  Genex Assistance
Introduction to Genex Assistance
Tempus Telcosys
 
Approximating Attack Surfaces with Stack Traces [ICSE 15]
Approximating Attack Surfaces with Stack Traces [ICSE 15]Approximating Attack Surfaces with Stack Traces [ICSE 15]
Approximating Attack Surfaces with Stack Traces [ICSE 15]
Chris Theisen
 
An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...
An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...
An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...
Mohammad Jafar Mashhadi
 
Using Robots for App Testing
Using Robots for App Testing Using Robots for App Testing
Using Robots for App Testing
Shauvik Roy Choudhary, Ph.D.
 
Boosting AM Quality & Economics - In-Situ Monitoring
Boosting AM Quality & Economics - In-Situ MonitoringBoosting AM Quality & Economics - In-Situ Monitoring
Boosting AM Quality & Economics - In-Situ Monitoring
Sigma Labs
 
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Dmitriy Gumeniuk
 
Simulation of Advanced Processes: Welding & Additive Manufacturing Cases Studies
Simulation of Advanced Processes: Welding & Additive Manufacturing Cases StudiesSimulation of Advanced Processes: Welding & Additive Manufacturing Cases Studies
Simulation of Advanced Processes: Welding & Additive Manufacturing Cases Studies
GeonX S.A.
 

Similar to Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods (20)

Test Amplification in Python — An Industrial Experience Report
       Test Amplification in Python — An Industrial Experience Report       Test Amplification in Python — An Industrial Experience Report
Test Amplification in Python — An Industrial Experience Report
 
Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...
 
Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...Optimization of sealing casting by identifying solidification defect and impr...
Optimization of sealing casting by identifying solidification defect and impr...
 
Comparative Study on Analysis of G+10 Regular Residential Building subjected ...
Comparative Study on Analysis of G+10 Regular Residential Building subjected ...Comparative Study on Analysis of G+10 Regular Residential Building subjected ...
Comparative Study on Analysis of G+10 Regular Residential Building subjected ...
 
AI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience ReportsAI For Software Engineering: Two Industrial Experience Reports
AI For Software Engineering: Two Industrial Experience Reports
 
Technical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-UpsTechnical Debt in Start-ups / Scale-Ups
Technical Debt in Start-ups / Scale-Ups
 
A technique for parallel gui testing of android applications
A technique for parallel gui testing of android applicationsA technique for parallel gui testing of android applications
A technique for parallel gui testing of android applications
 
What is the best approach to tdd
What is the best approach to tddWhat is the best approach to tdd
What is the best approach to tdd
 
MTech- Viva_Voce
MTech- Viva_VoceMTech- Viva_Voce
MTech- Viva_Voce
 
Csmr13b.ppt
Csmr13b.pptCsmr13b.ppt
Csmr13b.ppt
 
STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)
STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)
STAF 在自動化測試上的延伸應用 -- TMSTAF (TrendMicro STAF)
 
SOLIDWORKS Simulation Standard
SOLIDWORKS Simulation StandardSOLIDWORKS Simulation Standard
SOLIDWORKS Simulation Standard
 
Mdd test qa_test2014_bryan_bakker
Mdd test qa_test2014_bryan_bakkerMdd test qa_test2014_bryan_bakker
Mdd test qa_test2014_bryan_bakker
 
Introduction to Genex Assistance
Introduction to  Genex AssistanceIntroduction to  Genex Assistance
Introduction to Genex Assistance
 
Approximating Attack Surfaces with Stack Traces [ICSE 15]
Approximating Attack Surfaces with Stack Traces [ICSE 15]Approximating Attack Surfaces with Stack Traces [ICSE 15]
Approximating Attack Surfaces with Stack Traces [ICSE 15]
 
An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...
An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...
An Empirical Study On Practicality Of Specification Mining Algorithms On A Re...
 
Using Robots for App Testing
Using Robots for App Testing Using Robots for App Testing
Using Robots for App Testing
 
Boosting AM Quality & Economics - In-Situ Monitoring
Boosting AM Quality & Economics - In-Situ MonitoringBoosting AM Quality & Economics - In-Situ Monitoring
Boosting AM Quality & Economics - In-Situ Monitoring
 
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...Test Gap Analysis and regression minimization with Drill4j. Observability on ...
Test Gap Analysis and regression minimization with Drill4j. Observability on ...
 
Simulation of Advanced Processes: Welding & Additive Manufacturing Cases Studies
Simulation of Advanced Processes: Welding & Additive Manufacturing Cases StudiesSimulation of Advanced Processes: Welding & Additive Manufacturing Cases Studies
Simulation of Advanced Processes: Welding & Additive Manufacturing Cases Studies
 

More from University of Antwerp

MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
University of Antwerp
 
Social Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant ForksSocial Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant Forks
University of Antwerp
 
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
University of Antwerp
 
VST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdfVST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdf
University of Antwerp
 
Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software ...
Keynote VST2020 (Workshop on  Validation, Analysis and Evolution of Software ...Keynote VST2020 (Workshop on  Validation, Analysis and Evolution of Software ...
Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software ...
University of Antwerp
 
Saner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblindSaner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblind
University of Antwerp
 

More from University of Antwerp (6)

MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
 
Social Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant ForksSocial Coding Platforms Facilitate Variant Forks
Social Coding Platforms Facilitate Variant Forks
 
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
Finding Bugs, Fixing Bugs, Preventing Bugs - Exploiting Automated Tests to In...
 
VST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdfVST2022SmallAmpAmpyfier.pdf
VST2022SmallAmpAmpyfier.pdf
 
Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software ...
Keynote VST2020 (Workshop on  Validation, Analysis and Evolution of Software ...Keynote VST2020 (Workshop on  Validation, Analysis and Evolution of Software ...
Keynote VST2020 (Workshop on Validation, Analysis and Evolution of Software ...
 
Saner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblindSaner open steeringcommittee2018campobassodoubleblind
Saner open steeringcommittee2018campobassodoubleblind
 

Recently uploaded

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
ISH Technologies
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
Max Andersen
 
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket ManagementUtilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
Alina Yurenko
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
Ayan Halder
 
Launch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in MinutesLaunch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in Minutes
Roshan Dwivedi
 
Graspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code AnalysisGraspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code Analysis
Aftab Hussain
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Neo4j
 
GraphSummit Paris - The art of the possible with Graph Technology
GraphSummit Paris - The art of the possible with Graph TechnologyGraphSummit Paris - The art of the possible with Graph Technology
GraphSummit Paris - The art of the possible with Graph Technology
Neo4j
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
kalichargn70th171
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
Hironori Washizaki
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Łukasz Chruściel
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
mz5nrf0n
 

Recently uploaded (20)

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
 
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket ManagementUtilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
 
Launch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in MinutesLaunch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in Minutes
 
Graspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code AnalysisGraspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code Analysis
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
 
GraphSummit Paris - The art of the possible with Graph Technology
GraphSummit Paris - The art of the possible with Graph TechnologyGraphSummit Paris - The art of the possible with Graph Technology
GraphSummit Paris - The art of the possible with Graph Technology
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
 

Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods

  • 1. Universiteit Antwerpen Reproducible Crashes: 
 Fuzzing Pharo by Mutating the Test Methods Mehrdad Abdi — Henrique Rocha — Serge Demeyer VST 2021 (March 9th)
  • 2. VST2021 © Serge Demeyer • Background - Fuzzing - Mutation Coverage - Test Amplification • Fuzzing by amplification - Motivating examples - Proposed approach - Challenge • Conclusions Outline 2
  • 4. VST2021 © Serge Demeyer Testing 4 Program 
 Under Test Valid Input Expected output Software Testing is the process of executing a program or system with the intent of finding errors. (Myers, Glenford J., The art of software testing. Wiley, 1979
  • 5. VST2021 © Serge Demeyer Fuzz Testing 5 Program 
 Under Test Unexpected Input Crash/Freeze Fuzzing (or Fuzz Testing) is an automated testing technique to verify the robustness of a program-under-test. Valid input is replaced by random values with the goal to force the program-under-test into unexpected exceptional behavior.
  • 6. VST2021 © Serge Demeyer Black Box Fuzzing 6 Access to execution Mutating valid inputs
  • 7. VST2021 © Serge Demeyer White Box Fuzzing 7 Access to code, models, specs, … Program Analysis
  • 8. VST2021 © Serge Demeyer Grey Box Fuzzing 8 Limited Access (bytecode, trace, …) Reverse Engineering
  • 9. VST2021 © Serge Demeyer Code Coverage 9 Program 
 Under Test Valid Input Expected output coverage
  • 10. VST2021 © Serge Demeyer Line Coverage (Statement / Branch / Path / …) 10 c++ java
  • 11. VST2021 © Serge Demeyer Mutation Testing 11 int compare(int v1, int v2) {if (v1 <v2) return 1; return -1 ; } int compare(int v1, int v2) {if (v1 >=v2) return 1; return -1 ; } 🙂 🙁
  • 12. VST2021 © Serge Demeyer Mutation Coverage 12
  • 13. VST2021 © Serge Demeyer Test Amplification 13 Program 
 Under Test Valid Input Expected output coverage +coverage Extra Input Extra Input +Extra Input +Extra output
  • 14. VST2021 © Serge Demeyer Test amplification Definition 14 [Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit Baudry. 2019. A snowballing literature study on test ampli fi ca ti on. Journal of Systems and Software. Test amplification consists of exploiting the knowledge of a large number of test cases, in which developers embed meaningful input data and expected properties in the form of oracles, in order to enhance these manually written tests with respect to 
 an engineering goal. [Danglot-19] mutation coverage
  • 15. VST2021 © Serge Demeyer Fuzzing by Amplification 15 [Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit Baudry. 2019. A snowballing literature study on test ampli fi ca ti on. Journal of Systems and Software. Test amplification consists of exploiting the knowledge of a large number of test cases, in which developers embed meaningful input data and expected properties in the form of oracles, in order to enhance these manually written tests with respect to 
 an engineering goal. [Danglot-19] Detecting unexpected terminations 
 (crashes, freezes, …)
  • 16. VST2021 © Serge Demeyer Fuzzing by Amplification 16 Image from: https://www.enwild.com/snow-peak-colored-titanium-spork.html Fuzzing? Amplification?
  • 17. VST2021 © Serge Demeyer Motivating example (1/2) 17
  • 18. VST2021 © Serge Demeyer Motivating example (2/2) 18
  • 19. VST2021 © Serge Demeyer Proposed approach 19 Profile Test Method
  • 20. VST2021 © Serge Demeyer Proposed approach 20 (2) Assertion Removal
  • 21. VST2021 © Serge Demeyer Proposed approach 21 (2) Input Amplification Input Amplification Operators • Literal mutation • Nullify objects • Remove statements • Duplicate statements • Add new statements • Change helper methods • …
  • 22. VST2021 © Serge Demeyer Proposed approach 22 (3) Create new tests + install in the test suite
  • 23. VST2021 © Serge Demeyer Proposed approach 23 (4) Execute
  • 24. VST2021 © Serge Demeyer Proposed approach 24 (5) Sandboxing
  • 25. VST2021 © Serge Demeyer Proposed approach 25 Observation
  • 26. VST2021 © Serge Demeyer Proposed approach 26 (6) Reporting
  • 27. VST2021 © Serge Demeyer Challenge 27 If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
  • 28. VST2021 © Serge Demeyer Challenge 28 Explicit Types Profiling White Box Fuzzing
  • 29. VST2021 © Serge Demeyer Smalltalk Community 29 Cool dude! Mehrdad Abdi, Henrique Rocha, and Serge Demeyer. Reproducible crashes: Fuzzing pharo by mutating the test methods. In Proceedings IWST 2020 (International Workshop on Smalltalk Technologies). ESUG, 2020.
  • 30. VST2021 © Serge Demeyer Testing Community 30 Related Work? • Fuzzing by mutating test code • Fuzzing / Test Amplification - dynamically typed languages 
 (Python, Javascript, …) feedback and/or suggestions serge.demeyer@uantwerpen.be