Successfully reported this slideshow.
Your SlideShare is downloading. ×

Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 30 Ad

Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods

Download to read offline

Fuzzing (or Fuzz Testing) is a technique to verify the robustness of a program-under-test. Valid input is replaced by random values with the goal to force the program-under-test into unresponsive states. In this position paper, we propose a white box Fuzzing approach by transforming (mutating) existing test methods. We adopt the mechanisms used for test amplification to generate crash inducing tests, which developers can reproduce later. We provide anecdotal evidence that our approach towards Fuzzing reveals crashing issues in the Pharo environment.

Fuzzing (or Fuzz Testing) is a technique to verify the robustness of a program-under-test. Valid input is replaced by random values with the goal to force the program-under-test into unresponsive states. In this position paper, we propose a white box Fuzzing approach by transforming (mutating) existing test methods. We adopt the mechanisms used for test amplification to generate crash inducing tests, which developers can reproduce later. We provide anecdotal evidence that our approach towards Fuzzing reveals crashing issues in the Pharo environment.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods (20)

Advertisement

Recently uploaded (20)

Advertisement

Reproducible Crashes: Fuzzing Pharo by Mutating the Test Methods

  1. 1. Universiteit Antwerpen Reproducible Crashes: 
 Fuzzing Pharo by Mutating the Test Methods Mehrdad Abdi — Henrique Rocha — Serge Demeyer VST 2021 (March 9th)
  2. 2. VST2021 © Serge Demeyer • Background - Fuzzing - Mutation Coverage - Test Amplification • Fuzzing by amplification - Motivating examples - Proposed approach - Challenge • Conclusions Outline 2
  3. 3. Fuzz Testing Mutation Coverage Test Amplification
  4. 4. VST2021 © Serge Demeyer Testing 4 Program 
 Under Test Valid Input Expected output Software Testing is the process of executing a program or system with the intent of finding errors. (Myers, Glenford J., The art of software testing. Wiley, 1979
  5. 5. VST2021 © Serge Demeyer Fuzz Testing 5 Program 
 Under Test Unexpected Input Crash/Freeze Fuzzing (or Fuzz Testing) is an automated testing technique to verify the robustness of a program-under-test. Valid input is replaced by random values with the goal to force the program-under-test into unexpected exceptional behavior.
  6. 6. VST2021 © Serge Demeyer Black Box Fuzzing 6 Access to execution Mutating valid inputs
  7. 7. VST2021 © Serge Demeyer White Box Fuzzing 7 Access to code, models, specs, … Program Analysis
  8. 8. VST2021 © Serge Demeyer Grey Box Fuzzing 8 Limited Access (bytecode, trace, …) Reverse Engineering
  9. 9. VST2021 © Serge Demeyer Code Coverage 9 Program 
 Under Test Valid Input Expected output coverage
  10. 10. VST2021 © Serge Demeyer Line Coverage (Statement / Branch / Path / …) 10 c++ java
  11. 11. VST2021 © Serge Demeyer Mutation Testing 11 int compare(int v1, int v2) {if (v1 <v2) return 1; return -1 ; } int compare(int v1, int v2) {if (v1 >=v2) return 1; return -1 ; } 🙂 🙁
  12. 12. VST2021 © Serge Demeyer Mutation Coverage 12
  13. 13. VST2021 © Serge Demeyer Test Amplification 13 Program 
 Under Test Valid Input Expected output coverage +coverage Extra Input Extra Input +Extra Input +Extra output
  14. 14. VST2021 © Serge Demeyer Test amplification Definition 14 [Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit Baudry. 2019. A snowballing literature study on test ampli fi ca ti on. Journal of Systems and Software. Test amplification consists of exploiting the knowledge of a large number of test cases, in which developers embed meaningful input data and expected properties in the form of oracles, in order to enhance these manually written tests with respect to 
 an engineering goal. [Danglot-19] mutation coverage
  15. 15. VST2021 © Serge Demeyer Fuzzing by Amplification 15 [Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit Baudry. 2019. A snowballing literature study on test ampli fi ca ti on. Journal of Systems and Software. Test amplification consists of exploiting the knowledge of a large number of test cases, in which developers embed meaningful input data and expected properties in the form of oracles, in order to enhance these manually written tests with respect to 
 an engineering goal. [Danglot-19] Detecting unexpected terminations 
 (crashes, freezes, …)
  16. 16. VST2021 © Serge Demeyer Fuzzing by Amplification 16 Image from: https://www.enwild.com/snow-peak-colored-titanium-spork.html Fuzzing? Amplification?
  17. 17. VST2021 © Serge Demeyer Motivating example (1/2) 17
  18. 18. VST2021 © Serge Demeyer Motivating example (2/2) 18
  19. 19. VST2021 © Serge Demeyer Proposed approach 19 Profile Test Method
  20. 20. VST2021 © Serge Demeyer Proposed approach 20 (2) Assertion Removal
  21. 21. VST2021 © Serge Demeyer Proposed approach 21 (2) Input Amplification Input Amplification Operators • Literal mutation • Nullify objects • Remove statements • Duplicate statements • Add new statements • Change helper methods • …
  22. 22. VST2021 © Serge Demeyer Proposed approach 22 (3) Create new tests + install in the test suite
  23. 23. VST2021 © Serge Demeyer Proposed approach 23 (4) Execute
  24. 24. VST2021 © Serge Demeyer Proposed approach 24 (5) Sandboxing
  25. 25. VST2021 © Serge Demeyer Proposed approach 25 Observation
  26. 26. VST2021 © Serge Demeyer Proposed approach 26 (6) Reporting
  27. 27. VST2021 © Serge Demeyer Challenge 27 If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
  28. 28. VST2021 © Serge Demeyer Challenge 28 Explicit Types Profiling White Box Fuzzing
  29. 29. VST2021 © Serge Demeyer Smalltalk Community 29 Cool dude! Mehrdad Abdi, Henrique Rocha, and Serge Demeyer. Reproducible crashes: Fuzzing pharo by mutating the test methods. In Proceedings IWST 2020 (International Workshop on Smalltalk Technologies). ESUG, 2020.
  30. 30. VST2021 © Serge Demeyer Testing Community 30 Related Work? • Fuzzing by mutating test code • Fuzzing / Test Amplification - dynamically typed languages 
 (Python, Javascript, …) feedback and/or suggestions serge.demeyer@uantwerpen.be

×