This document describes Phuzzer, a fuzzer for testing Pharo code. It begins by introducing the author and their background and interests. It then discusses the essence of testing, how fuzz testing originated, and provides an example of fuzz testing the String>>asDate method in Pharo. Various fuzzing strategies are explored, including random, grammar-based, and backtracking approaches. The goal is to generate inputs that may uncover bugs in a way that looks syntactically and semantically valid to the code being tested.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Top 7 Unique WhatsApp API Benefits | Saudi ArabiaYara Milbes
Discover the transformative power of the WhatsApp API in our latest SlideShare presentation, "Top 7 Unique WhatsApp API Benefits." In today's fast-paced digital era, effective communication is crucial for both personal and professional success. Whether you're a small business looking to enhance customer interactions or an individual seeking seamless communication with loved ones, the WhatsApp API offers robust capabilities that can significantly elevate your experience.
In this presentation, we delve into the top 7 distinctive benefits of the WhatsApp API, provided by the leading WhatsApp API service provider in Saudi Arabia. Learn how to streamline customer support, automate notifications, leverage rich media messaging, run scalable marketing campaigns, integrate secure payments, synchronize with CRM systems, and ensure enhanced security and privacy.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
2. First: About Me
2
Evref
fervE
• Keywords: compilers, testing, test generation
• Ph.D.: Re
fl
ection, debloating, dynamic updates
• Interests: tooling, benchmarking, 日本語, board games, concurrency
Talk to me!
Or: guillermo.polito@inria.fr
guillermo.polito@inria.fr
@guillep
3. The Essence of Testing
“Program testing can be used to show the presence of
bugs, but never to show their absence”
- E. Dijkstra
3
4. What is a Good Test?
“A good test is a test that catches bugs”
- me
4
5. The Fuzz Generator
1988, Barton Miller observed random
crashes in UNIX utilities.
Shouldn’t it be more
robust?
5
6. An Empirical Study on the Reliability of
UNIX Utilities
• Call utilities with random inputs
• ~90 tested utilities
• 25-33% crashed
6
Communications of the ACM’90
fuzz 100000 -o outfile | deqn
8. Let’s Test some Parser
• Pharo 11
• String>>asDate
8
PASS "7 February 2039"
FAIL "DateError: day is after month ends"
PASS "1 June 2002"
PASS "5 August 13836"
FAIL "DateError: day may not be zero or negative"
PASS "1 January 2004"
FAIL "#isAlphaNumeric was sent to nil"
FAIL "SubscriptOutOfBounds: 0"
PASS "7 June 2009"
PASS "3 April 2001"
FAIL "DateError: day may not be zero or negative"
FAIL "SubscriptOutOfBounds: 72"
FAIL "SubscriptOutOfBounds: 0"
FAIL "DateError: day is after month ends"
PASS "3 April 1991"
PASS "3 October 2001"
FAIL "DateError: day is after month ends"
FAIL "SubscriptOutOfBounds: 0"
FAIL "DateError: day may not be zero or negative"
FAIL "SubscriptOutOfBounds: 0"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
f run: r times: 20.
9. Let’s Test some Parser
• Pharo 11
• String>>asDate
• 12/20 = 60% of failures?
9
PASS "7 February 2039"
FAIL "DateError: day is after month ends"
PASS "1 June 2002"
PASS "5 August 13836"
FAIL "DateError: day may not be zero or negative"
PASS "1 January 2004"
FAIL "#isAlphaNumeric was sent to nil"
FAIL "SubscriptOutOfBounds: 0"
PASS "7 June 2009"
PASS "3 April 2001"
FAIL "DateError: day may not be zero or negative"
FAIL "SubscriptOutOfBounds: 72"
FAIL "SubscriptOutOfBounds: 0"
FAIL "DateError: day is after month ends"
PASS "3 April 1991"
PASS "3 October 2001"
FAIL "DateError: day is after month ends"
FAIL "SubscriptOutOfBounds: 0"
FAIL "DateError: day may not be zero or negative"
FAIL "SubscriptOutOfBounds: 0"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
f run: r times: 20.
10. Refining the Results
• Pharo 11
• String>>asDate
• DateError is an expected error!
10
PASS "DateError: day is after month ends"
PASS "28 April 2006"
PASS "7 September 2029"
PASS "9 March 1995"
FAIL "SubscriptOutOfBounds: 73"
PASS "DateError: day is after month ends"
FAIL "SubscriptOutOfBounds: 0"
PASS "DateError: day is after month ends"
PASS "6 January 2007"
PASS "9 January 1986"
FAIL "SubscriptOutOfBounds: 0"
FAIL "#isAlphaNumeric was sent to nil"
PASS "DateError: day is after month ends"
PASS "1 September 1989"
PASS "DateError: day is after month ends"
PASS "DateError: day may not be zero or negative"
PASS "5 January 0228"
PASS "DateError: day may not be zero or negative"
PASS "7 September 1996"
PASS "2 January 2008"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
r expectedException: DateError.
f run: r times: 20.
11. Refining the Results
• Pharo 11
• String>>asDate
• DateError is an expected error!
• 4/20 = 5% of errors
11
PASS "DateError: day is after month ends"
PASS "28 April 2006"
PASS "7 September 2029"
PASS "9 March 1995"
FAIL "SubscriptOutOfBounds: 73"
PASS "DateError: day is after month ends"
FAIL "SubscriptOutOfBounds: 0"
PASS "DateError: day is after month ends"
PASS "6 January 2007"
PASS "9 January 1986"
FAIL "SubscriptOutOfBounds: 0"
FAIL "#isAlphaNumeric was sent to nil"
PASS "DateError: day is after month ends"
PASS "1 September 1989"
PASS "DateError: day is after month ends"
PASS "DateError: day may not be zero or negative"
PASS "5 January 0228"
PASS "DateError: day may not be zero or negative"
PASS "7 September 1996"
PASS "2 January 2008"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
r expectedException: DateError.
f run: r times: 20.
13. Fuzzer ratio over 100 runs
13
r := PzBlockRunner on: [ :e | e asDate ].
r expectedException: DateError.
f run: r times: 100.
Fuzzer Pass Expected Fail Fail
Weird Chars 49 % 29 % 22 %
Large Char set 0 % 0 % 100 %
Alphanumeric 0 % 0 % 100 %
14. Random Inputs Fail Easily
• We could expect to break something with fully random inputs
• This could be solved with input sanitizing
• What if we have almost correct inputs?
• Looks like a date, quacks like a date, parses as a date?
14
15. We need to generate syntactically
and semantically valid inputs
15
16. We need to generate syntactically
and semantically valid inputs
16
18. Grammars as Input Descriptions
18
• Grammars describe languages
• Usually used for parsing purposes, but…
• Key idea (from the 60s) => structured fuzzing with grammars
21. • Simple Date grammar fuzzing has a high success ratio
21
f := PzGrammarFuzzer on: PzDateGrammar new.
r := PzBlockRunner on: [ :e | e asDate ].
r expectedException: DateError.
f run: r times: 100.
Pass 81 %
Expected-Fail 10 %
Fail 9 %
Let’s test some parser (bis)
22. Looking at the bugs
• Out of 135 bugs fuzzing 1000 cases
22
method not understood during parsing 83 %
Out of bounds during parsing 13 %
Validation with generic error during parsing 4 %
23. Back-tracking a Bit
• Pharo 11
• String>>asDate
23
PASS "DateError: day is after month ends"
PASS "28 April 2006"
PASS "7 September 2029"
PASS "9 March 1995"
FAIL "SubscriptOutOfBounds: 73"
PASS "DateError: day is after month ends"
FAIL "SubscriptOutOfBounds: 0"
PASS "DateError: day is after month ends"
PASS "6 January 2007"
PASS "9 January 1986"
FAIL "SubscriptOutOfBounds: 0"
FAIL "#isAlphaNumeric was sent to nil"
PASS "DateError: day is after month ends"
PASS "1 September 1989"
PASS "DateError: day is after month ends"
PASS "DateError: day may not be zero or negative"
PASS "5 January 0228"
PASS "DateError: day may not be zero or negative"
PASS "7 September 1996"
PASS "2 January 2008"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
f run: r times: 20.
24. Back-tracking a Bit
• Pharo 11
• String>>asDate
24
PASS "DateError: day is after month ends"
PASS "28 April 2006"
PASS "7 September 2029"
PASS "9 March 1995"
FAIL "SubscriptOutOfBounds: 73"
PASS "DateError: day is after month ends"
FAIL "SubscriptOutOfBounds: 0"
PASS "DateError: day is after month ends"
PASS "6 January 2007"
PASS "9 January 1986"
FAIL "SubscriptOutOfBounds: 0"
FAIL "#isAlphaNumeric was sent to nil"
PASS "DateError: day is after month ends"
PASS "1 September 1989"
PASS "DateError: day is after month ends"
PASS "DateError: day may not be zero or negative"
PASS "5 January 0228"
PASS "DateError: day may not be zero or negative"
PASS "7 September 1996"
PASS "2 January 2008"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
f run: r times: 20.
How do we decide:
what is a PASS,
what is a FAIL?
25. When Dates Should Parse
• DateError is an expected error
• Malformed inputs should fail!
•
•
25
PASS "DateError: day is after month ends"
PASS "28 April 2006"
PASS "7 September 2029"
PASS "9 March 1995"
FAIL "SubscriptOutOfBounds: 73"
PASS "DateError: day is after month ends"
FAIL "SubscriptOutOfBounds: 0"
PASS "DateError: day is after month ends"
PASS "6 January 2007"
PASS "9 January 1986"
FAIL "SubscriptOutOfBounds: 0"
FAIL "#isAlphaNumeric was sent to nil"
PASS "DateError: day is after month ends"
PASS "1 September 1989"
PASS "DateError: day is after month ends"
PASS "DateError: day may not be zero or negative"
PASS "5 January 0228"
PASS "DateError: day may not be zero or negative"
PASS "7 September 1996"
PASS "2 January 2008"
f := PzRandomFuzzer new.
r := PzBlockRunner on: [ :e | e asDate ].
f run: r times: 20.
How do we decide:
what is a PASS,
what is a FAIL?
.+!;/./852"%7?3720("/)"!*43<,"4@>)>'(,"0(+7?
;% *:(41)215>/1890)@ 3"@3.35+6
26. Given a program and an input,
How can we distinguish correct from
incorrect behavior?
26
The Oracle Problem
27. Given a program and an input,
How can we automatically distinguish
correct from incorrect behavior?
27
The Oracle Problem
28. Remember Assertions
in this context
when this happens
then this should happen
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: 1.
28
29. Comparisons against known values
in this context
when this happens
then this should happen
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: 1.
29
30. Assertions against similar values?
in this context
when this happens
then this should happen
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: ?????.
30
31. Can we use another parser?
SetTest >> testAdd
| aSet |
"Context"
aSet := Set new.
“Stimuli"
aSet add: 5.
aSet add: 5.
"Check"
self assert: aSet size equals: ?????.
31
• (dis)agreement is evidence!
• Agreement: parsers have same behavior
• Disagreement: is there a bug?
32. Date>>fromString vs DateParser
32
• DateParser is a structured parser based on a speci
fi
ed format
• Should be more strict than asDate
• What kind of comparisons are fair/safe/legal?
DateParser readFrom: string readStream pattern: ‘m-d-yyyy'
34. Differential Parser Testing
34
runnerA := PzBlockRunner on: [ :e | e asDate ].
runnerA expectedException: DateError.
runnerB := PzBlockRunner on: [ :e |
(Date readFrom: e readStream pattern: 'm-d-yyyy') ].
runnerB expectedException: DateError.
diffRunner := PzDifferentialRunner new
runnerA: runnerA;
runnerB: runnerB;
yourself.
f := PzGrammarFuzzer on: PzDateMDYYYYGrammar new.
f run: diffRunner times: 100.
35. Results
• fuzz 100 times
• 3/100 errors!
35
FAIL 74-0-8 FAIL 74-0-8 SubscriptOutOfBounds: 0 PASS-FAIL 74-0-8 DateError
FAIL 3-2-6 PASS 3-2-6 2 March 2006 PASS-FAIL 3-2-6 DateError
PASS 5-515-1 PASS-FAIL 5-515-1 DateError: day is after month ends PASS-FAIL
FAIL 63-2-1 PASS 63-2-1 1 February 2063 PASS-FAIL 63-2-1 DateError
FAIL 4220-05-1 PASS 4220-05-1 1 May 4220 PASS-FAIL 4220-05-1 DateError
PASS 8-71-3 PASS-FAIL 8-71-3 DateError: day is after month ends PASS-FAIL 8
FAIL 6-7-34 PASS 6-7-34 7 June 2034 PASS-FAIL 6-7-34 DateError
FAIL 2-2-9 PASS 2-2-9 2 February 2009 PASS-FAIL 2-2-9 DateError
FAIL 29-2-0 FAIL 29-2-0 Error: Month out of bounds: 29. PASS-FAIL 29-2-0 Da
PASS 4-00-94 PASS-FAIL 4-00-94 DateError: day may not be zero or negative P
PASS 41150-8-0 PASS-FAIL 41150-8-0 DateError: day may not be zero or negati
FAIL 128-8-6 PASS 128-8-6 6 August 0128 PASS-FAIL 128-8-6 DateError
PASS 6-50054228-3 PASS-FAIL 6-50054228-3 DateError: day is after month ends
FAIL 39-4318-675 FAIL 39-4318-675 SubscriptOutOfBounds: 4318 PASS-FAIL 39-4
FAIL 0-7-9 FAIL 0-7-9 SubscriptOutOfBounds: 0 PASS-FAIL 0-7-9 DateError
FAIL 0-069848-2 FAIL 0-069848-2 SubscriptOutOfBounds: 0 PASS-FAIL 0-069848-
FAIL 9635-14-56 FAIL 9635-14-56 SubscriptOutOfBounds: 14 PASS-FAIL 9635-14-
FAIL 6461-5-8 PASS 6461-5-8 8 May 6461 PASS-FAIL 6461-5-8 DateError
FAIL 07-8-49 PASS 07-8-49 8 July 2049 PASS-FAIL 07-8-49 DateError
FAIL 61-74-51 FAIL 61-74-51 SubscriptOutOfBounds: 74 PASS-FAIL 61-74-51 Dat
FAIL 36-41-8 FAIL 36-41-8 SubscriptOutOfBounds: 41 PASS-FAIL 36-41-8 DateEr
FAIL 65-9-2 PASS 65-9-2 2 September 2065 PASS-FAIL 65-9-2 DateError
FAIL 321-3246-2 FAIL 321-3246-2 SubscriptOutOfBounds: 3246 PASS-FAIL 321-32
PASS 6-495-1 PASS-FAIL 6-495-1 DateError: day is after month ends PASS-FAIL
FAIL 2-9-74 PASS 2-9-74 9 February 1974 PASS-FAIL 2-9-74 DateError
FAIL 9158-909-0 FAIL 9158-909-0 SubscriptOutOfBounds: 909 PASS-FAIL 9158-90
PASS 7-41203-518 PASS-FAIL 7-41203-518 DateError: day is after month ends P
PASS 7-55-8 PASS-FAIL 7-55-8 DateError: day is after month ends PASS-FAIL 7
FAIL 0-78-9 FAIL 0-78-9 SubscriptOutOfBounds: 0 PASS-FAIL 0-78-9 DateError
FAIL 4-9-7 PASS 4-9-7 9 April 2007 PASS-FAIL 4-9-7 DateError
PASS 6-1242376-1 PASS-FAIL 6-1242376-1 DateError: day is after month ends P
PASS 2-169728-6327 PASS-FAIL 2-169728-6327 DateError: day is after month en
FAIL 99133-0-023 FAIL 99133-0-023 SubscriptOutOfBounds: 0 PASS-FAIL 99133-0
FAIL 08-4-64 PASS 08-4-64 4 August 2064 PASS-FAIL 08-4-64 DateError
FAIL 523-55-0 FAIL 523-55-0 SubscriptOutOfBounds: 55 PASS-FAIL 523-55-0 Dat
PASS 5-696-8 PASS-FAIL 5-696-8 DateError: day is after month ends PASS-FAIL
FAIL 77-6-8 PASS 77-6-8 8 June 1977 PASS-FAIL 77-6-8 DateError
FAIL 38-946-6 FAIL 38-946-6 SubscriptOutOfBounds: 946 PASS-FAIL 38-946-6 Da
FAIL 2615-5-7 PASS 2615-5-7 7 May 2615 PASS-FAIL 2615-5-7 DateError
36. Results
• fuzz 100 times
• 3/100 errors!
• Dates are mostly correct!
36
FAIL 74-0-8 FAIL 74-0-8 SubscriptOutOfBounds: 0 PASS-FAIL 74-0-8 DateError
FAIL 3-2-6 PASS 3-2-6 2 March 2006 PASS-FAIL 3-2-6 DateError
PASS 5-515-1 PASS-FAIL 5-515-1 DateError: day is after month ends PASS-FAIL
FAIL 63-2-1 PASS 63-2-1 1 February 2063 PASS-FAIL 63-2-1 DateError
FAIL 4220-05-1 PASS 4220-05-1 1 May 4220 PASS-FAIL 4220-05-1 DateError
PASS 8-71-3 PASS-FAIL 8-71-3 DateError: day is after month ends PASS-FAIL 8
FAIL 6-7-34 PASS 6-7-34 7 June 2034 PASS-FAIL 6-7-34 DateError
FAIL 2-2-9 PASS 2-2-9 2 February 2009 PASS-FAIL 2-2-9 DateError
FAIL 29-2-0 FAIL 29-2-0 Error: Month out of bounds: 29. PASS-FAIL 29-2-0 Da
PASS 4-00-94 PASS-FAIL 4-00-94 DateError: day may not be zero or negative P
PASS 41150-8-0 PASS-FAIL 41150-8-0 DateError: day may not be zero or negati
FAIL 128-8-6 PASS 128-8-6 6 August 0128 PASS-FAIL 128-8-6 DateError
PASS 6-50054228-3 PASS-FAIL 6-50054228-3 DateError: day is after month ends
FAIL 39-4318-675 FAIL 39-4318-675 SubscriptOutOfBounds: 4318 PASS-FAIL 39-4
FAIL 0-7-9 FAIL 0-7-9 SubscriptOutOfBounds: 0 PASS-FAIL 0-7-9 DateError
FAIL 0-069848-2 FAIL 0-069848-2 SubscriptOutOfBounds: 0 PASS-FAIL 0-069848-
FAIL 9635-14-56 FAIL 9635-14-56 SubscriptOutOfBounds: 14 PASS-FAIL 9635-14-
FAIL 6461-5-8 PASS 6461-5-8 8 May 6461 PASS-FAIL 6461-5-8 DateError
FAIL 07-8-49 PASS 07-8-49 8 July 2049 PASS-FAIL 07-8-49 DateError
FAIL 61-74-51 FAIL 61-74-51 SubscriptOutOfBounds: 74 PASS-FAIL 61-74-51 Dat
FAIL 36-41-8 FAIL 36-41-8 SubscriptOutOfBounds: 41 PASS-FAIL 36-41-8 DateEr
FAIL 65-9-2 PASS 65-9-2 2 September 2065 PASS-FAIL 65-9-2 DateError
FAIL 321-3246-2 FAIL 321-3246-2 SubscriptOutOfBounds: 3246 PASS-FAIL 321-32
PASS 6-495-1 PASS-FAIL 6-495-1 DateError: day is after month ends PASS-FAIL
FAIL 2-9-74 PASS 2-9-74 9 February 1974 PASS-FAIL 2-9-74 DateError
FAIL 9158-909-0 FAIL 9158-909-0 SubscriptOutOfBounds: 909 PASS-FAIL 9158-90
PASS 7-41203-518 PASS-FAIL 7-41203-518 DateError: day is after month ends P
PASS 7-55-8 PASS-FAIL 7-55-8 DateError: day is after month ends PASS-FAIL 7
FAIL 0-78-9 FAIL 0-78-9 SubscriptOutOfBounds: 0 PASS-FAIL 0-78-9 DateError
FAIL 4-9-7 PASS 4-9-7 9 April 2007 PASS-FAIL 4-9-7 DateError
PASS 6-1242376-1 PASS-FAIL 6-1242376-1 DateError: day is after month ends P
PASS 2-169728-6327 PASS-FAIL 2-169728-6327 DateError: day is after month en
FAIL 99133-0-023 FAIL 99133-0-023 SubscriptOutOfBounds: 0 PASS-FAIL 99133-0
FAIL 08-4-64 PASS 08-4-64 4 August 2064 PASS-FAIL 08-4-64 DateError
FAIL 523-55-0 FAIL 523-55-0 SubscriptOutOfBounds: 55 PASS-FAIL 523-55-0 Dat
PASS 5-696-8 PASS-FAIL 5-696-8 DateError: day is after month ends PASS-FAIL
FAIL 77-6-8 PASS 77-6-8 8 June 1977 PASS-FAIL 77-6-8 DateError
FAIL 38-946-6 FAIL 38-946-6 SubscriptOutOfBounds: 946 PASS-FAIL 38-946-6 Da
FAIL 2615-5-7 PASS 2615-5-7 7 May 2615 PASS-FAIL 2615-5-7 DateError
37. We need to generate syntactically
and semantically valid inputs
37
38. We need to generate syntactically
and semantically valid inputs
But Slightly Wrong
38
39. Mutations as Fuzzers
39
abc
Apply Mutation
original
input
mutated
input
Execute Program
mutation
axc
Result OK
Result NOK
E.g., change one + by a -
40. Random String Mutator
40
f := PzMutationFuzzer new
seed: { ‘abcd’ };
yourself.
(1 to: 10) collect: [ :e | f fuzz ]
3ou
AbC|dM
aEbcN`
bc
a`c$#
bcc
abc$
aabcd
!cbb~d
;
41. String Mutations
• Insert a random character in a random position of the string
• Delete a character in a random position of the string
• Replace a character by a random character in a random position of the string
41
42. Chaining Fuzzers
• Mutating a correct value
• pre-existent or grammar-fuzzed
• produces probably correct values
• and probably incorrect too
42
Grammar
Grammar Fuzzer
Structured value
Mutation Fuzzer
Probably structured value
DateGrammar
23-13-987
23/13-a87
43. Results by crash location
(signaler context method + pc)
• 68% disagreements - 6826 out of 10k fuzzings
• new errors!
Error # %
Stream still with data 2847 41,71 %
Input doesn’t match pattern 1534 22,47 %
Wrong year digits 1379 20,20 %
Wrong day/month 654 9,58 %
No error! 50 0,73 %
Day after month end 9 0,13 %
Day zero or negative 5 0,07 %
43
44. Results by crash location
(signaler context method + pc)
• 68% disagreements - 6826 out of 10k fuzzings
• new errors!
Error # %
Stream still with data 2847 41,71 %
Input doesn’t match pattern 1534 22,47 %
Wrong year digits 1379 20,20 %
Wrong day/month 654 9,58 %
No error! 50 0,73 %
Day after month end 9 0,13 %
Day zero or negative 5 0,07 %
44
45. When asDate was ok (50/10000)
asDate
DateParser Result OK
Result NOK
’10-30-s9 6’
45
46. When asDate was ok (50/10000)
’10-30-s9 6’
asDate
DateParser Result OK
Result NOK
DateParser accepts (and ignores) non numeric characters in year
46
47. Takeaways
• Simple random inputs can unveil bugs
• but, random inputs get random outputs!
• Adding some domain knowledge makes fuzzing more e
ffi
cient
• grammars, mutations, expected exceptions…
• Two programs following the same speci
fi
cation can test each other
• Yet, maybe neither holds the ground truth
47 * Supported by AlaMVic Action Exploratoire INRIA
Evref
fervE
https://github.com/Alamvic/phuzzer
https://github.com/Alamvic/gnocco
Also thanks to …
50. Material
• The Fuzzing Book. Fuzzer Chapter. A. Zeller et al
https://www.fuzzingbook.org/html/Fuzzer.html
• Fuzzing – Brute Force Vulnerability Discovery.
• Fuzzing for Software Security and Quality Assurance.
Takanen et al. 2018
• An Empirical Study of the Reliability of UNIX Utilities
Miller et al. Communications of the ACM’90
• Fuzz project assignment
https://pages.cs.wisc.edu/~bart/fuzz/CS736-Projects-f1988.pdf
50
51. Material
• The Fuzzing Book. Grammars Chapter. A. Zeller et al
https://www.fuzzingbook.org/html/Grammars.html
• Gnocco
https://github.com/Alamvic/gnocco/
51
52. Material
• The Oracle Problem in Software Testing: A Survey.
Barr et al. IEEE Transactions.’15
52
54. Material
• The Fuzzing Book. Mutation Chapter. A. Zeller et al
https://www.fuzzingbook.org/html/MutationFuzzer.html
• Binary Fuzzing Strategies in AFL — Blog
https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-
works.html
54
55. Building a Random Fuzzer
55
PzRandomFuzzer >> fuzz
| stringLength |
stringLength := random
nextIntegerBetween: minLength
and: maxLength + 1.
^ String streamContents: [ : str |
stringLength timesRepeat: [
str nextPut: (random
nextIntegerBetween: charStart asciiValue
and: charStart asciiValue + charRange)
asCharacter ] ]
• Choose a random size
• Choose random chars
in a range
• Build up a string
• + sensible default values
56. Analysis II
• Some inputs PASS but do not respect the contract
• Parser is too permissive
• Our runner is too permissive too => we should detect this as an error!
56
"Answer an instance of created from a string with format
mm.dd.yyyy or mm-dd-yyyy or mm/dd/yyyy"
'?(2/=-@=@:4?/(3$3(8"&,!-2/&6&&' asDate.
>> 4 February 2003
65. Domain-specific mutations
• E.g., swap day and month
• E.g., change the schema of a URL (http by ftp)
• E.g., change the a smic operator by another (+ by -)
65
f := PzMutationFuzzer new
seed: { ’00-11-22’ };
mutations: { PzDayMonthSwapMutation new }
yourself.
67. Mutation Analysis vs Mutation Fuzzing
• Mutation analysis evaluates test suite quality
• High Mutation Score => good tests
• Surviving mutants => show missing tests, or are equivalent
• Mutation fuzzing creates small variants
• There is no notion of score
• Equivalent mutants could be valuable!
67