The 16th Intl. Workshop on Search-Based and Fuzz Testing 14th of May 2023, Melbourne, Australia https://sbft23.github.io/

1. Search-Based and Fuzz Testing (SBFT) '23
May 14, 2023, 9:15am UTC+11
https://sbft23.github.io/
@sbstworkshop
Alessio Gambi, Giovani Guizzo, and Sebastiano Panichella

2. Twitch stream → https://www.twitch.tv/koxp
(screen mirroring for students)
Twitter → @sbftworkshop
On Zoom all day!
Welcome to Melbourne!
Sebastiano

3. Thanks to our sponsors and supporters!
Google Open Source Security Team
Google Cloud credits for running the tool competition
experiments
IEEE Technical Community on Software Engineering
(TCSE)
ACM Special Interest Group on Software Engineering
(SIGSOFT)
Providing a physics engine for authentic vehicle
simulation
Giovani
Sebastiano

4. SBST SBFT
Rebranding (refactoring maybe?) of our workshop
Direct result of last year’s discussion panel on SBST vs Fuzzing
Great opportunity for us to share knowledge
Introduction of the Fuzzing tool competition (already a success!)
2 SBST papers and 3 Fuzzing papers
Giovani
Giovani

5. General format
Participants:
Check your email for your Zoom invitations
Please remain muted until it is your turn to speak!
In-person:
Room 105
Non-registered participants:
Feel free to join our Twitch stream!
https://www.twitch.tv/koxp
Everything on Twitch will be recorded and uploaded to YouTube!
Giovani

6. Schedule / UTC+11
9:15 → Opening
9:30 → Keynote:
Truth or Dare: Real-World Fuzz Testing of UAVs in Flight
by Jane Cleland-Huang
10:30 → Break
11:00 → Discussion Panel
Testing and Security for Cyber-Physical Systems
Aitor Arrieta, Annibale Panichella, Jane Cleland-Huang,
Lionel Briand, Mohammad R. Mousavi, Shaukat Ali
12:00 → Research Track Session 1 - SBST (2 papers)
12:30 → Lunch Break
13:30 → Keynote:
Revisiting the Notion of Diversity in Software Testing
by Lionel Briand
14:30 → Research Track Session 2 - Fuzzing (3 papers)
15:15 → Break
15:45 → Tool Competition
18:15 → Closing (and Awards)
Sebastiano
https://conf.researchr.org/program/icse-2023/program-icse-2023/ https://sbft23.github.io/program/

7. Social Event/Dinner
Sunday 14 May Starting at 19:00
Meat Market, South Wharf
Sebastiano
2 course meal with a 3 hour
beverage package (includes wine,
beer and soft drink)

8. Keynote (09:30 - 10:30)
Sebastiano
Jane Cleland-Huang
Department of Computer Science and Engineering at
the University of Notre Dame.
Truth or Dare:
Real-World Fuzz Testing of UAVs in Flight

9. Truth or Dare: Real-World Fuzz Testing of UAVs in Flight
In this talk, Jane Cleland-Huang will draw upon her own
real-world experiences of developing and validating software
for Small Unmanned Aerial Systems (sUAS) applications, and
will explore the `truths’ and ‘dares’ of Fuzzing in the Field. The
truth is that in the normal course of running physical flight tests,
accidental erroneous input values can cause dramatic failures
and crashes, but can also be incredibly helpful at revealing
points of fragility in the system where modifications are
needed. However, accidental fuzzing falls far short of the
systematic goals of true fuzzing, introducing the non-trivial
dilemma of how to safely deploy Fuzz Testing in the field. This
talk explores this dare, by exploring a systematic approach for
fuzz testing physical sUAS systems, thereby empowering
testers to identify real-world weaknesses in edge-cases that
could have been missed in simulation. The end result is
increased robustness in real-world sUAS systems.
Bio:
Jane Cleland-Huang is the Frank M. Freimann Professor of Computer Science
Chair and Department Chair of Computer Science and Engineering at the
University of Notre Dame. Her research interests focus on Requirements
Engineering, Software and Systems Traceability, and Safety Assurance for
Cyber-Physical Systems (CPS). She is the Project Lead on the Drone Response
project which was initially developed as a research platform for supporting
Software Engineering research in multi-agent CPS, but is now the core platform
for a commercial system for deploying Small Unmanned Aerial Systems (sUAS) in
Emergency Response scenarios. Jane has served as Program Chair for several
international conferences (RE, ICSE, ESEC/FSE, and more), as Associate Editor of
IEEE TSE and on the IEEE Software editorial board. Currently, Jane serves as
Chair of IFIP 2.9 Working group on Requirements Engineering, and on the
Editorial Boards for Communications of the ACM and Springer Verlag
Requirements Engineering journal. Along with members of her research group
she has been the recipient of seven ACM SIGSOFT Distinguished Paper awards,
and the Mannfred Paul award for Excellence in Software Theory and Practice.
Jane is committed to supporting a diverse, equitable, and inclusive community of
Software Engineering researchers and envisions a future in which our research
community is truly reflective of the population around us. She is also passionate
about impacting the world in a positive way through technology transfer that
takes research into practice, and as a result is currently engaged in two spin-off
companies – DroneResponse and SAFA.”
Keynote (09:30 - 10:30)
Jane Cleland-Huang
Sebastiano

10. Testing and Security for Cyber-Physical Systems (11:00 - 12:00)
Sebastiano
DISCUSSION PANEL
Annibale Panichella
Aitor Arrieta Lionel Briand
Jane Cleland-Huang Mohammad Reza Mousavi Shaukat Ali

11. Summary of the Discussion Panel
- Challenges:
- Testing under uncertainty
- Execution time
- Multiple components, languages, etc.
- How to guarantee safety requirements? Proving wrong is not enough.
- Constrained resources make it harder to test
- Complex control layers
- Fidelity of simulation
- Human in-the-loop testing
- How to test ML models in this context (e.g., uncertainty they produce)
- Heterogeneity of CPS systems
- The oracle problem
- Hardware in-the-loop (expensive to test and crash - do you dare?)
- Solutions:
- Surrogate models to reduce cost
- Better understanding of vulnerabilities
- Balance cost and fidelity
- Peer-review with multiple expert simulators (e.g., co-simulation)
- Metamorphic testing can help with the oracle problem
- ML based testing oracles
- Trial and error. Experiment with simulators and real-world executions.
We need to be more creative :)
Sebastiano

12. Paper Session 1 - SBST (12:00 - 12:30)
Alessio & Vincenzo
On the Strengths of Pure Evolutionary Algorithms in Generating Adversarial Examples
Antony Bartlett, Cynthia C. S. Liem and Annibale Panichella
Automatic Generation of Smell-free Unit Tests
João Afonso and José Campos
All papers (15 minutes):
- 10 minutes for talk
- 5 minutes for questions

13. Lunch session (12:30 - 13:30)
You can socialise over ICSE lunch.
Stay in the Zoom meeting for chatting during that time.
Alessio, Giovani, Sebastiano

14. Keynote (13:30 - 14:30)
Gunel
Lionel Briand
EECS Department, University of Ottawa &
Centre for ICT Security, Reliability, and Trust (SnT), University of Luxembourg
Revisiting the Notion of
Diversity in Software Testing

15. Revisiting the Notion of Diversity in Software Testing
The notion of diversity has been used to automate various
software testing tasks, for example test selection and
minimisation. Intuitively, it is clear that more diverse test inputs
and outputs are likely to detect more faults by more extensively
exercising the software under test. However, measuring
diversity usually comes at a significant cost, alternatives based
on different information sources need to be considered, and
trade-offs are required. The way diversity is measured
therefore varies significantly depending on the context of
application and scalability considerations. This presentation will
reflect on many years of experience during which that concept
has been used in test automation, across various application
contexts, to help devise practical and scalable testing solutions.
Bio:
Lionel C. Briand is professor of software engineering and has shared
appointments between (1) The University of Ottawa, Canada and (2) The SnT
centre for Security, Reliability, and Trust, University of Luxembourg. In
collaboration with colleagues, over 25 years, he has run many collaborative
research projects with companies in the automotive, satellite, aeropsace, energy,
financial, and legal domains. Lionel has held various engineering, academic, and
leading positions in six countries. He was one of the founders of the ICST
conference (IEEE Int. Conf. on Software Testing, Verification, and Validation, a
CORE A event) and its first general chair. He was also EiC of Empirical Software
Engineering (Springer) for 13 years and led, in collaboration with first Victor Basili
and then Tom Zimmermann, the journal to the top tier of the very best publication
venues in software engineering.
Lionel was elevated to the grades of IEEE Fellow and ACM Fellow for his work on
software testing and verification. He was granted the IEEE Computer Society
Harlan Mills award, the ACM SIGSOFT outstanding research award, and the IEEE
Reliability Society engineer-of-the-year award, respectively in 2012, 2022, and
2013. He received an ERC Advanced grant in 2016 — on the topic of modelling
and testing cyber-physical systems — which is the most prestigious individual
research award in the European Union. He currently holds a Canada Research
Chair (Tier 1) on "Intelligent Software Dependability and Compliance". His
research interests include: software testing and verification, applications of AI in
software engineering, model-driven software development, requirements
engineering, and empirical software engineering.
Keynote (13:30 - 14:30)
Lionel Briand
Sebastiano

16. Paper Session 2 - Fuzzing (14:30 - 15:15)
PASTIS: A Collaborative Approach to Combine Heterogeneous Software Testing Techniques
Robin David, Richard Abou Chaaya and Christian Heitman
Continuous Fuzzing: A Study of the Effectiveness and Scalability of Fuzzing in CI/CD
Pipelines
Thijs Klooster, Fatih Turkmen, Gerben Broenink, Ruben Ten Hove and Marcel Böhme
Grammar-Based Evolutionary Fuzzing for JSON-RPC APIs
Lisette Veldkamp, Mitchell Olsthoorn and Annibale Panichella
Sebastiano & Dongge
All papers (15 minutes):
- 10 minutes for talk
- 5 minutes for questions

17. Fuzzing Tool Competition
Freshman
CPS Tool Competition
Junior
Java Tool Competition
Senior/Old School
Tools Competitions (15:45 - 18:00)
Tool Chairs - Dongge
All papers (5 minutes)

18. AFLSmart++ - Smarter Greybox Fuzzing
Van-Thuan Pham
AFLrustrust - A LibAFL-based AFL++ prototype
Andrea Fioraldi, Dominik Maier, Dongjia Zhang and Addison Crump
LibAFL_libFuzzer - LibFuzzer on top of LibAFL
Addison Crump, Andrea Fioraldi, Dominik Maier and Dongjia Zhang
HasteFuzz - Full-Speed Fuzzing
Zhengjie Du and Yuekang Li
R-Fuzz at SBFT'2023
Ju Chen, Chengyu Song and Heng Yin
Fuzzing Tools (15:45 - 16:30)
Tool Chairs

19. Java Tools (16:30 - 17:01)
Tool Chairs
EvoSuite at the SBFT 2023 Tool Competition
Sebastian Schweikl, Gordon Fraser and Andrea Arcuri
Kex at the SBFT 2023 Java Tool Competition
Azat Abdullin and Marat Akhin
UTBot at the SBFT 2023 Java Tool Competition
Dmitry Ivanov, Maxim Pelevin, Alexey Menshutin, Denis Fokin, Yury Kamenev, Sergey Pospelov, Egor
Kulikov, Nikita Stroganov and Ivan Volkov

20. CPS Tools (17:01 - 17:53)
Tool Chairs
CRAG at the SBFT 2023 Tool Competition - Cyber-Physical Systems Track
Paolo Arcaini and Ahmet Cetinkaya
EvoMBT at the 2023 SBFT Tool Competition
Raihana Ferdous, Chia-kang Hung, Fitsum Kifetew, Davide Prandi and Angelo Susi
RIGAA at the SBFT 2023 Tool Competition - Cyber-Physical Systems Track
Dmytro Humeniuk, Foutse Khomh, and Giuliano Antoniol
RoadSign at the SBFT 2023 Tool Competition - Cyber-Physical Systems Track
Jon Ayerdi, Aitor Arrieta, and Miren Illaramendi
Spirale at the SBFT 2023 Tool Competition - Cyber-Physical Systems Track
Domenico De Vivo and Anna Rita Fasolino
WOGAN at the SBFT 2023 Tool Competition - Cyber-Physical Systems Track
Jesper Winsten and Ivan Porres

21. Tools Competition Reports/Results (18:00 - 18:15)
Tool Chairs
All reports (5 minutes)

22. Closing (18:15)
Thanks to:
- All of you for joining!
- Our PC for their support in
reviewing papers!
- Our steering committee for their
support in moving the workshop
online!
- Our sponsor (Google, BeamNG,
TCSE, and SIGSOFT)
47 registered participants specifically for SBFT
65+ people joined (online + in-person)
42+ (peak) in-person participants at SBFT
9 (peak) attending via Twitch
21 (peak) concurrent online viewers
Sebastiano

23. Awards
Sebastiano

24. Sebastiano

26. Thanks to: Our Tool Competition Co-chairs
for organizing an exciting and relevant tool competitions!
Sebastiano
Abhishek Arya Dongge Liu Gunel Jahangirova Jarkko Peltomäki Jonathan Metzman
Marcel Böhme Matteo Biagiola Oliver Chang Stefan Klikovits Valerio Terragni Vincenzo Riccio

27. Thanks to: our Web-chair
for preparing our beautiful website!
Sebastiano
Rebecca Moussa

28. Thanks to: the Program Committee members
for their support in reviewing papers!
Aldeida Aleti, Monash University, Australia
Aymeric Blot, Université du Littoral Côte D'opale
Byron DeVries, Grand Valley State University, USA
Erik Fredericks, Oakland University, USA
Fiorella Zampetti, University of Sannio, Italy
Gordon Fraser, University of Passau, Germany
Gregory Gay, Chalmers and the University of Gothenburg, Sweden
Gregory M. Kapfhammer, Allegheny College, USA
Jie Zhang, King's College London, UK
José Campos, University of Washington, USA
Kate Bowers, Oakland University, USA
Matteo Biagiola, Università della Svizzera italiana, Switzerland
Mitchell Olsthoorn, Delft University of Technology, Netherlands
Nazareno M. Aguirre, Universidad Nacional de Río Cuarto, Argentina
Silvia Regina Vergilio, Universidade Federal do Paraná, Brazil
Simone do Rocio Senger de Souza, University of São Paulo, Brazil
Thomas Vogel, Humboldt-Universität zu Berlin, Germany
Sebastiano

29. Thanks to: our Viewers
For making this workshop interesting all around!
You are the best :)
Sebastiano

30. Disclaimer
Sebastiano
Recordings of our Workshop will be made available on Twitch and YouTube
Integration of Inputs from the Panel discussion for the SIGSOFT newsletter (informal
quarterly newsletter on Software Engineering), linking to the video of the discussion
We can remove your recording upon request.

31. What’s Next?
Special issue at Science of Computer Programming 2023:
SBFT’23: Search-Based and Fuzz Testing Tools
https://www.sciencedirect.com/journal/science-of-computer-programming/about/call
-for-papers#sbft-23-search-based-and-fuzz-testing-tools
Short papers with a great focus on software and replication packages
Submission Dates: 1st of June, 2023 to 1st of October, 2023
Sebastiano

32. What’s Next?
Find more sponsors to support with cash prize(s) for the winners of the tool competition? 🤑
Coordinate with similar academic and industrial workshops or venues (e.g., FuzzCon, Fuzzing
Workshop @ NDSS) in other SE and non-SE venues (e.g., Cyber-physical systems) to
continuously foster research in the field.
Establishment of complementary competitions on Python and other Cyber-physical domains.
We are also open to ideas or new tool/SBFT challenges (contact us)!
Sebastiano

33. SBFT 2024
Farewell and Thank you !
Giovani Guizzo Sebastiano Panichella
Alessio Gambi

34. SBFT 2024
Welcome on Board!
Gunel Jahangirova Vincenzo Riccio
Alessio Gambi

35. SBFT 2024 - AI and Generative Models Testing
Keynote on challenges concerning the development,
Testing, and fairness of AI and Generative Models
Promote discussion panels around contemporary testing
challenges of AI and Generative Models with industrial
and academic researchers (Chat GPT? Bert?, etc.)
Encourage the design, implementation, and public
availability of usable and high-quality tools to deal with
SBFT-related challenges.
Two years plan to transform SBFT as a Symposium (with
two days workshop as intermediate step)?
Sebastiano
https://github.com/Mooler0410/LLMsPracticalGuide

36. Social Event/Dinner
Sunday 14 May Starting at 19:00
Meat Market, South Wharf
Sebastiano
2 course meal with a 3 hour
beverage package (includes wine,
beer and soft drink)

37. Thank you all for participating!
See you next year in Lisbon
at SBFT 2024!