Boolean test input generation is the process of finding sets of values for variables of a logical expression such that a given coverage criterion is achieved. This paper presents a formal framework in which evaluating an expression produces a tree structure, and where a coverage criterion is expressed as equivalence classes induced by a particular transformation over these trees. It then defines many well-known coverage criteria as particular cases of this framework. The paper describes an algorithm to generate test suites by a reduction through a graph problem; this algorithm works in the same way regardless of the criterion considered. An experimental evaluation of this technique shows that it produces test suites that are in many cases smaller than existing tools.
A Configurable CEGAR Framework with Interpolation-Based RefinementsAkos Hajdu
Presentation of our paper at the the 36th IFIP International Conference on Formal Techniques for Distributed Objects, Components and Systems (FORTE 2016). Heraklion, Greece
Graph Methods for Generating Test Cases with Universal and Existential Constr...Sylvain Hallé
We introduce a generalization of the t -way test case generation problem, where parameter t is replaced by a set Φ of Boolean conditions on attribute values. We then present two reductions of this problem to graphs; first, to graph colouring, where we link the the minimal number of tests to the chromatic number of some graph; second, to hypergraph vertex covering. This latter formalization allows us to handle problems with constraints of two kinds: those that must be true for every generated test case, and those that must be true for at least one test case. Experimental results show that the proposed solution produces test suites of slightly smaller sizes than a range of existing tools, while being more general: to the best of our knowledge, our work is the first to allow existential constraints over test cases.
Test Sequence Generation with Cayley Graphs (Talk @ A-MOST 2021)Sylvain Hallé
The paper presents a theoretical foundation for test sequence generation based on an input specification. The set of possible test sequences is first partitioned according to a generic "triaging" function, which can be created from a state-machine specification in various ways. The notion of coverage metric is then expressed in terms of the categories produced by this function. Many existing test generation problems, such as t-way state or transition coverage, become particular cases of this generic framework. We then present algorithms for generating sets of test sequences providing guaranteed full coverage with respect to a metric, by building and processing a special type of graph called a Cayley graph. An implementation of these concepts is then experimentally evaluated against existing techniques, and shows it provides better performance in terms of running time and test suite size.
This paper presents a novel SAT-based approach for the computation of extensions in abstract argumentation, with focus on preferred semantics, and an empirical evaluation of its performances. The approach is based on the idea of reducing the problem of computing complete extensions to a SAT problem and then using a depth-first search method to derive preferred extensions. The proposed approach has been tested using two distinct SAT solvers and compared with three state-of-the-art systems for preferred extension computation. It turns out that the proposed approach delivers significantly better performances in the large majority of the considered cases.
A Configurable CEGAR Framework with Interpolation-Based RefinementsAkos Hajdu
Presentation of our paper at the the 36th IFIP International Conference on Formal Techniques for Distributed Objects, Components and Systems (FORTE 2016). Heraklion, Greece
Graph Methods for Generating Test Cases with Universal and Existential Constr...Sylvain Hallé
We introduce a generalization of the t -way test case generation problem, where parameter t is replaced by a set Φ of Boolean conditions on attribute values. We then present two reductions of this problem to graphs; first, to graph colouring, where we link the the minimal number of tests to the chromatic number of some graph; second, to hypergraph vertex covering. This latter formalization allows us to handle problems with constraints of two kinds: those that must be true for every generated test case, and those that must be true for at least one test case. Experimental results show that the proposed solution produces test suites of slightly smaller sizes than a range of existing tools, while being more general: to the best of our knowledge, our work is the first to allow existential constraints over test cases.
Test Sequence Generation with Cayley Graphs (Talk @ A-MOST 2021)Sylvain Hallé
The paper presents a theoretical foundation for test sequence generation based on an input specification. The set of possible test sequences is first partitioned according to a generic "triaging" function, which can be created from a state-machine specification in various ways. The notion of coverage metric is then expressed in terms of the categories produced by this function. Many existing test generation problems, such as t-way state or transition coverage, become particular cases of this generic framework. We then present algorithms for generating sets of test sequences providing guaranteed full coverage with respect to a metric, by building and processing a special type of graph called a Cayley graph. An implementation of these concepts is then experimentally evaluated against existing techniques, and shows it provides better performance in terms of running time and test suite size.
This paper presents a novel SAT-based approach for the computation of extensions in abstract argumentation, with focus on preferred semantics, and an empirical evaluation of its performances. The approach is based on the idea of reducing the problem of computing complete extensions to a SAT problem and then using a depth-first search method to derive preferred extensions. The proposed approach has been tested using two distinct SAT solvers and compared with three state-of-the-art systems for preferred extension computation. It turns out that the proposed approach delivers significantly better performances in the large majority of the considered cases.
This course provides a strong background about JAVA programming language in the field of computing. The course begins with an introductory overview of the Computer and programs, with distinguishes the terms API, IDE and JDK, and gives a comprehensive knowledge about Java development kits and Java integrative development environments like eclipse and NetBeans. Furthermore, the course prepares student to write, compile, run and develop Java applications which are used to find out the solution for several real life problems, in conjunction with using GUI to obtain input, process and display outputs like message dialog boxes, input dialog boxes, confirmation dialog and so on.
JAVA is a computer programming language that is concurrent, class-based, object-oriented, and specifically designed to have as few implementation dependencies as possible.
The aim of this course is to explore Java programming fundamentals related to write, compile, run and develop Java applications that are used to discover the solution for several real life problems.
The official learning outcome for this course is: Upon successful completion of the course the students:
• Must know the basic concepts related JAVA programming language.
• Must know how to write, compile, run and develop java applications.
A combination of lectures and practical sessions will be used in this course in order to achieve the aim of the course.
By MSc. Karwan Mustafa Kareem
Solution to the practice test ch 10 correlation reg ch 11 gof ch12 anovaLong Beach City College
Please Subscribe to this Channel for more solutions and lectures
http://www.youtube.com/onlineteaching
Elementary Statistics Practice Test 5
Module 5
Chapter 10: Correlation and Regression
Chapter 11: Goodness of Fit and Contingency Tables
Chapter 12: Analysis of Variance
This is the TEST BANK for Absolute Java by Savitch 5th edition. Comes with multiple choice, true false, essay questions and code to help you solve complex Java problems. Email lauriewest24@gmail.com if you are interested.
For undergraduate students in Computer Science and Computer Programming courses.
Praised for providing an engaging balance of thoughtful examples and explanatory discussion,¿best-selling author Walter Savitch explains concepts and techniques in a straightforward style using understandable language and code enhanced by a suite of pedagogical tools.¿ Absolute Java is appropriate for both introductory and intermediate programming courses introducing Java.
Email lauriewest24@gmail.com if you are interested in teh full file. Thanks.
Monitoring Business Process Compliance Across Multiple Executions with Stream...Sylvain Hallé
Slides of a talk given at EDOC 2023, November 3rd, 2023.
Abstract: Compliance checking is the operation that consists of assessing whether every execution trace of a business process satisfies a given correctness condition. The paper introduces the notion of hyperquery, which is a calculation that involves multiple traces from a log at the same time. A particular case of hyperquery is a hypercompliance condition, which is a correctness requirement that involves the whole log instead of individual process instances. A formalization of hyperqueries is presented, along with a number of elementary operations to express hyperqueries on arbitrary logs. An implementation of these concepts in an event stream processing engine allows users to concretely evaluate hyperqueries in real time.
A Stream-Based Approach to Intrusion DetectionSylvain Hallé
Slides of a presentation given to promote the book "CyberSecurity in a DevOps Environment" (2023).
Abstract:
Integrating security in the development and operation of information systems is the cornerstone of SecDevOps. From an operational perspective, one of the key activities for achieving such an integration is the detection of incidents (such as intrusions), especially in an automated manner. However, one of the stumbling blocks of an automated approach to intrusion detection is the management of the large volume of information typically produced by this type of solution. Existing works on the topic have concentrated on the reduction of volume by increasing the precision of the detection approach, thus lowering the rate of false alarms. However, another, less explored possibility is to reduce the volume of evidence gathered for each alarm raised.
This chapter explores the concept of intrusion detection from the angle of complex event processing. It provides a formalization of the notion of pattern matching in a sequence of events produced by an arbitrary system, by framing the task as a runtime monitoring problem. It then focuses on the topic of incident reporting, and proposes a technique to automatically extract relevant elements of a stream that explain the occurrence of an intrusion. These relevant elements generally amount to a small fraction of all the data ingested for an alarm to be triggered, and thus help reduce the volume of evidence that needs to be examined by manual means. The approach is experimentally evaluated on a proof-of-concept implementation of these principles.
More Related Content
Similar to Test Suite Generation for Boolean Conditions with Equivalence Class Partitioning
This course provides a strong background about JAVA programming language in the field of computing. The course begins with an introductory overview of the Computer and programs, with distinguishes the terms API, IDE and JDK, and gives a comprehensive knowledge about Java development kits and Java integrative development environments like eclipse and NetBeans. Furthermore, the course prepares student to write, compile, run and develop Java applications which are used to find out the solution for several real life problems, in conjunction with using GUI to obtain input, process and display outputs like message dialog boxes, input dialog boxes, confirmation dialog and so on.
JAVA is a computer programming language that is concurrent, class-based, object-oriented, and specifically designed to have as few implementation dependencies as possible.
The aim of this course is to explore Java programming fundamentals related to write, compile, run and develop Java applications that are used to discover the solution for several real life problems.
The official learning outcome for this course is: Upon successful completion of the course the students:
• Must know the basic concepts related JAVA programming language.
• Must know how to write, compile, run and develop java applications.
A combination of lectures and practical sessions will be used in this course in order to achieve the aim of the course.
By MSc. Karwan Mustafa Kareem
Solution to the practice test ch 10 correlation reg ch 11 gof ch12 anovaLong Beach City College
Please Subscribe to this Channel for more solutions and lectures
http://www.youtube.com/onlineteaching
Elementary Statistics Practice Test 5
Module 5
Chapter 10: Correlation and Regression
Chapter 11: Goodness of Fit and Contingency Tables
Chapter 12: Analysis of Variance
This is the TEST BANK for Absolute Java by Savitch 5th edition. Comes with multiple choice, true false, essay questions and code to help you solve complex Java problems. Email lauriewest24@gmail.com if you are interested.
For undergraduate students in Computer Science and Computer Programming courses.
Praised for providing an engaging balance of thoughtful examples and explanatory discussion,¿best-selling author Walter Savitch explains concepts and techniques in a straightforward style using understandable language and code enhanced by a suite of pedagogical tools.¿ Absolute Java is appropriate for both introductory and intermediate programming courses introducing Java.
Email lauriewest24@gmail.com if you are interested in teh full file. Thanks.
Similar to Test Suite Generation for Boolean Conditions with Equivalence Class Partitioning (20)
Monitoring Business Process Compliance Across Multiple Executions with Stream...Sylvain Hallé
Slides of a talk given at EDOC 2023, November 3rd, 2023.
Abstract: Compliance checking is the operation that consists of assessing whether every execution trace of a business process satisfies a given correctness condition. The paper introduces the notion of hyperquery, which is a calculation that involves multiple traces from a log at the same time. A particular case of hyperquery is a hypercompliance condition, which is a correctness requirement that involves the whole log instead of individual process instances. A formalization of hyperqueries is presented, along with a number of elementary operations to express hyperqueries on arbitrary logs. An implementation of these concepts in an event stream processing engine allows users to concretely evaluate hyperqueries in real time.
A Stream-Based Approach to Intrusion DetectionSylvain Hallé
Slides of a presentation given to promote the book "CyberSecurity in a DevOps Environment" (2023).
Abstract:
Integrating security in the development and operation of information systems is the cornerstone of SecDevOps. From an operational perspective, one of the key activities for achieving such an integration is the detection of incidents (such as intrusions), especially in an automated manner. However, one of the stumbling blocks of an automated approach to intrusion detection is the management of the large volume of information typically produced by this type of solution. Existing works on the topic have concentrated on the reduction of volume by increasing the precision of the detection approach, thus lowering the rate of false alarms. However, another, less explored possibility is to reduce the volume of evidence gathered for each alarm raised.
This chapter explores the concept of intrusion detection from the angle of complex event processing. It provides a formalization of the notion of pattern matching in a sequence of events produced by an arbitrary system, by framing the task as a runtime monitoring problem. It then focuses on the topic of incident reporting, and proposes a technique to automatically extract relevant elements of a stream that explain the occurrence of an intrusion. These relevant elements generally amount to a small fraction of all the data ingested for an alarm to be triggered, and thus help reduce the volume of evidence that needs to be examined by manual means. The approach is experimentally evaluated on a proof-of-concept implementation of these principles.
Event Stream Processing with BeepBeep 3Sylvain Hallé
Event logs and event streams can be found in software systems of very diverse kinds. Analyzing the wealth of information contained in these logs can serve multiple purposes. In this tutorial, the audience will learn about BeepBeep, a versatile Java library intended to make the processing of event streams (either offline or in realtime) both fun and simple. BeepBeep is the result of more than a decade of research led by a team at Laboratoire d’informatique formelle at Université du Québec à Chicoutimi (Canada). Over the past few years, BeepBeep has been involved in a variety of case studies, and provides built-in support for writing domain-specific languages. Recently, a complete textbook has been published on BeepBeep, testifying to the maturity that the system has acquired.
Smart Contracts-Enabled Simulation for Hyperconnected LogisticsSylvain Hallé
(Slides for a talk given in the journal-first track of the EDOC 2022 conference.)
The combination of the Internet of Things and blockchain-based technologies represents a real opportunity for supply chain and logistics protagonists, who need more dynamic, trustworthy and transparent tracking systems in order to improve their efficiency and strengthen customer confidence. In parallel, hyperconnected logistics promise more efficient and sustainable goods handling and delivery. This chapter shows how the Ethereum blockchain and smart contracts can be used to implement a shareable and secured tracking system for hyperconnected logistics. A simulation using the well-known AnyLogic software tool provides insights on the monitoring of properties depicting shipment lifecycle constraints through a stream of blockchain log events processed by BeepBeep 3, an open source stream processing engine.
Synthia: a Generic and Flexible Data Structure Generator (Long Version)Sylvain Hallé
Synthia is a versatile, modular and extensible Java-based data structure generation library. It is centered on the notion of "pickers", which are objects producing values of a given type on demand. Pickers are stateful and can be given as input to other pickers; this chaining principle can generate objects whose structure follows a complex pattern. The paper describes the core principles and key features of the library, including test input shrinking, provenance tracking, and object mutation.
Efficient Offline Monitoring of LTL with Bit Vectors (Talk at SAC 2021)Sylvain Hallé
We describe an extension of the BeepBeep stream processing library for the offline verification of arbitrary expressions of Linear Temporal Logic using bitmap manipulations. Experimental results show that, for complex LTL formulæ containing up to 20 operators, event traces can be evaluated at a throughput of millions of events per second and provide a considerable speed-up compared to the current implementation of the tool.
Detecting Responsive Web Design Bugs with Declarative SpecificationsSylvain Hallé
Responsive Web Design (RWD) is a concept that is born from the need to provide users with a positive and intuitive experience, no matter what device they use. Complex Cascading Style Sheets (CSS) are used in RWD to smoothly change the appearance of a website based on the window width of the device being used. The paper presents an automated approach for testing these dynamic web applications, where a combination of dynamic crawling and back-end testing is used to automatically detect RWD bugs.
Streamlining the Inclusion of Computer Experiments in Research PapersSylvain Hallé
To run experiments on a computer, you probably write command-line scripts for various tasks: generate your data, save it into files, process and display them as plots or tables to include in a paper. But soon enough, your handful of “quick and dirty” files becomes a bunch of poorly documented scripts that generate and pass around all kinds of obscure temporary files. LabPal is a library that allows you to set up an environment for running experiments, collating their results and processing them.
Writing Domain-Specific Languages for BeepBeepSylvain Hallé
This paper describes a plug-in extension of the BeepBeep 3 event stream processing engine. The extension allows one to write a custom grammar defining a particular specification language on event traces. A built-in interpreter can then convert expressions of the language into chains of BeepBeep processors through just a few lines of code, making it easy for users to create their own domain-specific languages.
Real-Time Data Mining for Event StreamsSylvain Hallé
Information systems produce different types of event logs; in many situations, it may be desirable to look for trends inside these logs. We show how trends of various kinds can be computed over such logs in real time, using a generic framework called the trend distance workflow. Many common computations on event streams turn out to be special cases of this workflow, depending on how a handful of workflow parameters are defined. This process has been implemented and tested in a real-world event stream processing tool, called BeepBeep.
Technologies intelligentes d'aide au développement d'applications web (WAQ 2018)Sylvain Hallé
Surfez sur le web quelques heures, et vous trouverez sûrement un site ou une application qui contient un bogue dans son interface graphique. Que ce soit un bouton qui ne fonctionne pas, un rectangle partiellement caché ou mal positionné, il semble que les problèmes d’interface soient notoirement difficiles à traquer! Des technologies intelligentes pourraient-elles aider les concepteurs et les développeurs à mieux tester leurs interfaces? Au Laboratoire d’informatique formelle de l’Université du Québec à Chicoutimi, nous le croyons. Nous avons développé Cornipickle, un logiciel permettant à un développeur d’énoncer, dans un langage simple et lisible, une foule de contraintes sur le positionnement et le contenu des éléments d’une interface web. Nous verrons comment Cornipickle peut s’intégrer à une application existante, détecter des problèmes, identifier les éléments qui sont fautifs et même suggérer des correctifs. Ceci permet donc à un développeur de gagner beaucoup de temps lors de la recherche des problèmes.
A "Do-It-Yourself" Specification Language with BeepBeep 3 (Talk @ Dagstuhl 2017)Sylvain Hallé
This talk reviews the basic principles behind the BeepBeep 3 event stream processing engine, and the facilities it provides to help you design you own, domain-specific query language.
Event Stream Processing with Multiple ThreadsSylvain Hallé
We present an extension to the BeepBeep 3 event stream engine that allows the use of multiple threads during the evaluation of a query. Compared to the single-threaded version of BeepBeep, the allocation of just a few threads to specific portions of a query provides improvement in terms of throughput.
Solving Equations on Words with Morphisms and AntimorphismsSylvain Hallé
Word equations are combinatorial equalities between strings of symbols, variables and functions, which can be used to model problems in a wide range of domains. While some complexity results for the solving of specific classes of equations are known, currently there does not exist a systematic equation solver. We present in this paper a reduction of the problem of solving word equations to Boolean satisfiability, and describe the implementation of a general-purpose tool that leverages existing SAT solvers for this purpose. Our solver will prove useful in the resolution of word equations, and in the computer-based exploration of various combinatorial conjectures.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Test Suite Generation for Boolean Conditions with Equivalence Class Partitioning
1. S. Hallé
Sylvain Hallé
Université du Québec à Chicoutimi
CANADA
Test Suite Generation for
Boolean Conditions with
Equivalence Class Partitioning
CRSNG
NSERC
FormaliSE, May 2022
2. S. Hallé
Boolean Conditions
Software systems are filled with conditions that
modulate their behavior.
SELECT name FROM employees
WHERE YEAR(dob) < 1990 AND rank = 3;
if (a % 3 == 0 || (b > c + 6 && o.isReady())
{
...
}
Italic
Bold
+
Underline
OK
Reversed
+
c1
a
(8/1)
=b ≠b
(4/2)
<2 >2
⊤
⊤
=2
c3
(7/2)
=0 ≠0
(3/1)
⊤
⊤
a
≠c =c
(10/3)
⊤
(7/2)
=0 ≠0
(3/1)
⊤
⊤
c2
3. S. Hallé
Boolean Conditions
Test input generation is the problem of
(automatically) generating values given to a system
if (a % 3 == 0 || (b > c + 6 && o.isReady())
{
...
}
4. S. Hallé
Boolean Conditions
Test input generation is the problem of
(automatically) generating values given to a system
if (a % 3 == 0 || (b > c + 6 && o.isReady())
{
...
}
a = 3
b = 2
c = 1
o.isReady() = true
test case
5. S. Hallé
Boolean Conditions
Test input generation is the problem of
(automatically) generating values given to a system
if (a % 3 == 0 || (b > c + 6 && o.isReady())
{
...
}
a = 3
b = 2
c = 1
o.isReady() = true
test case
1
1
6. S. Hallé
Boolean Conditions
Test input generation is the problem of
(automatically) generating values given to a system
if (a % 3 == 0 || (b > c + 6 && o.isReady())
{
...
}
a = 3
b = 2
c = 1
o.isReady() = true
test case
else
{
...
}
2
a = 1
b = 2
c = 2
o.isReady() = false
2
Different values send the
execution on different paths
1
1
7. S. Hallé
...but what if the condition
is incorrect?
Boolean Conditions
Test input generation is the problem of
(automatically) generating values given to a system
if (a % 3 == 0 || (b > c + 6 && o.isReady())
{
...
}
a = 3
b = 2
c = 1
o.isReady() = true
test case
else
{
...
}
2
a = 1
b = 2
c = 2
o.isReady() = false
2
Different values send the
execution on different paths
1
1
8. S. Hallé
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
9. S. Hallé
a = 0
b = 6
c = 0
o.isReady() = true
4
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
10. S. Hallé
a = 0
b = 6
c = 0
o.isReady() = true
4
⊤ ⊥ ⊤
⊤ ⊤ ⊤
⊤
⊤
+
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
11. S. Hallé
a = 0
b = 6
c = 0
o.isReady() = true
a = 1
b = 7
c = 1
o.isReady() = false
+
4 5
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
12. S. Hallé
a = 0
b = 6
c = 0
o.isReady() = true
a = 1
b = 7
c = 1
o.isReady() = false
+
4 5
⊥ ⊥ ⊥
⊥ ⊤ ⊥
⊥
⊥
+
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
13. S. Hallé
a = 0
b = 6
c = 0
o.isReady() = true
a = 1
b = 7
c = 1
o.isReady() = false
+
+
a = 1
b = 7
c = 1
o.isReady() = true
4 5 6
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
14. S. Hallé
a = 0
b = 6
c = 0
o.isReady() = true
a = 1
b = 7
c = 1
o.isReady() = false
+
+
a = 1
b = 7
c = 1
o.isReady() = true
4 5 6
⊥ ⊥ ⊤
⊥ ⊤ ⊤
⊥
⊤
✓
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
15. S. Hallé
Making the condition true/false once is not
sufficient to reveal errors
The components of the condition must evaluate to
true/false in various combinations
⇒ Boolean condition coverage
We must generate a test input for which the
condition sends the execution in the wrong branch
Boolean Conditions
a % 3 == 0 || (b > c + 6 && o.isReady())
a % 3 == 0 || (b ≥ c + 6 && o.isReady())
ACTUAL
EXPECTED
16. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
17. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
Predicate coverage
Each variable has a test case where it is true, and
another test case where it is false
18. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
Combinatorial ("t-way") coverage
Each t-tuple of variables has test cases for all
combinations of their true/false values
Example: for t=2
19. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
Clause coverage
Each clause of the condition has a test case where
it is true and another test case where it is false
{
clause
20. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
Modified condition/decision coverage (MC/DC)
Predicate coverage + clause coverage + every
clause is shown to independently affect the
outcome
21. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
Modified condition/decision coverage (MC/DC)
Predicate coverage + clause coverage + every
clause is shown to independently affect the
outcome
⊥
⊥ ⊥ ⊥
x₀ = ⊥ x₁ = ⊥ x₂ = ⊥ x₃ = ⊥
23. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
MUMCUT Coverage
3 conditions expressed in terms of:
24. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
MUMCUT Coverage
3 conditions expressed in terms of:
Unique true point (UTP): test case that makes a
single clause evaluate to ⊤
25. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
MUMCUT Coverage
3 conditions expressed in terms of:
Unique true point (UTP): test case that makes a
single clause evaluate to ⊤ x₀ = ⊥ x₁ = ⊤ x₂ = ⊥ x₃ = ⊤
⊥ ⊤ ⊥
26. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
MUMCUT Coverage
3 conditions expressed in terms of:
Unique true point (UTP): test case that makes a
single clause evaluate to ⊤
Near false point (NFP): test case where the
condition is ⊥, but flipping a single variable
changes its value
27. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
MUMCUT Coverage
3 conditions expressed in terms of:
Unique true point (UTP): test case that makes a
single clause evaluate to ⊤
Near false point (NFP): test case where the
condition is ⊥, but flipping a single variable
changes its value
⊥ ⊥ ⊥
x₀ = ⊥ x₁ = ⊥ x₂ = ⊥ x₃ = ⊤
28. S. Hallé
Several coverage criteria for Boolean conditions
have been proposed
Coverage Criteria
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
MUTP: each variable not in a UTP must be true/
false in at least one test
MNFP: each variable not in a NFP must be true/
false in at least one test
CUTP-NFP: all pairs of UTP and NFP for the same
clause that differ by a single variable flip must be
present
MUMCUT Coverage
1
2
3
29. S. Hallé
Each coverage criterion is expressed informally;
no uniform formal notation to define them all
Each criterion comes with its own algorithm to
generate test cases; the correctness/optimality of
these algorithms is often not demonstrated
Concrete implementations of these algorithms are
not easy to find (research papers); switching
criteria means switching programs
Must restart from scratch if developing a new
coverage criterion
In the current state of things...
Coverage Criteria
30. S. Hallé
Foundations based on concepts of algebra
Existing criteria become particular cases of this
model
Provides a test generation algorithm that works of
any criterion
Comes with a concrete and freely available
implementation
Goal: define a formal model of Boolean condition
coverage
Contribution
31. S. Hallé
Algebraic Definition of Coverage
Let X be a set of arbitrary symbols (variables). A
valuation is a total function ν : X → {⊤,⊥}. We
define as N the set of all valuations.
We denote by φ[ν] the result of evaluating φ by
setting its variables to the values defined by ν.
Let Φ be the set of Boolean formulas with variables
in X. A valuation ν ∈ N is a specific way of
evaluating a condition φ ∈ Φ.
Example: X = {x₀,x₁}
ν(x₀) = ⊤, ν(x₁) = ⊥
φ = x₀ ∧ ¬x₁
φ[ν] = ⊤
32. S. Hallé
Algebraic Definition of Coverage
Let τ : N → C be a total function mapping each
valuation ν to an element of a set C. An element c ∈
C is called a category.
Intuitively, τ classifies valuations into categories;
we call it a categorization function. Two valuations
ν and ν' are in the same equivalence class if
τ(ν) = τ(ν'). We note this ν ~ ν'.
The kernel of τ is the partition of N induced by the
quotient N/~. Each subset contains the valuations
that belong to the same category.
algebra!
33. S. Hallé
Algebraic Definition of Coverage
Example
X = {x₀,x₁}
There are 4 valuations in N:
ν₀ = {x₀ ↦ ⊤, x₁ ↦ ⊤}
ν₁ = {x₀ ↦ ⊤, x₁ ↦ ⊥}
ν₂ = {x₀ ↦ ⊥, x₁ ↦ ⊤}
ν₃ = {x₀ ↦ ⊥, x₁ ↦ ⊥}
34. S. Hallé
Define τ : N → {0,1} as
τ(ν) ={0 if ν(x₀) = ⊤
1 otherwise
Algebraic Definition of Coverage
Example
X = {x₀,x₁}
There are 4 valuations in N:
ν₀ = {x₀ ↦ ⊤, x₁ ↦ ⊤}
ν₁ = {x₀ ↦ ⊤, x₁ ↦ ⊥}
ν₂ = {x₀ ↦ ⊥, x₁ ↦ ⊤}
ν₃ = {x₀ ↦ ⊥, x₁ ↦ ⊥}
35. S. Hallé
Define τ : N → {0,1} as
τ(ν) ={0 if ν(x₀) ≠ ν(x₁)
1 otherwise
Algebraic Definition of Coverage
Example
X = {x₀,x₁}
There are 4 valuations in N:
ν₀ = {x₀ ↦ ⊤, x₁ ↦ ⊤}
ν₁ = {x₀ ↦ ⊤, x₁ ↦ ⊥}
ν₂ = {x₀ ↦ ⊥, x₁ ↦ ⊤}
ν₃ = {x₀ ↦ ⊥, x₁ ↦ ⊥}
36. S. Hallé
Define τ : N → {0,1} as
τ(ν) = the number of true variables in ν
2
1
0
Algebraic Definition of Coverage
Example
X = {x₀,x₁}
There are 4 valuations in N:
ν₀ = {x₀ ↦ ⊤, x₁ ↦ ⊤}
ν₁ = {x₀ ↦ ⊤, x₁ ↦ ⊥}
ν₂ = {x₀ ↦ ⊥, x₁ ↦ ⊤}
ν₃ = {x₀ ↦ ⊥, x₁ ↦ ⊥}
38. S. Hallé
Set of all possible
test cases
ECP: methodology where test cases are divided in
partitions
Equivalence Class Partitioning
39. S. Hallé
Set of all possible
test cases
Equivalence classes
ECP: methodology where test cases are divided in
partitions
Equivalence Class Partitioning
40. S. Hallé
Set of all possible
test cases
Equivalence classes
ECP: methodology where test cases are divided in
partitions
Equivalence Class Partitioning
Chosen test cases
41. S. Hallé
ECP: methodology where test cases are divided in
partitions
Equivalence Class Partitioning
Set of all possible
valuations (N)
Equivalence classes induced by τ
Representatives
42. S. Hallé
Algebraic Definition of Coverage
A test suite is a set of valuations (i.e. an element
V ∈ 2 ).
N
Define 𝜏(V) = ⋃
𝜈 ∈ V
{𝜏(𝜈)}
"the set of categories
present in the test suite"
A Boolean condition coverage criterion is called
algebraic if...
for every Boolean formula 𝜑,
there exists a function 𝜏,
such that a test suite V satisfies the criterion
if and only if 𝜏(V) = 𝜏(N). "all categories are
represented in V"
43. S. Hallé
𝜌(V) ∈ [0,1], and full coverage means 𝜌(V) = 1.
Algebraic Definition of Coverage
If a test suite V achieves partial coverage, it can
easily be quantified:
𝜌(V) =
𝜏(V)
𝜏(N)
coverage ratio of V fraction of all
categories that are
present in V
Note that we did not need to specify what specific
criterion we are talking about!
44. S. Hallé
Evaluation Trees
∧
∨ ¬
a ¬ c
b
∨
¬ b
c
φ = (a ∨ ¬b ∨ c) ∧ ¬(¬c ∨ b)
A valuation ν applied on a formula φ induces a
structure called an evaluation tree.
45. S. Hallé
ν = {a ↦ ⊤, b ↦ ⊥, c ↦ ⊤}
Each valuation "colors" the
structure differently
Each combination of ν and φ
produces a unique tree
Define as eν(φ) the function that produces
the evaluation tree of φ[ν].
Evaluation Trees
∧
∨ ¬
a ¬ c
b
∨
¬ b
c
φ = (a ∨ ¬b ∨ c) ∧ ¬(¬c ∨ b)
A valuation ν applied on a formula φ induces a
structure called an evaluation tree.
46. S. Hallé
Tree Transformation
Let T be the set of trees. We define T* as the set of
trees where nodes can be labeled with .
A tree transformation is a function 𝜏 : T* → T* that
turns an evaluation tree into another one.
Since trees are obtained by evaluating formulas, we
let 𝜏ν : Φ → T* be the function such that
?
^
𝜏ν(φ) = 𝜏(eν(φ))
^
i.e. for a valuation ν, 𝜏ν gets the evaluation tree for
φ and applies the transformation 𝜏 to it.
^
48. S. Hallé
Tree Transformation
We are free to define 𝜏 as we want.
∨
∧ ∧ ∧
a ¬ c
b
¬ ¬
a b
¬ b ¬
a c
∨
∧
Keep only the root and its immediate
children.
∧ ∧
49. S. Hallé
Tree Transformation
We are free to define 𝜏 as we want.
∨
∧ ∧ ∧
a ¬ c
b
¬ ¬
a b
¬ b ¬
a c
∨
? ∧ ?
Turn every child of the root, except
the second, into , and trim the
descendants of the second node.
?
50. S. Hallé
Tree Transformation
We are free to define 𝜏 as we want.
∨
∧ ∧ ∧
a ¬ c
b
¬ ¬
a b
¬ b ¬
a c
Put under the root a leaf labelled a
and a leaf labelled b, in that order.
∨
a b
51. S. Hallé
Tree Transformation
We are free to define 𝜏 as we want.
∨
∧ ∧ ∧
a ¬ c
b
¬ ¬
a b
¬ b ¬
a c
Under every connective node,
keep only subtrees until the
first that determines its color
∨
∧ ∧ ∧
a ¬ c
b
¬ ¬
a b
¬ b ¬
a c
52. S. Hallé
Tree Transformation
For a given formula φ, some tree transformations
map more than one valuation to the same tree.
∧
∨ ¬
a ¬ c
b
∨
¬ b
c
φ
𝜏 = keep only the root and its immediate children
= (a ∨ ¬b ∨ c) ∧ ¬(¬c ∨ b)
53. S. Hallé
Tree Transformation
For a given formula φ, some tree transformations
map more than one valuation to the same tree.
∧
∨ ¬
a ¬ c
b
∨
¬ b
c
φ
𝜏 = keep only the root and its immediate children
= (a ∨ ¬b ∨ c) ∧ ¬(¬c ∨ b)
∧
∨ ¬
{a ↦ ⊥, b ↦ ⊤, c ↦ ⊥}
^
𝜏
54. S. Hallé
Tree Transformation
For a given formula φ, some tree transformations
map more than one valuation to the same tree.
∧
∨ ¬
a ¬ c
b
∨
¬ b
c
φ
𝜏 = keep only the root and its immediate children
= (a ∨ ¬b ∨ c) ∧ ¬(¬c ∨ b)
∧
∨ ¬
{a ↦ ⊥, b ↦ ⊤, c ↦ ⊥}
^
𝜏
{a ↦ ⊥, b ↦ ⊥, c ↦ ⊤}
{a ↦ ⊤, b ↦ ⊥, c ↦ ⊤}
∧
∨ ¬
^
𝜏
55. S. Hallé
Tree Transformation
For a given formula φ, some tree transformations
map more than one valuation to the same tree.
∧
∨ ¬
a ¬ c
b
∨
¬ b
c
φ
𝜏 = keep only the root and its immediate children
= (a ∨ ¬b ∨ c) ∧ ¬(¬c ∨ b)
∧
∨ ¬
{a ↦ ⊥, b ↦ ⊤, c ↦ ⊥}
^
𝜏
{a ↦ ⊥, b ↦ ⊥, c ↦ ⊤}
{a ↦ ⊤, b ↦ ⊥, c ↦ ⊤}
∧
∨ ¬
^
𝜏
∧
∨ ¬
^
𝜏
{a ↦ ⊥, b ↦ ⊥, c ↦ ⊥}
{a ↦ ⊥, b ↦ ⊤, c ↦ ⊤}
{a ↦ ⊤, b ↦ ⊥, c ↦ ⊥}
{a ↦ ⊤, b ↦ ⊤, c ↦ ⊥}
{a ↦ ⊤, b ↦ ⊤, c ↦ ⊤}
56. S. Hallé
Tree Transformation
Key observation: for a given formula, a tree
transformation partitions the set of valuations
according to the tree each is mapped to...
∧
∨ ¬
∧
∨ ¬
{⊥⊥⊤,⊤⊥⊤} ∅
∧
∨ ¬
∧
∨ ¬
{⊥⊥⊥,⊥⊤⊤,
⊤⊥⊥,⊤⊤⊥,⊤⊤⊤}
{⊥⊤⊥}
⇒ it defines an algebraic coverage criterion!
57. S. Hallé
1.
Questions
Are the coverage criteria mentioned earlier
algebraic?
2. If so, can we find a tree transformation for each
that results in the same partitioning of the test
input space?
58. S. Hallé
1.
Questions
Are the coverage criteria mentioned earlier
algebraic?
2. If so, can we find a tree transformation for each
that results in the same partitioning of the test
input space?
YES (except MUMCUT*)
*An over-approximation is shown to be algebraic
59. S. Hallé
1.
Questions
Are the coverage criteria mentioned earlier
algebraic?
2. If so, can we find a tree transformation for each
that results in the same partitioning of the test
input space?
YES (except MUMCUT*)
*An over-approximation is shown to be algebraic
YES (all details in the paper)
60. S. Hallé
Define 𝜏ₙ as:
An Example
Turn the root and every child of the root, except
the n-th, into , and trim the descendants of the
n-th node.
?
61. S. Hallé
Define 𝜏ₙ as:
An Example
Turn the root and every child of the root, except
the n-th, into , and trim the descendants of the
n-th node.
?
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
Valuations are classified by 𝜏ₙ depending on the
value they give to the n-th clause.
n=2
? ∧ ? ? ∧ ?
? ?
?
∧ ? ?
∧ ?
? ?
n=1
? ∧
? ? ∧
?
? ?
n=3
62. S. Hallé
Define 𝜏ₙ as:
An Example
Turn the root and every child of the root, except
the n-th, into , and trim the descendants of the
n-th node.
?
(x₀ ∧ ¬x₁ ∧ x₂) ∨ (x₁ ∧ x₃) ∨ (¬x₀ ∧ x₂ ∧ x₃)
Valuations are classified by 𝜏ₙ depending on the
value they give to the n-th clause.
n=2
? ∧ ? ? ∧ ?
? ?
?
∧ ? ?
∧ ?
? ?
n=1
? ∧
? ? ∧
?
? ?
n=3
One valuation for each tree ⇒ clause coverage
63. S. Hallé
We define appropriate transformations for all
coverage criteria shown earlier... except MUMCUT.
A Word About MUMCUT
MUTP: each variable not in a UTP must be true/
false in at least one test
MNFP: each variable not in a NFP must be true/
false in at least one test
CUTP-NFP: all pairs of UTP and NFP for the same
clause that differ by a single variable flip must be
present
MUMCUT Coverage
1
2
3
64. S. Hallé
✓
algebraic
✓
algebraic
not algebraic? ?
We define appropriate transformations for all
coverage criteria shown earlier... except MUMCUT.
A Word About MUMCUT
MUTP: each variable not in a UTP must be true/
false in at least one test
MNFP: each variable not in a NFP must be true/
false in at least one test
CUTP-NFP: all pairs of UTP and NFP for the same
clause that differ by a single variable flip must be
present
MUMCUT Coverage
1
2
3
65. S. Hallé
✓
algebraic
✓
algebraic
not algebraic? ?
CUTP-NFP is defined on pairs of valuations, and not
on individual valuations
We define appropriate transformations for all
coverage criteria shown earlier... except MUMCUT.
A Word About MUMCUT
MUTP: each variable not in a UTP must be true/
false in at least one test
MNFP: each variable not in a NFP must be true/
false in at least one test
CUTP-NFP: all pairs of UTP and NFP for the same
clause that differ by a single variable flip must be
present
MUMCUT Coverage
1
2
3
66. S. Hallé
Generating Test Suites
A test suite for any algebraic criterion can be
obtained using the equivalence class partitioning
method. A possible way is using a hypergraph.
67. S. Hallé
Generating Test Suites
A test suite for any algebraic criterion can be
obtained using the equivalence class partitioning
method. A possible way is using a hypergraph.
⊥⊥⊤
⊥⊥⊥
⊥⊤⊤
⊤⊥⊥
⊤⊤⊥
⊤⊤⊤
⊥⊤⊥
⊤⊥⊤
1. Create one node for each valuation
68. S. Hallé
Generating Test Suites
A test suite for any algebraic criterion can be
obtained using the equivalence class partitioning
method. A possible way is using a hypergraph.
∧
∨
1
∧
∨
2
∧
∨
3 a 4 a 5
3
4
4
4
5
5
5
1
2
2
2
2
2. Add one hyperedge linking
the nodes mapped to
each tree induced by 𝜏
⊥⊥⊤
⊥⊥⊥
⊥⊤⊤
⊤⊥⊥
⊤⊤⊥
⊤⊤⊤
⊥⊤⊥
⊤⊥⊤
1. Create one node for each valuation
69. S. Hallé
Generating Test Suites
A test suite for any algebraic criterion can be
obtained using the equivalence class partitioning
method. A possible way is using a hypergraph.
∧
∨
1
∧
∨
2
∧
∨
3 a 4 a 5
3
4
4
4
5
5
5
1
2
2
2
2
2. Add one hyperedge linking
the nodes mapped to
each tree induced by 𝜏
⊥⊥⊤
⊥⊥⊥
⊥⊤⊤
⊤⊥⊥
⊤⊤⊥
⊤⊤⊤
⊥⊤⊥
⊤⊥⊤
1. Create one node for each valuation
3. Solve the hypergraph vertex
cover problem
70. S. Hallé
Generating Test Suites
A test suite for any algebraic criterion can be
obtained using the equivalence class partitioning
method. A possible way is using a hypergraph.
∧
∨
1
∧
∨
2
∧
∨
3 a 4 a 5
3
4
4
4
5
5
5
1
2
2
2
2
2. Add one hyperedge linking
the nodes mapped to
each tree induced by 𝜏
⊥⊥⊤
⊥⊥⊥
⊥⊤⊤
⊤⊥⊥
⊤⊤⊥
⊤⊤⊤
⊥⊤⊥
⊤⊥⊤
1. Create one node for each valuation
3. Solve the hypergraph vertex
cover problem
The selected nodes form the
test suite.
71. S. Hallé
Generating Test Suites
Cons
The hypergraph has 2|X|
nodes
The vertex cover problem has a high complexity
Pros
Guarantees full coverage for any algebraic
criterion
Optimal hypergraph cover ⇔ optimal test suite
Upper bound on complexity
Can solve for multiple criteria at once (instead of
separately + merging)
72. S. Hallé
Boolean Coverage Toolkit
These principles have been concretely implemented
in an open source Java library:
https://github.com/liflab/boolean-coverage-toolkit
Operator o = Or(And(a, Not(b), c),
And(Not(a), Not(b)), And(Not(a), b, Not(c)));
HologramNode n1 = o.evaluate(new Valuation(false,
false, true));
Truncation t = new FailFast(2);
HologramNode n2 = t.applyTo(n1);
Set<Truncation> set = new HashSet<>();
set.add(new KeepTop(2));
set.add(new KeepVariable(a));
HypergraphGenerator g = new HypergraphGenerator();
Hypergraph h = g.getGraph(o, set);
Iterator<?> it =
HittingSetRunner.runHittingSet(h).iterator();
while (it.hasNext()) {
Valuation v = g.getValuation(it.next());
}
73. S. Hallé
Create Boolean condition
Boolean Coverage Toolkit
These principles have been concretely implemented
in an open source Java library:
https://github.com/liflab/boolean-coverage-toolkit
Operator o = Or(And(a, Not(b), c),
And(Not(a), Not(b)), And(Not(a), b, Not(c)));
HologramNode n1 = o.evaluate(new Valuation(false,
false, true));
Truncation t = new FailFast(2);
HologramNode n2 = t.applyTo(n1);
Set<Truncation> set = new HashSet<>();
set.add(new KeepTop(2));
set.add(new KeepVariable(a));
HypergraphGenerator g = new HypergraphGenerator();
Hypergraph h = g.getGraph(o, set);
Iterator<?> it =
HittingSetRunner.runHittingSet(h).iterator();
while (it.hasNext()) {
Valuation v = g.getValuation(it.next());
}
74. S. Hallé
Create Boolean condition
Obtain an expression tree
and apply a transformation
Boolean Coverage Toolkit
These principles have been concretely implemented
in an open source Java library:
https://github.com/liflab/boolean-coverage-toolkit
Operator o = Or(And(a, Not(b), c),
And(Not(a), Not(b)), And(Not(a), b, Not(c)));
HologramNode n1 = o.evaluate(new Valuation(false,
false, true));
Truncation t = new FailFast(2);
HologramNode n2 = t.applyTo(n1);
Set<Truncation> set = new HashSet<>();
set.add(new KeepTop(2));
set.add(new KeepVariable(a));
HypergraphGenerator g = new HypergraphGenerator();
Hypergraph h = g.getGraph(o, set);
Iterator<?> it =
HittingSetRunner.runHittingSet(h).iterator();
while (it.hasNext()) {
Valuation v = g.getValuation(it.next());
}
75. S. Hallé
Create Boolean condition
Obtain an expression tree
and apply a transformation
Create tree transformations
Boolean Coverage Toolkit
These principles have been concretely implemented
in an open source Java library:
https://github.com/liflab/boolean-coverage-toolkit
Operator o = Or(And(a, Not(b), c),
And(Not(a), Not(b)), And(Not(a), b, Not(c)));
HologramNode n1 = o.evaluate(new Valuation(false,
false, true));
Truncation t = new FailFast(2);
HologramNode n2 = t.applyTo(n1);
Set<Truncation> set = new HashSet<>();
set.add(new KeepTop(2));
set.add(new KeepVariable(a));
HypergraphGenerator g = new HypergraphGenerator();
Hypergraph h = g.getGraph(o, set);
Iterator<?> it =
HittingSetRunner.runHittingSet(h).iterator();
while (it.hasNext()) {
Valuation v = g.getValuation(it.next());
}
76. S. Hallé
Create Boolean condition
Obtain an expression tree
and apply a transformation
Create tree transformations
Generate and solve
hypergraph
Boolean Coverage Toolkit
These principles have been concretely implemented
in an open source Java library:
https://github.com/liflab/boolean-coverage-toolkit
Operator o = Or(And(a, Not(b), c),
And(Not(a), Not(b)), And(Not(a), b, Not(c)));
HologramNode n1 = o.evaluate(new Valuation(false,
false, true));
Truncation t = new FailFast(2);
HologramNode n2 = t.applyTo(n1);
Set<Truncation> set = new HashSet<>();
set.add(new KeepTop(2));
set.add(new KeepVariable(a));
HypergraphGenerator g = new HypergraphGenerator();
Hypergraph h = g.getGraph(o, set);
Iterator<?> it =
HittingSetRunner.runHittingSet(h).iterator();
while (it.hasNext()) {
Valuation v = g.getValuation(it.next());
}
77. S. Hallé
Create Boolean condition
Obtain an expression tree
and apply a transformation
Create tree transformations
Generate and solve
hypergraph
Iterate over valuations of
the solution
Boolean Coverage Toolkit
These principles have been concretely implemented
in an open source Java library:
https://github.com/liflab/boolean-coverage-toolkit
Operator o = Or(And(a, Not(b), c),
And(Not(a), Not(b)), And(Not(a), b, Not(c)));
HologramNode n1 = o.evaluate(new Valuation(false,
false, true));
Truncation t = new FailFast(2);
HologramNode n2 = t.applyTo(n1);
Set<Truncation> set = new HashSet<>();
set.add(new KeepTop(2));
set.add(new KeepVariable(a));
HypergraphGenerator g = new HypergraphGenerator();
Hypergraph h = g.getGraph(o, set);
Iterator<?> it =
HittingSetRunner.runHittingSet(h).iterator();
while (it.hasNext()) {
Valuation v = g.getValuation(it.next());
}
78. S. Hallé
Experiments
We can compare the hypergraph-based approach
with existing tools and techniques:
Random selection
ACTS (combinatorial coverage)
mcdc (MC/DC coverage)
and results published in past papers
(implementation unavailable).
79. S. Hallé
Experiments
Test suite size
0
2
4
6
8
10
12
14
2 4 6 8 10 12 14
Hypergraph
MCDC
SAT
0
200
400
600
800
1000
1200
1400
1600
1800
0 200 400 600 800 1000 1200 1400 1600 1800
Hypergraph
Chen
G-CUN
MC/DC MUMCUT
The hypergraph approach generates test suites of
comparable size for MC/DC, but larger for
MUMCUT (over-approximated)
80. S. Hallé
Experiments
Solving for two criteria at once
Criteria Size ratio Time ratio
MC/DC + Clause
MC/DC + Predicate
MC/DC + MUMCUT
MC/DC + 2-way
MC/DC + 3-way
Clause + 2-way
Clause + 3-way
0.87
0.85
0.99
0.76
0.83
0.83
0.87
1.27
1.29
1.38
1.20
1.11
1.24
1.11
The hypergraph approach takes more time, but
produces smaller test suites than when two
independent solutions are merged
81. S. Hallé
Conclusion
Uniform theoretical framework to define coverage
criteria as equivalence classes induced by a tree
transformation
Generic algorithm to produce test suites according
to any algebraic coverage criterion
Opens the way to experiments with new coverage
criteria and benchmarking of existing ones.
Future work: extension to other logics, such as
temporal/description logics