The document surveys 159 papers on test suite minimization, regression test selection, and test case prioritization techniques. It finds that the majority of studies used small codebases with under 10,000 lines of code and fewer than 1,000 test cases. Graph-walking is identified as the most predominant regression test selection technique. Prioritization approaches focus on coverage-based and history-based methods. Future work opportunities include integrating regression testing with test data generation, considering other domains beyond white-box testing, and providing more tool support.
As testers, we know that we can define many more test cases than we will ever have time to design, execute, and report. The key problem in testing is choosing a small, “smart” subset from the almost infinite number of tests available that will find a large percentage of the defects. Join Lee Copeland to discover how to design test cases using formal black-box techniques, including equivalence class testing, boundary value testing, decision tables, and state-transition diagrams. Explore examples of each of these techniques in action. Don’t just pick test cases at random. Rather, learn to selectively choose a set of test cases that maximizes your effectiveness and efficiency to find more defects in less time. Then, learn how to use the test results to evaluate the quality of both your products and your testing. Discover the test design techniques that will make your testing more productive.
Whitepaper Test Case Design and Testing Techniques- Factors to ConsiderRapidValue
Software testing is an essential and important technique for assessing the quality of a particular software product/service. In software testing, test cases and scenarios play an inevitable and a pivotal role. A good strategic design and technique help to improve the quality of the software testing process.
This whitepaper provides information about test case design activities, test analysis, quality risks, testing techniques, phases of test development. The paper also, explains the factors that need to be considered while choosing the right testing techniques and provides a checklist of test cases based on our rich experience of testing mobile apps.
This is the presentation describing different techniques used to write test cases for software testing. You can have overview with detailed example for test case techniques. After reading this, You'll able to assume which technique can be more useful to you software testing.
In this quality assurance training, you will learn Test Case Design and Technique. Topics covered in this session are:
• Test Case Design and Techniques
• Black-box: Three major approaches
• Steps for drawing cause-Effect Diagram:
• Behavior Testing
• Random Testing
• White Box Techniques
• Path Testing
• Statement Coverage
• Data Flow Testing
For more information, visit this link: https://www.mindsmapped.com/courses/quality-assurance/software-testing-training-beginners-and-intermediate-level/
In this session you will learn:
Test Case Design and Techniques
Black-box: Three major approaches
Steps for drawing cause-Effect Diagram:
Behavior Testing
Random Testing
White Box Techniques
Path Testing
Statement Coverage
Data Flow Testing
For more information: https://www.mindsmapped.com/courses/quality-assurance/qa-software-testing-training-for-beginners/
As testers, we know that we can define many more test cases than we will ever have time to design, execute, and report. The key problem in testing is choosing a small, “smart” subset from the almost infinite number of tests available that will find a large percentage of the defects. Join Lee Copeland to discover how to design test cases using formal black-box techniques, including equivalence class testing, boundary value testing, decision tables, and state-transition diagrams. Explore examples of each of these techniques in action. Don’t just pick test cases at random. Rather, learn to selectively choose a set of test cases that maximizes your effectiveness and efficiency to find more defects in less time. Then, learn how to use the test results to evaluate the quality of both your products and your testing. Discover the test design techniques that will make your testing more productive.
Whitepaper Test Case Design and Testing Techniques- Factors to ConsiderRapidValue
Software testing is an essential and important technique for assessing the quality of a particular software product/service. In software testing, test cases and scenarios play an inevitable and a pivotal role. A good strategic design and technique help to improve the quality of the software testing process.
This whitepaper provides information about test case design activities, test analysis, quality risks, testing techniques, phases of test development. The paper also, explains the factors that need to be considered while choosing the right testing techniques and provides a checklist of test cases based on our rich experience of testing mobile apps.
This is the presentation describing different techniques used to write test cases for software testing. You can have overview with detailed example for test case techniques. After reading this, You'll able to assume which technique can be more useful to you software testing.
In this quality assurance training, you will learn Test Case Design and Technique. Topics covered in this session are:
• Test Case Design and Techniques
• Black-box: Three major approaches
• Steps for drawing cause-Effect Diagram:
• Behavior Testing
• Random Testing
• White Box Techniques
• Path Testing
• Statement Coverage
• Data Flow Testing
For more information, visit this link: https://www.mindsmapped.com/courses/quality-assurance/software-testing-training-beginners-and-intermediate-level/
In this session you will learn:
Test Case Design and Techniques
Black-box: Three major approaches
Steps for drawing cause-Effect Diagram:
Behavior Testing
Random Testing
White Box Techniques
Path Testing
Statement Coverage
Data Flow Testing
For more information: https://www.mindsmapped.com/courses/quality-assurance/qa-software-testing-training-for-beginners/
In this quality assurance training session, you will learn Test case design. Topics covered in this course are:
• Test Case Design and Techniques
• Black-box: Three major approaches
• Steps for drawing cause-Effect Diagram:
• Behavior Testing
• Random Testing
• White Box Techniques
• Path Testing
• Statement Coverage
• Data Flow Testing
To know more, visit this link: https://www.mindsmapped.com/courses/quality-assurance/software-testing-quality-assurance-qa-training-with-hands-on-exercises/
This slide deck is for all the QA members who want to understand the methodology of test case design. These slides are not theoretical gyan but designed based on experience.
A Test Analysis Method for Black Box Testing Using AUT and Fault Knowledge.Tsuyoshi Yumoto
With a rapid increase in size and complexity of software today, the scope of software testing is also expanding. The efficiency of software testing needs to be improved in order to ensure the appropriate delivery deadline and cost of software development. For improving efficiency of software testing, the test needs to be designed in a way that the number of test cases is sufficient and appropriate in quantity. Test analysis is the activity to refine Application Under Test (AUT) into proper size that test design techniques can be applied to. It is for designing the test properly. However, the classification for proper size depends on individual’s own judgments. This paper proposes a test analysis method for the black box testing using a test category that is the classification based on fault and AUT knowledge.
In this quality assurance training session, you will learn Test case design. Topics covered in this course are:
• Test Case Design and Techniques
• Black-box: Three major approaches
• Steps for drawing cause-Effect Diagram:
• Behavior Testing
• Random Testing
• White Box Techniques
• Path Testing
• Statement Coverage
• Data Flow Testing
To know more, visit this link: https://www.mindsmapped.com/courses/quality-assurance/software-testing-quality-assurance-qa-training-with-hands-on-exercises/
This slide deck is for all the QA members who want to understand the methodology of test case design. These slides are not theoretical gyan but designed based on experience.
A Test Analysis Method for Black Box Testing Using AUT and Fault Knowledge.Tsuyoshi Yumoto
With a rapid increase in size and complexity of software today, the scope of software testing is also expanding. The efficiency of software testing needs to be improved in order to ensure the appropriate delivery deadline and cost of software development. For improving efficiency of software testing, the test needs to be designed in a way that the number of test cases is sufficient and appropriate in quantity. Test analysis is the activity to refine Application Under Test (AUT) into proper size that test design techniques can be applied to. It is for designing the test properly. However, the classification for proper size depends on individual’s own judgments. This paper proposes a test analysis method for the black box testing using a test category that is the classification based on fault and AUT knowledge.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Real time implementation of the software system requires being more versatile. In the maintenance phase, the modified system under regression testing must assure that the existing system remains defect free. Test case prioritization technique of regression testing includes code as well as model based methods of prioritizing the test cases. System model based test case prioritization can detect the severe faults early as compare to the code based test case prioritization. Model based prioritization techniques based on requirements in a cost effective manner has not been taken for study so far. Model based testing used to test the functionality of the software system based on requirement. An effective model based approach is defined for prioritizing test cases and to generate the effective test sequence. The test cases are rescheduled based on requirement analysis and user view analysis. With the use of weighted approach the overall cost is estimated to test the functionality of the model elements. Here, the genetic approach has been applied to generate efficient test path. The regression cost in terms of effort has been reduced under model based prioritization approach.
Find out more about quality assurance training and specifically about Test Case Design and Techniques. Topics covered in this session are:
Test Case Design and Techniques
Black-box: Three major approaches
Steps for drawing cause-Effect Diagram:
Behavior Testing
Random Testing
White Box Techniques
Path Testing
Statement Coverage
Data Flow Testing
For more information, visit: https://www.mindsmapped.com/courses/quality-assurance/quality-assurance-training-learn-manual-and-automation-testing/
In this session you will learn:
Test Case Design and Techniques
Black-box: Three major approaches
Steps for drawing cause-Effect Diagram:
Behavior Testing
Random Testing
White Box Techniques
Path Testing
Statement Coverage
Data Flow Testing
For more information, click here:
https://www.mindsmapped.com/courses/quality-assurance/software-testing-tutorial/
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test.Software Testing Tutorial for beginners - Learn what is software testing and different testing types and methods and associated estimation techniques.
Software testing is a vital part in developing software.Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test.
Configuration Navigation Analysis Model for Regression Test Case Prioritizationijsrd.com
Regression testing has been receiving increasing attention nowadays. Numerous regression testing strategies have been proposed. Most of them take into account various metrics like cost as well as the ability to find faults quickly thereby saving overall testing time. In this paper, a new model called the Configuration Navigation Analysis Model is proposed which tries to consider all stakeholders and various testing aspects while prioritizing regression test cases.
Similar to The Current State of the Art of Regression Testing (20)
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
3. Introduction
3
Survey of 159 papers on test suite minimization, regression test selection, and test
case prioritization.
Intention is not to undertake a systematic review, but rather to provide a broad
state-of-the-art view on these related fields.
Note: I’m going to go back and forth between spelling minimization and
prioritisation with s’ and z’s
4. Introduction
4
Regression Testing: Provide confidence that the newly introduced changes do not
obstruct the behaviors of the existing, unchanged part of the software.
Difficulties Include:
• Black-box development with 3rd party applications
• Agile development
Note: Most straightforward approach is “retest-all”, but may not be viable in all
scenarios
5. Introduction
5
A number of different approaches have been studied to aid the regression testing
process. Three major branches include:
Test Suite Minimization: Process that seeks to identify and then eliminate the
obsolete or redundant test cases from the test suite.
Test Case Selection: Select a subset of test cases that will be used to test the
changed parts of the software.
Test Case Prioritization: Identify the “ideal” ordering of test cases that maximizes the
desirable properties, such as early fault detection.
6. Overview
6
1. Motivation
2. Background
3. Test Case Selection
4. Test Suite Minimization
5. Test Case Prioritization
6. Summary and Conclusion
7. Suggestions
8. Lessons Learned
7. Motivation
7
Why is this the right set of topics for a survey?
• Each topic is related by a common thread of optimization of already existing test
cases.
• All differ from areas that focus on test case generation.
• Intimate relationship between the topics (e.g. minimization could be performed
by prioritizing a set of cases and choosing the first N).
Is there already a recent survey in this area?
• Most similar paper was a survey on Regression Test Selection techniques in 1996.
• No previous survey paper that consider Prioritization, Selection, and Minimization
collectively.
8. Background
8
Redefine regression testing and further elaborate on the distinction of each
optimization technique defined in the introduction.
Classification of Test Cases
Reusable – Only execute parts of the program that remain unchanged. Not valuable
for new changes, but assist with future regression checks.
Retestable – Test cases that are still valid after a set of changes and can validate if
any regression has occurred.
Obsolete – Could be rendered obsolete due to: input/output is no longer relevant
and/or no longer test the desired specification (i.e. requirements changed).
10. Test Case Selection
10
Compare Test Case Selection vs. Test Case Minimisation
• Very similar to one another; both revolve around choosing a subset of test cases
from the test suite.
• Test suite minimization often based on metrics (e.g. code coverage) of an entire
application.
• Test case selection based on finding relevant tests to be run.
11. Test Case Selection
11
Integer Programming Approach
Optimization program in which all of the variables are restricted to integers.
• Heavily relies on two matrices that describe the relation between program
segments and test cases. (program segment can be defined as a single-entry,
single exit block of code)
• Matrix function represented as am1x1 + am2x2 >= bm (aij equal to 1 if the segment-
test case relation exists, 0 otherwise)
• Results in a decision vector (subset of selected test cases) < x1, … , xn > where xi is
equal to 1 if the ith test case is included.
Problematic with control flow changes. Entire test suite has to be run again.
12. Test Case Selection
12
Data-flow Analysis Approach
Technique for gathering information about the possible set of values calculated at
various points in a computer program. (i.e. How does input flow through the
application)
Seek to identity new, modified or deleted definition-use pairs in the new version of
the program; then select those cases that exercise these pairs. (Does the new code
impact the test data being used?)
Problematic with modifications that are unrelated to data-flow change. These test
scenarios will not be selected for testing.
13. Test Case Selection
13
Symbolic Execution Approach
A means of analyzing a program to determine what inputs cause each part of a
program to execute.
function(f) {
if f == 2 then return fail();
else return success();
}
1. Find all input partitions.
2. Produce test cases so that each input partition is executed at least once.
3. Given information on where the code has been modified (e.g. a diff), return
modified code segments and the test cases that execute these segments.
Drawback is the algorithmic complexity of symbolic execution as well as how
expensive it can be to execute.
14. Test Case Selection
14
Graph-Walk Approach
1. Parse P and P’ into graph data structures.
2. Traverse each graph and compare the nodes
3. If a node in P is not the same as the node in P’, select all the test cases that
execute the code within that node.
Problematic since there is no data dependence, the approach could include test
cases that provide little to no value.
15. Test Case Selection
15
Textual Difference Approach
A very similar approach to the Graph-Walk approach
• Uses the diff tool provided by Unix.
• Code sanitized to remove any characters that would not introduce change (e.g.
whitespace)
16. Test Case Selection
16
Path Analysis
• Construct exemplar paths from P and P’
• Paths in P’ are categorized as new, modified, cancelled, or unmodified.
• Since all test cases and the paths they execute in P are known, the test cases that
traverse the modified paths in P’ are selected.
The authors had a poor definition of “modified”. Test cases that executed new or
cancelled code was not chosen. However, these paths could lead to regression.
17. Test Case Selection
17
Modification-based Technique
Yet another similar approach to Graph-Walking
• Introduced a testing framework called TestTube.
• Partitions P and P’ into program entities (nodes), then monitors the test cases to
find out the code that each test case executes.
• Those entities that were different are selected.
Since the entities include not only functions, but variables, any test case that
executes modified functions will be selected.
This differs from the data-independent Graph-Walking approach described
previously. Modification-based technique encompasses data as well.
18. Test Case Selection
18
Firewall Approach
Draw a firewall around the modules of the system that need to be retested.
A given module M can be represented as:
• No Change NoCh(M)
• Only Code CodeCh(M)
• Spec Change SpecCh(M)
Considering integrations between module A and module B
• Ignore NoCh(A) ^ NoCh(B)
• If A and B are modified in either code or in spec (CodeCh or SpecCh respectively)
the tests should be rerun.
19. Test Case Selection
19
Design-based Approach
• Black-box, design level regression test selection that used UML-based designs.
• Requires traceability between design and test cases
• Leveraged obsolete, retestable, and reusable as highlighted in the background
Possible to select test cases that provide no value as a UML diagram does not
encapsulate all code interactions. (e.g. change a method, but diagram doesn’t
dictate it is ever called, just exists)
21. Test Suite Minimization
21
Heuristics
Essential test cases
• If a test requirement can be satisfied by one and only one test case
Redundant test cases
• If a test case satisfies only a subset of the test requirements satisfied by another
test case.
GE Heuristic
Select the test case that satisfies the maximum number of unsatisfied test
requirements.
GRE Heuristic
Remove all redundant test cases in the test suite (which may make some test cases
essential). Then run the GE heuristic.
22. Test Suite Minimization
22
Heuristics
Empirical evidence suggests no single approach is better
• Concerned with heuristics more so than preciseness.
Vast majority of presented literature focused on the minimal hitting set problem.
Most minimization techniques are based on coverage criteria, there were
exceptions.
• Minimizing the test itself (start with a failed test).
• Black-box approach to program input/output (research in state machines).
Different inputs may not flow through different branches.
24. Test Case Prioritisation
24
Coverage-based Prioritisation (code)
Structural coverage often used as prioritization criterion. The more code a test
executes, the higher chance of finding a fault.
Approaches include:
• Branch-total (number of branches covered by test cases)
• Branch-additional (number of additional branches a test case would execute)
• Statement-total
• Statement-additional
25. Test Case Prioritisation
25
Interaction Testing (black box)
Necessary when the system under test involves multiple combinations of different
components. (consider the application environment, Operating System)
Research focused on findings those interactions that impact a higher user base. (e.g.
prioritize Windows testing over Linux).
Additional research done in GUI-based programs.
• Take a sequence of inputs and find the case that executes the most code.
• Consider user interaction data for prioritisation (heat map).
26. Test Case Prioritisation
26
Distribution-based Approach
Profile test cases based on a dissimilarity metric, a real number representing the
degree of dissimilarity between two inputs.
Cluster test cases according to their similarities which can reveal:
• Similar profiles may indicate a group of redundant test cases
• Isolated clusters may contain test cases in unusual conditions (fault-proneness)
27. Test Case Prioritisation
27
History-based Approach
• Based on association clusters of software artifacts.
• If two files are often modified together, they will be clustered together.
• Each file is also associated with test cases that impact or execute it.
Non-source file (e.g. media, documentation) defects can be as severe as source
code defects.
28. Test Case Prioritisation
28
Requirement-based Approach
• Test cases are mapped to software requirements
• Prioritisation mapped by customer-assigned priority and/or implementation
complexity.
Makes the prioritization very subjective (customers will have conflicting priorities)
29. Test Case Prioritisation
29
Model-based Approach
• Test cases classified into a high priority set TSH and a low priority set TSL
• Initial prioritization was randomly assigned
• Test case is assigned high priority if it is relevant to the modification made in the
model.
Similar approach to the UML based approach when selecting test cases.
30. Test Case Prioritisation
30
Session-based Approach
• Recorded user sessions from the previous version of the (web) application.
• Thought to be ideal for testing web applications as it reflects actual use.
• Metrics such as number of HTTP requests, frequency of visits.
• Better than random selection, but no single prioritization criterion is always the
best.
31. Test Case Prioritisation
31
Cost-Aware Approach
Typical prioritization approaches assume equal fault level and cost.
Areas of focus similarly categorized:
• Time based (tests that take a long time, need a way to fit X tests in Y units of time)
• Fault level (prioritize most catastrophic tests first, not necessarily any fault)
32. Meta-Empirical Studies
32
• Empirical evaluation considered post-hoc, knowledge of faults is known. Without
previous knowledge of faults, not possible to perform a controlled experiment.
• Studies done in regards to seeded vs. real faults (concluded seeded faults can be
safely used in place of hand-seeded faults).
• Frequency of regression testing has a significant impact of the cost-effectiveness
of RTS techniques. The longer the window between tests, the more tests are
selected, lowering the value-add.
• Research efforts attempting to apply an RTS technique based on the type of
program (no silver bullet; Session-based for web applications, Model-based that
had its source generated from UML)
34. Analysis of Current Global Trends
34
Consider the graph as not a representation of the number of publications, but
trends of research popularity (single publication can count towards two categories)
35. Analysis of Current Global Trends
35
• 60% of studies included less than 10,000 lines of code.
• 70% of studies included less than 1,000 test cases.
37. State of the Art
37
• Among the class of RTS techniques, the graph walk approach is the most
predominant. Intuitive and incredibly generic.
• Two ideas played essential roles: test case classification and safe regression test
selection (if a modification occurred, it will be selected).
• Greedy algorithms are prominent in the selected literature (as much as possible,
as soon as possible).
38. Trends
38
• Emphasis on models (early adoption was very code focused)
• Increased domains (e.g. web applications)
• Cost-awareness – more and more literature are starting to consider test time and
amount of fault.
39. Issues / Limitations
39
Limited subjects (60% from the SIR repository). Hard to prove the proposed research
techniques can be generalized.
Solutions
• Design a method that will allow a realistic simulation of real software faults.
• Engage with open source and Industry
Technology Transfer observations of the literature suggests the community may have
reached maturity and its time to transfer.
Out of 159 papers, only 31 of them have an author involved in industry.
Out of 159 papers, only 12 consider industrial software.
40. Future Direction
40
Orchestrating regression testing techniques with test data generation
• Self healing tests
Multi-Objective Regression Testing
• Group tests requiring a given environment together, reduce cost.
Consideration of Other Domains
• Most were white-box
Tool Support
• No readily available tools means practical adoption will remain limited
41. Conclusion
41
Trends in literature show..
• The community is focused on prioritization, especially Graph-Walking.
• The community it moving towards assessment of complex trade offs (cost and
value)
• More are becoming interested in the research area. Number of publications
continue to rise.
42. Suggestions
42
• More literature on Minimization and/or clearer content.
• Briefly describe the references used. Felt a lot of the references forced you to
read the paper.
For a paper meant to give an overview of the state of the art.. it did just that
43. Lessons Learned
43
• How big of an area of research regression testing is
• Symbolic execution
• Consider binaries to be a source of fault