This document provides a retrospective on 50 years of research in software testing techniques. It examines how testing techniques have matured from ad hoc methods to a more systematic discipline. The document outlines the evolution of testing concepts over time and how this has guided research. It then summarizes several major theoretical and methodological contributions that have advanced the field, such as research establishing test data adequacy criteria and coverage-based models. The document uses frameworks to analyze how testing techniques have progressed from early formulation to broader adoption according to paradigms of technology maturation and software engineering research.
This document provides an overview of software testing techniques and their maturation over time. It examines the major research results that have contributed to the growth of testing as an area. The document defines testing goals and categories, including functional vs structural testing and static vs dynamic analysis. It also discusses testing at different stages of the software lifecycle from unit to system level. The technology maturation model and research paradigms framework are used to analyze how testing techniques have evolved from initial ideas to broader solutions and changes in research questions and strategies over time.
This document provides an overview of software testing. It discusses the background of software testing including definitions, terminology, and history. It presents a framework for software testing that includes the inputs, outputs, and processes. It also discusses test case generation, test execution, evaluation, and some challenges around automation. Finally, it touches on taxonomy and benchmarks for evaluating automated testing techniques.
The document discusses Object Oriented Analysis and Design (OOAD) and the Rational Unified Process (RUP). It explains that OOAD involves analyzing a problem domain to identify objects and then designing how software objects will collaborate to meet requirements. RUP is an iterative software development process that uses use cases, UML diagrams, and other artifacts. It has phases like inception, elaboration, construction, and transition where requirements are gathered, designs are created and refined, and software is implemented through iterations.
This document provides an overview of software testing techniques and best practices covered in a course on the topic. It discusses the purpose of software testing, including verification, error detection, and validation. It then surveys common software testing methodologies like white box testing, black box testing, and unit testing. The document also includes two case studies, one on test automation and one on testing an intranet system. Finally, it provides a template for a software test plan and discusses several papers on software testing methods and techniques.
Prioritizing Test Cases for Regression Testing A Model Based ApproachIJTET Journal
The document summarizes a model-based approach to prioritizing regression test cases. It involves generating test cases from UML models, prioritizing them based on the number of states and transitions covered, and clustering them by severity using a dendrogram approach. This helps decrease the time and cost of regression testing by focusing testing efforts on the most important and affected areas first. The proposed approach constructs models from requirements, identifies states, prioritizes flows, generates test cases, and prioritizes the test cases based on severity to improve regression testing efficiency.
The document discusses object-oriented testing strategies and techniques. It covers unit testing of individual classes, integration testing of groups of classes, validation testing against requirements, and system testing. Interclass testing focuses on testing collaborations between classes during integration. Test cases should uniquely identify the class under test, state the test purpose and steps, and list expected states, messages, exceptions, and external dependencies.
Information hiding based on optimization technique for Encrypted ImagesIRJET Journal
This document summarizes a research paper on reversible data hiding in encrypted images using an optimization technique. The paper proposes an algorithm that first identifies the area of interest in an encrypted image and then uses a Bat Algorithm to find noisy pixel coordinates for embedding text data. Any remaining data is embedded in the image border areas. The research aims to securely protect embedded data against attacks while maintaining efficiency. It discusses related work on separable reversible data hiding techniques and the need for reversible data hiding in encrypted images to maintain confidentiality while allowing lossless image recovery.
One of the core quality assurance feature which combines fault prevention and fault detection, is often known as testability approach also. There are many assessment techniques and quantification method evolved for software testability prediction which actually identifies testability weakness or factors to further help reduce test effort. This paper examines all those measurement techniques that are being proposed for software testability assessment at various phases of object oriented software development life cycle. The aim is to find the best metrics suit for software quality improvisation through software testability support. The ultimate objective is to establish the ground work for finding ways reduce the testing effort by improvising software testability and its assessment using well planned guidelines for object-oriented software development with the help of suitable metrics.
This document provides an overview of software testing techniques and their maturation over time. It examines the major research results that have contributed to the growth of testing as an area. The document defines testing goals and categories, including functional vs structural testing and static vs dynamic analysis. It also discusses testing at different stages of the software lifecycle from unit to system level. The technology maturation model and research paradigms framework are used to analyze how testing techniques have evolved from initial ideas to broader solutions and changes in research questions and strategies over time.
This document provides an overview of software testing. It discusses the background of software testing including definitions, terminology, and history. It presents a framework for software testing that includes the inputs, outputs, and processes. It also discusses test case generation, test execution, evaluation, and some challenges around automation. Finally, it touches on taxonomy and benchmarks for evaluating automated testing techniques.
The document discusses Object Oriented Analysis and Design (OOAD) and the Rational Unified Process (RUP). It explains that OOAD involves analyzing a problem domain to identify objects and then designing how software objects will collaborate to meet requirements. RUP is an iterative software development process that uses use cases, UML diagrams, and other artifacts. It has phases like inception, elaboration, construction, and transition where requirements are gathered, designs are created and refined, and software is implemented through iterations.
This document provides an overview of software testing techniques and best practices covered in a course on the topic. It discusses the purpose of software testing, including verification, error detection, and validation. It then surveys common software testing methodologies like white box testing, black box testing, and unit testing. The document also includes two case studies, one on test automation and one on testing an intranet system. Finally, it provides a template for a software test plan and discusses several papers on software testing methods and techniques.
Prioritizing Test Cases for Regression Testing A Model Based ApproachIJTET Journal
The document summarizes a model-based approach to prioritizing regression test cases. It involves generating test cases from UML models, prioritizing them based on the number of states and transitions covered, and clustering them by severity using a dendrogram approach. This helps decrease the time and cost of regression testing by focusing testing efforts on the most important and affected areas first. The proposed approach constructs models from requirements, identifies states, prioritizes flows, generates test cases, and prioritizes the test cases based on severity to improve regression testing efficiency.
The document discusses object-oriented testing strategies and techniques. It covers unit testing of individual classes, integration testing of groups of classes, validation testing against requirements, and system testing. Interclass testing focuses on testing collaborations between classes during integration. Test cases should uniquely identify the class under test, state the test purpose and steps, and list expected states, messages, exceptions, and external dependencies.
Information hiding based on optimization technique for Encrypted ImagesIRJET Journal
This document summarizes a research paper on reversible data hiding in encrypted images using an optimization technique. The paper proposes an algorithm that first identifies the area of interest in an encrypted image and then uses a Bat Algorithm to find noisy pixel coordinates for embedding text data. Any remaining data is embedded in the image border areas. The research aims to securely protect embedded data against attacks while maintaining efficiency. It discusses related work on separable reversible data hiding techniques and the need for reversible data hiding in encrypted images to maintain confidentiality while allowing lossless image recovery.
One of the core quality assurance feature which combines fault prevention and fault detection, is often known as testability approach also. There are many assessment techniques and quantification method evolved for software testability prediction which actually identifies testability weakness or factors to further help reduce test effort. This paper examines all those measurement techniques that are being proposed for software testability assessment at various phases of object oriented software development life cycle. The aim is to find the best metrics suit for software quality improvisation through software testability support. The ultimate objective is to establish the ground work for finding ways reduce the testing effort by improvising software testability and its assessment using well planned guidelines for object-oriented software development with the help of suitable metrics.
Real time implementation of the software system requires being more versatile. In the maintenance phase, the modified system under regression testing must assure that the existing system remains defect free. Test case prioritization technique of regression testing includes code as well as model based methods of prioritizing the test cases. System model based test case prioritization can detect the severe faults early as compare to the code based test case prioritization. Model based prioritization techniques based on requirements in a cost effective manner has not been taken for study so far. Model based testing used to test the functionality of the software system based on requirement. An effective model based approach is defined for prioritizing test cases and to generate the effective test sequence. The test cases are rescheduled based on requirement analysis and user view analysis. With the use of weighted approach the overall cost is estimated to test the functionality of the model elements. Here, the genetic approach has been applied to generate efficient test path. The regression cost in terms of effort has been reduced under model based prioritization approach.
The document defines various types of software testing techniques and terms, including:
- Audit testing which assesses compliance with specifications, standards, or agreements.
- Acceptance testing conducted by customers to determine if a system meets acceptance criteria.
- Alpha and beta testing which involve customer testing in controlled or live environments.
- Boundary value analysis which tests at boundaries and limits of input/output domains.
- Branch coverage which requires each code branch be tested at least once.
This study empirically evaluated the use of mutation testing to improve test quality for safety-critical software. Mutation testing was applied to two airborne software systems developed using rigorous development processes and guidelines. The study identified an effective subset of mutant types, categorized the root causes of test failures, and examined relationships between program characteristics and mutation survival. The results showed how mutation testing could be useful for finding issues missed by traditional structural coverage analysis and manual peer review. Industry feedback also provided perspectives on integrating mutation testing into typical verification lifecycles for airborne software.
Software testing is an activity which is aimed for evaluating quality of a program and also for improving it, by identifying defects and problems. Software testing strives for achieving its goal (both implicit and explicit) but it has certain limitations, still testing can be done more effectively if certain established principles are to be followed. In spite of having limitations, software testing continues to dominate other verification techniques like static analysis, model checking and proofs. So it is indispensable to understand the goals, principles and limitations of software testing so that the effectiveness of software testing could be maximized.
This document discusses different levels of software testing including unit, integration, system, and acceptance testing. It focuses on unit testing and describes how to plan and prepare for unit testing. Key points include:
- Unit testing involves testing individual software components like functions, methods, and classes. Good practice is to plan unit tests to target specific defect types.
- Preparing for unit tests involves planning the approach, designing test cases and procedures, and developing any auxiliary code needed to run the tests. Test cases should cover normal and abnormal behavior.
- When testing classes as units, testers focus on interactions between methods in the class. Choosing classes or individual methods as the test unit involves tradeoffs around complexity and cost
The document discusses software testing, which involves executing a program or application to evaluate properties like meeting requirements, responding correctly to inputs, performing functions in an acceptable time, usability, installability, and achieving stakeholder desires. Software testing aims to find bugs and provide information about quality and failure risk. It discusses roles, methods, and economics of software testing.
Test Case Optimization and Redundancy Reduction Using GA and Neural Networks IJECEIAES
More than 50% of software development effort is spent in testing phase in a typical software development project. Test case design as well as execution consume a lot of time. Hence, automated generation of test cases is highly required. Here a novel testing methodology is being presented to test objectoriented software based on UML state chart diagrams. In this approach, function minimization technique is being applied and generate test cases automatically from UML state chart diagrams. Software testing forms an integral part of the software development life cycle. Since the objective of testing is to ensure the conformity of an application to its specification, a test “oracle” is needed to determine whether a given test case exposes a fault or not. An automated oracle to support the activities of human testers can reduce the actual cost of the testing process and the related maintenance costs. In this paper, a new concept is being presented using an UML state chart diagram and tables for the test case generation, artificial neural network as an optimization tool for reducing the redundancy in the test case generated using the genetic algorithm. A neural network is trained by the backpropagation algorithm on a set of test cases applied to the original version of the system.
This is chapter 2 of ISTQB Advance Test Manager certification. This presentation helps aspirants understand and prepare the content of the certification.
The document discusses the V-Model approach to software testing and how it relates different types of testing to stages of development. It provides an overview of the V-Model stages including requirements, objectives, external specification, system design, and module interface specification. It also discusses how test objectives can be defined for each testing level to avoid redundancy. While the V-Model provides a constrained model, the types of testing it identifies are still relevant for more iterative processes.
The document describes various software testing methods and techniques. It discusses black box and white box testing methods. Black box testing evaluates software based on requirements without knowledge of internal structure, while white box testing uses knowledge of internal structure. Key black box techniques include equivalence partitioning, boundary value analysis, cause-effect graphing and comparison testing. White box techniques covered are basis path testing, loop testing and control structure testing. The document also discusses other techniques like gray box testing, fuzz testing and model-based testing.
Software testing and introduction to qualityDhanashriAmbre
The document provides an overview of software testing and quality assurance. It defines software testing as a process to investigate quality and find defects between expected and actual results. Testing is necessary to ensure software is defect-free per customer specifications and increases reliability. The document then discusses types of errors like ambiguous specifications, misunderstood specifications, and logic/coding errors. It outlines the software development life cycle including phases like planning, analysis, design, coding, testing, implementation, and maintenance. Each phase is described in 1-2 sentences.
Software Quality Analysis Using Mutation Testing SchemeEditor IJMTER
The software test coverage is used measure the safety measures. The safety critical analysis is
carried out for the source code designed in Java language. Testing provides a primary means for
assuring software in safety-critical systems. To demonstrate, particularly to a certification authority, that
sufficient testing has been performed, it is necessary to achieve the test coverage levels recommended or
mandated by safety standards and industry guidelines. Mutation testing provides an alternative or
complementary method of measuring test sufficiency, but has not been widely adopted in the safetycritical industry. The system provides an empirical evaluation of the application of mutation testing to
airborne software systems which have already satisfied the coverage requirements for certification.
The system mutation testing to safety-critical software developed using high-integrity subsets of
C and Ada, identify the most effective mutant types and analyze the root causes of failures in test cases.
Mutation testing could be effective where traditional structural coverage analysis and manual peer
review have failed. They also show that several testing issues have origins beyond the test activity and
this suggests improvements to the requirements definition and coding process. The system also
examines the relationship between program characteristics and mutation survival and considers how
program size can provide a means for targeting test areas most likely to have dormant faults. Industry
feedback is also provided, particularly on how mutation testing can be integrated into a typical
verification life cycle of airborne software. The system also covers the safety and criticality levels of
Java source code.
Chapter 1 - The Technical Test Analyst Tasks in Risk Based TestingNeeraj Kumar Singh
This is chapter 1 of ISTQB Advance Technical Test Analyst certification. This presentation helps aspirants understand and prepare the content of the certification.
Manual software testing interview questions and answers are provided. Key points include:
- The difference between a bug, error, and defect is explained. A bug or defect is a flaw that causes failure, while an error is a human mistake.
- White box testing and the V-model framework are described. White box testing uses internal structure, while the V-model integrates testing into each development phase.
- Stubs and drivers are parts of incremental testing used in bottom-up and top-down approaches. Stubs replace dependent components during testing.
The document discusses leadership lessons from Mahendra Singh Dhoni's captaincy of the Indian cricket team. It describes how Dhoni insisted on having the team he wanted, optimally utilized available resources despite injuries, and backed players who were low on confidence by giving them challenges to boost their confidence in him and the team. It highlights how Dhoni remained calm under pressure and gave credit to his team after winning.
FB, SS (4 files), Twitter http://q.4rd.ca/aaacmD #Sorenikhilawareness
This document provides a table of contents for "A Survey of French Literature Volume I: The Middle Ages and The Sixteenth Century". The table of contents lists 10 chapters that will cover major works and authors from these time periods, including La Chanson de Roland, Tristan et Iseut, Aucassin et Nicolette, Le Roman de la Rose, medieval theater, lyric poetry, Calvin, Rabelais, Renaissance poets like Marot, du Bellay and Ronsard, and Montaigne. Selections and excerpts from many of these works are translated into modern French.
The document discusses leadership lessons from Mahendra Singh Dhoni's captaincy of the Indian cricket team. It describes how Dhoni insisted on having the team he wanted, optimally utilized available resources despite injuries, and backed players who were low on confidence by giving them challenges to boost their confidence in him and the team. It highlights how Dhoni remained calm under pressure and let his team take the credit for their successes.
The document discusses leadership lessons from Mahendra Singh Dhoni's captaincy of the Indian cricket team. It describes how Dhoni insisted on having the team he wanted, optimally utilized available resources despite injuries, and backed players who were low on confidence by giving them challenges to boost their confidence in him and the team. It highlights how Dhoni remained calm under pressure and gave credit to his team after winning.
This document provides instructions for creating a PDF file using Adobe Acrobat, including three exercises:
1. Create a PDF using PDFMaker within Microsoft Word.
2. Create a PDF using the Print command.
3. Combine multiple PDF files into a single document.
Reg: Content with 8 channel types! http://awaren.us/xY1iFPnikhilawareness
Etiam eu lacus ligula, sit amet porta lacus. Aenean dolor nibh, condimentum non consectetur quis, dignissim ac dui. In non lorem risus, at tristique est. Pellentesque nulla lorem, pretium eu blandit eget, imperdiet ut ante. Integer posuere, purus at ultricies porta, libero nunc interdum sapien, eu bibendum lectus mauris ac nunc. Morbi orci urna, tempus eu rutrum dignissim, malesuada in libero. Suspendisse vitae nunc libero, sed vulputate urna. Nullam interdum suscipit sodales. Suspendisse sed dui vitae lacus dictum feugiat. Vivamus ut neque odio, quis ultrices lectus. Cras tortor metus, aliquam quis commodo quis, condimentum non urna. Duis tempor velit a arcu auctor laoreet. Cras eleifend, lorem vitae tempus consequat, turpis dolor feugiat odio, et lacinia purus libero non dui. Maecenas ac scelerisque ipsum. Fusce vulputate tincidunt leo, et eleifend ipsum luctus in. Nam varius dignissim mi vel mollis.
Nulla felis enim, viverra eu hendrerit id, suscipit sit amet metus. Ut eros risus, aliquet ac tempus eu, fringilla non urna. Suspendisse non leo sed velit pellentesque aliquam. Cras feugiat, diam ut mattis pulvinar, tellus libero rhoncus lectus, id rhoncus quam purus et lacus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque fringilla felis sed mi interdum sit amet faucibus enim ullamcorper. Suspendisse potenti.
Morbi rutrum gravida massa, ut suscipit mi faucibus id. Mauris dignissim euismod tellus ac pretium. Proin sit amet eros id orci posuere aliquam eu faucibus tortor. Nullam eu turpis eu neque laoreet fringilla. Praesent diam risus, auctor vitae fermentum et, cursus ac orci. Etiam varius, erat sit amet imperdiet consectetur, nisl dui ornare nunc, vitae venenatis quam ipsum lacinia velit. Duis bibendum rhoncus vehicula. Integer in sapien leo, sed pellentesque odio. Aliquam erat volutpat. Pellentesque lorem lorem, interdum a dapibus dignissim, fringilla sit amet enim. Ut sed neque sapien, venenatis malesuada metus. Ut eget massa felis, eu condimentum velit. Maecenas posuere mi at metus convallis aliquam. Aenean gravida turpis suscipit metus pellentesque ullamcorper.
This document is a presentation about India's struggle for independence from British rule, culminating in freedom on August 15, 1947. It discusses key figures in the independence movement like Gandhi, Nehru, and Bose. It outlines the periods of British East India Company rule and direct British government rule over India. It highlights events like the Salt March and formation of the Indian National Congress that advanced the independence movement. The presentation concludes with India gaining independence in 1947 after the partition into India and Pakistan, though violence and refugee crises ensued.
Haven't done this in a long time! http://awaren.us/NMH0Cnnikhilawareness
This document provides the Tamil text and English translation of the first chapter of the Thirukkural, a classic Tamil text of couplets on ethics and morality. It begins with an introduction noting the individuals who input the Tamil text and translated it into English. It thanks the copyright holders for permission and provides information on viewing the Tamil text properly. The chapter itself contains verses praising God and extolling the virtues of asceticism and the power of virtue.
Real time implementation of the software system requires being more versatile. In the maintenance phase, the modified system under regression testing must assure that the existing system remains defect free. Test case prioritization technique of regression testing includes code as well as model based methods of prioritizing the test cases. System model based test case prioritization can detect the severe faults early as compare to the code based test case prioritization. Model based prioritization techniques based on requirements in a cost effective manner has not been taken for study so far. Model based testing used to test the functionality of the software system based on requirement. An effective model based approach is defined for prioritizing test cases and to generate the effective test sequence. The test cases are rescheduled based on requirement analysis and user view analysis. With the use of weighted approach the overall cost is estimated to test the functionality of the model elements. Here, the genetic approach has been applied to generate efficient test path. The regression cost in terms of effort has been reduced under model based prioritization approach.
The document defines various types of software testing techniques and terms, including:
- Audit testing which assesses compliance with specifications, standards, or agreements.
- Acceptance testing conducted by customers to determine if a system meets acceptance criteria.
- Alpha and beta testing which involve customer testing in controlled or live environments.
- Boundary value analysis which tests at boundaries and limits of input/output domains.
- Branch coverage which requires each code branch be tested at least once.
This study empirically evaluated the use of mutation testing to improve test quality for safety-critical software. Mutation testing was applied to two airborne software systems developed using rigorous development processes and guidelines. The study identified an effective subset of mutant types, categorized the root causes of test failures, and examined relationships between program characteristics and mutation survival. The results showed how mutation testing could be useful for finding issues missed by traditional structural coverage analysis and manual peer review. Industry feedback also provided perspectives on integrating mutation testing into typical verification lifecycles for airborne software.
Software testing is an activity which is aimed for evaluating quality of a program and also for improving it, by identifying defects and problems. Software testing strives for achieving its goal (both implicit and explicit) but it has certain limitations, still testing can be done more effectively if certain established principles are to be followed. In spite of having limitations, software testing continues to dominate other verification techniques like static analysis, model checking and proofs. So it is indispensable to understand the goals, principles and limitations of software testing so that the effectiveness of software testing could be maximized.
This document discusses different levels of software testing including unit, integration, system, and acceptance testing. It focuses on unit testing and describes how to plan and prepare for unit testing. Key points include:
- Unit testing involves testing individual software components like functions, methods, and classes. Good practice is to plan unit tests to target specific defect types.
- Preparing for unit tests involves planning the approach, designing test cases and procedures, and developing any auxiliary code needed to run the tests. Test cases should cover normal and abnormal behavior.
- When testing classes as units, testers focus on interactions between methods in the class. Choosing classes or individual methods as the test unit involves tradeoffs around complexity and cost
The document discusses software testing, which involves executing a program or application to evaluate properties like meeting requirements, responding correctly to inputs, performing functions in an acceptable time, usability, installability, and achieving stakeholder desires. Software testing aims to find bugs and provide information about quality and failure risk. It discusses roles, methods, and economics of software testing.
Test Case Optimization and Redundancy Reduction Using GA and Neural Networks IJECEIAES
More than 50% of software development effort is spent in testing phase in a typical software development project. Test case design as well as execution consume a lot of time. Hence, automated generation of test cases is highly required. Here a novel testing methodology is being presented to test objectoriented software based on UML state chart diagrams. In this approach, function minimization technique is being applied and generate test cases automatically from UML state chart diagrams. Software testing forms an integral part of the software development life cycle. Since the objective of testing is to ensure the conformity of an application to its specification, a test “oracle” is needed to determine whether a given test case exposes a fault or not. An automated oracle to support the activities of human testers can reduce the actual cost of the testing process and the related maintenance costs. In this paper, a new concept is being presented using an UML state chart diagram and tables for the test case generation, artificial neural network as an optimization tool for reducing the redundancy in the test case generated using the genetic algorithm. A neural network is trained by the backpropagation algorithm on a set of test cases applied to the original version of the system.
This is chapter 2 of ISTQB Advance Test Manager certification. This presentation helps aspirants understand and prepare the content of the certification.
The document discusses the V-Model approach to software testing and how it relates different types of testing to stages of development. It provides an overview of the V-Model stages including requirements, objectives, external specification, system design, and module interface specification. It also discusses how test objectives can be defined for each testing level to avoid redundancy. While the V-Model provides a constrained model, the types of testing it identifies are still relevant for more iterative processes.
The document describes various software testing methods and techniques. It discusses black box and white box testing methods. Black box testing evaluates software based on requirements without knowledge of internal structure, while white box testing uses knowledge of internal structure. Key black box techniques include equivalence partitioning, boundary value analysis, cause-effect graphing and comparison testing. White box techniques covered are basis path testing, loop testing and control structure testing. The document also discusses other techniques like gray box testing, fuzz testing and model-based testing.
Software testing and introduction to qualityDhanashriAmbre
The document provides an overview of software testing and quality assurance. It defines software testing as a process to investigate quality and find defects between expected and actual results. Testing is necessary to ensure software is defect-free per customer specifications and increases reliability. The document then discusses types of errors like ambiguous specifications, misunderstood specifications, and logic/coding errors. It outlines the software development life cycle including phases like planning, analysis, design, coding, testing, implementation, and maintenance. Each phase is described in 1-2 sentences.
Software Quality Analysis Using Mutation Testing SchemeEditor IJMTER
The software test coverage is used measure the safety measures. The safety critical analysis is
carried out for the source code designed in Java language. Testing provides a primary means for
assuring software in safety-critical systems. To demonstrate, particularly to a certification authority, that
sufficient testing has been performed, it is necessary to achieve the test coverage levels recommended or
mandated by safety standards and industry guidelines. Mutation testing provides an alternative or
complementary method of measuring test sufficiency, but has not been widely adopted in the safetycritical industry. The system provides an empirical evaluation of the application of mutation testing to
airborne software systems which have already satisfied the coverage requirements for certification.
The system mutation testing to safety-critical software developed using high-integrity subsets of
C and Ada, identify the most effective mutant types and analyze the root causes of failures in test cases.
Mutation testing could be effective where traditional structural coverage analysis and manual peer
review have failed. They also show that several testing issues have origins beyond the test activity and
this suggests improvements to the requirements definition and coding process. The system also
examines the relationship between program characteristics and mutation survival and considers how
program size can provide a means for targeting test areas most likely to have dormant faults. Industry
feedback is also provided, particularly on how mutation testing can be integrated into a typical
verification life cycle of airborne software. The system also covers the safety and criticality levels of
Java source code.
Chapter 1 - The Technical Test Analyst Tasks in Risk Based TestingNeeraj Kumar Singh
This is chapter 1 of ISTQB Advance Technical Test Analyst certification. This presentation helps aspirants understand and prepare the content of the certification.
Manual software testing interview questions and answers are provided. Key points include:
- The difference between a bug, error, and defect is explained. A bug or defect is a flaw that causes failure, while an error is a human mistake.
- White box testing and the V-model framework are described. White box testing uses internal structure, while the V-model integrates testing into each development phase.
- Stubs and drivers are parts of incremental testing used in bottom-up and top-down approaches. Stubs replace dependent components during testing.
The document discusses leadership lessons from Mahendra Singh Dhoni's captaincy of the Indian cricket team. It describes how Dhoni insisted on having the team he wanted, optimally utilized available resources despite injuries, and backed players who were low on confidence by giving them challenges to boost their confidence in him and the team. It highlights how Dhoni remained calm under pressure and gave credit to his team after winning.
FB, SS (4 files), Twitter http://q.4rd.ca/aaacmD #Sorenikhilawareness
This document provides a table of contents for "A Survey of French Literature Volume I: The Middle Ages and The Sixteenth Century". The table of contents lists 10 chapters that will cover major works and authors from these time periods, including La Chanson de Roland, Tristan et Iseut, Aucassin et Nicolette, Le Roman de la Rose, medieval theater, lyric poetry, Calvin, Rabelais, Renaissance poets like Marot, du Bellay and Ronsard, and Montaigne. Selections and excerpts from many of these works are translated into modern French.
The document discusses leadership lessons from Mahendra Singh Dhoni's captaincy of the Indian cricket team. It describes how Dhoni insisted on having the team he wanted, optimally utilized available resources despite injuries, and backed players who were low on confidence by giving them challenges to boost their confidence in him and the team. It highlights how Dhoni remained calm under pressure and let his team take the credit for their successes.
The document discusses leadership lessons from Mahendra Singh Dhoni's captaincy of the Indian cricket team. It describes how Dhoni insisted on having the team he wanted, optimally utilized available resources despite injuries, and backed players who were low on confidence by giving them challenges to boost their confidence in him and the team. It highlights how Dhoni remained calm under pressure and gave credit to his team after winning.
This document provides instructions for creating a PDF file using Adobe Acrobat, including three exercises:
1. Create a PDF using PDFMaker within Microsoft Word.
2. Create a PDF using the Print command.
3. Combine multiple PDF files into a single document.
Reg: Content with 8 channel types! http://awaren.us/xY1iFPnikhilawareness
Etiam eu lacus ligula, sit amet porta lacus. Aenean dolor nibh, condimentum non consectetur quis, dignissim ac dui. In non lorem risus, at tristique est. Pellentesque nulla lorem, pretium eu blandit eget, imperdiet ut ante. Integer posuere, purus at ultricies porta, libero nunc interdum sapien, eu bibendum lectus mauris ac nunc. Morbi orci urna, tempus eu rutrum dignissim, malesuada in libero. Suspendisse vitae nunc libero, sed vulputate urna. Nullam interdum suscipit sodales. Suspendisse sed dui vitae lacus dictum feugiat. Vivamus ut neque odio, quis ultrices lectus. Cras tortor metus, aliquam quis commodo quis, condimentum non urna. Duis tempor velit a arcu auctor laoreet. Cras eleifend, lorem vitae tempus consequat, turpis dolor feugiat odio, et lacinia purus libero non dui. Maecenas ac scelerisque ipsum. Fusce vulputate tincidunt leo, et eleifend ipsum luctus in. Nam varius dignissim mi vel mollis.
Nulla felis enim, viverra eu hendrerit id, suscipit sit amet metus. Ut eros risus, aliquet ac tempus eu, fringilla non urna. Suspendisse non leo sed velit pellentesque aliquam. Cras feugiat, diam ut mattis pulvinar, tellus libero rhoncus lectus, id rhoncus quam purus et lacus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque fringilla felis sed mi interdum sit amet faucibus enim ullamcorper. Suspendisse potenti.
Morbi rutrum gravida massa, ut suscipit mi faucibus id. Mauris dignissim euismod tellus ac pretium. Proin sit amet eros id orci posuere aliquam eu faucibus tortor. Nullam eu turpis eu neque laoreet fringilla. Praesent diam risus, auctor vitae fermentum et, cursus ac orci. Etiam varius, erat sit amet imperdiet consectetur, nisl dui ornare nunc, vitae venenatis quam ipsum lacinia velit. Duis bibendum rhoncus vehicula. Integer in sapien leo, sed pellentesque odio. Aliquam erat volutpat. Pellentesque lorem lorem, interdum a dapibus dignissim, fringilla sit amet enim. Ut sed neque sapien, venenatis malesuada metus. Ut eget massa felis, eu condimentum velit. Maecenas posuere mi at metus convallis aliquam. Aenean gravida turpis suscipit metus pellentesque ullamcorper.
This document is a presentation about India's struggle for independence from British rule, culminating in freedom on August 15, 1947. It discusses key figures in the independence movement like Gandhi, Nehru, and Bose. It outlines the periods of British East India Company rule and direct British government rule over India. It highlights events like the Salt March and formation of the Indian National Congress that advanced the independence movement. The presentation concludes with India gaining independence in 1947 after the partition into India and Pakistan, though violence and refugee crises ensued.
Haven't done this in a long time! http://awaren.us/NMH0Cnnikhilawareness
This document provides the Tamil text and English translation of the first chapter of the Thirukkural, a classic Tamil text of couplets on ethics and morality. It begins with an introduction noting the individuals who input the Tamil text and translated it into English. It thanks the copyright holders for permission and provides information on viewing the Tamil text properly. The chapter itself contains verses praising God and extolling the virtues of asceticism and the power of virtue.
The document discusses leadership lessons from Mahendra Singh Dhoni's captaincy of the Indian cricket team. It describes how Dhoni insisted on having the team he wanted, optimally utilized available resources despite injuries, and backed players who were low on confidence by giving them challenges to boost their confidence in him and the team. It highlights how Dhoni remained calm under pressure and let his team take the credit for their successes.
This document provides an overview of American football and discusses opportunities for statistical analysis and research. It describes the basic structure of the game and notes that while detailed play-by-play data exists, it has not been easily accessible for academic research. The document then discusses several areas where statistical methods could be applied, including evaluating individual player and team performance, developing models to assess strategy around decisions like fourth downs and extra points, and creating statistical ratings of teams. It argues more research is needed to better understand the game and inform coaching decisions.
The document discusses India's struggle for independence from British colonial rule, highlighting key events and individuals involved. It notes that India became independent on August 15, 1947 after years of resistance led by prominent figures like Gandhi, Nehru, and Bose. Many ordinary Indians also sacrificed greatly for the freedom movement. The establishment of an independent India marked the end of British rule and the beginning of India as the world's largest democracy.
This document is a presentation about India's struggle for independence from British rule, culminating in freedom on August 15, 1947. It discusses key figures in the independence movement like Gandhi, Nehru, and Bose. It outlines the periods of British East India Company rule and direct British government rule over India. It highlights events like the Salt March and formation of the Indian National Congress that advanced the independence movement. The presentation concludes with India gaining independence in 1947 after the partition into India and Pakistan, though violence and refugee crises ensued.
This document contains recipes for 51 Indian dishes including appetizers, main courses, desserts, and snacks. The recipes cover both savory dishes like aloo palak (potato spinach curry) and gobi manchurian (cauliflower fritters) as well as sweet treats such as gulab jamun (milk balls in rose-flavored syrup), kaju barfi (cashew fudge), and boondi (chickpea flour fritters). The document provides ingredients and instructions for each recipe so readers can make a variety of traditional Indian food at home.
This document provides an overview of software testing techniques and the history of research in this area. It discusses:
1) The goals and types of software testing, including functional vs structural techniques.
2) Major milestones in the evolution of testing concepts from the 1950s to present day, shifting from debugging to prevention.
3) Key theoretical and methodological contributions to testing techniques from the 1970s onward, including work on path coverage, data flow testing, and model-based approaches.
4) How research in testing techniques has matured over time based on a technology maturation model, moving from ad hoc practices to a systematic discipline grounded in theory.
What is the Objective of Software Testing?seojayeshts
Software testing is the process of evaluating the functionality of a software application with an intent to find whether the developed software met the specified requirements and to identify defects to ensure that the product is free from errors. It involves the execution of software/system components using manual or automated tools to evaluate one or more properties of interest. The necessity of software testing arises from the critical role that software plays in our lives and the potential harm that software defects can cause to the economy, security, and well-being of users.
This document discusses various types of software testing methods and strategies. It describes black box testing as treating software as a black box without knowledge of internal implementation. White box testing involves working with internal data structures and algorithms. Unit testing validates individual units of source code. The document outlines the aims, processes, advantages, and limitations of these different testing approaches.
This document discusses software testing principles and concepts. It defines key terms like validation, verification, defects, failures, and metrics. It outlines 11 testing principles like testing being a creative task and test results needing meticulous inspection. The roles of testers are discussed in collaborating with other teams. Defect classes are defined at different stages and types of defects are provided. Quality factors, process maturity models, and defect prevention strategies are also summarized.
Interview questions for manual testing technology.Vinay Agnihotri
INTERVIEW QUESTIONS FOR MANUAL TESTING. THERE IS SOME IMPORTANT MANUAL TESTING INTERVIEW QUESTION WHICH IS VERY HELPFULL FOR FRESHERS AND EXPERIENCE CANDIDATE.
Testing is the process of identifying bugs and ensuring software meets requirements. It involves executing programs under different conditions to check specification, functionality, and performance. The objectives of testing are to uncover errors, demonstrate requirements are met, and validate quality with minimal cost. Testing follows a life cycle including planning, design, execution, and reporting. Different methodologies like black box and white box testing are used at various levels from unit to system. The overall goal is to perform effective testing to deliver high quality software.
The document discusses different types of software testing techniques, including black box testing and white box testing. It provides details on black box testing, which focuses on external inputs and outputs without knowledge of internal code. Key aspects of black box testing covered include independence from internal structure and focus on requirements. The document also discusses different types of functional testing like unit, integration, system, and acceptance testing. It then covers various non-functional testing techniques such as performance, security, usability, compatibility, and reliability testing. Specific black box testing techniques like equivalence partitioning and boundary value analysis are also summarized.
This document discusses various topics related to software testing and verification and validation (V&V). It begins with an overview of test plan creation and different types of testing such as unit, integration, system, and object-oriented testing. It then defines the key differences between verification and validation. The rest of the document provides more details on V&V techniques like static and dynamic verification, software inspections, and testing. It also covers testing fundamentals, principles, testability factors, and different testing techniques like black-box and white-box testing.
How to Make the Most of Regression and Unit Testing.pdfAbhay Kumar
Understanding the differences between regression testing and unit testing is paramount for maintaining the robustness of any software. Although both serve the purpose of vulnerability scanning to detect threats, they vary in terms of their test objectives, stages in the development process, and the scope of the code they cover. Let's delve into these distinctions to grasp their roles better.
Difference in Test Objectives:
- While similar, Unit and regression testing have distinct targets within your code. Unit testing, conducted by programmers, assesses individual components, validating the correct functionality of each variable, function, and object.
- On the other hand, regression testing (often termed QA testing) occurs after programmers complete work on specific features. It acts as a system-wide check, ensuring untouched components function as expected. While unit tests provide the precision of individual functions and variables, regression tests collaborate to ascertain that the entire system functions optimally.
Difference in the Development Phase:
- The timing of unit and regression tests sets them apart. Unit tests are conducted during the development phase, where developers run them after implementing changes to confirm no adverse impacts.
- Conversely, regression testing is performed before the feature's production release. It comprises unit tests, integration tests, and various other testing types. Testers are responsible for executing regression testing. Automated regression testing, a key step in continuous integration/continuous delivery, quickly detects if recent code changes have disrupted the existing code.
Difference in Code Coverage:
- A unit test concentrates on a single unit, method, or function, examining one element at a time. It doesn't account for how these units interact, necessitating integration tests. This approach provides swift feedback due to its focused testing nature.
- In contrast, regression tests validate if alterations to existing functionalities have affected other parts of the system by testing against predefined scenarios, ensuring correct integration of units. Given the comprehensive testing involved, it generally consumes more time.
Unit and regression testing are vital pillars in the software development journey. Regular execution of these tests is key to minimizing bugs and refining code quality.
Regression testing conducted post-unit testing before a software release, ensures system integrity despite changes. On the other hand, unit testing meticulously validates new functions, ensuring precise code execution.
Manual testing is the process of manually testing software for defects by playing the role of an end user and using all features of the application to ensure correct behavior, following a written test plan. The document discusses various concepts related to manual testing including software quality assurance, verification, validation, software life cycles, testing techniques like black box testing and white box testing, unit testing, alpha testing, beta testing, system testing, volume testing, stress testing, and security testing. It provides brief definitions and purposes of these concepts.
The document discusses various aspects of software testing such as the definitions of testing, different testing methodologies like black box and white box testing, testing levels from unit to acceptance testing, and performance testing types including stress, recovery, and compatibility testing. It also covers testing tools, test plans, test cases, and the software development life cycle.
This is the power point presentation on Software Testing. Software Testing is the process of finding error or bug in the developed software product based on the client requirement.
This power point presentation give the basic knowledge about the software testing.
Learn more at blog : --
https://solutionbyexpert.blogspot.com/2020/08/become-expert-secret-of-success-ii.html
for mathematics classes visit the below link ---
https://www.youtube.com/watch?v=g07wTZYYzKo&t=188s
https://www.youtube.com/watch?v=KleKFXSXGPY&t=853s
for physics classes visit the below link --
https://www.youtube.com/watch?v=6ha1sxMy4mU
https://www.youtube.com/watch?v=2k5uI6Gm-8Y
our facebook link --
https://www.facebook.com/Online-Smart-Classes-108395901487258
#coding
#coding development skill program
#java
System testing evaluates a complete integrated system to determine if it meets specified requirements. It tests both functional and non-functional requirements. Functional requirements include business rules, transactions, authentication, and external interfaces. Non-functional requirements include performance, reliability, security, and usability. There are different types of system testing, including black box testing which tests functionality without knowledge of internal structure, white box testing which tests internal structures, and gray box testing which is a combination. Input, installation, graphical user interface, and regression testing are examples of different types of system testing.
Manual testing is the process of manually testing software for defects by playing the role of an end user and using all features of the application to ensure correct behavior, following a written test plan. The document discusses various concepts in manual testing including software quality assurance, verification, validation, software life cycles, why testing is done, and different testing techniques like black box testing, white box testing, regression testing, unit testing, alpha testing, beta testing, system testing, volume testing, stress testing, and security testing.
The document discusses strategies for software testing. It defines different levels of testing including unit testing, integration testing, system testing, and validation testing. It also discusses different testing approaches such as test-driven development, behavior-driven development, and agile testing. The document provides details on unit testing, integration testing, system testing, and validation testing. It discusses testing strategies, testing methods including black box testing and white box testing, and the differences between black box and white box testing.
International Journal of Engineering Research and Development (IJERD)IJERD Editor
This document discusses test case prioritization, which aims to execute test cases in an order that increases their effectiveness at detecting faults. It describes regression testing and different prioritization techniques, including genetic algorithms. Genetic algorithms represent test cases as chromosomes and use selection, crossover and mutation to evolve solutions. The document concludes that prioritization techniques can improve fault detection rates but their effectiveness varies across programs and test suites, so practitioners must choose techniques carefully for different testing scenarios.
1) The document discusses software testing principles, lifecycles, limitations and methods. It describes the different phases of software testing like requirements study, test case design, test execution, test closure and test process analysis.
2) It also discusses different levels of testing including unit testing, integration testing, system testing and acceptance testing. Unit testing checks individual program modules, integration testing verifies interface connections, system testing checks full application functionality, and acceptance testing gets customer approval.
3) The document provides objectives and features of good test cases and objectives of a software tester. It also outlines principles of testing like testing for failures, starting early, defining test plans, and testing for valid and invalid conditions.
Testing is the process of evaluating a system or its components to identify differences between expected and actual results. It involves executing a system to identify gaps, errors, or missing requirements. There are different types of testing such as unit testing, integration testing, system testing, and acceptance testing. Testing can be manual, automated, functional, or non-functional. Functional testing evaluates system functionality while non-functional testing evaluates attributes like performance and security. Common non-functional test types are load testing and stress testing which analyze a system's behavior under different load conditions.
Similar to Staging's channles are being tested (20)
1. Software Testing Techniques
Technology Maturation and Research Strategy
Class Report for 17-939A
Lu Luo
Institute for Software Research International
Carnegie Mellon University
Pittsburgh, PA15232
USA
2. Software Testing Techniques
Technology Maturation and Research Strategies
Lu Luo
School of Computer Science
Carnegie Mellon University
1 Introduction1
Software testing is as old as the hills in the history of digital computers. The testing of software is an
important means of assessing the software to determine its quality. Since testing typically consumes
40~50% of development efforts, and consumes more effort for systems that require higher levels of
reliability, it is a significant part of the software engineering. With the development of Fourth generation
languages (4GL), which speeds up the implementation process, the proportion of time devoted to testing
increased. As the amount of maintenance and upgrade of existing systems grow, significant amount of
testing will also be needed to verify systems after changes are made [12] . Despite advances in formal
methods and verification techniques, a system still needs to be tested before it is used. Testing remains the
truly effective means to assure the quality of a software system of non-trivial complexity [13], as well as
one of the most intricate and least understood areas in software engineering [19]. Testing, an important
research area within computer science is likely to become even more important in the future.
This retrospective on a fifty-year of software testing technique research examines the maturation of the
software testing technique research by tracing the major research results that have contributed to the growth
of this area. It also assesses the change of research paradigms over time by tracing the types of research
questions and strategies used at various stages. We employ the technology maturation model given by
Redwine and Riddle [15] as the framework of our studies of how the techniques of software testing first get
the idea formulated, preliminarily used, developed, and then extended into a broader solution. Shaw gives
a very good framework of software engineering research paradigms in [17], which classifies the research
settings, research approaches, methods, and research validations that have been done by software
researchers. Shaw’s model is used to evaluate the research strategies for testing techniques used in our
paper.
2 The Taxonomy of Testing Techniques
Software testing is a very broad area, which involves many other technical and non-technical areas, such as
specification, design and implementation, maintenance, process and management issues in software
engineering. Our study focuses on the state of the art in testing techniques, as well as the latest techniques
which representing the future direction of this area. Before stepping into any detail of the maturation study
of these techniques, let us have a brief look at some technical concepts that are relative to our research.
2.1 The Goal of Testing
In different publications, the definition of testing varies according to the purpose, process, and level of
testing described. Miller gives a good description of testing in [13]:
The general aim of testing is to affirm the quality of software systems by systematically exercising the
software in carefully controlled circumstances.
Miller’s description of testing views most software quality assurances activities as testing. He contends
that testing should have the major intent of finding errors. A good test is one that has a high probability of
finding an as yet undiscovered error, and a successful test is one that uncovers an as yet undiscovered error.
This general category of software testing activities can be further divided. For purposes of this paper,
1
Jointly written by Paul Li
1
3. testing is the dynamic analysis of a piece of software, requiring execution of the system to produce results,
which are then compared to expected outputs.
2.2 The Testing Spectrum
Testing is involved in every stage of software life cycle, but the testing done at each level of software
development is different in nature and has different objectives.
Unit Testing is done at the lowest level. It tests the basic unit of software, which is the smallest
testable piece of software, and is often called “unit”, “module”, or “component” interchangeably.
Integration Testing is performed when two or more tested units are combined into a larger structure.
The test is often done on both the interfaces between the components and the larger structure being
constructed, if its quality property cannot be assessed from its components.
System Testing tends to affirm the end-to-end quality of the entire system. System test is often based
on the functional/requirement specification of the system. Non-functional quality attributes, such as
reliability, security, and maintainability, are also checked.
Acceptance Testing is done when the completed system is handed over from the developers to the
customers or users. The purpose of acceptance testing is rather to give confidence that the system is
working than to find errors.
2.3 Static Analysis and Dynamic Analysis
Based on whether the actual execution of software under evaluation is needed or not, there are two major
categories of quality assurance activities:
Static Analysis focuses on the range of methods that are used to determine or estimate software quality
without reference to actual executions. Techniques in this area include code inspection, program
analysis, symbolic analysis, and model checking.
Dynamic Analysis deals with specific methods for ascertaining and/or approximating software quality
through actual executions, i.e., with real data and under real (or simulated) circumstances. Techniques
in this area include synthesis of inputs, the use of structurally dictated testing procedures, and the
automation of testing environment generation.
Generally the static and dynamic methods are sometimes inseparable, but can almost always discussed
separately. In this paper, we mean dynamic analysis when we say testing, since most of the testing
activities (thus all the techniques studied in this paper) require the execution of the software.
2.4 Functional Technique and Structural Technique
The information flow of testing is shown in Figure 1. As we can see, testing involves the configuration of
proper inputs, execution of the software over the input, and the analysis of the output. The “Software
Configuration” includes requirements specification, design specification, source code, and so on. The
“Test Configuration” includes test cases, test plan and procedures, and testing tools.
Based on the testing information flow, a testing technique specifies the strategy used in testing to select
input test cases and analyze test results. Different techniques reveal different quality aspects of a software
system, and there are two major categories of testing techniques, functional and structural.
Functional Testing: the software program or system under test is viewed a a “black box”. The
s
selection of test cases for functional testing is based on the requirement or design specification of the
software entity under test. Examples of expected results, some times are called test oracles, include
2
4. requirement/design specifications, hand calculated values, and simulated results. Functional testing
emphasizes on the external behavior of the software entity.
Structural Testing: the software entity is viewed as a “white box”. The selection of test cases is based
on the impl ementation of the software entity. The goal of selecting such test cases is to cause the
execution of specific spots in the software entity, such as specific statements, program branches or
paths. The expected results are evaluated on a set of coverage criteria. Examples of coverage criteria
include path coverage, branch coverage, and data-flow coverage. Structural testing emphasizes on the
internal structure of the software entity.
Software
Configuration
Corrections
Debug
Errors
Test
Results
Testing Evaluation
Test
Configuration Expected
Error Rate Reliability
Results
Data Model
Predicted
Reliability
Figure 1. Testing Information Flow
3 Scope of the Study
3.1 Technical Scope
In this paper, we focus on the technology maturation of testing techniques, including these functional and
structural techniques that have been influential in the academic world and widely used in practice. We are
going to examine the growth and propagation of the most established strategy and methodology used to
select test cases and analyze test results. Research in software testing techniques can be roughly divided
into two branches: theoretical and methodological, and the growth in both branches push the growth of
testing technology together. Inhibitors of maturation, which explains why the in-depth research hasn’t
brought revolutionary advantage in industry testing practice, are also within our scope of interest.
There are many other interesting areas in software testing. We limit the scope of our study within the range
of testing techniques, although some of the areas maybe inseparable from our study. Specifically, we are
not going to discuss:
• How testing is involved in the software development cycle
• How different levels of testing are performed
• Testing process models
• Testing policy and management responsibilities, and
• Stop criteria of testing and software testability
3.2 Goal and standard of progress
The ultimate goal of software testing is to help designers, developers, and managers construct systems with
high quality. Thus research and development on testing aim at efficiently performing effective testing – to
find more errors in requirement, design and implementation, and to increase confidence that the software
has various qualities. Testing technique research leads to the destination of practical testing methods and
3
5. tools. Progress toward this destination requires fundamental research, and the creation, refinement,
extension, and popularization of better methods.
The standard of progress for the research of testing techniques include:
• Degree of acceptance of the technology inside and outside the research community
• Degree of dependability on other areas of software engineering
• Change of research paradigms in response to the maturation of software development technologies
• Feasibility of techniques being used in a widespread practical scope, and
• Spread of technology – classes, trainings, management attention
4. The History of Testing Techniques
4.1 Concept Evolution
Software has been tested as early as software has been written. The concept of testing itself evolved with
time. The evolution of definition and targets of software testing has directed the research on testing
techniques. Let’s briefly review the concept evolution of testing using the testing process model proposed
by Gelperin and Hetzel [6] before we begin study the history of testing techniques.
Phase I. Before 1956: The Debugging-Oriented Period
– Testing was not separated from debugging
In 1950, Turing wrote the famous article that is considered to be the first on program testing. The
article addresses the question “How would we know that a program exhibits intelligence?” Stated in
another way, if the requirement is to build such a program, then this question is a special case of “How
would we know that a program satisfies its requirements?” The operational test Turing defined
required the behavior of the program and a reference system (a human) to be indistinguishable to an
interrogator (tester). This could be considered the embryotic form of functional testing. The concepts
of program checkout, debugging and testing were not clearly differentiated by that time.
Phase II. 1957~78: The Demonstration-Oriented Period
– Testing to make sure that the software satisfies its specification
It was not until 1957 was testing, which was called program checkout by that time, distinguished from
debugging. In 1957, Charles Baker pointed out that “program checkout” was seen to have two goals:
“Make sure the program runs” and “Make sure the program solves the problem.” The latter goal was
viewed as the focus of testing, since “make sure” was often translated into the testing goal of satisfying
requirements. As we’ve seen in Figure 1, debugging and testing are actually two different phases. The
distinction between testing and debugging rested on the definition of success. During this period
definitions stress the purpose of testing is to demonstrate correctness: “An ideal test, therefore,
succeeds only when a program contains no errors.” [5]
The 1970s also saw the widespread idea that software could be tested exhaustively. This led to a series
of research emphasis on path coverage testing. As is said in Goodenough and Gerhart’s 1975 paper
“exhaustive testing defined either in terms of program paths or a program’s input domain.” [5]
Phase III. 1979~82: The Destruction-Oriented Period
– Testing to detect implementation faults
In 1979, Myers wrote the book The Art of Software Testing, which provided the foundation for more
effective test technique design. For the first time software testing was described as “the process of
executing a program with the intent of finding errors.” The important point was made that the value of
test cases is much greater if an error is found. As in the demonstration-oriented period, one might
unconsciously select test data that has a low probability of causing program failures. If testing intends
4
6. to show that a program has faults, then the test cases selected will have a higher probability of
detecting them and the testing is more successful. This shift in emphasis led to early association of
testing and other verification/validation activ ities.
Phase IV. 1983~87: The Evaluation-Oriented Period
– Testing to detect faults in requirements and design as well as in implementation
The Institute for Computer Sciences and Technology of the National Bureau of Standards published
Guideline for Lifecycle Validation, Verification, and Testing of Computer Software in 1983, in which a
methodology that integrates analysis, review, and test activities to provide product evaluation during
the software lifecycle was described. The guideline gives the belief that a carefully chosen set of
VV&T techniques can help to ensure the development and maintenance of quality software.
Phase V. Since 1988: The Prevention-Oriented Period
– Testing to prevent faults in requirements, design, and implementation
In the significant book Software Testing Techniques [2], which contains the most complete catalog of
testing techniques, Beizer stated that “the act of designing tests is one of the most effective bug
preventers known,” which extended the definition of testing to error prevention as well as error
detection activities. This led to a classic insight into the power of early testing.
In 1991, Hetzel gave the definition that “Testing is planning, designing, building, maintaining and
executing tests and test environments.” A year before this, Beizer gave four stages of thinking about
testing: 1. to make software work, 2, to break the software, 3, to reduce risk, and 4, a state of mind, i.e.
a total life-cycle concern with testability. These ideas led software testing to view emphasize the
importance of early test design throughout the software life cycle.
The prevention-oriented period is distinguished from the evaluation-oriented by the mechanism,
although both focus on software requirements and design in order to avoid implementation errors. The
prevention model sees test planning, test analysis, and test design activities playing a major role, while
the evaluation model mainly relies on analysis and reviewing techniques other than testing.
4.2 Major Technical Contributions
In general, the research on testing techniques can be roughly divided into two categories: theoretical and
methodological. Software testing techniques are based in an amalgam of methods drawn from graph
theory, programming language, reliability assessment, reliable-testing theory, etc. In this paper, we focus
on those significant theoretical research results, such as test data adequacy and testing criteria, which
provide a sound basis for creating and refining methodologies in a rational and effective manner. Given a
solid theoretical basis, a systematic methodology seeks to employ rational techniques to force sequences of
actions that, in aggregate, accomplish some desired testing-oriented effect.
We are going to start with the major technical contributions of theoretical researches as well as milestone
methodologies of testing techniques. In next section, the Redwine/Riddle maturity model will be used to
illustrate how testing techniques have matured from an intuitive, ad hoc c ollection of methods into an
integrated, systematic discipline. Figure 2 shows the concept formation of testing, milestone technical
contributions on testing techniques, including the most influential theoretical and method research. Note
that the principle for paper selection is significance. A research is chosen and its idea shown in Figure 2
because it defines, influences, or changes the technology fundamentally in its period. It does not
necessarily have to be the first published paper on similar topics.
5
7. Time Concept Theory & Method
Functional Structural
2001: Integrated technique for
component-based software
2000
2000: UML based integration testing
1997: Probablistic Functional Testing 1997: Integration Testing based on
Architectural Description
1994: Coverage based model for
realiability estimation
1991: Integrate test
1990 activities during software 1990: Logic-based Testing
life-cycle
1989: Intgrating spec-based and impl-
based testing using formal methods
1985: Data Flow Oriented Strategy
1983: Testing to prevent
errors in VV&T
1980 1980: Functional Testing and Design 1980: Domain Strategy
1979: Association of Abstractions
Testing to fault detection
1976: Path Approach to Test
Selection
1975: Fundamental 1975: Edge Approach to Test
Theory of Test Data Selection, Probe Insertion
Selection
1970
1960
1957: Distinguish
debugging from testing
1950 1950: Testing to know a
program satisfies
requirements
Figure 2. Major research results in the area of software testing techniques.
This diagram shows the most influential ideas, theory, methods and practices in the area of software testing,
emphasizing on testing techniques. There are two columns in the diagram marked “Concept” and “Theory
&Methods”. The items appear in “Concept” column contains the leading technical viewpoints about the goals of
testing software since the year 1950. The two sub-columns of “Theory & Method” classify theoretical and
methodological research results in functional-based and structural-based.
6
8. Before the year 1975, although software testing was widely performed as an important part of software
development, it remained an intuitive, somehow ad hoc collection of methods. People used principle
functional and structural techniques in their testing practices, but there was little systematic research on
methods or theories of these techniques.
In 1975, Goodenough and Gerhart gave the fundamental theorem of testing in their paper Toward a Theory
of Test Data Selection [5]. This is the first published research attempting to provide a theoretical
foundation for testing, which characterizes that a test data selection strategy is completely effective if it is
guaranteed to discover any error in a program. As is mentioned in section 2, this gave testing a direction to
uncover errors instead of not finding them. The limitation of the ideas in this paper is also analyzed in
previous section. Their research led to a series of successive research on the theory of testing techniques.
In the same year, Huang pointed out that the common test data selection criterion – having each and every
statement in the program executed at least once during the test – leaves some important classes of errors
undetected [10]. As a refinement of statement testing criterion, edge strategy, was given. The main idea
for this strategy is to exercise every edge in the program diagraph at least once. Probe insertion, a very
useful testing technique in later testing practices was also given in this research.
Another significant test selection strategy, the path testing approach, appeared in 1976. In his research,
Howden gave the strategy that test data is selected so that each path through a program is traversed at least
once [8]. Since the set of program paths is always infinite, in practice, only a subset of the possibly infinite
set of program paths can be tested. Studies of the reliability of path testing are interesting since they
provide an upper bound on the reliability of strategies that call for the testing of a subset of a program’s
paths.
The year 1980 saw two important theoretical studies for testing techniques, one on functional testing and
one on s tructural.
Although functional testing had been widely used and found useful in academic and industry practices,
there was little theoretical research on functional testing. The first theoretical approach towards how
systematic design methods can be used to construct functional tests was given in 1980 [9]. Howden
discussed the idea of design functions, which often correspond to sections of code documented by
comments, which describe the effect of the function. The paper indicates how systematic design methods,
such as structured design methodology, can be used to construct functional tests.
The other 1980 research was on structural testing. If a subset of a program’s input domain causes the
program to follow an incorrect path, the error is called a domain error. Domain errors can be caused by
incorrect predicates in branching statements or by incorrect computations that affect variables in branch
statement predicates. White and Cohen proposed a set of constraints under which to select test data to find
domain errors [18]. The paper also provides useful insight into why testing succeeds or fails and indicates
directions for continued research.
The dawn of data flow analysis for structural testing was in 1985. Rapps and Weyuker gave a family of
test data selection criteria based on data flow analysis [16]. They contend the defect of path selection
criteria is that some program errors can go undetected. A family of path selection criteria is introduced
followed by a discussion of the interrelationships between these criteria. This paper also addresses the
problem of appropriate test data selection. This paper founded the theoretical basis for data-flow based
program testing techniques.
In 1989, Richardson and colleagues proposed one of the earliest approaches focusing on utilizing
specifications in selecting test cases [14]. In traditional specification-based functional testing, test cases are
selected by hand based on a requirement specification, thus makes functional testing consist merely
heuristic criteria. Structural testing, on the other hand, has the advantage of that the applications can be
automated and the satisfaction determined. The authors extended implementation-based techniques to be
applicable with formal specification languages and to provide a testing methodology that combines
7
9. specification-based and implementation-based techniques. This research appeared to be the first to
combine the idea of structural testing, functional testing, and formal specifications.
Another research towards specification-based testing appeared in 1990. Boolean algebra is the most basic
of all logic systems and many logical analysis tools incorporate methods to simplify, transform, and check
specifications. Consistency and completeness can be analyzed by using Boolean algebra, which can also be
used as a basis for test design. The functional requirements of many programs can be specified by decision
tables, which provide a useful basis for program and test design. Boolean algebra is trivialized by using
Karnaugh-Veitch charts. Beizer described the method of using decision tables and Karnaugh-Veitch
charts to specify functional requirements in his book [2]. This was among the earliest approaches to select
test based on existing, well-known Boolean algebra logic in test data selection.
In the early 1990s, there was an increasing interest in estimating and predicting the reliability of software
systems. Before Jalote and colleagues’ study in 1994 [11], many existing reliability models employed
functional testing techniques and predicted the reliability based on the failure data observed during testing.
The application of these models requires a fair amount of data collection, computation, and expertise and
computation for interpreting the results. The authors propose a new approach based on the coverage
history of the program, by which a software system is modeled as a graph, and the reliability of a node is
assumed to be a function of the number of times it gets executed during testing-the larger the number of
times a node gets executed, the higher its reliability. The reliability of the software system is then
computed through simulation by using the reliabilities of the individual nodes. With such a model,
coverage analysis tools can easily be extended to compute the reliability also, thereby fully automating
reliability estimation.
The year 1997 saw good results in both functional and structural testing techniques. In this year a
framework for probabilistic functional testing was proposed. Bernot and colleagues introduce the
formulation of the testing activity, which guarantees a certain level of confidence into the correctness of the
system under test, and gives estimate of the reliability [1]. They also explain how one can generate
appropriate distributions for data domains including most common domains such as intervals of integers,
unions, Cartesian products, and inductively defined sets.
Another interesting research in 1997 used formal architectural description for rigorous, automatable method
for integration test of complex systems [4]. In this paper the authors propose to use the formal specification
language CHAM to model the behavior of interest of the systems. Graph of all the possible behaviors of
the system in terms of the interactions between its components is derived and further reduced. A suitable
set of reduced graphs highlights specific architectural properties of the system, and can be used for the
generation of integration tests according to a coverage strategy, analogous to the control and data flow
graphs in structural testing. This research is among the trend of using formal methods in testing techniques
since later 1980s.
From late 1990s, Commercial Off The Shelf (COTS) software and Unified Modeling Language (UML)
UML have been used by increasing number of software developers. This trend in development therefore
calls the corresponding testing techniques for the UML components. In their 2000 paper [7], Hartmann and
colleagues at Siemens addressed the issue of testing components by integrating test generation and test
execution technology with commercial UML modeling tools such as Rational Rose. The authors present
their approach to modeling components and interactions, describe how test cases are derived from these
component models and then executed to verify their conformant behavior. The T environment of
nT
Siemens is used to evaluate the approach by examples. Test cases are derived from annotated Statecharts.
Another most recent research is also an approach to test component-based software. In 2001, Beydeda and
colleagues proposed a graphical representation of component-based software flow graph [3]. Testing is
made complicated with features, such as the absence of component source code, that are specific to
component-based software. The paper proposes a technique combining both black-box and white-box
strategies. A graphical representation of component software, called component-based software flow graph
(CBSFG), which visualizes information gathered from both specification and implementation, is described.
It can then be used for test case identification based on well-known structural techniques. Traditional
8
10. structural testing techniques can be applied to this graphical representation to identify test cases, using data
flow criterion. The main components are still tested with functional techniques.
5 Technology maturation
The maturation process of how the testing techniques for software have evolved from an ad hoc, intuitive
process, to an organized, systematic technology discipline is shown in Figure 3. Three models are used to
help analyzing and understanding the maturation process of testing techniques. A brief introduction on
these models followed.
5.1 Redwine/Riddle software technology maturation model
The Redwine/Riddle is the backbone of this maturation study. In 1985, Redwine and Riddle viewed the
growth and propagation of a variety of software technologies in an attempt to discover natural
characteristics of the process as well as principles and techniques useful in transitioning modern software
technology into widespread use. They looked at the technology maturation process by which a piece of
technology is first conceived, then shaped into something usable, and finally “marketed” to the point that it
is found in the repertoire of a majority of professionals. Six phases are described in Redwine/Riddles
model:
Phase 1: Basic research.
Investigate basic ideas and concepts, put initial structure on the problem, frame critical research
questions.
For testing techniques, the basic research phase ended in the mid to late 1950s, when the targets of
testing were set to make sure the software satisfies its specification. Initial researches began to try to
formulate the basic principles for testing.
Phase 2: Concept formulation.
Circulate ideas informally, develop a research community, converge on a compatible set of ideas,
publish solutions to specific subproblems.
The concept formulation process of testing focused between late 1950s and mid 1970s, when the
arguments of the real target of testing came to an end and theoretical researches began. The milestone
of the end of this phase is the publication of Goodenough and Gerhart’s paper on testing data selection.
Phase 3: Development and extension.
Make preliminary use of the technology, clarify underlying ideas, and generalize the approach.
The years between mid 1970s and late 1980s are the development and extension phase of testing
techniques. Various testing strategies are proposed and evaluated for both functional and structural
testing during this period. Principles of testing techniques were built up gradually during this period.
Phase 4: Internal enhancement and exploration.
Extend approach to another domain, use technology for real problems, stabilize technology, develop
training materials, show value in results.
Starting from late 1980s, researches on software testing techniques entered the internal exploration
stage. The principles defined during the last period were accepted widely and used in larger scales of
real problems, technologies began to stabilize and expand. People can now hire testing specialists or
9
11. companies to do the testing job for them. More and more courses and training programs are given in
universities and companies.
Phase 5: External enhancement and exploration.
Similar to internal, but involving a broader community of people who weren’t developers, show
substantial evidence of value and applicability.
Testing techniques started their prosperity in mid 1990s. Along with the advance of programming
languages, development methods, abstraction granularity and specification power, testing became
better facilitated and challenged with more complex situations and requirements of different kinds of
systems. Effective specification and development methods are borrowed employed during testing.
Phase 6: Popularization.
Develop production-quality, supported versions of the technology, commercialize and market
technology, expand user community.
Although the testing activities have been carried out everywhere and all the time, we don’t think
research on testing techniques have entered its popularization period. Testing techniques vary with
different systems and development methods. Even if specialist companies can be hired to do testing
job, there lacks a common basis for the industry to share and utilize academic research result, as well
as to use good practices from other enterprises. We are going to discuss this in future sections.
5.2 Brief History of Software Engineering
The evolution of software testing cannot be separated with activities in other areas of software engineering,
such as programming language, formal specification, and software architecture. In her paper, Shaw
summarizes the significant shift in research attention of software engineering, which can be viewed as a
brief history of this broad area [12]. Starting from mid 1950s, the change of software engineering research
paradigms can be described as shown in Table 1.
Table 1. Significant shifts in research attention of software engineering
Mnemonics, precise use of prose
1960 ¹ 5 Emphasis on small programs
Programming-any-which-way Representing structure, symbolic information
Elementary understanding of control flow
Simple input-output specifications
Emphasis on algorithms
1970 ¹ 5
Programming-in-the -small Data structures and types
Programs execute once and terminate
Systems with complex specifications
Emphasis on system integration, management
1980 ¹ 5
Programming-in-the -large Long-lived databases
Program assemblies execute continually
Software integrated with hardware
Emphasis on management process improvement, system structure
1990 ¹ 5
Programming-in-the -world Abstractions for system design (client-server, object…)
Heavily interactive systems, multimedia
10
12. Time Functional Structural Time
2001: spec + impl.based for cmponent-based software
2000 2000: UML based 2000
1997: Probablistic 1997: Architectural
1988~ Approach Emphasis: Program
Preventi Specification ming-in-
1994: Coverage
on- & System the-
Model for Reliability
Oriented world
1991: Integrate test activities during software life-cycle
1990 1990: Logic-based 1990
1989: spec-based and impl-based using formal methods
1983~87
Evalutati
on- 1985: Data Flow Strategy
Oriented
1979~82
Destructi Program
on- 1980 1980: Design Functions 1980: Domain Strategy 1980 ming-in-
Oriented 1979: Association of Testing to fault detection Emphasis: the-large
Implementation
& Program
1976: Path Strategy
1975: Edge Strategy
Program
ming-in-
1970 1970
1957~78 the-
Demons small
tration-
Oriented
Program
ming-
1960 1960 any-
which-
Ad Hoc way
1957: Distinguish debugging from testing
~1956
Debuggi
ng- 1950 1950: Testing to know a program satisfies requirements 1950
Oriented
Evolution of Redwine/Riddles Testing Technique Technology History of
Testing Concept Technology Maturity Big Ideas Evolution Software Engineering
Model
Key: Stages of Redwine/Riddles Model
External Exploration Internal Exploration Development&Extension Concept Formation Basic Research
Figure 3. Technology maturation analysis of software testing techniques
11
13. 5.3 Testing Process Models
The third reference model is the major testing goals model given by Gelperin and Hetzel in [6]. We have
discussed this in detail in our previous section of testing concept evolution. Understanding how the goals
of testing has changed helps us a lot to understand the intellectual history of testing technique research,
how new ideas are based on old ideas from the same area or on the studies from other areas.
The Redwine/Riddle model provides benchmarks for judging where a research stands in the maturation
process, and how a process can migrate to the next one. Shaw’s model sets up the global background
where studies of technology maturation can be put into and compared with the rest of the world of software
engineering. Gelperin and Hetzel’s model is specific in testing so that the objectives of a testing technique
research are made clear. Each model assists us to understand the problem from a different point of view,
but none of them alone fully explains the unique phenomena in the technology history of test techniques.
We put our study in the framework built with these models, and give our own viewpoints.
5.4 The Major Stages of Research & Development Trends
Generally, we see three major stages of the research and development of testing techniques, each with a
different trend. By trend we mean the how mainstream of research and development activities find the
problems to solve, and how they solve the problems. As is shown in the column of “Technology
Evolution” in Figure 3, testing technique technologies, thus the ways of selecting test data have developed
from ad hoc, experienced implementation-based phase, and is focusing on specification-based now.
1950 – 1970: Ad Hoc
From Figure 3 we can see that between the years 1950 and 1970, there were few research results on
testing techniques except for the conceptual ideas of testing goals. It’s possible that research results
before 1970 are too old to be in the reach of current bibliography collections. To avoid being
influenced by this factor, we looked at many testing survey papers in the 1980s, which should have had
the “ancient” studies in hand by the time they performed their study. We suppose their surveys at least
addressed the most important technical contributes before their time, and we can build our research for
the decades before 1970 on theirs.
Based on above assumption, we define the period between 1950 and 1970 as being ad hoc. During this
period, major research interest focuses on the goal of testing, and there are quite a few discussions on
how to evaluate if a test is good. Meanwhile, testing had become gradually independent from part of
debugging activities, to a necessary way to demonstrate that a program satisfies its requirements, as is
seen in the GH88 model. At the same time, if we look from Shaw’s view, we can see that the whole
world of software engineering was in its programming-in-any-which-way stage. It’s very natural that
testing stayed in its ad hoc stage, where test data is selected randomly and in an unorganized,
undirected way.
1971 – 1985: Emphasize on Implementation and Single Program
Beginning from the mid 1960s to the mid 1980s, the whole software engineering research community
shifted it paradigms to the program-in-the-small stage, and then started the program-in-the-large stage.
The main changes this migration brought to software development were that the characteristic
problems changed from small programs, to larger programs and algorithms, and were on the way to
developing more complex problems. In response to this significant change, researches on testing
techniques began their prosperity.
On the structural side, in 1975, 1976, 1980, and 1985, four significant papers were published, each
proposing a very important structural testing strategy, and all of which were adopted as the classic
criteria for later researches followed them. From Figure 3 we can see that almost all the important
12
14. theoretical structural testing researches appeared in this period. Although the whole software
engineering community was facing the challenge of switching the gear of developing from comparably
simple programs to complex large systems, it took time for testing community to react to the change,
specifically, in approximately 5 years.
From the figure we also find that only one significant result for functional testing appeared in this
period. The reason is obvious. Functional testing is based on requirements and has consisted merely
of heuristic criteria. It is difficult to determine when and if such criteria are satisfied without being
able to exp ress the requirements in an efficient, rigorous, unambiguous way. This was in part the
motivation for developing implementation-based testing techniques; they have the advantage that their
application can be automated and their satisfaction determined. F ortunately the research appeared
during this period set up a very good tone of successive researches, since it moved emphasis from the
simple input/output specifications that testers often used in this period to a higher level – the design of
the system.
In this period, how to test a “program”, instead of a “system”, still drew the attention of researchers
and practitioners. However the whole software engineering had begun to get ready for moving from
the stage of programming-in-the-large to a higher level.
1986 – current: Emphasize on Specification and System
As software become more and more pervasive, the engineering for this area experienced the shift from
programming-in-the-large to programming-in-the-world, starting from the mid 1980s. The
characteristic problems changed from algorithms, to system structures, and component interfaces.
Systems have been specified in more complex ways. Studies in software architecture and formal
methods have brought a lot of facilities as well as inspiration to the way people specifying their
systems. Based on these studies, software system now can be specified in more rigorous,
understandable, automatable ways, which has brought great chances to improve functional testing
techniques. Meanwhile, software development is no longer limited to standalone systems, in reality,
there have been more and more needs to develop distributed, object-oriented, and component based
systems. The researchers in testing community have responded this trend and move their emphasis
accordingly.
Starting from the late 1980s, many researches have made use of the achievements of formal methods
and logical analysis. There is still limitation in the specification capabilities so that researchers have
been calling for better specification methods to improve their results. Both functional and structural
testing techniques have benefited from the enhancement of software specification technologies.
The widespread developing and using of object-oriented technologies, COTS software and component-
based systems has brought a great density of testing researches on these kinds of systems. The earliest
OO testing studies appeared in the early 1990s. Most of them use traditional functional and/or
structural techniques on the components, i.e. classes and so on. Researchers have proposed new
problems and solutions on testing the connections and inheritances among components. Both
structural and functional techniques are hired in their approaches, and it has proven to be an effective
method to integrate the two techniques for testing complex systems.
5.5 The Test Gap
In present, testing techniques have gradually involved from the practice of single programmers or small
development teams into a systematic, managed engineering discipline. Not only have there been numerous
researches on testing techniques, but also more and more considerable industry practices. There are testing
classes taught in universities. There have been special testing teams, test managers, and tester job positions
open to professional testers; there have been training programs and complete procedures for testing in large
enterprises; and there are increasing number of companies and vendors doing testing work for other
companies.
13
15. However, despite the numerous research results (quite a lot of them are really sound) testing remains an
awkward, time -consuming, cost-ineffective chunk of work that is always not very satisfying in most
industry practices. Only a small number of the research results have been utilized successfully in industry
practices so that the test process can be greatly improved or automated. The most common testing
exercises in industry are static analysis including code inspections, peer reviews, walkthroughs. Not
enough testing tools can be applied directly on industry projects and products without being largely
modified and even re-developed. Test plans are still written by hand, while test environment remained
simple and crude. It is always the case that testing ends up being a must-end activity because the project
runs out of budget and is beyond deadline. This inconsistency of testing research and practice has been
called the “testing gap.”
Provided many other issues involved in the testing gap, such as process and management, further
researches in testing technique are among the most significant solutions that will work for the problem.
Fundamental researches in techniques need to:
• Demonstrate effectiveness of existing techniques
• Address the need in new areas
• Create new adaptive techniques
• Facilitate transferring technology to industry
5.6 Research Strategies for Testing Techniques
We have studied the research strategies of twelve influential papers since the year 1975 and tried to find out
the common form and successful examples of research settings that offer concrete guidance for future
research work. The results are listed in Table 2. It’s not hard to find out that most of researches on testing
techniques are motivated by questioning if there is a better method of doing something. The question is
answered by inventing, implementing, combining, refining, evaluating, alternating or proposing new ways
to do this task, and the result gets analyzed through analysis most of the times. Combined with the test gap
mentioned in last section, we contend that fundamental researches should address the challenges testing
techniques are facing in the real world, generalize them, and pursue practical solutions for them. Research
should be carried out with industry partners on real world problems, instead of simple toy systems.
Researchers in academic community and in industry should talk often to address the need for each other.
Table 2. Paradigms of testing technique researches
Paper Research Paradigm
Ref. Year Age Idea
Question Result Validation
GG75 1975 26 Fundamental theorem Evaluation Analytic Model Analysis
Huang75 1975 26 Edge approach, probe insertion Method/Means Technique Analysis
Howden76 1976 25 Path approach and its reliability Characterization Technique Analysis
WC80 1980 21 Domain testing strategy Method/Means Technique Analysis
Howden80 1980 21 Functional design abstraction Method/Means Technique Persuasion
RW85 1985 16 Data flow strategy Method/Means Technique Analysis
ROT89 1989 12 Integrate spec. and impl. testing Method/Means Technique Analysis
JM94 1994 7 Coverage reliability estimation Method/Means Technique Analysis
BBL97 1997 4 Probabilistic functional testing Method/Means Technique Analysis
BIMR97 1997 4 Testing based on architectural Method/Means Technique Persuasion
HIM00 2000 1 UML based testing Method/Means Technique Experience
BG01 2001 0 Component based testing Method/Means Technique Analysis
6 Conclusion
Testing has been widely used as a way to help engineers develop high-quality systems, and the techniques
for testing have evolved from an ad hoc activities means of small group of programmers to an organized
discipline in software engineering. However, the maturation of testing techniques has been fruitful, but not
adequate. Pressure to produce higher-quality software at lower cost is increasing and existing techniques
used in practice are not sufficient for this purpose. Fundamental research that addresses the challenging
14
16. problems, development of methods and tools, and empirical studies should be carried out so that we can
expect significant improvement in the way we test software. Researchers should demonstrate the
effectiveness of many existing techniques for large industrial software, thus facilitating transfer of these
techniques to practice. The successful use of these techniques in industrial software development will
validate the results of the research and drive future research. The pervasive use of software and the
increased cost of validating it will motivate the creation of partnerships between industry and researchers to
develop new techniques and facilitate their transfer to practice. Development of efficient testing techniques
and tools that will assist in the creation of high-quality software will become one of the most important
research areas in the near future.
7 Annotated bibliography
[1] G. Bernet, L. Bouaziz, and P. LeGall, “A Theory of Probabilistic Functional Testing,” Proceedings of
the 1997 International Conference on Software Engineering, 1997, pp. 216 –226
[BBL97] A framework for probabilistic functional testing is proposed in this paper. The authors
introduce the formulation of the testing activity, which guarantees a certain level of confidence into the
correctness of the system under test. They also explain how one can generate appropriate distributions
for data domains including most common domains such as intervals of integers, unions, Cartesian
products, and inductively defined sets. A tool assisting test case generation according to this theory is
proposed. The method is illustrated on a small formal specification.
Question: Method/Means
Result: Technique
Validation: Analysis
[2] B. Beizer, “Software Testing Techniques,” Second Edition, Van Nostrand Reinhold Company Limited,
1990, ISBN 0-442-20672-0
[Beizer90] This book gives a fairly comprehensive overview of software testing that emphasizes
formal models for testing. The author gives a general overview of the testing process and the reasons
and goals for testing. In the second chapter of this book, the author classifies the different types of
bugs that could arise in program development. The notion of path testing, transaction flowgraphs,
data-flow testing, domain testing, and logic-based testing are introduced in detail in the chapters
followed. The author also introduces several attempts to quantify program complexity, and more
abstract discussion involving paths, regular expression, and syntax testing. How to implement
software testing based on the strategies is also discussed in the book.
[3] S. Beydeda and V. Gruhn, “An integrated testing technique for component-based software,” ACS/IEEE
International Conference on Computer Systems and Applications, June 2001, pp 328 – 334
[BG01] Testing is made complicated with features, such as the absence of component source code, that
are specific to component-based software. The paper proposes a technique combining both black-box
and white-box strategies. A graphical representation of component software, called component-based
software flow graph (CBSFG), which visualizes information gathered from both specification and
implementation, is described. It can then be used for test case identification based on well-known
structural techniques.
Question: Method/Means
Result: Technique
Validation: Analysis
[4] A. Bertolino, P. Inverardi, H. Muccini, and A. Rosetti, “An approach to integration testing based on
architectural descriptions,” Proceedings of the IEEE ICECCS- 97, pp. 77-84
15
17. [BIMR97] In this paper the authors propose to use formal architectural descriptions (CHAM) to model
the behavior of interest of the systems. Graph of all the possible behaviors of the system in terms of
the interactions between its components is derived and further reduced. A suitable set of reduced
graphs highlights specific architectural properties of the system, and can be used for the generation of
integration tests according to a coverage strategy, analogous to the control and data flow graphs in
structural testing.
Question: Method/Means
Result: Technique
Validation: Persuasion
[5] J.B. Good Enough and S. L. Gerhart, “Toward a Theory of Test Data Selection,” IEEE Transactions on
Software Engineering, June 1975, pp. 156-173
[GG75] This paper is the first published paper, which attempted to provide a theoretical foundation for
testing. The “fundamental theorem of testing” brought up by the authors characterizes the properties
of a completely effective test selection strategy. The authors think a test selection strategy is
completely effective if it is guaranteed to discover any error in a program. As an example, the
effectiveness of branch and path testing in discovering errors is compared. The use of decision table (a
mixture of requirements and design-based functional testing) as an alternative method is also proposed.
Question: Evaluation
Result: Analytic Model
Validation: Analysis
[6] D. Gelperin and B. Hetzel, “The Growth of Software Testing”, Communications of the ACM, Volume
31 Issue 6, June 1988, pp. 687-695
[GH88] In this article, the evolution of software test engineering is traced by examining changes in the
testing process model and the level of professionalism over the years. Two phase models, the
demonstration and destruction models, and two life cycle models, the evolution and prevention models
are given to characterize the growth of software testing with time. Based on the models a prevention
oriented testing technology is introduced and analyzed in detail.
Question: Characterization
Result: Descriptive Model
Validation: Persuasion
[7] J. Hartmann, C. Imoberdorf, and M.Meisinger, “UML-Based Integration Testing,” Proceedings of the
International Symposium on Software Testing and Analysis, ACM SIGSOFT Software Engineering Notes,
August 2000
[HIM00] Unified Modeling Language (UML) is widely used for the design and implementation of
distributed, component-based applications. In this paper, the issue of testing components by
integrating test generation and test execution technology with commercial UML modeling tools such
as Rational Rose is addressed. The authors present their approach to modeling components and
interactions, describe how test cases are derived from these component models and then executed to
verify their conformant behavior. The TnT environment of Siemens is used to evaluate the approach
by examples
Question: Method/Means
Result: Technique
Validation: Experience
[8] W. E. Howden, “Reliability of the Path Analysis Testing Strategy”, IEEE Transactions on Software
Testing, September 1976, pp. 208-215
16
18. [Howden76] The reliability of path testing provides an upper bound for the testing of a subset of a
program’s paths, which is always the case in reality. This paper begins by showing the impossibility
of constructing a test strategy that is guaranteed to discover all errors in a program. Three commonly
occurring classes of errors, computations, domain, and subcase, are characterized. The reliability
properties associated with these errors affect how path testing is defined.
Question: Characterization
Result: Technique
Validation: Analysis
[9] W. E. Howden, “Functional Testing and Design Abstractions,” The Journal of System and Software,
Volum 1, 1980, pp. 307-313
[Howden80] The usual practice of functional testing is to identify functions that are implemented by a
system or program from requirements specifications. In this paper, the necessity of testing design as
well as requirement functions is discussed. The paper indicates how systematic design methods, such
as Structured design and the Jackson design can be used to construct functional tests. Structured
design can be used to identify the design functions that must be tested in the code, while the Jackson
method can be used to identify the types of data which should be used to construct tests for those
functions.
Question: Method/Means
Result: Technique
Validation: Persuasion
[10] J. C. Huang, “An Approach to Program Testing,” ACM Computing Surveys, September 1975, pp.113-
128
[Huang75] This paper introduces the basic notions of dynamic testing based on detailed path analysis
in which full knowledge of the contents of the source program being tested is used during the testing
process. Instead of the common test criteria by which to have every statement in the program executed
at least once, the author suggested and demonstrated by an example, that a better criterion is to require
that every edge in the program diagraph be exercised at least once. The process of manipulating a
program by inserting probes along each segment in the program is suggested in this paper.
Question: Method/Means
Result: Technique
Validation: Analysis
[11] P. Jalote and Y. R. Muralidhara, “A coverage based model for software reliability estimation,”
Proceedings of First International Conference on Software Testing, Reliability and Quality Assurance,
1994, pp. 6 – 10 (IEEE)
[JM94] There exist many models for estimating and predicting the reliability of software systems, most
of which consider a software system as a black box and predict the reliability based on the failure data
observed during testing. In this paper a reliability model based on the software structure is proposed.
The model uses the number of times a particular module is executed as the main input. A software
system is modeled as a graph, and the reliability of a node is assumed to be a function of the number of
times it gets executed during testing – the larger the number of times a node gets executed, the higher
its reliability. The reliability of the software system is then computed through simulation by using the
reliabilities of the individual nodes.
Question: Method/Means
Result: Technique
Validation: Analysis
17
19. [12] J. J. Marciniak, “Encyclopedia of software engineering”, Volume 2, New York, NY: Wiley, 1994, pp.
1327-1358
[Marciniak94] A book intended for software engineers, this books gives introductions, overviews, and
technical outlines of the major areas in software engineering. A review in to test generators is given
where the major types of test case generators are given and their intended purpose and principles are
discussed. A review on the testing process is given where the entire process of testing is discussed
from planning to execution to achieving to maintenance retesting. All the common terms and ideas are
discussed. A review of testing tools is given where the testing tools for each purpose is discussed and
a couple for state of the art systems are given.
[13] E. F. Miller, “Introduction to Software Testing Technology,” Tutorial: Software Testing & Validation
Techniques, Second Edition, IEEE Catalog No. EHO 180-0, pp. 4-16
[Miller81] This article serves as the one of the introductory sections of the book Tutorial: Software
Testing & Validation Techniques. A cross section of program testing technology before and around
the year 1980 is provided in this book, including the theoretical foundations of testing, tools and
techniques for static analysis and dynamic analysis, effectiveness assessment, management and
planning, and research and development of soft ware testing and validation. The article briefly
summarizes each of the major sections. The article also gives good view of the motivation forces, the
philosophy and principles of testing, and the relation of testing to software engineering.
[14] D. Richardson, O. O’Malley and C. Tittle, “Approaches to specification-based testing”, ACM
SIGSOFT Software Engineering Notes, Volume 14 , Issue 9, 1989, pp. 86 – 96
[ROT89] This paper proposes one of the earliest approaches focusing on utilizing specifications in
selecting test cases. In traditional specification-based functional testing, test cases are selected by hand
based on a requirement specification, thus makes functional testing consist merely heuristic criteria.
Structural testing has the advantage of that the applications c be automated and the satisfaction
an
determined. The authors propose approaches to specification-based testing by extending a wide
variety of implementation-based testing techniques to be applicable to formal specification languages,
and demonstrate these approaches for the Anna and Larch specification languages.
Question: Method/Means
Result: Technique
Validation: Analysis
[15] S. Redwine & W. Riddle, “Software technology maturation,” Proceedings of the Eighth International
Conference on Software Engineering, May 1985, pp. 189-200
[RR85] In this paper, a variety of software technologies are reviewed. The technology maturation
process by which a piece of technology first gets the idea formulated and preliminarily used, then is
developed and extended into a broader solution, and finally is enhanced to product-quality applications
and marketed to the public. The time required for a piece of technology to mature is studied, and the
actions that can accelerate the maturation process are addressed. This paper serves as a very good
framework for technology maturation study.
Question: Characterization
Result: Empirical Model
Validation: Analysis
[16] S. Rapps and E. J. Weyuker, “Selecting Software Test Data Using Data Flow Information,” IEEE
Transactions on Software Engineering¸ April 1985, pp. 367-375
18
20. [RW85] A family of test data selection criteria based on data flow analysis is defined in this paper.
The authors contend that data flow criteria are superior to currently path selection criteria being used in
that using the latter strategy program errors can go undetected. Definition/use graph is introduced and
compared with a program graph based on the same program. The interrelationships between these data
flow criteria are also discussed.
Question: Method/Means
Result: Technique
Validation: Analysis
[17] M. Shaw, “Prospects for an engineering discipline of software,” IEEE Software, November 1990, pp.
15-24
[Shaw90] Software engineering is still on its way of being a true engineering discipline. This article
studies the model for the evolution of an engineering discipline and applies it to software technology.
Five basic steps are suggested to the software profession to take towards a true engineering discipline:
to understand the nature of expertise, to recognize different ways to get information, to encourage
routine practice, to expect professional specializations, and to improve the coupling between science
and commercial practice. The significant shifts in research attention of software engineering since the
1960s are also given in this article.
Question: Characterization
Result: Descriptive Model
Validation: Persuasion
[18] L. J. White and E. I. Cohen, “A Domain Strategy for Computer Program Testing,” IEEE Transactions
on Software Engineering, May 1980, pp. 247-257
[WC80] Domain errors are in the subset of the program input domain, and can be caused by incorrect
predicates in branching statements or incorrect computations that affect variables in branching
statements. In this paper a set of constraints under which it’s possible to reliably detect domain errors
is introduced. The paper develops the idea of linearly bounded domains. The practical limitations of
the approach are also discussed, of which the most severe is that of generating and then developing test
points for all boundary segments of all domains of all program paths.
Question: Method/Means
Result: Technique
Validation: Analysis
[19] J. A. Whittaker, “What is Software Testing? And Why Is It So Hard?” IEEE Software, January 2000,
pp. 70-79
[Whit00] Being a practical tutorial article, the paper answers questions from developers how bugs
escape from testing. Undetected bugs come from executing untested code, difference of the order of
executing, combination of untested input values, and untested operating environment. A four-phase
approach is described in answering to the questions. By carefully modeling the software’s
environment, selecting test scenarios, running and evaluating test scenarios, and measuring testing
progress, the author offers testers a structure of the problems they want to solve during each phase.
Question: Characterization
Result: Qualitative & Descriptive Model
Validation: Persuasion
19