Software testing

Uploaded on


  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Software Testing Testing is the activity where the errors remaining from all the previous phases must be detected. Hence testing plays a critical role for ensuring quality. Testing is of two types: • Software inspections. Concerned with analysis of the static system representation to discover problems (static verification) – May be supplement by tool-based document and code analysis • Software testing. Concerned with exercising and observing product behaviour (dynamic verification) – The system is executed with test data and its operational behaviour is observed Static and dynamic testing S ftw re o a ins e ns p ctio Rq e uire e m nts Hig ve h-le l Fo a rm l De ile ta d P g m ro ra se p cifica n tio ds n e ig se p cification ds n e ig P to e ro typ P gra ro m te ting s Types of testing • Defect testing – Tests designed to discover system defects. – A successful defect test is one which reveals the presence of defects in a system. – Covered in Chapter 23 • Validation testing – Intended to show that the software meets its requirements. – A successful test is one that shows that a requirements has been properly implemented. Statistical Use testing It is a part of Validation testing which amounts to testing software the way users intend to use it Procedure: • The specification for each increment is analyzed to define a set of stimuli (inputs or events) that cause the software to change it’s behavior • Test cases are generated for each set of stimuli according to the usage probability distribution
  • 2. Testing and debugging • Defect testing and debugging are distinct processes. • Verification and validation is concerned with establishing the existence of defects in a program. • Debugging is concerned with locating and repairing these errors. • Debugging involves formulating a hypothesis about program behaviour then testing these hypotheses to find the system error. The debugging process Ts et Te t s Se p cifica n tio re ults s ca e ss Lo te ca De ig s n R p ir ea R te t e s e o rr r e o re a rr r p ir e o rr r p gr m ro a Software inspections • These involve people examining the source representation with the aim of discovering anomalies and defects. • Inspections not require execution of a system so may be used before implementation. • They may be applied to any representation of the system (requirements, design, configuration data, test data, etc.). • They have been shown to be an effective technique for discovering program errors. Inspection success • Many different defects may be discovered in a single inspection. In testing, one defect , may mask another so several executions are required. • The reuse domain and programming knowledge so reviewers are likely to have seen the types of error that commonly arise. The inspection process P nning la Ov rvie e w F llow o -up Ind vid l i ua R w rk e o p p ra n re a tio Ins e n p ctio mee ting
  • 3. Inspection procedure • System overview presented to inspection team. • Code and associated documents are distributed to inspection team in advance. • Inspection takes place and discovered errors are noted. • Modifications are made to repair discovered errors. • Re-inspection may or may not be required. Inspection checklists • Checklist of common errors should be used to drive the inspection. • Error checklists are programming language dependent and reflect the characteristic errors that are likely to arise in the language. • In general, the 'weaker' the type checking, the larger the checklist. • Examples: Initialisation, Constant naming, loop termination, array bounds, etc. Inspections and testing • Inspections and testing are complementary and not opposing verification techniques. • Both should be used during the V & V process. • Inspections can check conformance with a specification but not conformance with the customer’s real requirements. • Inspections cannot check non-functional characteristics such as performance, usability, etc. Automated static analysis • Static analysers are software tools for source text processing. • They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team. • They are very effective as an aid to inspections - they are a supplement to but not a replacement for inspections. Stages of static analysis • Control flow analysis. Checks for loops with multiple exit or entry points, finds unreachable code, etc. • Data use analysis. Detects uninitialized variables, variables written twice without an intervening assignment, variables which are declared but never used, etc. • Interface analysis. Checks the consistency of routine and procedure declarations and their use • Information flow analysis. Identifies the dependencies of output variables. Does not detect anomalies itself but highlights information for code inspection or review • Path analysis. Identifies paths through the program and sets out the statements executed in that path. Again, potentially useful in the review process • Both these stages generate vast amounts of information. They must be used with care. Use of static analysis • Particularly valuable when a language such as C is used which has weak typing and hence many errors are undetected by the compiler, • Less cost-effective for languages like Java that have strong type checking and can therefore detect many errors during compilation. Verification and formal methods • Formal methods can be used when a mathematical specification of the system is produced.
  • 4. • They are the ultimate static verification technique. • They involve detailed mathematical analysis of the specification and may develop formal arguments that a program conforms to its mathematical specification. Dynamic Testing During dynamic testing, the software to be tested is executed with a set of test cases, and the behavior of the system for the test cases is evaluated to determine if the system is performing as expected. The behavior is evaluated using test oracles which are considered to have the correct answer. These can be humans who decide the correct behavior of the system ,or can be generated automatically using requirement specifications. Test cases are given to the test oracle (if it is a human) and the program under testing. The output of the two is compared to find if program behaved correctly for the test cases. test cases The V-model of development Rq e e e uir m nts S te ys m Sys m te De ile ta d se p cifica n tio se p cifica n tio ds n e ig ds n e ig S te ys m S -s te ub ys m Mo ulea d nd Acce ta p nce inte tio gra n inte tio gra n unit co e d te t p n s la te t p n s la te t p n s la a te t nd s Acce ta p nce Sys m te S -s te ub ys m S rvice e te t s inte tio te t gra n s inte a n te t gr tio s Psychology of testing The basic purpose of testing is to detect the errors that may be present in the program. Hence the intent should be to show that a program does not work, thus an intent of finding errors. That is why most of the organizations, require a product to be tested by people not involved with developing the program before finally delivering it to the customer. Also at times errors occur because the programmer did not understand the specification correctly. Thus testing by external person helps in resolving such errors. This approach is suitable for earlier stages, where the objective is to reveal errors. However last stages are meant for evaluating the product. Here test cases are selected to mimic user scenario. Objectives of testing 1. Software Quality Improvement. Quality means conformance to the specified software design requirements. 2. Verification and validation: To ensure conformity with requirements.
  • 5. 3. Software reliability estimation Test design Test design refers to understanding the sources of test cases, test coverage, how to develop and document test cases, and how to build and maintain test data. Two methods of test design are: 1. Black-box testing 2. White-box testing Black box testing In black box testing, the tester only knows the inputs that can be given to the system and what output the system should give. Also known as functional or behavioral testing, in black-box testing, the basis for deciding test cases in functional testing is the requirements or specifications of the system or module. To decide the test cases we use the following methods: 1. Equivalence class partitioning: It breaks classes into small parts and selects test cases from each class, such partitioned. An equivalence class is formed of the inputs for which the behavior is expected to be similar. Each group of inputs for which the behavior is expected to be different from others is considered a separate equivalence class. Equivalence classes are usually formed by considering each condition specified on an input as specifying a valid equivalence class and one or more invalid equivalence classes. Another approach is to consider any special case where behavior could be different. 2. Graph based testing methods or cause-effect testing is a technique that aids in selecting combinations of input conditions in a systematic way, such that the number of test cases does not become unmanageably large. Thus for n input conditions there will be 2n test cases. A cause is a distinct input condition, and an effect is a distinct output condition. The conditions are set such that they can be set to either true or false. A test case is generated for each combination of conditions, which make some effect true. 3. Boundary value analysis selects 7 cases for all boundary value variables. The seven values being min-1, min, min+1, max-1, max, max+1, nominal value. Thus for n variables there would be 7n test cases 4. Specialized testing where special cases are provided for. It also includes testing of documentation, real time systems, inter task testing etc. Advantages 1. Tester and programmer are independent 2. Testing from user’s point of view 3. Test cases can be prepared right after specifications are decided. Disadvantages i. Duplication of test cases by tester and programmer ii. Only sample testing can be done as all cases cannot be built and tested. White-Box testing
  • 6. White box testing, also known as structural testing, is concerned with the implementation of the requirements. It evaluates the programming structures and data structures used in the program. The criterion for deciding test cases for white-box testing are: 1. Control flow based criterion: The control flow graph of a program is considered and coverage of various aspects of the graph are specified. The coverage criterions are: 1. Statement coverage requires that the paths executed during testing include all the nodes in the graph. 2. Branch coverage requires that each edge in the control flow graph be traversed at least once. 3. Path coverage requires all possible paths in the control flow graph to be executed during testing. 2. Data flow based testing is conducted by creating a definition-use graph for the program. A variable occurs in one of the three ways: • def refers to the definition of the variable • C-use represents computational use of the variable • P-use represents predicate use, which is used for transfer of control Based on these various paths can be defines, such as • Global c-use • Def-clear • Dcu(x, i) set of nodes such that each node has x is in c-use • Dpu(x,i) set of nodes such that each node has x is in p-use • All-p-uses includes a def-clear path w.r.t x from I to dpu(x, i) • All-c-uses includes a def-clear path w.r.t x from I to dcu(x, i) • Advantages • Reveals errors in hidden code • Forces reason in implementation • Disadvantages • Expensive • Cases omitted can miss out code Mutation testing It is performed by creating many mutants in the code by making simple changes to the program. By testing we try to identify errors. If all the mutants planted are found, then we can assume that test cases are sufficient, otherwise more test cases are created unless we find all the mutants. Steps for mutant testing are as follows: • Generate mutants for P. Suppose there are N mutants • By executing each mutant and P on each test case in T, find how many mutants can be distinguished by T. Let D be the number of mutants distinguished or dead. • For each mutant that cannot be distinguished by T (live), find how many are equivalent to P. Let E be the number of equivalent mutants. • The mutation score is computed as D/(N-E) • Add more test cases to T and continue testing until the mutation score is 1. Test case specification
  • 7. Requirement number Condition to be tested Test data and settings Expected o Error, fault and failure • Error refers to the discrepancy between a computed, observed or measured value and the true, specified, or theoretically correct value. • Fault is a condition that causes a system to fail in performing its required function. It is the basic reason for the software malfunction and is synonymous with the term bug. • Failure is the inability of a system or component to perform a required function according to its specifications. Presence of an error implies that a failure must have occurred, and the observance of a failure implies a fault must be present. However, the presence of a fault does not imply that a failure must occur. During the testing process, only failures are observed, by which the presence of faults is deduced. The actual faults are identified by separate activities, commonly referred to as “debugging”. Defect logging and tracking Severity of defect can be categorized as follows: • Critical: Show stopper; affects a lot of users; can delay project • Major: Has a large impact but workaround exists • Minor: An isolated defect that manifests rarely and with little impact • Cosmetic: Small mistakes that don’t impact the correct working Defect analysis and prevention techniques • Pareto Analysis states that 80% of the problems come from 20% of the possible sources. Once these sources are identified, similar test cases can be prepared for the rest of the program. • Casual analysis are drawn using cause-effect diagram. After the problem is stated, the major categories of causes are identified. Then further brainstorming helps identify sub-causes to these major causes. • Develop and implement solution attacks the root cause of the defects. Preventive solutions devised are implemented as project activities, assigned to team members. The effect of such solutions is monitored.
  • 8. Types of testing • Unit testing is performed at the time of development. Errors found at this level can be debugged easily. Each module is tested against the specifications produced during design for the modules. It is typically done by the programmer. Some examples of unit testing are: • Black-box approach – Field level check – Field level validation – User interface check – Function level check • White box approach – Statement coverage – Decision coverage – Condition coverage – Path coverage Advantages of unit testing – Can be applied directly to object code – Performance profilers implement this measure Disadvantages – Not implementable to logical operators – Insensitive to loop operators (number of iterations) • Integration testing is performed to put together all the modules developed and unit tested. It is done in either top-down method or bottom up method • System testing is conducted after all the modules have been integrated and is followed by performance testing, where the actual performance of the system is tested. System testing is concerned with the compliance of requirements regarding the system. Techniques for selecting test cases for system testing are as follows: • Testing the system’s capabilities is more important than testing its components; thus focus shifts on catastrophic failure than minor irritants • Testing the usual is more important than testing the exotic • If we are testing after modification, we should test old capabilities rather than new ones. Testing techniques can be structural or functional. Some of these are discussed as follows Structural techniques for system testing: • Stress testing: It executes the system in a manner that demands resources in abnormal quantity, frequency or volume. It decides the higher limit parameters for the system. • Recovery testing forces the software to fail in a variety of ways and verifies that the system recovers in a proper manner. • Compliance testing is conducted to ensure that the system confirms to the standards laid down by the quality management system. • Operations testing is performed to ensure that the system fits correctly in the organizations ‘operations. Functional techniques for system testing:
  • 9. • Security testing verifies that protection mechanism built in the system will protect it from improper penetration. The role of system designer is to make penetration cost more than the value of the information that will be obtained. It can also be a part of structural techniques. • Manual support testing includes user documentation required by the clients and ensures correct support is available. • Control testing checks the control mechanism for correctness • Parallel testing is conducted to ensure that all the versions have the required functionalities and correctly display the changes incorporated in each version. • Acceptance testing is performed with realistic data of the client to demonstrate that the software is working satisfactorily. Testing here focuses on the external behavior of the system. There are two types of acceptance testing: • Alpha testing is performed at the developer’s site, but the user is present for testing • Beta testing is performed at the client site and the system is tested for acceptability Cleanroom software development • The name is derived from the 'Cleanroom' process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal. • This software development process is based on: – Incremental development; – Formal specification; – Static verification using correctness arguments; – Statistical testing to determine program reliability. Fo a y rm ll Err r re o o w rk se p cify s te ys m De fine C ns ct o tru F rm ll y o a Inte te gra s ftwa o re structured v rify e incre e m nt incre e m nts p g m ro ra co e d De e p v lo o e a na p r tio l De ig s n Ts et p file ro s tica tatis l inte te gra d te ts s s te ys m Cleanroom process characteristics • Formal specification using a state transition model. • Incremental development where the customer prioritises increments. • Structured programming - limited control and abstraction constructs are used in the program. • Static verification using rigorous inspections. • Statistical testing of the system
  • 10. Formal specification and inspections • The state based model is a system specification and the inspection process checks the program against this model • The programming approach is defined so that the correspondence between the model and the system is clear. • Mathematical arguments (not proofs) are used to increase confidence in the inspection process. Cleanroom process teams • Specification team. Responsible for developing and maintaining the system specification. • Development team. Responsible for developing and verifying the software. The software is NOT executed or even compiled during this process. • Certification team. Responsible for developing a set of statistical tests to exercise the software after development. Reliability growth models used to determine when reliability is acceptable. Certification Certification requires the creation of three models 1. Sampling model: Software testing executes m random test cases and is certified if no failures or a certified number of failures occur. 2. Component model: A system composed of n components is to be certified to determine the probability that component I will fail prior to completion. 3. Certification model: The overall reliability of the system is projected and certified. Cleanroom process evaluation • The results of using the Cleanroom process have been very impressive with few discovered faults in delivered systems. • Independent assessment shows that the process is no more expensive than other approaches. • There were fewer errors than in a 'traditional' development process. • However, the process is not widely used. It is not clear how this approach can be transferred to an environment with less skilled or less motivated software engineers.