UNIT IIIL E V E LS O F T E S T I N G After the individual components have been tested, and any necessary repairs made, they are integrated to build subsystems and clusters. Testers check for defects and adherence to specifications. Proper interaction at the component interfaces is of special interest at the integration level. In most cases black box design predominates, but often white box tests are used to reveal defects in control and data flow between the integrated modules. System test begins when all of the components have been integrated successfully; it usually requires the bulk of testing resources. At the system level the tester focus on evaluating performance, usability, reliability, and other quality-related requirements. If the system is being custom made for an individual client then the next step following system test is acceptance test. (Very important testing stage for developers) Final payment depends on the quality of the software as observed during the acceptance test.
Software developed for the mass market often goes through a series of tests Called alpha and beta tests. Alpha tests bring potential users to the developer’s site to use the software. Beta tests send the software out to potential users who use it under real-world Conditions and report defects to the developing organization.Levels of Testing and Software Development Paradigms The approach used to design and develop a software system has an impact on how testers plan and design suitable tests. Major approaches to system development – bottom-up & top-down Programming languages supported – procedural and object-oriented Hence, testers require to use different strategies to identify and test components and component groups. Procedural language supported systems – composed of passive data and active procedures Test cases focus on generating input data to pass to the procedures (or functions) in order to reveal defects. Object oriented systems – composed of active data along with allowed operations on that data, all encapsulated within a unit similar to an abstract data type. Testing means designing an order of calls to the operations using various parameter values in order to reveal defects. Levels of abstraction are also different for the two systems.
Testing approaches to procedural systems are well established. Object-oriented systems – still research is ongoing w.r.t. testing approaches. Encapsulation – hide details from testers, and that can lead to uncovered code. Inheritance – retesting of inherited methods when they are used by a subclass in a different context Difficult to define a unit – method (small in size and developing test harness is a large overhead) Message passing, Dynamic binding, state changes, and nonhierarchical calling relationships => makes testing more complex.Unit Test: Functions, Procedures, Classes, and Methods as Units A unit is the smallest possible testable software component. – Unit • Function or procedure implemented in a procedural language • Method and class/object in object-oriented systems • Small-sized COTS components • Simple module retrieved from an in-house reuse libraryUnit Test: The Need for Preparation Good testing practice calls for unit tests that are planned and public. Planning includes designing tests to reveal defects such as – Functional description defects, – Algorithmic defects, – Data defects, and
– Control, logic, and sequence defects. Resources should be allocated, and test cases should be developed, using both white and black box test design strategies. Independent tester Also reviewed by a team of reviewers, preferably before the unit test But, the testing approach may be adhoc and defects found are not recorded. Poor practice for mission or safety critical tasks. If defects escape detection in unit test because of poor unit testing practices, they are likely to show up during integration, system, or acceptance test where they are much more costly to locate and repair. To prepare for unit tests, the developer/tester must perform several tasks. These are, – plan the general approach to unit testing; – design the test cases, and test procedures (these will be attached to the test plan); – define relationships between the tests; – prepare the auxiliary code necessary for unit test.Unit test planning General unit test plan should be prepared. Developed in conjunction with the master test plan and the project plan for each project. Inputs for the unit test plan are the project plan, as well the requirements, specification, and design documents that describe the target units. IEEE Standard for Software Unit Testing – rich in information and is an excellent guide for test planner. Brief description of a set of development phases for unit test planning is found below:Phase 1: Describe Unit Test Approach and Risks – Test risks identified – Describing techniques for Test case design – Describing techniques for data validation and recording of test results – Describing requirements for test harness Also, planner identifies – Completeness requirements (what will be covered by the unit test & to what degree) – Termination conditions (special cases – abnormal termination of unit test)
– Resources needed for unit test, such as software, hardware, and staff and develops a tentative schedule under the constraints identified at that timePhase 2: Identify Unit Features to be tested – Functions, performance requirements, states, and state transitions, control structures, messages, and data flow patterns.Phase 3: Add Levels of Detail to the Plan – Existing test cases that can be reused for this project can be identified in this phase. – Test-related documents that will be required for this task, for example, test logs, and test incident reports, should be described, and references to standards for these documents are provided.Designing the Unit tests Part of the preparation work for unit test involves unit test design Important to specify (i) Test cases and (ii) Test procedures Test case data should be tabularized for ease of use, and reuse. For Object-Oriented test design, and the organization of test data, Berard has described a test case specification notation. The components of a test case are arranged into a semantic network with parts, - Object_ID, - Test_case_ID, - Purpose, and - List_of_Test_case_steps Test design specification also includes lists of relevant states, messages, exceptions, and interrupts. Testers should also describe the relationships between the tests. Test suites can be defined that bind related tests together as a group. All of this test design information is attached to the unit test plan. Test cases, test procedures, and test suites may be reused from past projects if the organization has been careful to store them so that they are easily retrievable and reusable. Both black and white-box test design approaches are useful for designing test cases, functions, procedures and member functions in class.
Use white-box test design as the unit is small in size. Tester has the opportunity to exercise logic structures and/or data flow sequences, or to use mutation analysis, all with the goal of evaluating the structural integrity of the unit. COTS components – black-box test design approach is the only option. Units that perform mission/safely/business critical functions, it is often useful and prudent to design stress, security, and performance tests at the unit level if possible.The Class as a Testable Unit: Special Considerations Unit – Choice of individual method or class as a whole. Methods – In the case of the method as the selected unit to test, it may call other methods within its own class to supports its functionality. - built additional codes (test harness) to represent the called methods within the class. - This test harness is costly but can be applied for mission or safety critical methods. Classes - The process of testing classes as units is sometimes called component test. - A class encapsulates multiple interacting methods operating on common data, so what we are testing is the intra-class interaction of the methods. Testing on the class level, detects the defects due to the nature of object-oriented systems, for example, defects due to encapsulation, inheritance, and polymorphism errors. Basic trade-offs in selecting the component (class) to be considered for a unit test in object- oriented systems is as follows:Issue 1: Adequately Testing Classes Many methods (20 to 30) in a class lead to high cost for testing individually. If class is selected as a unit, the cost will be reduced and the methods would serve as stubs and drivers for one another. This has the effect of lowering the complexity of test harness that has to be developed. In some cases, driver classes that represent outside classes using the methods of the class under test will have to be developed. When testers unit (or component) tests this class, they focus on the operation of each of the methods in the class and the interactions between them. To properly test this class, a sequence of calls to the methods needs to be specified as part
of component test design. For example, a test sequence for a stack that can hold three items might be: Create(s, 3), empty(s), push(s, item-1), push(s, item-2), push(s, item-3), full(s), show_top(s), pop(s, item), pop(s,item), pop(s,item), empty(s), . . . Many different sequences and combination of calls are possible even for this simple class. Exhaustively testing every possible sequence is usually not practical. The tester must select those sequences she believes will reveal the most defects in the class. Tester uses a combination of approaches and tests some critical methods as individual units and then testing the class as a whole.Issue 2: Observation of Object States and State Changes State-based testing is very useful for testing objects. The state of an object is represented by a specific set of values for its attributes or state variables. Methods will often modify the state of an object, and the tester must ensure that each state transition is proper. Test designer prepare a state table – states, the objects can assume, indicate sequence of messages and parameters that will cause the object to enter each state. When the tests are run, the tester can enter the results in this same type of table. Example: First call to the method push -> changes the state of the stack so that empty is no longer true. Also changes the value of the stack pointer variable, top. To determine if the method push is working properly the value of the variable top must be visible both before and after the invocation of this method. In this case the method show_top within the class may be called to perform this task. The methods full and empty also probe the state of the stack A sample augmented sequence of calls to check the value of top and the full/empty state of the three-item stack is: empty(s), push(s,item-1), show_top(s), push(s,item-2), show_top(s), push(s,item-3), full(s), show_top(s), pop(s,item), show_top(s), pop(s,item), show_top(s), empty(s), . . . Test sequences also need to be designed to try to push an item on a full stack and pop an
item from an empty stack. In the case of the stack class, the class itself contains methods that can provide information about state changes. If this is not the case then additional classes/methods may have to be created to show changes of state. (Test Harness) Another option is to include in each class, methods that allows state changes to be observable. Testers should have the option of going back to the designers and requesting changes that make a class more testableIssue 3: The Retesting of Classes—I Class (the unit) - built with a well-defined public interface that declares its services (available methods) to client classes. Implementation of services is private (encapsulation) and the clients are unaware of the implementation details. – As long as the interface is unchanged, making changes to the implementation should not affect the client classes. Tester would therefore conclude that only the class with implementation changes to its methods needs to be retested. This is not correct. In an object-oriented system, if a developer changes a class implementation that class needs to be retested as well as all the classes that depend on it. If a super class, for example, is changed, then it is necessary to retest all of its subclasses. – Antidecomposition testing axiomIssue 4: The Retesting of Classes—II Classes have existing inheritance relationships. Subclasses inherit methods from their super classes. Methods tested in super class need not be retested in sub classes that inherit it. In some cases the method is used in a different context by the subclass and will need to be retested. In addition, there may be an overriding of methods where a subclass may replace an inherited method with a locally defined method. Retest and add a set of new test cases
Example – Shape super class – Suppose the shape super class has a subclass, triangle, and triangle has a subclass, equilateral triangle. – Also suppose that the method display in shape needs to call the method color for its operation. This is a new context that must be retested; a set of new test cases should be developed. The tester must carefully examine all the relationships between members of a class to detect such occurrences.The Test Harness(i) call the target unit;(ii) do 1, and pass inputs parameters from a table;(iii) do 1, 2, and display parameters;(iv) do 1, 2, 3 and display results (output parameters).
The stubs could also exhibit different levels of functionality. A stub could:(i) display a message that it has been called by the target unit;(ii) do 1, and display any input parameters passed from the target unit;(iii) do 1, 2, and pass back a result from a table;(iv) do 1, 2, 3, and display result from table.Drivers and Stubs Developed as procedures and functions for traditional procedural systems. Developed as special classes for object-oriented systems. The test harness itself is a hierarchy of classes Test harness for individual classes – more complex than those needed for individual procedures and functions (more interactions has to be considered).
At any level of testing, – The tester should carefully record, review, and check test results. – The tester must determine from the results whether the unit has passed or failed the test.If the test is failed, the nature of the problem should be recorded in a test incident report.The most likely reason for the failure is a fault in the unit implementation (the code).Other likely causes that need to be carefully investigated by the tester are the following: – a fault in the test case specification – a fault in test procedure execution – a fault in the unit designThe causes of the failure should be recorded in a test summary report, which is asummary of testing activities for all the units covered by the unit test plan.At some situations, a unit may be given a conditional acceptance for integration test.This may occur when unit fails some tests, but the impact is not significant w.r.t its functionin a subsystem and the availability is critical for integration test to proceed on schedule.But, it faces risks. So, testers should evaluate risksUnits with a conditional pass must eventually be repaired.
Designing Integration Tests For procedural systems, both black-box and white-box test designs are used Many errors occur at unit interfaces – hence, test designers need to focus on exercising all input/output parameter pairs, and all calling relationships; also insure the parameters are of correct type and in correct order. Example program in fig. 6.9: – Procedure_a calls Procedure_b with two input parameters in3, in4. – Procedure_b uses those parameters and then returns a value for the output parameter out1. – The parameters could be involved in a number of def and/or use data flow patterns. – The actual usage patterns of the parameters must be checked at integration time – Data flow–based (def-use paths) and control flow (branch coverage) test data generation methods are useful here to insure that the input parameters, in3, in4, are used properly in Procedure_b. – Black box tests are useful for checking the behavior of the pair of procedures
For conventional systems – input/output parameters and calling relationships will appear in a structure chart built during detailed design – Test cases are designed so that all modules in the structure chart are called at least once, and all called modules are called by every caller -> coverage criteria for integration test. – When units are integrated and subsystems are to be tested as a whole, new tests will have to be designed to cover their functionality and adherence to performance and other requirements – Sources – requirements documents and user manuals• Black-box test – developed to insure proper functionality and ability to handle subsystem stress.Integration Test Planning Planning can begin when high-level design is complete. Other documents relevant to integration test planning are the requirements document, the user manual, and usage scenarios
These documents contain structure charts, state charts, data dictionaries, cross-referencetables, module interface descriptions, data flow descriptions, messages and eventdescriptions, all necessary to plan integration tests.For procedural systems – the order of integration of the units should be defined based onthe strategy selected.For object-oriented systems – a working definition of a cluster or similar construct must bedescribed, and relevant test cases must be specified.Testing resources and schedules for integration – included in the test plan.Cluster test plan: – clusters this cluster is dependent on; – a natural language description of the functionality of the cluster to be tested; – list of classes in the cluster; – a set of cluster test cases.Planning should begin at the requirements phase with the development of a master test planand requirements-based (black box) tests.Many components of the plan need to be prepared such as test approaches, costs, schedules,test cases, and test procedures. Hence, complicated task.Requires a large amount of resources, special laboratory equipment, and long test times,hence, it is usually performed by a team of testers. (Independent testing group)Evaluates both functional behavior and quality requirements such as reliability, usability,performance and securityEspecially useful for detecting external hardware and software interface defectsThen the software will be turned over to users for evaluation during acceptance test oralpha/beta test.Several types of system tests exist: – Functional testing – Performance testing – Stress testing – Configuration testing – Security testing – Recovery testing
– Reliability testing – Usability testingNot all software systems need to undergo all the types of system testing.Test planners need to decide on the type of tests applicable to a particular software system.For example, if multiple device configurations not required for a system – configurationtesting is not needed.As the system has been assembled from its component parts, many of the system tests havebeen implemented on the component parts and subsystems.However, during system test the testers can repeat these tests (considered as regressiontests) and design additional tests for the system as a whole.A careful effort at system test is essential for high software quality.Properly planned and executed system tests are excellent preparation for acceptance test.Beizer provides additional material on the different types of system tests – Paper and online forms – Requirements traceability matrix – Test logsImportant tool for implementing system tests – load generatorA load generator is essential for testing quality requirements such as performanceand stress.A load is a series of inputs that simulates a group of transactions.A transaction is a unit of work seen from the system user’s view. A transaction consists of aset of operations that may be performed by a person, software system, or a device that isoutside the system.A use case can be used to describe a transaction.Eg. System testing a telecommunication system – Simulate a series of phone calls of particular types and lengths arriving from different locationsLoad generators will generate test input data for system test. It also outputs a fixed set ofpredetermined transactionsUsers of the load generators can usually set various parameters.
For example, in a telecommunication system, the parameters set are – the mean arrival rate of the calls, – the call durations, – the number of wrong numbers and misdials, and – the call destinations. Usage profiles, and sets of use cases can be used to set up loads for use in performance, stress, security and other types of system test.Functional testing – To ensure the behavior of the system that holds to the requirements specification – All functional requirements must be achievable by the system. – Eg. Personal finance system – to allow users to setup accounts, add, modify, and delete entries, print reports. – Black box in nature – The focus is on the inputs and proper outputs for each function. Improper and illegal inputs must also be handled by the system – It should be designed at requirements time, and be included in the master and system test plans. – Equivalence class partitioning and boundary-value analysis are useful methods that can be used to generate test cases. State-based tests are also considered. – The tests should focus on the following goals. • All types or classes of legal inputs must be accepted by the software. • All classes of illegal inputs must be rejected. • All possible classes of system output must exercised and examined. • All effective system states and state transitions must be exercised and examined. • All functions must be exercised.
Test planning Test plan components - described in IEEE STD 829-1983 -Test plan identifier - Introduction - Items to be tested - Features to be tested - Approach - Pass/fail criteria - Suspension and resumption criteria - Test deliverables - Testing Tasks - Test environment - Responsibilities - Staffing and training needs - Scheduling
- Risks and contingencies - Testing costs - Approvals Test plan identifier– Unique, associated with a specific project and become a part of the project history.– Project history stored in project database or under the control of Configuration Management System.– Organizational standards should describe the format for the test plan Identifier; also test plan is likely to change.– If a configuration management system is used, the test plan identifier can serve to identify it as a configuration item. Introduction– Overall description of the project– High-level description of testing goals and the testing approaches to be used.Items to be tested– Listing of the entities to be tested and should include names, identifiers, and version/revision numbers for each entity.– Items listed could include procedures, classes, modules, libraries, subsystems, and systems.– References to the appropriate documents where these items and– Their behaviors are described such as requirements and design documents, and the user manual should be included in this component of the test plan.– These references support the tester with traceability tasks.– The focus of traceability tasks is to ensure that each requirement has been covered with an appropriate number of test cases.Testing costs – Test costs are usually allocated in the overall project management plan. – Project manager in consultation with developers and testers. – Estimates the costs of testing. – If the test plan is an independent document prepared by the testing group and has a cost component, the test planner will need tools and techniques to help estimate
test costs. – Test costs that should include in the plan are: • costs of planning and designing the tests; • costs of acquiring the hardware and software necessary for the tests (includes development of the test harnesses); • costs to support the test environment; • costs of executing the tests; • costs of recording and analyzing test results; • tear-down costs to restore the environment. – Other costs are, costs of training the testers, and the costs of maintaining the test database. When estimating testing costs, the test planner should consider organizational, project, and staff characteristics that impact on the cost of testing.Several key characteristics of test cost impact items are: – The nature of the organization; its testing maturity level, and general maturity. – The nature of the software product being developed - the testing scope for a business application will be smaller than one for a mission or safely critical system, since in case of the latter there is a strong possibility that software defects and/or poor software quality could result in loss of life or property. – The scope of the test requirements. – The level of tester ability. – Knowledge of the project problem domain. – The level of tool support. – Training requirements. Project planners have cost estimation models – COCOMO model (used to estimate overall project costs) At this time models of this type have not been designed specifically for test cost estimation. However, test planners often borrow cost estimation techniques and models from project planners and apply them to testing.
The utility of these data items will be demonstrated in the following paragraphs whichdescribe approaches to test cost estimation based on: (i) the COCOMO model and heuristics; (ii) use of test cost drivers; (iii) test tasks; (iv) tester/developer ratios; (v) expert judgment (Delphi).The test planner can use the COCOMO model to estimate total project costs, and thenallocate a fraction of those costs for test.Application of the COCOMO model is based on a group of project constants that dependon the nature of the project and items known as cost drivers.A cost driver can be described as a process or product factor that has an impact on overallproject costs.In the testing domain cost drivers are factors that contribute to testing costs.There should be a correlation between the factors and testing costs and also some causalconnection between them.To use the COCOMO model a project manager must first estimate the size of the newproject and identify its type.The simple COCOMO equation used for an initial estimate isE = a (size in KLOC) bE is estimated effort in man-months,
a and b are constants Cost drivers for the project include: – product attributes such as the required level of reliability; – hardware attributes such as memory constraints; – personnel attributes such as experience level; – project attributes such as use of tools and methods. The project cost drivers are rated on an ordinate scale and folded into what Boehm calls an effort adjustment factor (EAF). The results from equation (1) can be multiplied by the EAF to give a revised estimate. Unfortunately, no work has been done at this time to formalize them into a COCOMO- like model for testing. After the total project costs have been estimated with COCOMO, the test planner can then use a heuristic that estimates testing costs for a new project as some fraction of total project costs.Locating Test Items: The Test Item Transmittal Report• Tester is ready to run tests on an item on the date described in the test plan and locate the item and have knowledge of its current status. This is the function of the Test Item Transmittal Report.• This document is not a component of the test plan, but is necessary to locate and track the items that are submitted for test.• It has a unique identifier and should contain the following information for each item that is tracked (i) version/revision number of the item; (ii) location of the item; (iii) persons responsible for the item (e.g., the developer); (iv) references to item documentation and the test plan it is related to; (v) status of the item; (vi) approvals—space for signatures of staff who approve the transmittal. Reporting test results • Additional documents related to testing are prepared during and after execution of
the tests. • The IEEE Standard for Software Test Documentation describes the following documents:Test log • Prepared by the person executing the tests. • It is a diary of the events that take place during the test. • It supports the concept of a test as a repeatable experiment • The test log is invaluable for use in defect repair. • It gives the developer a snapshot of the events associated with a failure. • The test log, in combination with the test incident report which should be generated in case of anomalous behavior, gives valuable clues to the developer whose task it is to locate the source of the problem. • Prevent incorrect decisions based on incomplete or erroneous test results. • Retest that follows defect repair is also supported by the test log. • In addition, the test log is valuable for o regression testing that takes place in the development of future releases of a software product, and o circumstances where code from a reuse library is to be reused. • The IEEE Standard for Software Test Documentation describes the test log as a chronological record of all details relating to the execution of its associated tests. Test log has the following sections: Test Log Identifier Description - the tester should identify the items being tested, their version/revision number, and their associated Test Item/Transmittal Report. Activity and Event Entries - should provide dates and names of test log authors for each event and activity Execution description: Procedure results:
Environmental information: Anomalous events: Incident report identifiers:Test Incident report (problem report) – It should record any event that occurs during the execution of the tests that is unexpected, unexplainable, and that requires a follow-up investigation.The IEEE Standard recommends the following sections in the report: – Test Incident Report identifier – Summary – Incident description - should describe time and date, testers, observers, environment, inputs, expected outputs, actual outputs, anomalies, procedure step, environment, and attempts to repeat. – Impact - A severity rating should be inserted here.Test Summary Report- summary of the results of the testing efforts- Part of the project’s historical database and provides a basis for lessons learned as applied to future projects-The IEEE test documentation standard describes the following sections: Test Summary Report identifier Variances - Deviations and reasons for the deviation from the test plan, test procedures, and test designs are discussed. Comprehensiveness assessment - the document author discusses the comprehensiveness of the test effort as compared to test objectives and test completeness criteria as described in the test plan. Summary of results Evaluation Summary of activities - Resource consumption, actual task durations, and hardware and software tool usage should be recorded Approvals
The Role of the Three Critical Groups in Testing Planning and Test Policy Development Three groups were identified as critical players in the testing process Managers, Developers/Testers, and Users/Clients In TMM terminology they are called the three critical views (CV) Each group views the testing process from a different perspective that is related to their particular goals, needs, and requirements. The developers/testers work with client/user groups on quality-related activities and tasks that concern user-oriented needs. The focus is on soliciting client/user support, consensus, and participation in activities such as requirements analysis, usability testing, and acceptance test planning.
In the following paragraphs contributions to achievement of the managerial-orientedmaturity goals by the three critical views is discussed.For the TMM maturity goal, ―Develop Testing and Debugging Goals,‖ the TMMrecommends that project and upper management: Provide access to existing organizational goal/policy statements and sample testing policies , and from other sources. These serve as policy models for the testing and debugging domains. Provide adequate resources and funding to form the committees (team or task force) on testing and debugging. Committee makeup is managerial, with technical staff serving as co members. Support the recommendations and policies of the committee by: – distributing testing/debugging goal/policy documents to project managers, developers, and other interested staff, – appointing a permanent team to oversee compliance and policy change making.
Ensure that the necessary training, education, and tools to carry out defined testing/debugging goals is made available. Assign responsibilities for testing and debugging. As representatives of the technical staff developers must ensure that the policies reflect best testing practices, are implementable, receive management support, and support among technical personnel.The activities, tasks, and responsibilities for the developers/testers include: Working with management to develop testing and debugging policies and goals. Participating in the teams that oversee policy compliance and change management. Familiarizing themselves with the approved set of testing/debugging goals and policies, keeping up-to-date with revisions, and making suggestions for changes when appropriate. When developing test plans, setting testing goals for each project at each level of test that reflect organizational testing goals and policies. Carrying out testing activities that are in compliance with organizational policies. The goals and policies of users and clients reflect the organizations efforts to ensure customer/client/user satisfaction. ―Initiate a Test Planning Process,‖ the second management-oriented maturity goal at TMM level 2, also requires input from the three critical groups.Upper management supports this goal by: Establishing an organization wide test planning committee with funding. Ensuring that the testing policy statement and quality standards support test planning with commitment of resources, tools, templates, and training. Ensuring that the testing policy statement contains a formal mechanism for user input to the test planning process, especially for acceptance and usability testing. Ensuring that all projects are in compliance with the test planning policy. Ensuring that all developers/testers complete all the necessary post test documents such as test logs and test incident reports. Project managers support the test planning maturity goal by preparing the test plans for each project with inputs and support from developers. Developers who are experienced in testing support this maturity goal by participating in
test planning. From the user/client point of view support for test planning is in the form of articulating their requirements clearly, and supplying input to the acceptance test plan.Process and the Engineering Disciplines: One of our major focuses as engineers is on designing, implementing, managing, and improving the processes related to software development. Testing is such a process. An engineer can serve as the change agent, using his education in the area of testing to form a process group or to join an existing one. The engineer can initiate the implementation of a defined testing process by working with management and users/clients toward achievement of the technical and managerial-oriented maturity goals.