Software Testing Foundations #2Testing in the Software LifecycleNikita Knyshnknysh@gmail.comhttp://www.facebook.com/groups/istqb/
Agenda• Levels of Test: The V-model• Testing New Product Versions• Types of Testing• Testing Related to Changes
Development Activities• Req specification: needs & reqs of the customer, desired characteristics and features of the system.• Func system design: reqs are mapped onto system functions & dialogs.• Technical system design: implementation of system is designed (architecture, interfaces, etc.).• Component specification: task, behavior, inner structure & interfaces are defined for each subsystem.• Programming.
Levels of Test: The V-model Requirements Acceptance Definition Test Functional System Test System Design Technical Integration System Design Test Component Component Specification Test Implementation & Integration Validation Programming Verification
V-model: Test Activities #1• Component test: test functionality, robustness, efficiency and maintainability of components. Techniques: white box, black box, test-driven development.• Integration test: test of collaboration between components after assembling to expose faults in interfaces and interaction between components. Test drivers, stubs and monitors are used. Integration in the large – integration of several software systems. ▫ Integration strategy has to be designed by test manager taking into account system architecture, project plan and test plan constraints. Top-down, bottom-up, ad- hoc, backbone.
V-model: Test Activities #2• System test: check if the integrated product meets the specified requirements. System is examined from the point of view of customer and future user. Environment should be as close to the operation environment as possible, but this should NOT be the customer’s operational environment.• Acceptance test: “Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.” [After IEEE 610]. Customer must be involved.
V-model: Acceptance Test on Lower Levels• A commercial off-the-shelf software product (COTS) may be acceptance tested when it is installed or integrated.• Acceptance testing of new functionality may come before system testing (using a prototype).
V-model: Forms of Acceptance Test• According to contract• User acceptance testing• Operational testing• Field testing ▫ alpha – at the producer’s site ▫ beta – at the customer’s siteExtent of acceptance test depends on application risk.
Testing New Product Versions• Software maintenance: ▫ testing after maintenance (anything new or changed is tested), ▫ testing after change of environment, ▫ testing for retirement (data archiving or data migration into future system).• Release: full regression test (if possible) to find side effects of changes to the software.
Testing New Product Versions #2• Testing in incremental development: several V- models after each other, where every next "V" reuses existing test material and adds the tests for everything new, or for higher reliability require-ments.
Types of Testing: Overview• Functional• Non-functional• Testing of Software Structure• Testing related to Changes
Types of Testing: FunctionalTest what the system does. Tests are based onfunctional specs, check system’s input-output behaviorand use black-box methods.• Requirements-based testing focuses on single system functions.• Business-process-based testing focuses on whole process consisting of many steps.
Types of Testing: Non-functional #1Test how well the (partial) system carries out itsfunctions.[ISO 9126]: reliability, usability, efficiency,maintainability, portability.
Types of Testing: Non-functional #2• Load test: measuring system behavior when working with the number of users, number of transaction at a time.• Performance test: measuring system processing speed and response time (depending on increasing load).• Volume test: observation of system behavior on large amount of data (e.g. processing very large files).• Stress test: observation of system behavior when it is overloaded.• Security test: against unauthorized access, denial of service attacks, etc.• Stability (reliability) test during permanent operation, e.g. time between failures or failure rate within a given user profile.
Types of Testing: Non-functional #3• Robustness test for response to operating errors, wrong programming hardware failure, etc., as well as examination of exception handling and recovery.• Testing of compatibility (to other systems) and data conversion (import / export).• Testing of different configurations of the system, e.g. OS version, UI language, hardware platform, etc.• Usability test: ease of learning the system, ease and efficiency of operation, etc. with respect to a specific group of users.• Checking of documentation for compliance with the system behavior (e.g. user manual and GUI).• Checking of maintainability: is documentation understandable and up-to-date? Does the system have modular structure, etc.
Types of Testing: Testing SoftwareStructureUsing white box techniques to test object’s internalcode or architecture.• Goal is to cover all structure items.• Most used in component and integration testing.
Testing Related to Changes• Re-test of changed (modified or repaired) parts.• Regression test of whole program (part) after modification to ensure that no faults are introduced as side effects of changes.
Volume of Regression Test• Defect retest (confirmation test): rerunning of tests that detected faults that have been then fixed.• Testing of altered functionality: testing of all program parts that were changed or corrected.• Testing of new functionality: testing of all newly integrated program parts.• Complete regression test: testing of the whole system.
Shrinking Regression Test EffortDeciding which test cases we can omit:• Only high-priority test cases (according to test plan) are repeated.• In the functional test, certain variations (test cases) are omitted.• Restriction of the test to certain configurations only (on one OS only, in one language only, etc.)• Restriction of the test to certain subsystems or test levels.