Agenda• Classes of Test Tools ▫ Test Management and Control Tools ▫ Test Specification Tools ▫ Tools for Static Testing ▫ Tools for Dynamic Test ▫ Tools for Non-functional Test• Selecting Test Tools• Introducing Test Tools
Test Management and Control Tools• Administering test cases and results of their execution• Testing resource & schedule planning• Advanced test management tools support requirements-based testing; requirements management; incident management; configuration management• Tool integration (req / test mgmt <-> test execution <-> config mgmt)• Generating test reports and test documentation.
Test Specification Tools• Test (data) generators ▫ Database-based ▫ Code-based ▫ Interface-based ▫ Specification-based – the one generating expected outputs!
Tools for Static Testing• Review support• Static analysis• Model checking tools
Tools for Dynamic Test• Debuggers (not really a test tool)• Test drivers (test harnesses), test harness and stub generators• Simulators• Test robots (data-driven testing, command- or keyword-driven testing, interaction method – object-oriented management of test modules, so called interactions, in a DB)• Comparators (compare something static with expected results)• Dynamic analysis tools• Coverage analyzers (insert measurement code into the test object prior to test run – make this code ‘covered’).
Tools for Non-functional Tests• Load and performance test tools• Monitors• Security may be checked using even virus scanners and firewalls (their logs).
Basics of Tools Introduction #1• Automating chaos just gives faster chaos!• Recommended order of tool introduction (by type): ▫ Incident management ▫ Configuration management ▫ Test planning ▫ Test execution ▫ Test specification
Basics of Tools Introduction #2• Some time is necessary to learn the tool and establish its use so productivity may even decline. It is risky to introduce a new tool during ‘hot’ project phrases.• Without good test procedures or well-established test methods, tools do NOT lead to the desired cost reduction.
Test Tool Costs and Benefits• Make a cost-benefit analysis of a tool taking into account selection, acquisition and maintenance costs, as well as time frame needed for the tool to start paying back.• Benefits include not only time savings but also larger number of found bugs and higher trustworthiness of tests (so there are savings on development, support and maintenance).• Extra test programming effort does not pay back after one test run but only after many regression test runs (if tests are programmed for easy use and maintenance).• Some tests (like performance test) can only be run automatically.
Use of Tools• Creative test activities can be supported by tools.• Mechanical test execution can be automated.
Test Tool Selection Process1. Requirement specification for the tool application.2. Market research (creating an overview of possible candidates).3. Tool demonstrations and creating a short list.4. Evaluating the tools on the short list.5. Review results and selection of the tool.
Test Tool Introduction Success Factors• Stepwise introduction (including a pilot project).• Integration with processes.• User training and continuous coaching.• Make use experiences available to all users (hints, tricks, FAQs, etc.).• Monitoring of tool acceptance, gathering and evaluating cost-benefit data.
Test Tool Introduction Steps1. Execute a pilot project.2. Evaluate the pilot project experiences.3. Adapt the processes and implement rules for usage.4. Train the users.5. Introduce the tool stepwise.6. Offer accompanying coaching.