A beginners guide to testingPresentation Transcript
A beginners guide to testing Philip Johnson Collaborative Software Development Laboratory Information and Computer Sciences University of Hawaii Honolulu HI 96822
Understand the basic concepts, terminology, and approaches to software testing.
Be able to apply this information to improve the quality of your own testing efforts during software development.
Why is testing important?
First National Bank (1996)
An error at First National Bank of Chicago resulted in the balance of 800 customers being inflated by a total of $763 billion.
Inadequate testing. Bank updated ATM transaction software with new message codes. Message codes were not tested on all ATM protocols, resulting in some ATMs interpreting them as huge increases to customer balances.
Six people were overexposed during radiation treatments for cancer by the Therac-25 radiation therapy machine. Three people were believed to have died from the overdoses.
Inadequate testing. Hardware safety locks removed and replaced by software safety locks, which could be overcome by technician “type ahead”.
Ariane 5 (1996)
Ariane 5 rocket exploded on its maiden flight.
Inadequate testing. Navigation package inherited from Ariane-4 without proper testing. New rocket flew faster, resulting in larger values of some variables, resulting in an attempt to convert a 64-bit floating number into a 16 bit integer. Code was caught and action taken was to shut down navigation system.
ICS 314, Fall 2011
A student got a bad grade for ICS 314.
Inadequate testing. The student did not learn how to write good test cases for their code.
Testing fits into Validation and Verification
Establishing the fitness of a software product for its use.
“ Are we building the right product?”
Requires interaction with customers.
Establishing the correspondence between the software and its specification.
“ Are we building the product right?”
Requires interaction with software.
Static vs. Dynamic V&V
Static analysis of source code
Control/data flow analysis
Looks for errors in functionality
Looks for errors in scalability, performance, reliability
Is it “testing” if you simply run the program to see what it does?
Is validation testing done best with static or dynamic testing?
Is verification testing done best with static or dynamic testing?
Why is testing hard?
Execute program on all possible inputs and compare actual to expected behavior.
Could “prove” program correctness.
Not practical for any non-trivial program.
Select a tiny % of all possible tests.
Goal: executing the tiny % of tests will uncover a large % of defects present!
A “testing method” is essentially a way to decide which tiny % to pick.
How do we determine which tests to write? Selected examples:
Functional (black box) testing
Use specification to determine the tests
Structural (white box) testing
Use coverage to assess the quality of tests
Write tests as specification
Use tests to determine the implementation!
Combinations of methods are acceptable!
Black box testing
Specification based testing
A mapping between the possible “inputs” to the program and the corresponding expected “outputs”
Design a set of test cases to see if inputs actually map to outputs.
Does not require access to source code
Differences with White Box (coverage) testing:
Can catch errors of omission.
Effectiveness depends upon the quality of specification
Goal: Divide the possible inputs into categories such that testing one point in each category is equivalent to testing all points in the category.
Provide one test case for each point.
Equivalence class definition is usually an iterative process and goes on throughout development.
Use heuristics to get you started designing your test cases.
Unit test design heuristics
If input is a value:
If input is a sequence:
Max and min element values
Sequences of different sizes
If I/O specification contains conditions:
If I/O specification contains iterations:
> 1 time
Web app design heuristics
Every page is retrieved at least once
Prevent 404 errors.
Every link is followed at least once.
Prevent 404 errors.
All form input fields are tested with:
Always check response for appropriateness.
Tool support: JUnit
Tool Support: JUnitReport
White box testing
For a test case to uncover a defect, the statement containing the defect must be executed.
Therefore, a set of test cases which guarantees all statements are executed might uncover a large number of the defects present.
Whether or not the defects are actually uncovered depends upon the program state at the time each statement is executed.
Control flow coverages
Control flow coverage adds conditions to statement coverage to raise the odds of discovering defects.
Every conditional is evaluated as both true and false during testing.
Every loop must be executed both 0 times and more than 1 time.
All permutations of paths through the program are taken
JaCoCo: A Java Coverage Tool
JaCoCo: Drilldown to Package
JaCoCo: Drilldown to Class
JaCoCo: Red regions not tested
“ Coverage-driven test case design”
1. Write some test cases to exercise your code in new ways.
2. Run your coverage tool.
3. If coverage is not 100%, go to 1.
“ Coverage-driven test case design” is bad!
Using coverage data to drive the design of your tests often results in a poorly designed test case suite. (See readings.)
Instead, use coverage to assess the quality of your test method.
In a nutshell:
Good test design method -> Good coverage
But not the other way around.
Limitations of white-box testing
Can catch bugs in code that has been written.
Cannot catch bugs in code that has not been written!
Errors of omission: code that ought to have been written but wasn ’t
Missing boolean conditions in IF statements
Missing exception handlers
To catch bugs in code that has not been written, you must compare behavior of program to its specification.
Quality assurance in this class
You will use a combination of manual and automated quality assurance.
Use Checkstyle, PMD, FindBugs, and eliminate all warnings.
Use JUnit to develop tests, use Jacoco to check coverage.