Software Testing Foundations Part 4 - Black Box Testing
Software Testing Foundations #4Dynamic Analysis. Black Box Testing.Nikita Knyshnknysh@gmail.comhttp://www.facebook.com/groups/istqb/
Agenda• Dynamic Analysis Overview• [Generic] Test Steps of Performing Tests• Back Box Testing Techniques ▫ Equivalence Class Partitioning ▫ Boundary Value Analysis ▫ State Transition Testing ▫ Cause-effect Graphing, Decision Tables ▫ Use Case Testing ▫ More Techniques
Dynamic Analysis Overview• The test object (program) must be executable.• In component and integration testing, test bed (stubs and drivers) is used to make the test object executable.• Test bed generators can be used.
[Generic] Test Steps of Performing Tests• Determine conditions and preconditions for the test and the goals to be achieved.• Specify the individual test cases (determine input, expected output and post-conditions). Writing test script (most often in programming language or similar notation).• Determine how to execute test cases (chaining several test cases into test scenario / test sequence, define priorities of test cases).
Black Box Test Techniques• Test cases are designed only by using the specification or the requirements of the test object.• Point of Observation (PoO) is outside the test object. Point of Control (PoC) is outside the test object too (we only can choose adequate input test data).• Reasonable in component tests but usually applied in higher test levels.
Black Box. Equivalence Class Partitioning.• Equivalence Class – a group of values that are supposed to be processed by test object the same way.• Equivalence classes for incorrect input must be tested as well.• For every single equivalence class, a representative value (input value) should be chosen for testing. For every input value, expected output or reaction should be defined.
Black Box. Equivalence Class Partitioning #2• Minimum test cases: every representative value of an equivalence class appears in at least one test case.• Negative test cases: representatives of invalid classes should not be combined with representatives of other invalid classes. EC-coverage = (number of tested EC / total number of EC) * 100 %The method is used in component, integration andsystem testing.
Black Box. Boundary Value Analysis.Checks the ‘border’ of the equivalence classes. Onevery border, the exact boundary value and bothnearest adjacent values (inside and outside theequivalence class) are tested.It must be decided if it is sufficient to test a boundarywith two instead of pieces three test data. BV-Coverage = (number of tested BV / total number of BV) * 100 %
Black Box. Boundary Value Analysis #2Rules for choosing boundary values:• For ordered sets the first and last element is of special interest for the test.• If complex data structures are given as in- or output, for instance an empty list or zero matrixes can be considered a boundary value.• For numeric calculations, values that are close together as well as values that are far apart should be taken into consideration as boundary values.• For invalid equivalence classes, boundary value analysis is only useful when different exception handling for the test object is expected depending on an equivalence class boundary.• For lists and tables empty and full lists and the first and last element are of interest, as they often show failures due to wrong programming (Off-by-one problem).
Black Box. State Transition Testing.• State diagrams illustrate the dependence of test object on history. These diagrams are the basis for designing tests.• A finite-state machine consists of states (nodes), transitions (links), inputs and outputs.
Black Box. State Transition Testing #4• Test intensity levels / completion criteria: (1) every state has been reached at least once, (2) every transition has been executed at least once, (3) every transition violating the specification has been checked.• For highly critical applications also: (4) all combinations of transitions, (5) all transitions in any order with all possible states, multiple instances in succession.
Black Box. State Transition Testing #5• State transition testing should be used where functionality is influenced by the state of the test object.• Very useful in testing object-oriented systems.• Is also a good technique for system test (e.g. GUI test).
Black Box. Cause Effect Graphing.The technique uses the dependencies between inputsand their effects on outputs for identification of thetest cases.
Black Box. Cause Effect Graphing #3Each cause and effect should occur at least once with"yes" and "no" in the table.An optimized decision table does not contain allpossible combinations, but the impossible orunnecessary combinations are not entered.
Black Box. Cause Effect Graphing #5Test completion criteria: execute every column in thedecision table by at least one test case.The graph and the table may grow very fast. Thistechnique is not easily applicable without adequatetool support.
Black Box. Use Case Testing.• Use cases and use case diagrams are the basis for determining test cases. Typical use of the system (user-system interaction) is checked.• The technique is relevant for customer, developer and tester and is useful for system- and acceptance testing.
Black Box. Use Case Testing #3• Test cases: ▫ Each alternative (extension point) in the use case diagram should be covered by a test case, ▫ Concrete input data and expected results cannot be derived directly from use case – analysis needed to define them.• Test completion criteria: every use case (including alternatives and extensions) and every possible sequence of use cases is tested at least once.
Black Box. Use Case Testing #4• The approach is supported by test specification tools.• However, there is no systematic method to determine further test cases to test what is not shown on the use case diagram (need to use other techniques).
Black Box. More Techniques.• Syntax Testing is used for testing interpreters, command languages, compilers and protocol analyzers. May be applied if a formal spec of input syntax is available.• Random Testing is used for selection of test values based on given statistical distribution of input values. Makes it possible to use statistical models for predicting or certifying system reliability.• Smoke test is a ‘quick and dirty’ test which is primarily aimed at verifying a minimum reliability of the test object and concentrated on its main functions. Used to decide if the test object is mature enough to be tested further by more comprehensive test techniques. Used for first and fast test of software updates.
More about Black Box.• Faults in specification are not detected (when techniques are used slavishly). If common sense is not used in test design, wrong requirements will lead to wrong test cases. Reviews must be used to find problems in specifications.• Not required functionality is not detected or tested not enough because of absence of documentation and no influence on coverage criteria. Such extra functionality is often the cause of security problems.• Black box methods verify the functionality. Correct working of a software system has the highest priority so black box techniques should always be applied.