10_Testing_Strategies.ppt

1,127 views
1,025 views

Published on

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,127
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
65
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • From 1 to 2: Nodes 5 and 7 are eliminated using Rule 2 From 2 to 3 An edge from node2 to node 6 removed using rule 4
  • From 3 to 4 Node 2 is eliminated using rule 2 From 4 to 5 Node 6 is eliminated using rule 2 From 5 to 6 The edge from node 1 to node 8 is removed using rule 4 From 6 to 7 Node 8 is eliminated using rule 2
  • 10_Testing_Strategies.ppt

    1. 1. Course Notes Set 10: Testing Strategies Computer Science and Software Engineering Auburn University
    2. 2. Strategic Approach to Testing <ul><li>Testing begins at the unit level and works toward integrating the entire system </li></ul><ul><li>Various techniques for testing are appropriate at different times </li></ul><ul><li>Conducted by the developer and by independent test groups </li></ul>
    3. 3. Testing Strategies Requirements Specification Preliminary Design Detailed Design Coding Unit Testing Integration Testing System Testing [Adapted from Software Testing A Craftman’s Approach , by Jorgensen, CRC Press, 1995]
    4. 4. A Testing Strategy Unit Test Integration Test Validation Test System Test System engineering Requirements Design Code [Adapted from Software Engineering 4th Ed , by Pressman, McGraw-Hill, 1997]
    5. 5. Integration Testing <ul><li>After individual components have passed unit testing, they are merged together to form subsystems and ultimately one complete system. </li></ul><ul><li>Integration testing is the process of exercising this “hierarchically accumulating” system. </li></ul>
    6. 6. Integration Testing <ul><li>We will (normally) view the system as a hierarchy of components. </li></ul><ul><ul><li>Call graph </li></ul></ul><ul><ul><li>Structure chart </li></ul></ul><ul><ul><li>Design tree </li></ul></ul><ul><li>Integration testing can begin at the top of this hierarchy and work downward, or it can begin at the bottom of the hierarchy and work upwards. </li></ul><ul><li>It can also employ a combination of these two approaches. </li></ul>
    7. 7. Example Component Hierarchy A B C D E F G [Figure and associated examples adapted from Pfleeger 2001]
    8. 8. Integration Testing Strategies <ul><li>Big-bang integration </li></ul><ul><li>Bottom-up Integration </li></ul><ul><li>Top-down Integration </li></ul><ul><li>Sandwich Integration </li></ul>
    9. 9. Big-bang Integration <ul><li>All components are tested in isolation. </li></ul><ul><li>Then, the entire system is integrated in one step and testing occurs at the top level. </li></ul><ul><li>Often used (perhaps wrongly), particularly for small systems. </li></ul><ul><li>Does not scale. </li></ul><ul><li>Difficult or impossible to isolate faults. </li></ul>
    10. 10. Big-bang Integration Test A Test B Test C Test D Test E Test F Test A,B,C, D,E,F, G A B C D E F G
    11. 11. Bottom-up Integration <ul><li>Test each unit at the bottom of the hierarchy first. </li></ul><ul><li>Then, test the components that call the previously tested ones (one layer up in the hierarchy). </li></ul><ul><li>Repeat until all components have been tested. </li></ul><ul><li>Component drivers are used to do the testing. </li></ul>
    12. 12. Bottom-up Integration Test E Test F Test G Test B,E,F Test C Test D,G Test A,B,C, D,E,F, G A B C D E F G
    13. 13. Bottom-up Integration <ul><li>The manner in which the software was designed will influence the appropriateness of bottom-up integration. </li></ul><ul><li>While it is normally appropriate for object-oriented systems, bottom-up integration has disadvantages for functionally-decomposed systems: </li></ul><ul><ul><li>Top-level components are usually the most important, but the last to be tested. </li></ul></ul><ul><ul><li>The upper levels are more general while the lower levels are more specific. Thus, by testing from the bottom up the discover of major faults can be delayed. </li></ul></ul><ul><ul><li>Top-level faults are more likely to reflect design errors, which should obviously be discovered as soon as possible and are likely to have wide-ranging consequences. </li></ul></ul><ul><ul><li>In timing-based systems, the timing control is usually in the top-level components. </li></ul></ul>
    14. 14. Top-down Integration <ul><li>The top-level component is tested in isolation. </li></ul><ul><li>Then, all the components called by the one just tested are combined and tested as a subsytem. </li></ul><ul><li>This is repeated until all components have been integrated and tested. </li></ul><ul><li>Stubs are used to fill in for components that are called but are not yet included in the testing. </li></ul>
    15. 15. Top-down Integration Test A Test A,B, C,D Test A,B,C, D,E,F, G A B C D E F G
    16. 16. Top-down Integration <ul><li>Again, the design of the system influences the appropriateness of the integration strategy. </li></ul><ul><li>Top-down integration is obviously well-suited to systems that have been created through top-down design. </li></ul><ul><ul><li>When major system functions are localized to components, top-down integration allows the testing to isolate one function at a time and follow its control flow from the highest levels of abstraction to the lowest levels. </li></ul></ul><ul><ul><li>Also, design problems show up earlier rather than later. </li></ul></ul>
    17. 17. Top-down Integration <ul><li>A major disadvantage is the need of stubs. </li></ul><ul><ul><li>Writing stubs can be complex since they must function under the same conditions as their real counterpart. </li></ul></ul><ul><ul><li>The correctness of the stub will influence the validity of the test. </li></ul></ul><ul><ul><li>A large number of stubs could be required, particularly when there are a large number of general-purpose components in the lowest layer. </li></ul></ul><ul><li>Another criticism is the lack of individual testing on interior components. </li></ul><ul><ul><li>To address this concern, a modified top-down integration strategy can be used. Instead of incorporating an entire layer at once, each component in a given layer is tested individually before the integration of that layer occurs. </li></ul></ul><ul><ul><li>This introduces another problem, however: Now both stubs and component drivers are needed. </li></ul></ul>
    18. 18. Modified Top-down Integration Test A Test B Test C Test D Test A,B, C,D Test E Test F Test G Test A,B,C, D,E,F, G A B C D E F G
    19. 19. Sandwich Integration <ul><li>Top-down and bottom-up can be combined into what Myers calls “sandwich integration.” </li></ul><ul><li>The system is viewed as being composed of three major levels: the target layer in the middle, the layers above the target, and the layers below the target. </li></ul><ul><li>A top-down approach is used for the top level while a bottom-up approach is used for the bottom level. Testing converges on the target level. </li></ul>
    20. 20. Sandwich Integration Test E Test F Test G Test A Test B,E,F Test D,G Test A,B,C, D,E,F, G A B C D E F G
    21. 21. Measures for Integration Testing <ul><li>Recall v(G) is an upper bound on the number of independent/basis paths in a source module </li></ul><ul><li>Similarly, we would like to limit the number of subtrees in a structure chart or call graph </li></ul>
    22. 22. Subtrees in Architecture vs. Paths in Units <ul><li>A call graph (or equivalent) architectural representation corresponds to a design tree representation, just as the source code for a unit corresponds to a flowgraph. </li></ul><ul><li>Executing the design tree means it is entered at the root, modules in the subtrees are executed, and it eventually exits at the root. </li></ul><ul><li>Just as the program can have a finite (if it halts), but overwhelming, number of paths, a design tree can have an inordinately large number of subtrees as a result of selection and iteration. </li></ul><ul><li>We need a measure for design trees that is the analog of the basis set of independent paths for units. </li></ul>
    23. 23. Design Tree : Complexity of 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    24. 24. Design Tree : Complexity > 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    25. 25. Design Tree Notation M A B M A B M A B Possible Paths: Neither A B A B Possible Paths: A B Possible Paths: Neither A B
    26. 26. Subtrees vs. Paths Design Tree C M A B E X A B M’s Flowgraph [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    27. 27. Flowgraph Information <ul><li>Flowgraph symbols </li></ul><ul><ul><li>A black dot is a call to a subordinate module </li></ul></ul><ul><ul><li>A white dot is a sequential statement (or a collection of sequential statements) </li></ul></ul><ul><li>Rules for reduction </li></ul><ul><ul><li>Sequential black dot : may not be reduced </li></ul></ul><ul><ul><li>Sequential white dot : a sequential node may be reduced to a single edge </li></ul></ul><ul><ul><li>Repetitive white dots : a logical repetition without a black dot can be reduced to a single node </li></ul></ul><ul><ul><li>Conditional white dots : a logical decision with two paths without a black dot may be reduced to one path </li></ul></ul>
    28. 28. Reduction Rules 1. Sequential Black Dot 2. Sequential White Dot 3. Repetitive White Dot or 4. Conditional or Looping White Dot Decisions [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    29. 29. Example Reduction 1 2 3 4 5 6 7 8 9 1 2 3 4 6 8 9 1 2 3 4 6 8 9 [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    30. 30. Example Reduction 1 3 4 6 8 9 1 3 4 8 9 1 3 4 8 9 1 3 4 9 [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    31. 31. Architectural Design Measures <ul><li>Number of subtrees </li></ul><ul><ul><li>The set of all subtrees is not particularly useful, but a basis set would be. </li></ul></ul><ul><li>Module Design Complexity : iv(G) </li></ul><ul><ul><li>The cyclomatic complexity of the reduced flowgraph of the module </li></ul></ul><ul><li>Design Complexity: S 0 </li></ul><ul><ul><li>S 0 of a module M is </li></ul></ul><ul><ul><ul><li>S 0 = iv(G j ) </li></ul></ul></ul><ul><ul><ul><li>j D </li></ul></ul></ul><ul><ul><ul><li>where D is the set of descendants of M unioned with M </li></ul></ul></ul><ul><ul><li>Note: If a module is called several times, it is added only once </li></ul></ul>
    32. 32. Design Complexity Example S 0 =11 iv=3 S 0 =3 iv=2 S 0 =4 iv=2 S 0 =1 iv=1 S 0 =1 iv=1 S 0 =1 iv=1 S 0 =1 iv=1 [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    33. 33. Design Complexity Example S 0 =1 iv=1 S 0 =1 iv=1 S 0 =4 iv=2 S 0 =1 iv=1 S 0 =6 iv=2 S 0 =9 iv=2 M A B C D E S 0 (A) = iv(A) + iv(C) + iv(D) + iv(E) [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    34. 34. Architectural Design Measures <ul><li>Integration Complexity : S 1 </li></ul><ul><ul><li>Measure of the number of integration tests required </li></ul></ul><ul><ul><li>S 1 = S 0 - n + 1 </li></ul></ul><ul><ul><ul><li>S 0 is the design complexity </li></ul></ul></ul><ul><ul><ul><li>n is the number of modules </li></ul></ul></ul>
    35. 35. Integration Complexity S 1 =5 S 0 =9 iv=3 S 0 =5 iv=3 iv=1 iv=1 iv=1 S 1 =5 S 0 =9 iv=3 S 0 =5 iv=3 iv=1 iv=1 iv=1 M A B C D N S T U V [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    36. 36. Integrated Properties of M and N S 0 =18 iv=3 S 0 =5 iv=3 S 0 =10 iv=1 iv=1 iv=1 S 0 =9 iv=3 S 0 =5 iv=3 S 0 =1 iv=1 iv=1 iv=1 M A B C D N S T U V Integration Point
    37. 37. Integration Testing <ul><li>Module integration testing </li></ul><ul><ul><li>Scope is a module and its immediate subordinates </li></ul></ul><ul><ul><li>Testing Steps </li></ul></ul><ul><ul><ul><li>Apply reduction rules to the module </li></ul></ul></ul><ul><ul><ul><li>Cyclomatic complexity of the subalgorithm is the module design complexity of the original algorithm. This determines the number of required tests. </li></ul></ul></ul><ul><ul><ul><li>The baseline method applied to the subalgorithm yields the design subtrees and the module integration tests </li></ul></ul></ul>
    38. 38. Integration Testing <ul><li>Design integration testing </li></ul><ul><ul><li>Derived from integration complexity, which quantifies a basis set of integration tests </li></ul></ul><ul><ul><li>Testing steps </li></ul></ul><ul><ul><ul><li>Calculate iv and S 0 for each module </li></ul></ul></ul><ul><ul><ul><li>Calculate S 1 for the top module (number of basis subtrees required) </li></ul></ul></ul><ul><ul><ul><li>Build a path matrix (S 1 x n) to establish the basis set of subtrees </li></ul></ul></ul><ul><ul><ul><li>Identify and label each predicate in the design tree and place those labels above each column of the path matrix corresponding to the module it influences </li></ul></ul></ul><ul><ul><ul><li>Apply the baseline method to the design to complete the matrix (1 : the module is executed; 0 : the module is not executed) </li></ul></ul></ul><ul><ul><ul><li>Identify the subtrees in the matrix and the conditions which derive the subtrees </li></ul></ul></ul><ul><ul><ul><li>Build corresponding test cases for each subtree </li></ul></ul></ul>
    39. 39. Design Integration Example S 0 =8 iv=2 S 0 =3 iv=1 S 0 =4 iv=2 S 0 =1 iv=1 S 0 =1 iv=1 S 0 =1 iv=1 M A B C D E P 1 P 2 S 1 = S 0 - n + 1 = 8 - 6 + 1 = 3 P 1 : condition W = X P 2 : condition Y = Z [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    40. 40. Integration Path Test Matrix [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    41. 41. Integration Path Test Matrix [Adapted from McCabe and Butler, “Design Complexity Measurement and Testing,” CACM 32(12)]
    42. 42. with ModuleA, ModuleB; use ModuleA, ModuleB; procedure Main is begin S1; while CM loop ProcA; ProcB; end loop; end Main; with ModuleC; use ModuleC; package body ModuleA is procedure ProcA is begin S1; if CA then S1; else ProcC; end if; end ProcA; begin null; end ModuleA; Example with ModuleC; use ModuleC; package body ModuleB is procedure ProcB is begin S1; if CB then ProcC; else S2; end if; if CB2 then S3; end if; end ProcB; begin null; end ModuleB; package body ModuleC is procedure ProcC is begin S1; if CC then S2; else S3; end if; end ProcC; begin null; end ModuleC; What is an appropriate number of integration test cases and what are those cases?
    43. 43. Example Main [Adapted from Watson and McCabe, “Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric,” NIST 500-235, 1996] ProcA ProcB ProcC
    44. 44. System Testing <ul><li>The primary objective of unit and integration testing is to ensure that the design has been implemented properly; that is, that the programmers wrote code to do what the designers intended. (Verification) </li></ul><ul><li>The primary objective of system testing is very different: We want to ensure that the system does what the customer wants it to do. (Validation) </li></ul>[Some notes adapted from Pfleeger 2001]
    45. 45. System Testing <ul><li>Steps in system testing </li></ul><ul><ul><li>Function Testing </li></ul></ul><ul><ul><li>Performance Testing </li></ul></ul><ul><ul><li>Acceptance Testing </li></ul></ul><ul><ul><li>Installation Testing </li></ul></ul>Function Test Performance Test Acceptance Test Installation Test Integrated Modules Functioning System Verified, Validated Software Accepted System Delivered System
    46. 46. Function Testing <ul><li>Checks that an integrated system performs its functions as specified in the requirements. </li></ul><ul><li>Common functional testing techniques (cause-effect graphs, boundary value analysis, etc.) used here. </li></ul><ul><li>View the entire system as a black box. </li></ul>
    47. 47. Performance Testing <ul><li>Compares the behavior of the functionally verified system to nonfunctional requirements. </li></ul><ul><li>System performance is measured against the performance objectives set by the customer and expressed as nonfunctional requirements. </li></ul><ul><li>This may involve hardware engineers. </li></ul><ul><li>Since this stage and the previous constitute a complete review of requirements, the software is now considered validated. </li></ul>
    48. 48. Types of Performance Tests <ul><li>Stress tests </li></ul><ul><li>Configuration tests </li></ul><ul><li>Legacy Regression tests </li></ul><ul><li>Security tests </li></ul><ul><li>Timing tests </li></ul><ul><li>Environmental tests </li></ul><ul><li>Quality tests </li></ul><ul><li>Recovery tests </li></ul><ul><li>Documentation tests </li></ul><ul><li>Usability tests </li></ul>
    49. 49. Acceptance Testing <ul><li>Customer now leads testing and defines the cases to test. </li></ul><ul><li>The purpose of acceptance testing is to allow the customer and users to determine if the system that was built actually meets their needs and expectations. </li></ul><ul><li>Many times, the customer representative involved in requirements gathering will specify the acceptance tests. </li></ul>
    50. 50. Types of Acceptance Tests <ul><li>Benchmark tests </li></ul><ul><ul><li>Subset of users operate the system under a set of predefined test cases. </li></ul></ul><ul><li>Pilot tests </li></ul><ul><ul><li>Subset of users operate the system under normal or “everyday” situations. </li></ul></ul><ul><ul><li>Alpha testing if done at developer’s site </li></ul></ul><ul><ul><li>Beta testing if done at customer’s site </li></ul></ul><ul><li>Parallel tests </li></ul><ul><ul><li>New system operates in parallel with the previous version. Users gradually transition to the new system. </li></ul></ul>
    51. 51. Installation Testing <ul><li>Last stage of testing </li></ul><ul><li>May not be needed if acceptance testing was performed at the customer’s site. </li></ul><ul><li>The system is installed in the environment in which it will be used, and we verify that it works in the field as it did when tested previously. </li></ul>

    ×