Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. An Introduction to Software Testing CMSC 445 Shon Vick
  2. 2. Summary <ul><li>We are going to have an introduction to software testing </li></ul><ul><li>Chapters 16 and 17 of the Pressman deal with testing </li></ul><ul><ul><li>We will use this as a reference </li></ul></ul><ul><ul><li>A set of other references can be found at the end of this lecture </li></ul></ul>
  3. 3. Basic Ideas <ul><li>What is software testing? </li></ul><ul><li>Why test? </li></ul><ul><li>Test Verification versus validation. </li></ul><ul><li>How hard is program verification? </li></ul><ul><li>Testing versus proofs of correctness. </li></ul>
  4. 4. What is Testing? <ul><li>Most Fundamentally </li></ul><ul><ul><li>A phase of the software life-cycle that includes three stages: unit testing, integration testing, and system testing </li></ul></ul><ul><li>Its about </li></ul><ul><ul><li>Finding errors </li></ul></ul><ul><ul><li>Exercising code across range of inputs and conditions </li></ul></ul><ul><li>An activity aimed at evaluating an attribute or capability of a program/system to see it meets its required results </li></ul>
  5. 5. What is Testing? <ul><li>The solution assessment stage of the software process. </li></ul><ul><li>Delivers partial answer to the question: </li></ul><ul><li>&quot;How good is this software?'' </li></ul><ul><li>Any attempt to ensure that modules have been coded correctly. </li></ul><ul><li>An (oftentimes ad hoc ) activity to generate a set of inputs, called test data , which when applied to a program increases the confidence in the correctness of the program. </li></ul><ul><li>The purpose of testing is not to demonstrate that software performs correctly.Rather, testing demonstrates the existence of an error. Hence, success is finding an error. Failure is not finding an error. </li></ul><ul><li>Contrast testing with debugging , which resolves the error after its existence is known. </li></ul>
  6. 6. Testing and Debugging Are Not the Same Thing <ul><li>Testing and debugging are not synonymous. </li></ul><ul><ul><li>Testing: Establish the existence of defects . </li></ul></ul><ul><ul><li>Debugging : Locating and fixing these defects. </li></ul></ul><ul><li>A typical debugging process follows. </li></ul><ul><ul><li>Locate and repair defects in the code. </li></ul></ul><ul><ul><li>Repair program defect. </li></ul></ul><ul><ul><li>Re-Test </li></ul></ul>
  7. 7. Verification versus Validation <ul><li>Finding an error in software means that the software is incorrect . </li></ul><ul><li>Two senses of incorrect . </li></ul><ul><ul><li>Validation - subjective </li></ul></ul><ul><ul><li>Verification - objective </li></ul></ul>The benefit of defining the two V & V components separately is that the exercise demonstrates the narrow scope of objective correctness and the broad role of subjective judgment and experience. Sommerville
  8. 8. Validation <ul><li>Continual checking that the system meets the expectations of the user/procurer . </li></ul><ul><li>Validation often is a subjective decision by application user using domain knowledge. </li></ul><ul><li>This subjective evaluation of design decisions lasts throughout the software process. </li></ul>
  9. 9. Verification <ul><li>Involves checking that the program conforms to its specification . </li></ul><ul><li>Verification is more objective </li></ul><ul><li>Given some objective criteria specification of correctness one can always answer true or false. </li></ul><ul><ul><li>Finding an error with respect to this specification proves a implementation incorrect </li></ul></ul><ul><ul><li>Not finding an error does not prove the opposite </li></ul></ul>
  10. 10. Verification – The Bad News <ul><li>Program correctness is not an absolute property, but a relative one. </li></ul><ul><li>Important theoretical results in program testing </li></ul><ul><ul><li>No general purpose testing and analysis procedure can be used to prove program correctness. </li></ul></ul><ul><ul><li>Given any program P and its specification S no technique can determine if P is equivalent to S for all P </li></ul></ul><ul><ul><li>Equivalent to the Halting Problem . </li></ul></ul><ul><ul><li>Therefore practical, systematic procedure for selecting test data which can be used to prove correctness of a program. </li></ul></ul><ul><ul><li>It is possible to prove the correctness of a P but it may not cost effective. </li></ul></ul>
  11. 11. Testing and a Proof of Correctness <ul><li>Testing different than absolute correctness </li></ul><ul><li>Correctness </li></ul><ul><ul><li>A proof that one description of a function is equivalent to another. </li></ul></ul><ul><ul><li>Usually one is a state description in some formal language and the other is an algorithm described in a programming language. </li></ul></ul><ul><ul><li>Proving that a program is correct is a lofty goal. Much research has been done using automated tools to prove program correctness </li></ul></ul><ul><li>Testing </li></ul><ul><ul><li>An activity to generate a set of input test data for a program/system which when used by a program/system increases the confidence that a program/system is correct </li></ul></ul>
  12. 12. Finding Problems in a System <ul><li>Many errors in a software system are introduced early in development process </li></ul><ul><ul><li>Errors may persist until the implementation phase </li></ul></ul><ul><li>The cost to fix the errors grows as the life-cycle proceeds. </li></ul>
  13. 13. The Relative Costs 20.0 Maintenance 5.0 Acceptance Test 2.0 Unit Test 1.0 Coding 0.5 Design 0.1- 0.2 Requirements Relative Cost Stage
  14. 14. Methods of Software Testing <ul><li>White-box testing </li></ul><ul><li>Black-box testing </li></ul><ul><li>Exhaustive testing </li></ul>
  15. 15. White-Box Versus Black-Box Testing <ul><li>White-Box testing </li></ul><ul><ul><li>Assumes that the internal structure and logic of the module is known. </li></ul></ul><ul><ul><li>Tester can analyze the code and use knowledge of program structure to derive a test case. </li></ul></ul><ul><ul><li>Also called structural testing or glass-box testing . </li></ul></ul><ul><li>Black-box testing </li></ul><ul><ul><li>Relies only on input/output pairs for determination of existence of errors with no knowledge of the internal workings of system/program is used </li></ul></ul><ul><ul><li>Only the external behavior of the program is being tested (acceptance testing). </li></ul></ul><ul><ul><li>Also called functional testing . </li></ul></ul>
  16. 16. <ul><li>With functional testing, the component is treated as a black box whose behavior can only be determined by studying its inputs and related outputs. </li></ul><ul><li>Identify inputs which cause anomalous behavior in the outputs which reveal the existence of defects. </li></ul>Functional Testing
  17. 17. Functional Testing Concept Input Set Input i Output Set Anomalous Output i Component
  18. 18. <ul><li>Structural or white-box testing is complementary to black-box testing. </li></ul><ul><li>Here, the tester has knowledge of the actual code and the structure of the component to derive test cases. </li></ul><ul><ul><li>With black-box testing, the tester could only use the specification. </li></ul></ul><ul><li>Structural testing has certain advantages. </li></ul><ul><ul><li>Test cases can be systematically derived. </li></ul></ul><ul><ul><li>Test coverage can be measured. </li></ul></ul><ul><li>Structural testing can be used with black-box testing to refine the selection of test data. </li></ul>WhiteboxTesting
  19. 19. <ul><li>// Java example </li></ul><ul><li>boolean Contains(Enumeration e, Object key) </li></ul><ul><li>{ </li></ul><ul><li>try { </li></ul><ul><li>while (e.hasMoreElements()){ </li></ul><ul><li>Object element = e.nextElement(); </li></ul><ul><li>if(element.equals(key) return true; </li></ul><ul><li>} </li></ul><ul><li>catch (NoSuchElementException ex) { </li></ul><ul><li>return false; </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>Structural Testing Example
  20. 20. Exhaustive Testing On States <ul><li>Is it reasonable to do exhaustive testing? </li></ul><ul><ul><li>Usually not! </li></ul></ul><ul><li>Testing on possible variable states </li></ul><ul><li>Suppose that our program has n variables each with m values then the number of test required is m n </li></ul><ul><li>General formula </li></ul><ul><ul><li>If each variable vi and mi state then  i m i tests are required </li></ul></ul>
  21. 21. Exhaustive Path Testing <ul><li>Exhaustive Testing on paths also is usually not tenable </li></ul><ul><li>Combinatorial Explosion there as well </li></ul><ul><li>We’ll discuss this in more detail later </li></ul>
  22. 22. Tests to perform <ul><li>Test data based on structure. </li></ul><ul><ul><li>Inputs where the enumeration has a single element. </li></ul></ul><ul><ul><li>Inputs where numbers of elements in the enumeration is very large. </li></ul></ul><ul><ul><li>Inputs where the enumeration has no elements. </li></ul></ul><ul><ul><li>Inputs where Key is the first/last element of the array. </li></ul></ul><ul><ul><li>Inputs where the enumeration contains heterogeneous data </li></ul></ul>
  23. 23. Types of Testing <ul><li>Defect testing </li></ul><ul><ul><li>Tests designed to reveal the presence of defects in a system. </li></ul></ul><ul><li>Statistical testing </li></ul><ul><ul><li>Tests are selected so that if a program P and S agree on the tests then S and P are probabilistically equivalent </li></ul></ul><ul><ul><li>Tests are designed to reflect the frequency of actual user inputs. After running tests, estimate of operational reliability of system is made (Sommerville) </li></ul></ul><ul><ul><li>Assumes that there is an input/output oracle. </li></ul></ul>
  24. 24. Defect Testing Objectives <ul><li>Program testing has 2 objectives. </li></ul><ul><ul><li>Show that the system meets its requirements. (Validation) </li></ul></ul><ul><ul><li>Exercise the system to expose latent defects. (Verification) </li></ul></ul><ul><li>These objectives are distinct. </li></ul><ul><ul><li>Final system testing emphasizes validation. </li></ul></ul><ul><ul><li>Earlier components and integration testing emphasizes verification. </li></ul></ul>
  25. 25. Test Cases <ul><li>Test cases and test data are not the same thing. </li></ul><ul><ul><li>Test data are inputs used to test the system. </li></ul></ul><ul><ul><li>Test cases are input and output specifications plus a statement of the function under test. </li></ul></ul><ul><li>In principle defect testing could be exhaustive. </li></ul><ul><ul><li>Exercise every statement with every possible path combination. </li></ul></ul><ul><ul><li>This is generally impossible. </li></ul></ul><ul><ul><li>Instead, a set of test cases are developed using heuristics. </li></ul></ul>
  26. 26. Considerations in choosing Test Cases <ul><li>Emphasize Capabilities. </li></ul><ul><ul><li>Testing a system’s capabilities is as important as testing its components. </li></ul></ul><ul><ul><li>For example, screen corruption is less critical than defects which cause a loss of data or program termination. </li></ul></ul><ul><li>Old Capabilities. </li></ul><ul><ul><li>Testing old capabilities may be more important that testing new ones. </li></ul></ul><ul><ul><li>With a new revision, users are more concerned that the features they are already dependent on still work properly. </li></ul></ul><ul><ul><li>Regression Testing may help Identification </li></ul></ul>
  27. 27. More Considerations <ul><li>Boundary Conditions </li></ul><ul><ul><li>Always Test Boundary cases </li></ul></ul><ul><ul><li>Errors are likely there </li></ul></ul><ul><ul><li>Boundary conditions may be rare </li></ul></ul><ul><li>Don’t Neglect Typical Situations. </li></ul><ul><ul><li>Testing typical situations may be more important than testing boundary value cases. </li></ul></ul><ul><ul><li>It is important that the system works under normal conditions than occasional situations. </li></ul></ul>
  28. 28. Text References <ul><li>Software Engineering, A Holistic View , Bruce I. Blum, 1992. </li></ul><ul><li>Software Engineering , 4th ed., Ian Sommerville, 1992 . </li></ul><ul><li>Software Engineering, The Production of Quality Software, 2nd ed., Shari Lawrence Pfleeger, 1991. </li></ul><ul><li>Functional Program Testing and Analysis, William E. Howden, 1987. </li></ul>
  29. 29. Journal References <ul><li>Software Requirements: Analysis and Specification, Alan Davis, 1990. </li></ul><ul><li>The Software Life Cycle, 2nd ed. , Darrel Ince and Derek Andrews, 1990. </li></ul><ul><li>Design Complexity Measurement and Testing,'' Thomas J. McCabe and Charles W. Butler, IEEE Communications of the ACM, 32:12 (December 1989), 1415-1425. </li></ul>
  30. 30. More Journal References <ul><li>Hints on Test Data Selection: Help for the Practicing Programmer , Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward , IEEE Computer 11:4 pp.34-43 , 1978 . </li></ul><ul><li>Comparing the Effectiveness of SoftwareTesting Strategies, Victor R. Basili and Richard W. Selby, IEEE Transactions on Software Engineering SE-13 :12 (December 1987), 1278-1296. </li></ul>
  31. 31. Online Resources <ul><li>Newsgroup: </li></ul><ul><li> </li></ul><ul><li> </li></ul>