01 Software Testing Introduction


Published on

  • Be the first to comment

01 Software Testing Introduction

  1. 1. Software Testing Introduction
  2. 2. Introduction <ul><li>An introduction to software testing. </li></ul><ul><li>Serves as a tutorial to formal testing of software </li></ul><ul><li>Topics covered </li></ul><ul><ul><li>definitions of testing </li></ul></ul><ul><ul><li>validation and verification </li></ul></ul><ul><ul><li>levels of testing from unit testing through to acceptance testing </li></ul></ul><ul><ul><li>relationship with requirements and design specifications and </li></ul></ul><ul><ul><li>test documentation. </li></ul></ul>
  3. 3. What is Software Testing? <ul><li>Many definitions of software testing </li></ul><ul><li>All definitions boil down to: </li></ul><ul><ul><li>software testing is the process of executing software in a controlled manner, in order to answer the question &quot;Does the software behave as specified?&quot;. </li></ul></ul><ul><li>Software testing is often used in association with the terms verification and validation. </li></ul><ul><ul><li>Verification is the checking or testing of items, including software, for conformance and consistency with an associated specification. </li></ul></ul><ul><ul><li>Software testing is just one kind of verification, which also uses techniques such as reviews, analysis, inspections and walkthroughs. </li></ul></ul>
  4. 4. What is Software Testing? <ul><ul><li>Validation is the process of checking that what has been specified is what the user actually wanted. </li></ul></ul><ul><li>Validation: Are we doing the right job? </li></ul><ul><li>Verification: Are we doing the job right? </li></ul><ul><li>“ bug” refers to a problem or fault in a computer. </li></ul><ul><li>Software testing should not be confused with debugging. </li></ul><ul><ul><li>Debugging is the process of analyzing and locating bugs when software does not behave as expected. </li></ul></ul><ul><li>A methodical approach to software testing is a much more thorough means of identifying bugs. </li></ul>
  5. 5. What is Software Testing? <ul><li>Debugging supports testing, but cannot replace testing. </li></ul><ul><li>However, no amount of testing can guarantee discovery of all bugs. </li></ul><ul><li>Other activities often associated with software testing are: </li></ul><ul><ul><li>static analysis </li></ul></ul><ul><ul><li>dynamic analysis </li></ul></ul><ul><li>Static analysis investigates the source code, looking for problems and gathering metrics without executing the code. </li></ul><ul><li>Dynamic analysis examines the behavior of executing software . </li></ul>
  6. 6. The Cost of Testing <ul><li>It is well known that most of an iceberg is hidden under water. </li></ul><ul><li>The same is true of software. </li></ul><ul><li>The number of lines of code in the product often describes the size of a software project, but that's just the tip of the iceberg. </li></ul><ul><li>The amount of software that needs to be written to test the product can be staggering. </li></ul><ul><li>Often the number of lines of code of test software will exceed the lines of code in the product software. </li></ul>
  7. 7. Cost <ul><ul><li>Suppose you have to write three lines of test software for every new line of software you deliver. </li></ul></ul><ul><ul><li>Then suppose management says that all your test software must be fully tested. </li></ul></ul><ul><ul><li>So for every line of test software you write, you need to write 3 lines of software that tests the software that tests your product. </li></ul></ul><ul><ul><li>If X is the number of lines in your product, you will have to write 3X lines of test software, and 9X lines of code that tests the test software. </li></ul></ul><ul><ul><li>Your job has just increased by a factor of 12! </li></ul></ul>
  8. 8. Cost <ul><li>What generally happens is that people write as much test software as they can in the time left over at the end of the project. </li></ul><ul><li>The test software is inadequate, so the product gets shipped with some bugs in it. </li></ul><ul><li>Testing is a big job. </li></ul><ul><li>You have to plan money, people, and most of all, TIME for it. </li></ul><ul><li>Don't expect to write and debug all your test code in the last month before the critical design review. </li></ul>
  9. 9. Cost <ul><li>Even if you could, it wouldn't do you much good. </li></ul><ul><li>By that time the design is cast in concrete, so nobody is going to change anything unless your test programs find catastrophic errors. </li></ul><ul><li>If you find moderate errors, people will say, &quot;That's a shame, but we can't do anything about it now. Why didn't you catch these errors sooner?&quot; </li></ul><ul><li>You have to code and test early in the development cycle while there is still time to take corrective action. </li></ul>
  10. 10. Software Specifications and Testing <ul><li>Validation and verification activities, such as software testing, cannot be meaningful unless there is a specification for the software. </li></ul><ul><li>Depending on the size of the development, specification of software can range from a single document to a complex hierarchy of documents. </li></ul><ul><ul><li>A hierarchy of software specifications will typically contain three or more levels of software specification documents . </li></ul></ul><ul><li>Requirements Specifications specify what the software is required to do and may also specify constraints on how this may be achieved. </li></ul>
  11. 11. Software Specifications and Testing <ul><li>Architectural Design Specifications describe the architecture of a design which implements the requirements. Components within the software and the relationship between them. </li></ul><ul><li>Detailed Design Specifications describe how each component in the software, down to individual units, is to be implemented. </li></ul>
  12. 12. Software Specifications and Testing
  13. 13. Software Specifications and Testing <ul><li>A hierarchy of specifications, makes possible testing software at various stages of the development. </li></ul><ul><li>The levels of testing which correspond to the hierarchy of software specifications: </li></ul><ul><ul><li>Unit Testing : each unit (basic component) of the software is tested </li></ul></ul><ul><ul><li>Integration Testing : progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a whole. </li></ul></ul><ul><ul><li>System Testing , the software is integrated to the overall product and tested to show that all requirements are met. </li></ul></ul>
  14. 14. Software Specifications and Testing <ul><ul><li>Acceptance Testing : upon which acceptance of the completed software is based. This will often use a subset of the system tests, witnessed by the customers for the software or system. </li></ul></ul><ul><li>Once each level of software specification has been written, the next step is to design the tests. </li></ul><ul><li>Tests should be designed before the software is implemented. </li></ul><ul><ul><li>if the software was implemented first, it would be tempting to test the software against what it is observed to do rather than against what it is specified to do. </li></ul></ul>
  15. 15. Software Specifications and Testing <ul><li>Within each level of testing, test results are evaluated. </li></ul><ul><li>If a problem is encountered, either the tests are revised and reapplied, or the software is fixed and the tests reapplied. </li></ul><ul><li>This is repeated until no problems are encountered, at which point development can proceed to the next level of testing. </li></ul><ul><li>Testing does not end at the conclusion of acceptance testing. </li></ul><ul><li>Software has to be maintained to fix problems which show up during use and to accommodate new requirements. </li></ul>
  16. 16. Software Specifications and Testing <ul><li>Software tests have to be repeated, modified and extended. </li></ul><ul><li>Revision and repetition of tests forms a major part of the overall cost of developing and maintaining software. </li></ul><ul><li>The term regression testing is used to refer to the repetition of earlier successful tests in order to make sure that changes to the software have not introduced side effects. </li></ul>
  17. 17. Test Design Documentation <ul><li>Tests design is subject to same engineering principles as the design of software. </li></ul><ul><li>Good design consists of a number of stages which progressively elaborate the design of tests from a high level strategy to detailed test procedures. </li></ul><ul><li>These stages are: </li></ul><ul><ul><li>test strategy, </li></ul></ul><ul><ul><li>test planning, </li></ul></ul><ul><ul><li>test case design, and </li></ul></ul><ul><ul><li>test procedure design. </li></ul></ul>
  18. 18. Test Design Documentation <ul><li>The design of tests is driven by the specification of the software. </li></ul><ul><li>At the highest level tests will be designed to verify that the software faithfully implements the Requirements Specification. </li></ul><ul><li>At lower levels tests will be designed to verify that items of software implement all design decisions made in the Architectural Design Specification and Detailed Design Specifications. </li></ul><ul><li>As with any design process, each stage of the test design process should be subject to informal and formal review. </li></ul>
  19. 19. Test Strategy <ul><li>A test strategy is a statement of the overall approach to testing, identifying what levels of testing are to be applied and the methods, techniques and tools to be used. </li></ul><ul><li>A test strategy should be applicable to all of an organizations software developments. </li></ul><ul><li>Developing a test strategy which efficiently meets the needs of an organization is critical to the success of software development within the organization. </li></ul><ul><li>The application of a test strategy to a software development project should be detailed in the projects software quality plan . </li></ul>
  20. 20. Test Plans <ul><li>The next stage of test design is development of a test plan. </li></ul><ul><li>A test plan states the </li></ul><ul><ul><li>items to be tested are, </li></ul></ul><ul><ul><li>at what level they will be tested, </li></ul></ul><ul><ul><li>what sequence they are to be tested in, </li></ul></ul><ul><ul><li>how the test strategy will be applied to the testing of each item, </li></ul></ul><ul><ul><li>and describes the test environment. </li></ul></ul><ul><li>A test plan may be project wide, or a hierarchy of plans relating to the various levels of specification and testing: </li></ul>
  21. 21. Plans <ul><ul><li>An Acceptance Test Plan : describes the plan for acceptance testing of the software. Usually published as a separate document, but might be published with the system test plan. </li></ul></ul><ul><ul><li>A System Test Plan : describes the plan for system integration and testing. Usually published as a separate document, but might be published with the acceptance test plan. </li></ul></ul><ul><ul><li>A Software Integration Test Plan : describing the plan for integration of tested software components. May form part of the Architectural Design Specification. </li></ul></ul><ul><ul><li>Unit Test Plan(s): describes the plans for testing of individual units of software. May form part of the Detailed Design Specifications. </li></ul></ul>
  22. 22. Plans <ul><li>Each test plan provides a plan for verification that the software fulfills the requirements or design statements of the software specification. </li></ul><ul><li>In the case of acceptance testing and system testing, this means the Requirements Specification. </li></ul>
  23. 23. Test Plans <ul><li>Test Plan Outline (Adapted from IEEE Standard for Software Test Documentation (829) ) </li></ul><ul><ul><li>1.BACKGROUND </li></ul></ul><ul><ul><li>2.INTRODUCTION </li></ul></ul><ul><ul><li>3.ASSUMPTIONS </li></ul></ul><ul><ul><li>4.TEST ITEMS </li></ul></ul><ul><ul><li>List each of the items (programs) to be tested </li></ul></ul>
  24. 24. Plan <ul><ul><li>5.FEATURES TO BE TESTED </li></ul></ul><ul><ul><li>List each of the features (functions or requirements) which will be tested or demonstrated by the test. </li></ul></ul><ul><ul><li>6.FEATURES NOT TO BE TESTED </li></ul></ul><ul><ul><li>Explicitly lists each feature, function, or requirement which won't be tested and why not. </li></ul></ul><ul><ul><li>7.APPROACH </li></ul></ul><ul><ul><li>Describe the data flows and test philosophy. </li></ul></ul><ul><ul><li>Simulation or Live execution, Etc. </li></ul></ul><ul><ul><li>8.ITEM PASS/FAIL CRITERIA Blanket statement </li></ul></ul><ul><ul><li>Itemized list of expected output and tolerances </li></ul></ul>
  25. 25. Plan <ul><ul><li>9.SUSPENSION/RESUMPTION CRITERIA </li></ul></ul><ul><ul><li>Must the test run from start to completion? </li></ul></ul><ul><ul><li>Under what circumstances may it be resumed in the middle? </li></ul></ul><ul><ul><li>Establish check-points in long tests. </li></ul></ul><ul><ul><li>10.TEST DELIVERABLES </li></ul></ul><ul><ul><li>What, besides software, will be delivered? </li></ul></ul><ul><ul><li>Test report </li></ul></ul><ul><ul><li>Test software </li></ul></ul><ul><ul><li>11.TESTING TASKS Functional tasks (e.g., equipment set up) </li></ul></ul><ul><ul><li>Administrative tasks </li></ul></ul>
  26. 26. Plan <ul><ul><li>12.ENVIRONMENTAL NEEDS </li></ul></ul><ul><ul><li>Security clearance </li></ul></ul><ul><ul><li>Office space & equipment </li></ul></ul><ul><ul><li>Hardware/software requirements </li></ul></ul><ul><ul><li>13.RESPONSIBILITIES </li></ul></ul><ul><ul><li>Who does the tasks in Section 10? </li></ul></ul><ul><ul><li>What does the user do? </li></ul></ul><ul><ul><li>14.STAFFING & TRAINING </li></ul></ul><ul><ul><li>15.SCHEDULE </li></ul></ul><ul><ul><li>16.RESOURCES </li></ul></ul><ul><ul><li>17.RISKS & CONTINGENCIES </li></ul></ul><ul><ul><li>18.APPROVALS </li></ul></ul>
  27. 27. Test Case Design <ul><li>After the test plan stage, the next stage of test design is to specify a set of test cases for each item to be tested at that level. </li></ul><ul><li>A number of test cases will be identified for each item to be tested at each level of testing. </li></ul><ul><li>Each test case specifies how the implementation of a particular requirement or design decision is to be tested and the criteria for success of the test. </li></ul><ul><li>The test cases may be documented with the test plan, as a section of a software specification, or in a separate document called a test specification or test description. </li></ul>
  28. 28. Case Design <ul><li>The specification of the following test cases may be documented by a separate document or in the appropriate test plan: </li></ul><ul><ul><li>acceptance testing </li></ul></ul><ul><ul><li>system integration </li></ul></ul><ul><ul><li>for each stage of integration of tested software components </li></ul></ul><ul><ul><li>testing of individual units of software </li></ul></ul><ul><li>System testing and acceptance testing involve an enormous number of individual test cases. </li></ul><ul><ul><li>an index which cross references between requirements and test cases. Referred to as a Verification Cross Reference Index (VCRI) and is attached to the test specification. </li></ul></ul>
  29. 29. Case Design <ul><ul><li>Cross reference indexes may also be used with unit testing and software integration testing. </li></ul></ul><ul><li>Design test cases for both positive testing and negative testing. </li></ul><ul><ul><li>Positive testing checks that the software does what it should. </li></ul></ul><ul><ul><li>Negative testing checks that the software doesn't do what it shouldn't. </li></ul></ul>
  30. 30. Test Procedures <ul><li>The final stage of test design implements a set of test cases as a test procedure, specifying the exact process to be followed to conduct each of the test cases. </li></ul><ul><ul><li>For each item to be tested, at each level of testing, a test procedure specifies the process to be followed in conducting the appropriate test cases. </li></ul></ul><ul><ul><li>A test procedure cannot leave out steps or make assumptions. The level of detail must be such that the test procedure is deterministic and repeatable . </li></ul></ul><ul><ul><li>Test procedures should always be separate items, because they contain a great deal of detail which is irrelevant to software specifications. </li></ul></ul>
  31. 31. Results Documentation <ul><li>The outputs of each test execution should be recorded. </li></ul><ul><ul><li>Results are assessed against criteria in the test specification </li></ul></ul><ul><li>Each test execution should also be recorded in a test log. </li></ul><ul><ul><li>The test log contains records of </li></ul></ul><ul><ul><ul><li>when each test run, </li></ul></ul></ul><ul><ul><ul><li>outcome of each test execution, and </li></ul></ul></ul><ul><ul><ul><li>may include key observations made during test execution. </li></ul></ul></ul><ul><ul><li>A log may not be exist for unit and software integration tests. </li></ul></ul><ul><li>Test reports may be produced during the testing process. </li></ul><ul><ul><li>A test report summarizes the results and documents analysis. </li></ul></ul><ul><ul><li>An acceptance test report often is a contractual document. </li></ul></ul>
  32. 32. Summary <ul><li>List of rules to an aid effective software testing. </li></ul><ul><ul><li>Always test against a specification. </li></ul></ul><ul><ul><li>Document the testing process: specify tests and record test results. </li></ul></ul><ul><ul><li>Test hierarchically against each level of specification. Finding more errors earlier will ultimately reduce costs. </li></ul></ul><ul><ul><li>Plan verification and validation activities, particularly testing. </li></ul></ul><ul><ul><li>Complement testing with techniques such as static analysis. </li></ul></ul><ul><ul><li>Test positively that the software does what it should, but also negatively that it doesn't do what it shouldn't. </li></ul></ul>