(slide #34 of our presentation)


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

(slide #34 of our presentation)

  1. 1. SOFTWARE TESTING Presented to SENG 623 Software Quality Management 13-March-2003
  2. 2. Agenda <ul><li>Introductions </li></ul><ul><li>Software Testing Fundamentals </li></ul><ul><li>Software Testing Levels </li></ul><ul><li>Software Test Management </li></ul><ul><li>Concluding Remarks </li></ul><ul><li>Questions, Comments, & Great Thoughts </li></ul>
  3. 3. Software Testing Fundamentals
  4. 4. Software Testing Fundamentals Overview <ul><li>Software Testing is the process that is used to </li></ul><ul><ul><li>uncover errors before delivering the product to the customer, and </li></ul></ul><ul><ul><li>to give confidence that the product is working well. </li></ul></ul>
  5. 5. <ul><li>Fundamental Limitation of Testing </li></ul>Software Testing Fundamentals Overview Testing can only show the presence of an error; testing can never prove the absence of all errors.
  6. 6. Software Testing Fundamentals Test Design and Test Execution <ul><li>The Testing Activity is more than just running tests </li></ul><ul><li>In order for them to be run, they must be designed and written. </li></ul><ul><li>When should tests be designed? </li></ul><ul><li>As Late As Possible? </li></ul><ul><li>As Early as Possible? </li></ul>
  7. 7. Software Testing Fundamentals Testing Objectives <ul><li>A good test case is one that has high probability of finding an as-yet-undiscovered error. </li></ul><ul><li>A successful test is one that uncovers an as-yet-undiscovered error. </li></ul>
  8. 8. Software Testing Fundamentals Software Testing Techniques <ul><li>Software Testing Techniques provide systematic approaches for designing tests that </li></ul><ul><ul><li>Exercise the internal logic of software components </li></ul></ul><ul><ul><li>Exercise the input and output domains of the program to uncover errors in </li></ul></ul><ul><ul><ul><li>program function, </li></ul></ul></ul><ul><ul><ul><li>behavior and </li></ul></ul></ul><ul><ul><ul><li>performance </li></ul></ul></ul>
  9. 9. Software Testing Fundamentals Software Testing Techniques <ul><li>Dynamic Analysis versus Static Analysis </li></ul><ul><li>Black Box versus White (Glass) Box Testing </li></ul>
  10. 10. Software Testing Levels
  11. 11. Software Testing Levels Verification and Validation Requirements Analysis System Design Component Design Integration Testing Unit Testing System/Validation Testing Coding Integrated into Verified Against Validated Against Developed into
  12. 12. Software Testing Levels Unit Testing <ul><li>Unit Testing focuses the effort on the smallest unit of software design. </li></ul><ul><li>The Unit Test has a Dynamic, White-Box orientation. </li></ul>Test Cases Module ~~~~~ ~~~~~ ……… User Interface Local data structure Boundary conditions Independent paths Error handling paths
  13. 13. Software Testing Levels Unit Testing Metrics <ul><li>Defect Tracking Metrics </li></ul><ul><li>Code Complexity </li></ul><ul><li>Test Completeness Metrics </li></ul><ul><li>Code Coverage - Test Effectiveness Ratios </li></ul><ul><ul><li>Statement Coverage - TER 1 </li></ul></ul><ul><ul><li>Branch/Decision Coverage - TER 2 </li></ul></ul><ul><ul><li>Decision-condition Coverage </li></ul></ul><ul><ul><li>LCSAJ Coverage - TER 3 </li></ul></ul>
  14. 14. Software Testing Levels (Component) Integration Testing <ul><li>Integration Testing is a systematic technique for conducting tests to uncover the errors associated with interfacing. </li></ul><ul><li>As well, it will expose errors related to the larger functionality of the individual components under test. </li></ul><ul><li>Different incremental integration strategies: </li></ul><ul><ul><li>Top-Down Integration </li></ul></ul><ul><ul><li>Bottom-Up Integration </li></ul></ul><ul><li>Integration Testing is typically Dynamic, and is usually Black Box </li></ul><ul><li>Must also perform </li></ul><ul><ul><li>Regression Testing </li></ul></ul><ul><ul><li>Smoke Testing </li></ul></ul>
  15. 15. Software Testing Levels (Component) Integration Testing Metrics <ul><li>Error rates in design </li></ul><ul><li>Error rates in implementation </li></ul><ul><li>Error rates in test design </li></ul><ul><li>Test Execution Progress Metrics </li></ul><ul><li>Time/Cost Metrics </li></ul><ul><li>Requirements Churn Metrics </li></ul>
  16. 16. Software Testing Levels System Testing <ul><li>Software cannot be used alone, it has to be used under certain context. (e.g, hardware, people, information). The system testing is a series of different tests whose primary purpose is to fully exercise the computer-based system. </li></ul><ul><li>Types of system tests [Beizer, 1984] : </li></ul><ul><ul><li>Recovery testing </li></ul></ul><ul><ul><li>Security testing </li></ul></ul><ul><ul><li>Stress testing </li></ul></ul><ul><ul><li>Performance testing </li></ul></ul>
  17. 17. Software Testing Levels System Testing Metrics <ul><li>Test Completeness Metrics </li></ul><ul><li>Defect Arrival Rate </li></ul><ul><li>Cumulative Defects by Status </li></ul><ul><li>Defect Closure Rate </li></ul><ul><li>Reliability Predictions </li></ul><ul><li>Schedule Tracking Metrics </li></ul><ul><li>Staff and Resource Tracking Metrics </li></ul>
  18. 18. Software Testing Levels Validation Testing <ul><li>After System Testing, a final series of software tests – Validation Testing – begins. </li></ul><ul><li>The simple definition is that validation succeeds when software functions in a manner that can be reasonably expected by the customer. </li></ul><ul><li>Different approaches for Validation Testing: </li></ul><ul><ul><li>Acceptance Tests </li></ul></ul><ul><ul><li>Alpha Test </li></ul></ul><ul><ul><li>Beta Test </li></ul></ul>
  19. 19. Software Testing Levels Validation Testing Metrics <ul><li>Defect Arrival Rate </li></ul><ul><li>Cumulative Defects by Status </li></ul><ul><li>Defect Closure Rate </li></ul><ul><li>Defect Backlog by Severity </li></ul><ul><li>Cost of Quality Metrics </li></ul>
  20. 20. Software Testing Levels Quality Views <ul><li>Unit Testing </li></ul><ul><ul><li>Manufacturing View of Quality </li></ul></ul><ul><li>(Component) Integration Testing </li></ul><ul><ul><li>Manufacturing View of Quality </li></ul></ul><ul><li>System Testing </li></ul><ul><ul><li>Product View of Quality </li></ul></ul><ul><li>Validation Testing </li></ul><ul><ul><li>User View of Quality </li></ul></ul>
  21. 21. Software Testing Management
  22. 22. Software Testing Management Who Does the Testing? <ul><li>Development Team </li></ul><ul><li>Independent Test Group (ITG) </li></ul><ul><li>Customer / Clients </li></ul>
  23. 23. Software Testing Management Organizing for Software Testing <ul><li>Software Engineers create programs, documentation and related artifacts. </li></ul><ul><li>The Software Engineer is proud of what has been built and looks askance at anyone who attempts to tear it down. </li></ul><ul><li>The ITG attempts to “break” the thing that the Software Engineer has build. </li></ul><ul><li>ITG is paid to find the errors. </li></ul><ul><li>The Software Developer is always responsible for testing the individual units of the program. </li></ul><ul><li>In many cases, the developer also conducts integration testing. </li></ul><ul><li>The ITG is part of the Software Development Project Team. It becomes involved during the specification activity and stays involves throughout the project. </li></ul>
  24. 24. Software Testing Management Organizing for Software Testing <ul><li>The Software Engineer does not turn the software over to ITG and walk away. </li></ul><ul><li>They have to work together closely throughout a software project to ensure that thorough tests will be conducted. </li></ul><ul><li>While the testing is conducted, the developer must be available to correct errors that are uncovered. </li></ul>
  25. 25. Software Testing Management Organizing for Software Testing <ul><li>The customer is always involved in the validation testing. </li></ul><ul><li>The relationship between Software Project Team and customers in Validation Testing is different from the relationship between the Software Engineer and ITG. </li></ul>
  26. 26. Software Testing Management Responsibilities of Management <ul><li>Senior Management at the Organization Level </li></ul><ul><ul><li>Setting the testing policy, strategy and objectives for the company </li></ul></ul><ul><ul><li>Ensure that metrics for test effort and results are collected and used </li></ul></ul><ul><ul><li>Invest in Tool Support </li></ul></ul><ul><ul><li>Commit to improving the test process </li></ul></ul>
  27. 27. Software Testing Management Responsibilities of Management <ul><li>Test Management at the Project Level </li></ul><ul><ul><li>Assignment of Resources </li></ul></ul><ul><ul><li>Test Planning (before development) </li></ul></ul><ul><ul><li>Reviews and/or Inspections (throughout) </li></ul></ul><ul><ul><li>Test Design (before or in parallel with development) </li></ul></ul><ul><ul><li>Test Execution (after development) </li></ul></ul><ul><ul><li>Problem Resolution (whenever needed) </li></ul></ul>
  28. 28. Software Testing Management When To Stop Testing? <ul><li>Early Release </li></ul><ul><ul><li>may defects still left, with some at the “critical” level </li></ul></ul><ul><ul><li>reactive environment keeps team “fire fighting” rather than working on new product </li></ul></ul><ul><ul><li>often results in key employee turnover </li></ul></ul><ul><ul><li>frustrated customers </li></ul></ul>“ There is no single, valid, rational criterion for stopping. Furthermore, given any set of applicable criteria, how each is weighted depends very much on the product, the environment, the culture and the attitude to risk. [Beizer 1990]. <ul><li>Late Release </li></ul><ul><ul><li>team and users confident in quality of product </li></ul></ul><ul><ul><li>significant increase in product costs </li></ul></ul><ul><ul><li>organization experiences loss of revenue, smaller market share, project cancellations, but may gain reputation for quality </li></ul></ul><ul><ul><li>frustrated customers </li></ul></ul>
  29. 29. Software Testing Management When To Stop Testing? <ul><li>No single metric should be used to determine when to stop testing </li></ul><ul><ul><li>Defect Discovery Rate </li></ul></ul><ul><ul><li>Trend in Severity Level </li></ul></ul><ul><ul><li>Remaining Defects Estimation Criteria </li></ul></ul><ul><ul><li>Coverage Measurement </li></ul></ul><ul><ul><li>Reliability Growth Models </li></ul></ul><ul><ul><li>Running out of Resources </li></ul></ul>
  30. 30. Concluding Remarks
  31. 31. Concluding Remarks Inspections vrs Testing <ul><li>Both have their place </li></ul><ul><ul><li>Inspections prevent defects </li></ul></ul><ul><ul><li>Testing identifies defects </li></ul></ul>In an ancient land there was a family of healers, one of whom was known by all and was employed as a physician to a great lord. The physician was asked which of his family was the most skillful healer. He replied, &quot;I tend to the sick and dying with drastic and dramatic treatments, and on occasion someone is cured and my name gets out among the lords.&quot; &quot;My elder brother cures sickness when it just begins to take root, and his skills are known among the local peasants and neighbours.&quot; &quot;My eldest brother is able to sense the spirit of sickness and eradicate it before it takes form. His name is unknown outside our home.&quot;
  32. 32. Concluding Remarks Keys to Successful Testing <ul><li>In order to be cost effective, concentrate testing on the areas where it will be most effective. </li></ul><ul><li>Test the most critical parts (to the user) first </li></ul><ul><li>Concentrate the testing on the most difficult, most complex, hardest to write, least liked </li></ul><ul><li>Errors are “social” - test where historically the most errors have been found before. </li></ul><ul><li>Plan the testing so that when testing is stopped, the most effective testing in the time allotted has already been done. </li></ul>
  33. 33. Concluding Remarks Testing Pitfalls <ul><li>Absence of a Testing Policy </li></ul><ul><li>Failing to Plan Testing / Ineffective Testing </li></ul><ul><li>Lack of Schedule / Staff / Budget </li></ul><ul><li>To Little / To Much Test Data </li></ul><ul><li>Uneven Testing / Bad Coverage </li></ul><ul><li>Lack of Testing Tools / Belief of Test Tools </li></ul><ul><li>Lack of Testability of Software </li></ul>
  34. 34. <ul><li>The Testing Paradox </li></ul><ul><li>The Pesticide Paradox </li></ul>Concluding Remarks If testing finds errors in the product, how can it also build confidence that it is working well, since it was just shown that it wasn’t! Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual. [Biezer, 1983]
  35. 35. Questions, Comments, and Great Thoughts
  36. 36. Thank You!