Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Conquering Complex and Changing Systems


Published on

  • Be the first to comment

  • Be the first to like this

Conquering Complex and Changing Systems

  1. 1. Chapter 9, Testing
  2. 2. Terminology <ul><li>Reliability: The measure of success with which the observed behavior of a system confirms to some specification of its behavior. </li></ul><ul><li>Failure: Any deviation of the observed behavior from the specified behavior. </li></ul><ul><li>Error: The system is in a state such that further processing by the system will lead to a failure. </li></ul><ul><li>Fault (Bug): The mechanical or algorithmic cause of an error. </li></ul><ul><li>There are many different types of errors and different ways how we can deal with them. </li></ul>
  3. 3. What is this?
  4. 4. Erroneous State (“Error”)
  5. 5. How do we deal with Errors and Faults?
  6. 6. Verification?
  7. 7. Modular Redundancy?
  8. 8. Declaring the Bug as a Feature?
  9. 9. Patching?
  10. 10. Testing?
  11. 11. Examples of Faults and Errors <ul><li>Faults in the requirements specification </li></ul><ul><ul><li>Mismatch between what the clients need and what the developers offer </li></ul></ul><ul><ul><li>Mismatch between requirements and implementation </li></ul></ul><ul><li>Algorithmic Faults </li></ul><ul><ul><li>Missing initialization </li></ul></ul><ul><ul><li>Branching errors (too soon, too late) </li></ul></ul><ul><ul><li>Missing test for nil </li></ul></ul><ul><li>Errors </li></ul><ul><ul><li>Stress or overload errors </li></ul></ul><ul><ul><li>Capacity or boundary errors </li></ul></ul><ul><ul><li>Timing errors </li></ul></ul><ul><ul><li>Throughput or performance errors </li></ul></ul>
  12. 12. Dealing with Errors <ul><li>Avoid them: </li></ul><ul><ul><li>software quality assurance, best practices, on going reviews/inspections </li></ul></ul><ul><li>Verification (mathematical): </li></ul><ul><ul><li>Assumes hypothetical environment that does not match real environment </li></ul></ul><ul><ul><li>Proof might be buggy (omits important constraints; simply wrong) </li></ul></ul><ul><li>Modular redundancy: </li></ul><ul><ul><li>Expensive </li></ul></ul><ul><li>Declaring a bug to be a “feature” </li></ul><ul><ul><li>Bad practice </li></ul></ul><ul><li>Patching </li></ul><ul><ul><li>Slows down performance </li></ul></ul><ul><li>Testing (this lecture) </li></ul>
  13. 13. Some Observations <ul><li>It is impossible to completely test any nontrivial module or any system </li></ul><ul><li>Testing can only show the presence of bugs, not their absence (Dijkstra) </li></ul>
  14. 14. Testing takes creativity <ul><li>Testing is not always viewed as a glamorous activity </li></ul><ul><li>To develop an effective test, one must have: </li></ul><ul><ul><ul><li>Detailed understanding of the system </li></ul></ul></ul><ul><ul><ul><li>Knowledge of the testing techniques </li></ul></ul></ul><ul><ul><ul><li>Skill to apply these techniques in an effective and efficient manner </li></ul></ul></ul><ul><li>Testing is done best by independent testers </li></ul><ul><ul><li>We often develop a certain mental attitude that the program should in a certain way when in fact it does not. </li></ul></ul><ul><li>Programmer often uses a data set that makes the program work </li></ul><ul><li>A program often does not work when tried by somebody else. </li></ul><ul><ul><li>Don't let this be the end-user. </li></ul></ul>
  15. 15. Testing Activities Tested Subsystem Subsystem Code Functional Integration Unit Tested Subsystem Requirements Analysis Document System Design Document Tested Subsystem Test Test T est Unit T est Unit T est User Manual Requirements Specification Subsystem Code Subsystem Code All tests by developer Functioning System Integrated Subsystems
  16. 16. Testing Activities ctd Requirements Specification (non-functional) User’s understanding Tests by developer Performance Acceptance Client’s Understanding of Requirements Test Functioning System Test Installation User Environment Test System in Use Usable System Validated System Accepted System Tests by client
  17. 17. Component Testing <ul><li>Unit Testing: </li></ul><ul><ul><li>Individual subsystem </li></ul></ul><ul><ul><li>Carried out by developers </li></ul></ul><ul><ul><li>Goal: Confirm that subsystems is correctly coded and carries out the intended functionality </li></ul></ul><ul><li>Integration Testing: </li></ul><ul><ul><li>Groups of subsystems (collection of classes) and eventually the entire system </li></ul></ul><ul><ul><li>Carried out by developers </li></ul></ul><ul><ul><li>Goal: Test the interface among the subsystem </li></ul></ul>
  18. 18. System Testing <ul><li>System Testing: </li></ul><ul><ul><li>The entire system </li></ul></ul><ul><ul><li>Carried out by developers </li></ul></ul><ul><ul><li>Goal: Determine if the system meets the requirements (functional and non-functional) </li></ul></ul><ul><li>Acceptance Testing: </li></ul><ul><ul><li>Evaluates the system delivered by developers </li></ul></ul><ul><ul><li>Carried out by the client. May involve executing typical transactions on site on a trial basis </li></ul></ul><ul><ul><li>Goal: Demonstrate that the system meets customer requirements and is ready to use </li></ul></ul>
  19. 19. The 4 Testing Steps <ul><li>1. Select what has to be measured </li></ul><ul><li>Code tested for correctness with respect to: </li></ul><ul><ul><li>requirements </li></ul></ul><ul><ul><li>architecture </li></ul></ul><ul><ul><li>detailed design </li></ul></ul><ul><li>2. Decide how the testing is done </li></ul><ul><ul><li>Code inspection </li></ul></ul><ul><ul><li>Proofs </li></ul></ul><ul><ul><li>Black-box, white box, </li></ul></ul><ul><ul><li>Select integration testing strategy (big bang, bottom up, top down, sandwich) </li></ul></ul><ul><li>3. Develop test cases </li></ul><ul><ul><li>A test case is a set of test data or situations that will be used to exercise the unit (code, module, system) being tested or about the attribute being measured </li></ul></ul><ul><li>4. Create the test oracle </li></ul><ul><ul><li>An oracle contains of the predicted results for a set of test cases </li></ul></ul><ul><ul><ul><li>I.e., expected output for each test </li></ul></ul></ul><ul><ul><li>The test oracle has to be written down before the actual testing takes place </li></ul></ul>
  20. 20. System Testing <ul><li>Functional Testing </li></ul><ul><li>Performance Testing </li></ul><ul><li>Acceptance Testing </li></ul><ul><li>Installation Testing </li></ul><ul><li>Impact of model quality on system testing: </li></ul><ul><ul><li>The more explicit the requirements, the easier they are to test. </li></ul></ul><ul><ul><li>Quality of use cases determines the ease of functional testing </li></ul></ul><ul><ul><li>Quality of subsystem decomposition determines the ease of integration testing </li></ul></ul><ul><ul><li>Quality of detailed design determines the ease of unit testing </li></ul></ul>
  21. 21. Functional Testing <ul><li>Black box testing </li></ul><ul><li>Goal: Test functionality of system </li></ul><ul><li>Test cases are designed from the requirements analysis document (better: user manual) and centered around requirements and key functions (use cases) </li></ul><ul><li>The system is treated as black box. </li></ul><ul><li>user oriented test cases have to be developed </li></ul>.
  22. 22. Sample test case – typical content <ul><li>Name </li></ul><ul><li>Purpose </li></ul><ul><li>Pre-conditions for running the test </li></ul><ul><li>Define the input </li></ul><ul><li>Define the expected output </li></ul><ul><li>Test status </li></ul><ul><ul><li>To be run </li></ul></ul><ul><ul><li>Passed </li></ul></ul><ul><ul><li>Failed </li></ul></ul>
  23. 23. Sample test case – example <ul><li>Name: Test Case 42 (add supplier) </li></ul><ul><li>Purpose. The purpose of this test case is to test the normal flow of events for the Add Supplier Use Case </li></ul><ul><li>Pre-conditions for running the test. The Dental Office System has been initialized. </li></ul><ul><li>Define the input. </li></ul><ul><li>1. The user selects the Add Supplier menu option. </li></ul><ul><li>2. The user enters the following data into the form: </li></ul><ul><ul><li>Supplier name: Acme Floss </li></ul></ul><ul><ul><li>Supplier address: 2601 N. Floyd Rd </li></ul></ul><ul><ul><li>Supplier phone: 972 883 4216 </li></ul></ul><ul><ul><li>Supplier fax: 972 883 2349 </li></ul></ul><ul><ul><li>Supplier web: </li></ul></ul><ul><li>3. The user selects the OK button </li></ul><ul><li>Define the expected output. The system displays a message indicating the supplier “Acme Floss” has been added. </li></ul><ul><li>Test status </li></ul><ul><ul><li>To be run </li></ul></ul>
  24. 24. Acceptance Testing <ul><li>Goal: Demonstrate system is ready for operational use </li></ul><ul><ul><li>Choice of tests is made by client/sponsor </li></ul></ul><ul><ul><li>Acceptance test is performed by the client , not by the developer. </li></ul></ul><ul><li>Requirements specification forms the contract between the customer and the developers </li></ul><ul><li>Customer uses the requirements to derive their own test cases… </li></ul>
  25. 25. Test Team Test Analyst Team User Programmer too familiar with code Professional Tester Configuration Management Specialist System Designer
  26. 26. Summary <ul><li>Testing is still a black art, but many rules and heuristics are available </li></ul><ul><ul><li>active research area in software engineering </li></ul></ul><ul><li>Testing consists of component-testing ( unit testing, integration testing) and system testing </li></ul><ul><li>Design Patterns can be used for component-based testing </li></ul><ul><li>Testing has its own lifecycle </li></ul>