Software testing


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software testing

  1. 1. CPSC 333 / SENG 311: Foundations / Principles of SE Software Testing
  2. 2. Agenda <ul><li>Testing definition and introduction </li></ul><ul><li>Test case design </li></ul><ul><li>Blackbox testing </li></ul><ul><li>Whitebox testing </li></ul><ul><li>Path coverage testing </li></ul><ul><li>Loop testing </li></ul><ul><li>Equivalence partitioning </li></ul><ul><li>Boundary value analysis </li></ul>
  3. 3. What is testing? “ Testing is the process of executing a program with the intent of finding errors.” Glen Myers
  4. 4. Limits of software testing <ul><li>“Good” testing will find bugs </li></ul><ul><li>“Good” testing is based on requirements </li></ul><ul><li>BUT : Testing can only prove the presence of bugs— never their absence </li></ul>
  5. 5. Exhaustive testing There are 10 14 possible paths! If we execute one test per millisecond, it would take 3.170 years to test this program!! loop < 20x
  6. 6. <ul><li>Qualities of a testable program: </li></ul><ul><ul><li>operability - it operates cleanly </li></ul></ul><ul><ul><li>observability </li></ul></ul><ul><ul><ul><li>the results are easy to see </li></ul></ul></ul><ul><ul><ul><li>distinct output is generated for each input </li></ul></ul></ul><ul><ul><ul><li>incorrect output is easily identified </li></ul></ul></ul><ul><ul><li>controllability </li></ul></ul><ul><ul><ul><li>processing can be controlled </li></ul></ul></ul><ul><ul><ul><li>tests can be automated & reproduced </li></ul></ul></ul><ul><ul><li>decomposability - software modules can be tested independently </li></ul></ul><ul><ul><li>simplicity - no complex architecture and logic </li></ul></ul><ul><ul><li>stability - few changes are requested during testing </li></ul></ul><ul><ul><li>understandability - program is easy to understand </li></ul></ul>Testability = how easily a program can be tested
  7. 7. Who tests the software Understands the system, but will test “gently”, and is driven by “delivery” Must learn about the system, but will attempt to break it, and is driven by quality developer independent tester
  8. 8. Test cases <ul><li>Describe how to test a system/module/function </li></ul><ul><li>Description must identify </li></ul><ul><ul><li>system state before executing the test </li></ul></ul><ul><ul><li>function to be tested </li></ul></ul><ul><ul><li>parameter values for the test </li></ul></ul><ul><ul><li>expected outcome of the test </li></ul></ul>
  9. 9. Test case examples <ul><li>system state before executing the test </li></ul><ul><ul><li>vehicle not assigned to customer </li></ul></ul><ul><li>function/method to be tested </li></ul><ul><ul><li>rentVehicle(vehicleId) </li></ul></ul><ul><li>parameter values for the test </li></ul><ul><ul><li>vehicleId of the vehicle to be rented </li></ul></ul><ul><li>expected outcome of the test </li></ul><ul><ul><li>vehicle is allocated or reserved for customer </li></ul></ul><ul><li>system state before executing the test </li></ul><ul><ul><li>ResourcePool is not empty </li></ul></ul><ul><li>function to be tested </li></ul><ul><ul><li>removeAgent(anAgent) </li></ul></ul><ul><li>parameter values for the test </li></ul><ul><ul><li>anAgent is in NOT the ResourcePool </li></ul></ul><ul><li>expected outcome of the test </li></ul><ul><ul><li>AgentNotFoundException is thrown </li></ul></ul>
  10. 10. Test case design “ Bugs lurk in corners and congregate at boundaries…” Boris Beizer OBJECTIVE to uncover errors CRITERIA in a complete manner CONSTRAINT with a minimum of effort and time
  11. 11. “Adequate” set of test cases <ul><li>Test the interface of the components </li></ul><ul><ul><li>all public methods </li></ul></ul><ul><li>Is one test case per method enough? </li></ul><ul><li>How many do we need? </li></ul>
  12. 12. Black box testing requirements input events output
  13. 13. Black-box testing <ul><li>Ensure that external user interface is functioning properly. </li></ul><ul><ul><li>give correct inputs, check for expected outputs </li></ul></ul><ul><ul><li>create certain events, and check that system responds appropriately </li></ul></ul><ul><ul><li>ensure that all user requirements are satisfied by system’s user interface and externally observed behavior </li></ul></ul><ul><ul><li>black-box tester NOT looking at guts of system’s implementation </li></ul></ul>
  14. 14. White box testing … our goal is to ensure that all statements and conditions have been executed at least once...
  15. 15. Why bother with white box testing? <ul><li>Black box testing: </li></ul><ul><ul><li>Requirements fulfilled </li></ul></ul><ul><ul><li>Interfaces available and working </li></ul></ul><ul><li>Question: Why white box testing? </li></ul>
  16. 16. Why cover? <ul><li>logic errors and incorrect assumptions are inversely proportional to a path’s execution probability </li></ul><ul><li>we often believe that a path is not likely to be executed; in fact, reality is often counter intuitive </li></ul><ul><li>typographical errors are random; it’s likely that untested paths will contain some </li></ul>
  17. 17. Exhaustive testing There are 5 20 =10 14 (approx.) possible paths! If-then-else loop < 20x
  18. 18. Selective testing a selected path loop < = 20x
  19. 19. Path coverage testing <ul><li>1) Derive a logical complexity measure </li></ul><ul><li>2) Use it to define a basis set of execution paths </li></ul><ul><li>First, we compute the cyclomatic complexity: </li></ul><ul><li>Number of edges – number of nodes + 1 </li></ul><ul><li>(Compound decisions must be split into multiple nodes) </li></ul><ul><li>In this case, CC(G) = 4 </li></ul><ul><li>CC(G) provides a lower bound of tests that must be executed to guarantee coverage of all program statements </li></ul>
  20. 20. Cyclomatic complexity A number of industry studies have indicated that the higher CC(G), the higher the probability of errors. modules CC(G) modules in this range are more error prone
  21. 21. Path coverage set <ul><li>Path coverage set = set of paths that will execute all statements and all conditions in a program at least once </li></ul><ul><li>Goal: Define test cases for basis set </li></ul><ul><li>Path coverage set is not unique </li></ul>
  22. 22. Path coverage testing Next, we derive the independent paths: Since CC(G) = 4, there are four paths Path 1: 1,2,3,6,7,8 Path 2: 1,2,3,5,7,8 Path 3: 1,2,4,7,8 Path 4: 1,2,4,7,2,4…7,8 Finally, we derive test cases to exercise these paths. 1 2 4 7 8 3 6 5
  23. 23. Troubles with cyclomatic complexity for test coverage <ul><li>case A is </li></ul><ul><li>when “One” => i := 1; </li></ul><ul><li>when “Two” => i := 2; </li></ul><ul><li>when “Three” => i := 3; </li></ul><ul><li>when “Four” => i := 4; </li></ul><ul><li>when “Five” => i := 5; </li></ul><ul><li>end case; </li></ul>Str: array(1..5) of STRING := (“One”, “Two”, “Three”, “Four”, “Five”); i := 1; loop exit when Str(i) = A; i := i + 1; end loop; <ul><li>CC = 5 and 1 respectively, but same functionality => should be tested the same </li></ul>Example from Sommerville:
  24. 24. Documenting test cases <ul><li>Name </li></ul><ul><li>Number </li></ul><ul><li>Description of system state before running the test case </li></ul><ul><li>Values for the inputs </li></ul><ul><li>Expected outputs </li></ul><ul><li>Short description (if needed) </li></ul>
  25. 25. Loops <ul><li>Cornerstone of every program </li></ul><ul><li>Loops can lead to non-terminating programs </li></ul>
  26. 26. Loop testing simple loop nested loops concatenated loops unstructured loops
  27. 27. Loop testing: simple loops <ul><li>Minimum conditions - simple loops </li></ul><ul><ul><li>1. skip the loop entirely </li></ul></ul><ul><ul><li>2. only one pass through the loop </li></ul></ul><ul><ul><li>3. two passes through the loop </li></ul></ul><ul><ul><li>4. m passes through the loop m < n </li></ul></ul><ul><ul><li>5. (n-1), n, and (n+1) passes through the loop </li></ul></ul><ul><ul><li>where n is the maximum number of allowable passes </li></ul></ul>
  28. 28. Nested loops <ul><li>Just extending simple loop testing: number of tests explodes </li></ul><ul><li>Reduce the number of tests: </li></ul><ul><ul><li>start at the innermost loop; set all other loops to minimum values </li></ul></ul><ul><ul><li>conduct simple loop test; add out of range or excluded values </li></ul></ul><ul><ul><li>work outwards while keeping inner nested loops to typical values </li></ul></ul><ul><ul><li>continue until all loops have been tested </li></ul></ul>
  29. 29. Black box testing requirements input events output
  30. 30. Categories of errors in black box testing <ul><li>incorrect or missing functions </li></ul><ul><li>interface errors </li></ul><ul><li>errors in data structures or external database access </li></ul><ul><li>performance errors </li></ul><ul><li>initialization and termination errors </li></ul>
  31. 31. Equivalence partitioning user queries numerical data output format requests responses to prompts command key input mouse picks on menu Partitioning is based on input conditions
  32. 32. Equivalence classes (1) <ul><li>Valid data </li></ul><ul><ul><li>user supplied commands </li></ul></ul><ul><ul><li>responses to system prompts </li></ul></ul><ul><ul><li>file names </li></ul></ul><ul><ul><li>computational data physical parameters bounding values initiation values </li></ul></ul><ul><ul><li>output data formatting commands </li></ul></ul><ul><ul><li>responses to error messages </li></ul></ul><ul><ul><li>graphical data (e.g., mouse picks) </li></ul></ul>
  33. 33. Equivalence classes (2) <ul><li>Invalid data </li></ul><ul><ul><li>data outside bounds of the program </li></ul></ul><ul><ul><li>physically impossible data </li></ul></ul><ul><ul><li>proper value supplied in wrong place </li></ul></ul>
  34. 34. Defining equivalence classes <ul><li>Input condition is a range: one valid and two invalid classes are defined </li></ul><ul><li>Input condition requires specific value: one valid and two invalid classes are defined </li></ul><ul><li>Input condition is boolean: one valid and one invalid class are defined </li></ul>
  35. 35. Boundary value analysis output domain user queries numerical data output format requests responses to prompts command key input mouse picks on menu
  36. 36. Boundary value testing <ul><li>Range a..b </li></ul><ul><ul><li>test cases: a, b, just below a, just above b </li></ul></ul><ul><li>Number of values: </li></ul><ul><ul><li>test cases: max, min, just below min, just above max </li></ul></ul><ul><li>Output bounds should be checked </li></ul><ul><li>Boundaries of data structures shall be checked (e.g. arrays) </li></ul>
  37. 37. Automating software testing <ul><li>Manual software testing is time consuming </li></ul><ul><li>Software testing has to be repeated after every change (regression testing) </li></ul><ul><li>Write test drivers that can run automatically and produce a test report </li></ul>
  38. 38. We’re Available! <ul><li>Questions? </li></ul><ul><ul><li>if you have any questions about contents of this lecture or other course-related issues, please come by during our office hours, or send us email </li></ul></ul><ul><ul><li>Dr. Joshua: MTW, 12-1pm, ICT 548 </li></ul></ul><ul><ul><ul><li>[email_address] </li></ul></ul></ul><ul><ul><li>Dr. Walker: WF, 1-2pm, ICT 546 </li></ul></ul><ul><ul><ul><li>[email_address] </li></ul></ul></ul>