12 functional-system-testing


Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

12 functional-system-testing

  1. 1. Functional System Testing Written by Adam Carmi
  2. 2. Outline <ul><li>Goal of testing </li></ul><ul><li>Test cases, test suites and test data </li></ul><ul><li>What is functional system testing? </li></ul><ul><li>Coverage </li></ul><ul><li>Functional testing techniques: </li></ul><ul><ul><li>Functional analysis </li></ul></ul><ul><ul><li>Equivalence partitioning </li></ul></ul><ul><ul><li>Boundary value analysis </li></ul></ul>
  3. 3. The goal of software testing <ul><li>The process of uncovering evidence of defects in software systems </li></ul><ul><ul><li>Does not include efforts associated with tracking down bugs and fixing them </li></ul></ul><ul><li>No amount of testing will improve the quality of a computer program </li></ul><ul><ul><li>The more testing we do of a system, the more convinced we might be of its correctness </li></ul></ul><ul><ul><li>Testing cannot in general prove a system works 100% correctly </li></ul></ul>
  4. 4. Test cases <ul><li>The basic component of testing is a Test Case </li></ul><ul><li>In its most general form: ( inputs, expected-result) </li></ul><ul><ul><li>inputs include system state, user commands and data values to be processed </li></ul></ul><ul><ul><li>expected result includes visible/audible interface changes or changes in the system state </li></ul></ul><ul><li>Test cases are organized into Test Suites </li></ul><ul><ul><li>functionality, security, performance, … </li></ul></ul>
  5. 5. Test case execution <ul><li>A running of the software (under test) that provides the inputs specified in the test case and observing the results and comparing them to those specified by the test case </li></ul><ul><ul><li>If the actual result varies from the expected result, then a failure has been detected </li></ul></ul>
  6. 6. Test data <ul><li>An effective test strategy requires careful acquisition and preparation of test data prior to testing </li></ul><ul><ul><li>Testing can suffer if test data is poor </li></ul></ul><ul><li>Test data concerns: </li></ul><ul><ul><li>Depth : quantity and size of data </li></ul></ul><ul><ul><li>Breadth : variance of data values and data types </li></ul></ul><ul><ul><li>Scope : completeness, relevance and accuracy of data </li></ul></ul><ul><ul><ul><li>Result of a query should be valid for the specific purpose of the query, and not due to a missing or inappropriate value </li></ul></ul></ul><ul><ul><li>Conditions : data should reflect specific “conditions” in the domain </li></ul></ul><ul><ul><ul><li>Data that would otherwise arrive after performing specific operations over time </li></ul></ul></ul><ul><li>Test data and test results are expensive to construct </li></ul>
  7. 7. Example: Test data for TVRS … … … … 8700342 longLastN longFirstN 000345 8245731 Josef Dina 255245 1346329 Porat Dan 264683 237812 Dan Levi 6386541 243567 Rachel Josef 8700342 Violation ID Offender’s first name Offender’s last name Issuing policeman ID Description: Violation records designed for validating violation lookup Name: test1.db
  8. 8. Specification Vs. Implementation <ul><li>The basic approaches to testing software are based on its specification and implementation </li></ul><ul><li>White box testing – test cases and data are constructed based on the code that implements the software </li></ul><ul><ul><li>Quality and correctness of computations is validated </li></ul></ul><ul><ul><li>Will not be further discussed in this tutorial </li></ul></ul><ul><li>Black box testing – test cases and data are constructed based solely on the software’s specification </li></ul>
  9. 9. Functional System Testing <ul><li>Testing of a completed application to determine that it provides all of the behaviors required of it </li></ul><ul><ul><li>Testing of completed increments that provide some degree of end-user functionality </li></ul></ul><ul><li>Search for defects that are variances between the actual operation of the system and the requirements for the system </li></ul><ul><li>System is treated as a black box </li></ul>
  10. 10. How much testing is adequate? <ul><li>Completely validating IEEE 754 floating-point division requires 2 64 test-cases! </li></ul><ul><li>float divide(float x, float y) </li></ul><ul><li>From practical and economic perspectives, exhaustive testing is usually not possible </li></ul><ul><ul><li>Which software pieces should we test? </li></ul></ul><ul><ul><li>Which test cases should we choose? </li></ul></ul>
  11. 11. Coverage <ul><li>Coverage is a measure of how completely a test suite exercises the capabilities of a piece of software </li></ul><ul><ul><li>“ Each line of code should be executed at least once” </li></ul></ul><ul><ul><li>“ One test case should be constructed from each specified requirement” </li></ul></ul><ul><li>It is necessary to use testing techniques that narrow down the number of test cases allowing the broadest testing coverage with the least effort </li></ul>
  12. 12. Technique: Functional Analysis <ul><li>Analyze the expected behavior of the system according to its functional specification </li></ul><ul><li>Generate a test procedure for each of the possible usage scenarios </li></ul><ul><ul><li>Corresponds to use case scenarios </li></ul></ul><ul><ul><li>Analyze how a change in one part of the system affects other parts </li></ul></ul><ul><ul><li>“ Grand tour” test cases: the result of one test case produces the data that is the input to the next test case </li></ul></ul>
  13. 13. Example: Functional Analysis I <ul><li>Use Case: Remove Traffic Violation </li></ul><ul><ul><li>Supervisor calls for deletion of the chosen Traffic Violation </li></ul></ul><ul><ul><li>TVRS prompts Supervisor for confirmation </li></ul></ul><ul><ul><li>Supervisor confirms </li></ul></ul><ul><ul><li>TVRS requests OffendersDB to delete the Traffic Violation from the offender’s record </li></ul></ul><ul><ul><li>OffendersDB approves that the Traffic Violation has been deleted </li></ul></ul><ul><ul><li>TVRS allows Supervisor to look up a new Traffic Violation as described in the “Lookup Traffic Violation” UC </li></ul></ul>
  14. 14. Example: Functional Analysis II Defect diagnosis: Actual results: Passed Failed Test results A message dialog stating that violation “243567” is not stored in TVRS Enter “243567” at “Violation ID” text field and press the “Search” button “ Lookup Violation” dialog is displayed Press the “Yes” button Confirmation dialog is displayed Press “Delete” button Expected result Action Related use cases: “Lookup Traffic Violation”, “Remove Traffic Violation” Pre-conditions: 1. TVRS initialized with test1.db database 2. Violation 243567 displayed in the “Lookup Violation” dialog Test case ID: 134543
  15. 15. Example: Functional Analysis III Verify effects of change Filled when the test case is executed How do we know that violation 243567 is stored in the system? Can a tester diagnose the cause of a defect? In addition, a query could be run on the Offenders database
  16. 16. Technique: Equivalence Partitioning <ul><li>Identifies ranges of input and initial conditions that are expected to produce the same result </li></ul><ul><li>A group of test cases form an equivalence class if: </li></ul><ul><ul><li>They test the same feature/scenario </li></ul></ul><ul><ul><li>If one test reveals a fault, the other ones (probably) will too </li></ul></ul><ul><ul><li>If a test does not reveal a fault, the other ones (probably) will not either </li></ul></ul><ul><li>It is adequate to use only a single representative of the equivalence class </li></ul>
  17. 17. Example: Equivalence Partitioning I <ul><li>Input value specification for “Lookup Violation” form: </li></ul>[a-zA-Z]{0, 10} Offender’s last name [a-zA-Z]{0, 10} Offender’s first name [0-9]{0, 9} Violation ID Valid values Field name
  18. 18. Example: Equivalence Partitioning II … … … … “” Empty [email_address] Invalid character Rachel Single known violation Dan Many known violations -1, 1234567890 ID < 0 or ID > 999999999 00243567 Known violation Violation ID 23ab@ Non numeric ID 32456720 Unknown violation “” Empty Offender’s first name Unknown violation David Character# > 10 Hasalongname Field Valid equivalent classes Valid representative values Invalid equivalent classes Invalid representative values
  19. 19. Example: Equivalence Partitioning III <ul><li>The number of test cases to choose from is reduced to (3 + 2) × (4 + 2) × (4 + 2) = 180 </li></ul><ul><li>The actual number can be further limited </li></ul><ul><ul><li>Single invalid field per test case (3 × 4 × 4 + 6 = 54) </li></ul></ul><ul><ul><li>Importance of use case </li></ul></ul><ul><ul><li>Resources available </li></ul></ul><ul><ul><li>Most frequent input </li></ul></ul><ul><ul><li>Life-critical software </li></ul></ul><ul><ul><li>Infeasible test cases </li></ul></ul><ul><ul><li>Randomly </li></ul></ul><ul><ul><li>... </li></ul></ul>
  20. 20. Technique: Boundary Value Analysis <ul><li>Based on experience / heuristics </li></ul><ul><ul><li>Testing boundary conditions of equivalence classes is more effective </li></ul></ul><ul><li>Choose input boundary values as equivalence classes representatives </li></ul><ul><li>Choose inputs that invoke output boundary values </li></ul><ul><li>Examples: </li></ul><ul><ul><li>(0, 10] ⇒ validate using 0, 1, 2, 9, 10, 11 </li></ul></ul><ul><ul><li>Read up to 5 elements ⇒ validate reading 0, 1, 4, 5, 6 elements </li></ul></ul>
  21. 21. BVA as an equivalence partitioning extension <ul><li>Choose one (or more) arbitrary value(s) in each equivalence class </li></ul><ul><li>Choose valid values exactly on lower and upper boundaries of equivalence class </li></ul><ul><li>Choose invalid values immediately below and above each boundary (if applicable) </li></ul>
  22. 22. General purpose test-suite construction technique <ul><li>May be used to obtain reasonable coverage with little effort </li></ul><ul><ul><li>Use cautiously! </li></ul></ul><ul><ul><li>Unsuitable when values of different fields are related </li></ul></ul><ul><li>While test cases can be added </li></ul><ul><ul><li>Each new test case should include as many un-included valid non-boundary equivalence class representatives as possible </li></ul></ul><ul><li>While test cases can be added </li></ul><ul><ul><li>Each new test case should include as many un-included valid boundary equivalence class representatives as possible </li></ul></ul><ul><li>While test cases can be added </li></ul><ul><ul><li>Each new test case should include a single invalid equivalence class representative that has not been included before </li></ul></ul><ul><li>Manually replace/remove redundant or infeasible test-cases </li></ul>
  23. 23. Example: Country Club I Specification 45 15 50 30 15 35 [60, 120] [16, 60) 50 25 45 70 25 65 [0, 16) 25 10 20 35 10 30 Age (years) Admission fee Guest status Visitor Member Student Visitor Member Student Day Sunday - Thursday Friday - Saturday
  24. 24. Example: Country Club II A combo box is used for choosing the day and guest status 100, 60 , 120 [60, 120] Member Member Student Student Fri, Sat Fri - Sat [16, 60) 34, 16 , 59 Age < 0 or Age > 120 -1 , 121 Age [0, 16) 2, 0 , 15 Non-numeric value [email_address] Guest status Visitor Visitor Day Sun - Thu Mon, Sun, Thu Field Valid equivalent classes Valid representative values Invalid equivalent classes Invalid representative values
  25. 25. Example: Country Club III valid valid (boundary) invalid Invalid age 121 Visitor Fri 12 Invalid age -1 Student Fri 11 Invalid age [email_address] Member Mon 10 9 Thu Visitor 59 50 8 Sat Student 120 45 7 Sun Member 15 10 6 Thu Student 60 30 5 Sat Member 16 10 4 Sun Visitor 0 25 3 Mon Student 100 30 2 Fri Member 34 25 1 Mon Visitor 2 25 Test case ID Day Guest status Age Result