Testing Course - Birgitta


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Particullary important during regression testing and debugging
  • Non-determinstic testing -- studies performed in 1995 – 15 of 420 execution orders was observed in 50 000 runs… but if timing or synchronization behavior change in some threads, or if the scheduler behaves differently, the new orders may occur
  • Even if worst case w.r.t. timeliness is easy to find – timeliness needs to be verified
  • Behavior when assumptions about the environment are broken/changed..
  • Testing Course - Birgitta

    1. 1. Software Testing A Crash Course in SW testing Techniques and Concepts
    2. 2. Overview <ul><li>Introduction to testing concepts </li></ul><ul><li>Levels of testing </li></ul><ul><li>General test strategies </li></ul><ul><li>Testing concurrent systems </li></ul><ul><li>Testing of RT systems </li></ul>
    3. 3. Testing is necessary <ul><li>To gain a sufficient level of confidence for the system </li></ul><ul><ul><li>Risk information </li></ul></ul><ul><ul><li>Bug information </li></ul></ul><ul><ul><li>Process information </li></ul></ul><ul><li>Perfect development process infeasible </li></ul><ul><ul><li>Building without faults implies early testing </li></ul></ul><ul><li>Formal methods not sufficient </li></ul><ul><ul><li>Can only prove conformance to a model </li></ul></ul><ul><ul><li>Perfect requirements are cognitive infeasible </li></ul></ul><ul><ul><li>Error prone </li></ul></ul>
    4. 4. Testing for quality assurance <ul><li>Traditionally testing focuses on functional attributes </li></ul><ul><ul><li>E.g. correct calculations </li></ul></ul><ul><li>Non-functional attributes equally important </li></ul><ul><ul><li>E.g. reliability, availability, timeliness </li></ul></ul>
    5. 5. How much shall we test? <ul><li>Testing usually takes about half of development resources </li></ul><ul><li>Stop testing is a business decision </li></ul><ul><ul><li>There is always something more to test </li></ul></ul><ul><ul><li>Risk based decision. Tester provides risk estimation </li></ul></ul>
    6. 6. When do we test? <ul><li>The earlier a fault is found the less expensive it is to correct it </li></ul><ul><li>Testing is not only concerning code. </li></ul><ul><ul><li>Documents and models should also be subject to testing. </li></ul></ul><ul><li>As soon as a document is produced, testing can start. </li></ul>
    7. 7. Levels of Testing <ul><li>Component/Unit testing </li></ul><ul><li>Integration testing </li></ul><ul><li>System testing </li></ul><ul><li>Acceptance testing </li></ul><ul><li>Regression testing </li></ul>
    8. 8. Component Testing (1/2) <ul><li>Require knowledge of code </li></ul><ul><ul><li>High level of detail </li></ul></ul><ul><ul><li>Deliver thoroughly tested components to integration </li></ul></ul><ul><li>Stopping criteria </li></ul><ul><ul><li>Code Coverage </li></ul></ul><ul><ul><li>Quality </li></ul></ul>
    9. 9. Component Testing (2/2) <ul><li>Test case </li></ul><ul><ul><li>Input, expected outcome, purpose </li></ul></ul><ul><ul><li>Selected according to a strategy, e.g., branch coverage </li></ul></ul><ul><li>Outcome </li></ul><ul><ul><li>Pass/fail result </li></ul></ul><ul><ul><li>Log, i.e., chronological list of events from execution </li></ul></ul>
    10. 10. Integration Testing (1/2) <ul><li>Test assembled components </li></ul><ul><ul><li>These must be tested and accepted previously </li></ul></ul><ul><li>Focus on interfaces </li></ul><ul><ul><li>Might be interface problem although components work when tested in isolation </li></ul></ul><ul><ul><li>Might be possible to perform new tests </li></ul></ul>
    11. 11. Integration Testing (2/2) <ul><li>Strategies </li></ul><ul><ul><li>Bottom-up, start from bottom and add one at a time </li></ul></ul><ul><ul><li>Top-down, start from top and add one at a time </li></ul></ul><ul><ul><li>Big-bang, everything at once </li></ul></ul><ul><ul><li>Functional, order based on execution </li></ul></ul><ul><li>Simulation of other components </li></ul><ul><ul><li>Stubs receive output from test objects </li></ul></ul><ul><ul><li>Drivers generate input to test objects </li></ul></ul><ul><ul><li>Note that these are also SW, i.e., need testing etc. </li></ul></ul>
    12. 12. System Testing (1/2) <ul><li>Functional testing </li></ul><ul><ul><li>Test end to end functionality </li></ul></ul><ul><ul><li>Requirement focus </li></ul></ul><ul><ul><ul><li>Test cases derived from specification </li></ul></ul></ul><ul><ul><li>Use-case focus </li></ul></ul><ul><ul><ul><li>Test selection based on user profile </li></ul></ul></ul>
    13. 13. System Testing (2/2) <ul><li>Non-functional testing </li></ul><ul><li>Quality attributes </li></ul><ul><ul><li>Performance, can the system handle required throughput? </li></ul></ul><ul><ul><li>Reliability, obtain confidence that system is reliable </li></ul></ul><ul><ul><li>Timeliness, testing whether the individual tasks meet their specified deadlines </li></ul></ul><ul><ul><li>etc. </li></ul></ul>
    14. 14. Acceptance Testing <ul><li>User (or customer) involved </li></ul><ul><li>Environment as close to field use as possible </li></ul><ul><li>Focus on: </li></ul><ul><ul><li>Building confidence </li></ul></ul><ul><ul><li>Compliance with defined acceptance criteria in the contract </li></ul></ul>
    15. 15. Re-Test and Regression Testing <ul><li>Conducted after a change </li></ul><ul><li>Re-test aims to verify whether a fault is removed </li></ul><ul><ul><li>Re-run the test that revealed the fault </li></ul></ul><ul><li>Regression test aims to verify whether new faults are introduced </li></ul><ul><ul><li>Re-run all tests </li></ul></ul><ul><ul><li>Should preferably be automated </li></ul></ul>
    16. 16. Strategies <ul><li>Code coverage strategies, e.g. </li></ul><ul><ul><li>Decision coverage </li></ul></ul><ul><ul><li>Data-flow testing (Defines -> Uses) </li></ul></ul><ul><li>Specification-based testing, e.g. </li></ul><ul><ul><li>Equivalence partitioning </li></ul></ul><ul><ul><li>Boundary-value analysis </li></ul></ul><ul><ul><li>Combination strategies </li></ul></ul><ul><li>State-based testing </li></ul>
    17. 17. Code Coverage (1/2) <ul><li>Statement coverage </li></ul><ul><ul><li>Each statement should be executed by at least one test case </li></ul></ul><ul><ul><li>Minimum requirement </li></ul></ul>
    18. 18. Code Coverage (2/2) <ul><li>Branch/Decision coverage </li></ul><ul><ul><li>All decisions with true and false value </li></ul></ul><ul><ul><ul><li>Subsumes all statements coverage </li></ul></ul></ul><ul><ul><li>MC/DC for each variable x in a boolean condition B , let x decide the value of B and test with true and false value </li></ul></ul><ul><ul><ul><li>Example: </li></ul></ul></ul><ul><ul><ul><li>if( x 1 and x 2 ) { </li></ul></ul></ul><ul><ul><ul><li>S </li></ul></ul></ul><ul><ul><ul><li>} </li></ul></ul></ul><ul><ul><ul><li>Used for safety critical applications </li></ul></ul></ul>false true false true x 1 and x 2 false true true true true false true true x 2 x 1
    19. 19. Mutation testing <ul><li>Create a number of mutants, i.e., faulty versions of program </li></ul><ul><ul><li>Each mutant contains one fault </li></ul></ul><ul><ul><li>Fault created by using mutant operators </li></ul></ul><ul><li>Run test on the mutants (random or selected) </li></ul><ul><ul><li>When a test case reveals a fault, save test case and remove mutant from the set, i.e., it is killed </li></ul></ul><ul><ul><li>Continue until all mutants are killed </li></ul></ul><ul><li>Results in a set of test cases with high quality </li></ul><ul><li>Need for automation </li></ul>
    20. 20. Specification-based testing (1/2) <ul><li>Test cases derived from specification </li></ul><ul><li>Equivalence partitioning </li></ul><ul><ul><li>Identify sets of input from specification </li></ul></ul><ul><ul><ul><li>Assumption: if one input from set s leads to a failure, then all inputs from set s will lead to the same failure </li></ul></ul></ul><ul><ul><li>Chose a representative value from each set </li></ul></ul><ul><ul><li>Form test cases with the chosen values </li></ul></ul>
    21. 21. Specification-based testing (2/2) <ul><li>Boundary value analysis </li></ul><ul><ul><li>Identify boundaries in input and output </li></ul></ul><ul><ul><li>For each boundary: </li></ul></ul><ul><ul><ul><li>Select one value from each side of boundary (as close as possible) </li></ul></ul></ul><ul><ul><li>Form test cases with the chosen values </li></ul></ul>
    22. 22. Combination Strategies (1/5) <ul><li>Equivalence partitioning and boundary analysis give representative parameter values </li></ul><ul><li>Test case often contain more than one input parameter </li></ul><ul><li>How do we form efficient test suites, i.e., how do we combine parameter values? </li></ul>
    23. 23. Combination Strategies (2/5) <ul><li>Each choice </li></ul><ul><ul><li>Each chosen parameter value occurs in at least one test case </li></ul></ul><ul><li>Assume 3 parameters A, B, and C with 2, 2, and 3 values </li></ul>TC 3 TC 2 TC 1 Test case 3 any any 2 2 2 1 1 1 C B A
    24. 24. Combination Strategies (3/5) <ul><li>Pair-wise combinations </li></ul><ul><ul><li>Each pair of chosen values occurs in at least one test case </li></ul></ul><ul><ul><li>Efficiently generated by latin squares or a heuristic algorithm </li></ul></ul><ul><li>Cover failures due to pairs of input values </li></ul>Latin square 1 3 2 2 1 3 3 2 1 1 any any TC 9 2 2 any TC 8 3 1 any TC 7 3 any 2 TC 6 1 2 2 TC 5 2 1 2 TC 4 2 any 1 TC 3 3 2 1 TC 2 1 1 1 TC 1 C B A
    25. 25. Combination Strategies (4/5) <ul><li>All combinations, i.e., n -wise </li></ul><ul><ul><li>Each combination of chosen values occurs in at least one test case </li></ul></ul><ul><ul><li>Very expensive </li></ul></ul>3 2 2 TC 12 2 2 2 TC 11 1 2 2 TC 10 3 1 2 TC 9 2 1 2 TC 8 1 1 2 TC 7 3 2 1 TC 6 2 2 1 TC 5 1 2 1 TC 4 2 1 1 TC 3 2 1 1 TC 2 1 1 1 TC 1 C B A
    26. 26. Combination Strategies (5/5) <ul><li>Base choice </li></ul><ul><ul><li>For each parameter, define a base choice, i.e., the most likely value </li></ul></ul><ul><ul><li>Let this be base test case and vary one value at a time </li></ul></ul><ul><ul><li>In the example, let A-1, B-2, and C-2 be base choices </li></ul></ul>3 2 1 TC 5 1 2 1 TC 4 2 1 1 TC 3 2 2 2 TC 2 2 2 1 TC 1 C B A
    27. 27. State-Based Testing <ul><li>Model functional behavior in a state machine </li></ul><ul><li>Select test cases in order to cover the graph </li></ul><ul><ul><li>Each node </li></ul></ul><ul><ul><li>Each transition </li></ul></ul><ul><ul><li>Each pair of transitions </li></ul></ul><ul><ul><li>Each chain of transitions of length n </li></ul></ul>
    28. 28. Concurrency Problems <ul><li>(Logical) Parallelism often leads to non-determinism in actual execution </li></ul><ul><ul><li>E.g. Synchronization errors may occur in some execution orders </li></ul></ul><ul><ul><li>Order may influence the arithmetic results or the timing </li></ul></ul><ul><li>Explosion in number of required tests </li></ul><ul><ul><li>Need confidence for all execution orders that might occur </li></ul></ul>
    29. 29. Silly example of Race Conditions with shared data W(A,1) R(B, 1 ) W(B,1) R(A, 1 ) W(A,1) R(B,1) W(B,1) R(A, 0 ) W(A,1) R(B, 0 ) W(B,1) R(A,1)
    30. 30. Observability Issues <ul><li>Probe effect (Gait,1985) </li></ul><ul><ul><li>“ Heisenbergs's principle” - for computer systems </li></ul></ul><ul><ul><li>Common “solutions” </li></ul></ul><ul><ul><ul><li>Compensate </li></ul></ul></ul><ul><ul><ul><li>Leave probes in system </li></ul></ul></ul><ul><ul><ul><li>Ignore </li></ul></ul></ul><ul><li>Must observe execution orders </li></ul><ul><ul><li>Gain coverage </li></ul></ul>
    31. 31. Controllability Issues <ul><li>To be able to test correctness of a particular execution order we need control </li></ul><ul><ul><li>Input data to all tasks </li></ul></ul><ul><ul><ul><li>Initial state of shared data/buffers </li></ul></ul></ul><ul><ul><li>Scheduling decisions </li></ul></ul><ul><ul><ul><li>Order synchronization/communication between tasks </li></ul></ul></ul>
    32. 32. Few testing criteria exists for concurrent systems <ul><li>Number of execution orders grow exponentially with # synchronization primitives in tasks </li></ul><ul><ul><li>Testing criteria needed to bound and selecting subset of execution orders for testing </li></ul></ul><ul><li>E.g. Branch / Statement coverage not sufficient for concurrent software </li></ul><ul><ul><li>Still useful on serializations </li></ul></ul><ul><ul><li>Execution paths may require specific behavior from other tasks </li></ul></ul><ul><li>Data-flow based testing criteria has been adapted </li></ul><ul><ul><li>E.g. define-use pairs </li></ul></ul>
    33. 33. Summary : Determinism vs. Non-Determinism <ul><li>Deterministic systems </li></ul><ul><ul><li>Controllability is high </li></ul></ul><ul><ul><ul><li>input (sequence) suffice </li></ul></ul></ul><ul><ul><li>Coverage can be claimed after single test execution with inputs </li></ul></ul><ul><ul><li>E.g. Filters, Pure “table-driven” real-time systems </li></ul></ul><ul><li>Non-Deterministic systems </li></ul><ul><ul><li>Controllability is generally low </li></ul></ul><ul><ul><li>Statistical methods needed in combination with input coverage </li></ul></ul><ul><ul><li>E.g. </li></ul></ul><ul><ul><ul><li>Systems that use random heuristics </li></ul></ul></ul><ul><ul><ul><li>Behavior depends on execution times / race conditions </li></ul></ul></ul>
    34. 34. Test execution in concurrent systems <ul><ul><li>Non-deterministic testing </li></ul></ul><ul><ul><ul><li>“ Run, Run, Run and Pray” </li></ul></ul></ul><ul><ul><li>Deterministic testing </li></ul></ul><ul><ul><ul><li>Select a particular execution order and force it </li></ul></ul></ul><ul><ul><ul><li>E.g. Instrument with extra synchronizations primitives </li></ul></ul></ul><ul><ul><ul><ul><li>(No timing constraints make this possible) </li></ul></ul></ul></ul><ul><ul><li>Prefix-based Testing (and Replay) </li></ul></ul><ul><ul><ul><li>Deterministically run system to a specific (prefix) point </li></ul></ul></ul><ul><ul><ul><li>Start non-deterministic testing at that specific point </li></ul></ul></ul>
    35. 35. Real-time systems testing <ul><li>Inherits issues from concurrent systems </li></ul><ul><ul><li>Problems becomes harder due to time-constraints </li></ul></ul><ul><ul><ul><li>More sensitive to probe-effects </li></ul></ul></ul><ul><ul><ul><li>Timing/order of inputs become more significant </li></ul></ul></ul><ul><li>Adds new potential problems </li></ul><ul><ul><li>New failure types </li></ul></ul><ul><ul><ul><li>E.g. Missed deadlines, Too early responses… </li></ul></ul></ul><ul><ul><li>Test inputs  Execution times </li></ul></ul><ul><ul><li>Faults in real-time scheduling </li></ul></ul><ul><ul><ul><li>Algorithm implementation errors </li></ul></ul></ul><ul><ul><ul><li>Assumption about system wrong </li></ul></ul></ul>
    36. 36. Real-time systems testing <ul><li>Pure time-triggered systems </li></ul><ul><ul><li>Deterministic </li></ul></ul><ul><ul><li>Test-methods for sequential software usually apply </li></ul></ul><ul><li>Fixed priority scheduling </li></ul><ul><ul><li>Non-deterministic </li></ul></ul><ul><ul><ul><li>Limited set of possible execution orders </li></ul></ul></ul><ul><ul><li>Worst-case w.r.t timeliness can be found from analysis </li></ul></ul><ul><li>Dynamic (online) scheduled systems </li></ul><ul><ul><li>Non-deterministic </li></ul></ul><ul><ul><ul><li>Large set of possible execution orders </li></ul></ul></ul><ul><ul><li>Timeliness needs to be tested </li></ul></ul>
    37. 37. Testing Timeliness <ul><li>Aim : Verification of specified deadlines for individual tasks </li></ul><ul><ul><li>Test if assumptions about system hold </li></ul></ul><ul><ul><ul><li>E.g. worst-case execution time estimates, overheads, context switch times, hardware acceleration efficency, I/O latency, blocking times, dependency-assumptions </li></ul></ul></ul><ul><ul><li>Test system temporal behavior under stress </li></ul></ul><ul><ul><ul><li>E.g. Unexpected job requests, overload management, component failure, admission control scheme </li></ul></ul></ul><ul><li>Identification of potential worst-case execution orders </li></ul><ul><li>Controllability needed to test worst-case situations efficiently </li></ul>
    38. 38. Testing Embedded Systems <ul><li>System-level testing differs </li></ul><ul><ul><li>Performed on target platform to keep timing </li></ul></ul><ul><li>Closed-loop testing </li></ul><ul><ul><li>Test-cases consist of parameters sent to the environment simulator </li></ul></ul><ul><li>Open-loop testing </li></ul><ul><ul><li>Test-cases contain sequences of events that the system should be able to handle </li></ul></ul>Environment Simulator Real-time (control) system Test parameters Real-time (control) system Test Cases
    39. 39. Approach of TETReS <ul><li>Real-time database background </li></ul><ul><ul><li>Dynamic scheduling, flexibility requirements, mixed load… </li></ul></ul><ul><li>Two general approaches </li></ul><ul><ul><li>Investigate how architectural constraints impacts testability </li></ul></ul><ul><ul><ul><li>Keep flexibility – Limit/avoid testing problems </li></ul></ul></ul><ul><ul><ul><li>E.g. Database environment provide some nice properties </li></ul></ul></ul><ul><ul><li>Investigate methods for supporting automated testing of timeliness </li></ul></ul><ul><ul><ul><li>Testing criteria for timeliness </li></ul></ul></ul><ul><ul><ul><li>Generation of tests automatically </li></ul></ul></ul><ul><ul><ul><li>Automated, prefix-based test execution </li></ul></ul></ul>
    40. 40. Approach of TETReS test generation and execution <ul><li>Open-loop </li></ul><ul><li>System-level testing of timeliness </li></ul><ul><li>Model-based test-case generation </li></ul><ul><ul><li>Architecture information </li></ul></ul><ul><ul><ul><li>E.g. Locking policies, Scheduling etc… </li></ul></ul></ul><ul><ul><li>Timing assumptions/constraints of tasks </li></ul></ul><ul><ul><li>Assumptions about normal environment behavior </li></ul></ul><ul><li>Mutation-based testing criteria </li></ul><ul><li>Prefix-based Test-case execution </li></ul>
    41. 41. Summary <ul><li>Motivation </li></ul><ul><li>Test methods </li></ul><ul><ul><li>Examples on different test strategies has been presented </li></ul></ul><ul><li>Concurrency and Real-time issues </li></ul><ul><li>TETReS approach </li></ul>
    42. 42. Mutant Generator Model Checker Test Specs. Generator Test coordinator State Enforcer Test Driver Testing Criteria Formal modeling Task unit Testing RTS Spec. Mutated Specs. Counter Example Traces Test Specs Input Data to Tasks Parameterized event sequences Pre-State description