Integration Testing


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Integration Testing

  1. 1. Integration Testing Testing of Large Systems 1
  2. 2. [Abrain and Moore, 2004] Testing levels – Unit testing • Verifies isolated functioning of seperately testable software pieces – Integration testing • Verifying the interaction between software components • Architecture-driven – System testing • Concerned with the behavior of the full system • Functional and ”non-functional”/quality properties of the system Testing of Large Systems 2
  3. 3. War Story I In a project I was developing a component for creating UML sequence diagrams – The component worked perfectly well (presentation, editing, undo/redo, load/save, …) when tested by itself When combined and tested with a class diagram component it failed – The class diagram component might delete/change packages related to sequence diagram • So if a class was created, sequence diagrams containing objects instantiated from that class created, and the class deleted this would lead to a failure Testing of Large Systems 3
  4. 4. War Story II I was writing a component that should interface with a serial port temperature sensing device (TH-03) – It would read a temperature based on a protocol for TH-03, do some processing, and eventually give that temperature away Worked well… When connected to actual TH-03 device, nothing happened – Needed to set special serial port options to power device through serial cable… Testing of Large Systems 4
  5. 5. Integration defects Binder list typical interface defects… – “2/3 of defects are in integration” War stories are covered by it  Testing of Large Systems 5
  6. 6. Integration testing Integration testing deals with finding defects in the way individual parts work together We use composition at almost any level in software engineering – May argue (as does Binder [2000]) that integration testing occurs at all levels. Testing of Large Systems 6
  7. 7. Why integration testing? Why test component interactions??? If we provide – ‘sufficient’ unit testing and – ‘sufficient’ system testing is it not OK then??? Testing of Large Systems 7
  8. 8. Why integration testing? Binder relates an argument (on axiomatizing test case adequacy): Antidecomposition axiom: – There exists a program P and component Q [of P] such that T is adequate for P, T’ is the set of vectors of values that variables can assume on entrance to Q for some t of T, and T’ is not adequate for Q – System scope coverage does not necessarily achieve component coverage – Why? Examples? Anticomposition axiom: – There exist programs P and Q, and test set T, such that T is adequate for P, and the set of vectors of values that variables can assume on entrance to Q for inputs in T is adequate for Q, but T is not adequate for P;Q – Adequate testing at component level is not equivalent to adequate testing of a system of components – Why? Examples? Testing of Large Systems 8
  9. 9. Premise for integration testing At any given level: System consisting of components Testing interactions between components require that these are stable – Stable = sufficient dependability to allow integration testing – Threshold reflects practical considerations In case of unstable component – then what? Testing of Large Systems 9
  10. 10. Component stability/volatility An important aspect is of course to preserve our investment in testing Spending staff hours to test a class that is later fundamentally changed is a waste of time – In opposition to our wish for early integration testing – Postpone integration of volatile components! Testing of Large Systems 10
  11. 11. Integration and architecture Integration testing is highly coupled to architecture Definition by [Bass et al., 2003] The software architecture of a computing system is the structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them. Testing of Large Systems 11
  12. 12. Integration and architecture Thus the architecture defines the testing in terms of what are – the elements • (stable units that have already been unit tested adequately) – and the relations • interactions that contain defects we wish to reveal (Binder focuses on the module view of a software architecture) Testing of Large Systems 12
  13. 13. Dependencies Components may interact in a large number of ways using either explicit or implicit relations – composition and aggregation – delegation to objects / API calls – inheritance – global variables – instance variable access – objects as message parameters – RMI / socket communication Testing of Large Systems 13
  14. 14. Dependencies Many are however implicit dependencies that are much harder to spot – database access / persistent store access – initialization sequences – timing constraints – file formatting / coding of byte streams Binder states: – Integration testing does not focus on implicit dependencies –? Testing of Large Systems 14
  15. 15. Dependency analysis (Explicit) dependencies often dictate the sequence of testing – I.e. in which order are component interactions tested? Thus a very helpful tool is to do a dependency analysis – Similar to what build systems do to determine compilation order – Hard for implicit dependencies Testing of Large Systems 15
  16. 16. Coupling A measure of the strength of dependencies between two subsystems Which is preferable from a testing perspective? high coupling low coupling Cf. testing tactics Testing of Large Systems 16
  17. 17. Example from Binder Testing of Large Systems 17
  18. 18. Example from Binder Remove cycles, do a Here arrows are uses relations topological sort on the directed, acyclical dependency graph Root level – unit not used by any other unit in cluster under test – often there are several roots Leaf level – units that do not use any other units in cluster under test. Cycles – either be tested as a “unit” – or stubs introduced to break cycle Testing of Large Systems 18
  19. 19. Lessons learned Key lessons from early 1970s is that incremental integration and testing is most effective Integrating all components at the end of development is problematic: – Last-minute changes are necessary but no time for adequate testing – Testing often not systematic Testing of Large Systems 19
  20. 20. Lessons learned Advantages in incremental testing – Interfaces systematically exercised and shown stable before new unproven interfaces are exercised – Observed failures more likely to come from most recently added components making debugging more efficient. Testing of Large Systems 20
  21. 21. Stubs So, the doctrine says integration test often but how do we do integration with components that are not stable or not developed at all? Stub: partial or surrogate implementation of a component Testing of Large Systems 21
  22. 22. Reasons for stubs Stubs are powerful tools for many reasons Make testing of certain conditions possible – Hardware simulator that outputs data that are seldom occurring in practice • Example: specific patterns of temperature from temperature sensor – Replace random behavior with predictable behavior • Example: testing backgammon move validation (calling getNextMove() or similar to get move to validate) Make testing economic – Stub for a database that is costly to reset/set up Testing of Large Systems 22
  23. 23. Reasons for stubs Often stubs are relevant to decouple interactions – Stub for complex algorithm with ‘simple’ answer • Ensures that the defect found does not lie in the algorithm implementation – Stub for a component in a cycle There is no free lunch! – Stubs can be costly to produce – … and sometimes you chase defects in production code – that are actually embedded in the stubs  Morale: Keeps stubs simple! Testing of Large Systems 23
  24. 24. Integration Strategies Binder lists nine integration strategies, documented in pattern form, for doing integration testing We focus on the classic strategies: – Big Bang (actually an anti-pattern ) – Bottom-up – Top-down Testing of Large Systems 24
  25. 25. Big Bang Intent – demonstrate stability by attempting to exercise an entire system with a few test runs Context – bring all components together all at once. All interfaces tested in one go. – usually ends in ‘big-bang’ – system dies miserably… Entry criteria – all components have passed unit testing Exit criteria – Test suite passes Testing of Large Systems 25
  26. 26. Big Bang Consequences – Fails – then what? Failure diagnosis is very difficult. – Even if exit criteria met, many interface faults can still hide. – On the plus side: if it works, no effort has been spent on test drivers and writing stubs. – may be the course of action for • small systems with adequate unit testing • existing system with only minor modifications • system made from certified high quality reusable components Testing of Large Systems 26
  27. 27. Bottom Up Intent – Demonstrate system stability by adding components to the SUT in uses-dependency order, starting with component having the fewest dependencies Context – stepwise verification of tightly coupled components Testing of Large Systems 27
  28. 28. Bottom up Strategy: 1. stage: leafs Testing of Large Systems 28
  29. 29. Bottom up 2. stage.. Testing of Large Systems 29
  30. 30. Bottom up Last stage.. Testing of Large Systems 30
  31. 31. Bottom up Entry criteria – components pass unit tests Exit criteria – interface for each subcomponent has been exercised at least once – complete when all root-level components pass test suites Testing of Large Systems 31
  32. 32. Bottom up Consequences – Actually most unit tests in OO are partly integration tests • Bottom-up testing implicitly takes place in a bottom up development process Disadvantages – Driver development cost significant – Fix in lower level component may require revisions and retest up the chain – Interfaces only indirectly exercised – only the ones used by the component under test – Upper levels testing may require stubs in the bottom to test special conditions – High level testing very late in cycle, i.e., validation only possible late Testing of Large Systems 32
  33. 33. Bottom up Advantages – Parallel implementation and testing possible – Little need for stub writing Testing of Large Systems 33
  34. 34. Tools: JUnit Stubs? Drivers? Testing of Large Systems 34
  35. 35. Tools: Mock Objects / – Library for testing Java code using mock objects Mock objects – Given an interface create an advanced stub at runtime using reflection – May define expected values on mock objects using constraints Example... Testing of Large Systems 35
  36. 36. Tools: Using jMock Testing of Large Systems 36
  37. 37. Top Down Intent – Demonstrate stability by adding components to the SUT in control hierarchy order, beginning with the top-level control objects Strategy – 1 stage: • test control objects Testing of Large Systems 37
  38. 38. Top Down 2. stage Testing of Large Systems 38
  39. 39. Top Down Final Stage Testing of Large Systems 39
  40. 40. Top Down Entry – each component to be integrated passes unit test Exit – interface of each component has been exercised at least once – complete when leaf-level components passes system scope test suite Testing of Large Systems 40
  41. 41. Top Down Disadvantages – Large number of stubs necessary which is costly – Stubs are brittle – Fix in lower level component may require revisions up the chain – Difficult to get lower level components sufficiently exercised – Lower level components are exercised late Advantages – Low driver development costs – early demonstration of user-related behavior Testing of Large Systems 41
  42. 42. Variations There are many variations of these strategies – Sandwich testing • Moving from both top and bottom Bottom line – there is no substitute from being clever and utilize a combination of techniques that is most cost-efficient for the project at hand. Testing of Large Systems 42
  43. 43. High-frequency integration Binder also describes high-frequency integration that is a strategy whose characteristics lies in the timing, not the ordering, of testing – It is an intrinsic part of the process pattern daily build – At an extreme: continuous integration of eXtreme Programming fame Intent – Integrate new code with stabilized baseline frequently to prevent integration bugs from going undiscovered for a long time; and to prevent divergence from the baseline Testing of Large Systems 43
  44. 44. High-frequency integration Context – A stable baseline must be present; increments are the focus of high-frequency integration – Increments size must match integration frequency • daily builds + integrations -> increments must be deliverable in a day – Test code developed in parallel with code – Testing must be automated… – Software Configuration Management must be in place Testing of Large Systems 44
  45. 45. High-frequency integration Procedure: First – Revise code + test code on private branch – Desk check code and test – When all component testing passes, check-in to the integration branch Second – Integration tester builds system of increments – Testing using • smoke tests and as much additional as time permits Any increments that break HFI are corrected immediately Testing of Large Systems 45
  46. 46. High-frequency integration Disadvantages – Automated tests must be in place – High commitment to maintaining code as well as tests – Be aware of adequacy criteria – the suite that found the old bugs may not find the new ones Advantages – Focus on maintaining tests is an effective bug prevention strategy – Defects found early; debugging easier – Morale high as system works early and keeps doing it Testing of Large Systems 46
  47. 47. Integration test in the TMM (1/2) Basic integration test appears in level 2 (as does all basic test techniques/methods). Planning of the test process (incl. integration test) is also initiated at level 2. At level 3, integration test (and all other test activities) is integrated in the sw life cycle. – In modern iterative/agile development methods, this happens more or less automatically At level 4, the integration test process gets measured. At level 5 the integration test process is continuously optimized. Testing of Large Systems 47
  48. 48. Integration test in the TMM (2/2) At level 2, Managers – Should ensure that multilevel testing is part of policies, planned for and catered for via project schedules and resources Developers – Should help PM’s to make multilevel testing work, help develop use cases/acceptance criteria, implement tests at all levels Users/clients – Should help provide acceptance criteria, provide ample and due feedback, participate in tests (accept/beta) Testing of Large Systems 48
  49. 49. Summary Testing levels – Unit testing • Verifies isolated functioning of seperately testable software pieces – Integration testing • Verifying the interaction between software components • Architecture-driven – System testing • Concerned with the behavior of the full system • Functional and ”non-functional”/quality properties of the system Testing of Large Systems 49