Integration Testing


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Integration Testing

  1. 1. Integration Testing Thomas Bradley 368100 February 8, 2008 Abstract This document discusses integration testing which is an important stage in the overall testing processes. Various different approaches to integration testing will be discussed including their individual advan- tages and disadvantages plus there is a brief overview of two different software testing tools that can be used help automate testing tasks. 1
  2. 2. Contents 1 Introduction 3 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Decomposition-Based Integration Testing 5 2.1 Top-down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Bottom-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Sandwich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4 Big bang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3 Call Graph Based 13 3.1 Pair-Wise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2 Neighbourhood . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4 Software Testing Tools 18 4.1 JUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 4.2 Eggplant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5 Summary 22 2
  3. 3. 1 Introduction “testing is a vital component of a quality software process, and is one of the most challenging and costly activities carried out during software development and maintenance”[1] Software testing as a whole is an essential part of the software development processes (often accounting for over 50% of the total cost of a project)[2] it is however an area that is often underestimated during planning. If a project is under significant time pressure it is the testing stage of the development process that is affected most significantly[3]. It is therefore very important that software developers throughly understand the different stages of testing and the different techniques that are used. Integration Testing is the testing of software components after they have been integrated and this form of testing will be the focus of this dissertation. Integration testing assumes that the individual software components have been successfully unit tested1 and is an important form of testing because “Some bugs that can be discovered as the units are integrated are impossible to find when testing isolated units”[4]. It is also worth noting that often different software companies collaborate on projects and it is critical that once the individual companies software are integrated together it is tested throughly. 1.1 Motivation There is one well known example that is often used to explain why testing is important and it is perfect for illustrating what happens when integration testing is not performed correctly. The Mars Climate Orbiter was a spacecraft developed by NASA designed to be launched from Earth, travel to Mars, move into a stable orbit with a periapse2 altitude of 160km[5] and once there record data regarding the atmospheric conditions of the planet. The Orbiter was launched success- fully on 11 December 1998 however while moving into orbit around Mars (on September 23,1999) NASA lost contact with the spacecraft[6]. 1 Testing of individual software components. 2 The position in an orbital path that is closest to the primary body. 3
  4. 4. Figure 1: Mars Climate Orbiter The investigation board found that the cause of the accident was a sim- ple software error. Specifically one module outputted thruster performance data in imperial measurements however in the software documentation it was stated that this information should be in metric units. During the nine month journey to Mars, NASA made several propulsion ma- neuvers based upon data which was not in the correct format. This caused small errors to be introduced in the trajectory during the journey meaning that by the time the craft reached Mars, it was 170 km lower than was orig- inally planned. As a result of this the craft was quickly destroyed due to hitting the atmosphere of the planet.[6] This should never have happened, although the individual software units would appear to be working correctly, appropriate integration testing should have been able to discover the error between the units. This small error which was not picked up due to insufficient testing, cost NASA over $300 million.[7] 4
  5. 5. 2 Decomposition-Based Integration Testing Decomposition-based integration testing is the most popular form of integra- tion testing, it is widely publicised and in general when integration testing is mentioned in an article or book they are referring to this form of testing. This approach to integration testing is “based on the functional decomposition3 of the system tested”[8]. There are four major techniques that can be used in decomposition-based integration testing, specifically: top-down, bottom-up, sandwich and big bang. The major differences in all these approaches is the order that the individual units of the system are integrated and tested. As stated previously, integration testing assumes that the individual units have been successfully unit tested and therefore the goal of this testing is to dis- cover any errors in the integration of the individual units. This section will discuss these four different approaches to decomposition- based integration testing with use of the example backup program which is detailed in table 1 and shown as a decomposition tree in figure 2 Figure 2: Decomposition Tree Example 3 Resolving a functional relationship into its constituent parts. 5
  6. 6. Module Name Module Description Main This is the first module that will be run, it provides the initial interface. Backup Provides the interface for the various types of backup that the user can perform. USB Backup Called if the user requests a USB Backup, it starts the backup process. Mount USB Drive Mounts the USB Drive, once the mount- ing process has completed it calls the backup to USB module. Backup to USB Uploads all the necessary files to the USB drive Update Last USB Backup Updates the lastUSB file with the current date and time. Server Backup Called if the user requests a Server Backup, it starts the backup process. Remote Connect Connects to the remote server, once suc- cessfully connected it calls the Upload Backup module. Upload Backup Uploads all the necessary files to the server Update Last Server Backup Updates the lastServer file with the cur- rent date and time. Manual Backup This module is used if the user backups manually. Update Manual Backup Date Updates the lastManual file with the cur- rent date and time. Recent Backups Displays the date and time of all the last backups (by calling the appropriate mod- ules). Last USB Backup Returns the date and time stored in the lastUSB file. Last Manual Backup Returns the date and time stored in the lastManual file. Last Server Backup Returns the date and time stored in the lastServer file. Table 1: Backup Program Example 6
  7. 7. 2.1 Top-down The Top-down method begins by testing the main program unit (the root4 of the tree) with one lower level node. Any other lower level units/nodes that may be connected should created as a stub5 . In the example used here (see figure 3 for the colour key and figure 4) the ‘Main’ unit would first be tested with ‘Backup’ with stubs created for ‘Recent Backups’, ‘Manual Backup’, ‘USB Backup” and ‘Server Backup’. The stub files would be created with the absolute minimum amount of information needed. for example the ‘Re- cent Backup’ stub could be created to simply return a specific set of dates with no dynamic lookups at all. Figure 3: Colour Code Key Once the appropriate stub files have been created and everything has been integrated, the tester simply tests the program as if it were a stand-alone program therefore any appropriate testing techniques could be used to look for faults. If the tests are successful then the project can continue by pro- ducing real code to use in place of one of the stubs that were used in the previous test. There is no set way to traverse through the decomposition tree. Breadth-first or Depth-first are both perfectly acceptable methods. 4 The topmost node in a tree. 5 A piece of ‘throw-away’ code that emulates the unit in question. 7
  8. 8. Figure 4: Top Down Integration Testing: Stage 1 It is up to the tester to decide how many stubs should be replaced before the next test is performed (see figure 5). The more stubs that are replaced before a test, the harder it would be to localise a fault; however if a test was performed after every new unit was produced then the number of tests that have to be performed rises quickly. This rise in test numbers could cause problems due to scheduling or associated costs. The major advantage of this top-down method is the fact that a system prototype can be developed early on in the project process. This is a very attractive property of this integration testing technique as usually the client of a software company will have little or no software engineering knowledge and therefore if the client cannot see something tangible then it would be easy for the client to assume that little or no work is being completed. In addition to this, having an early prototype will give the client a better feel for how the final product will look which is useful as any changes that the client might think of will appear early in the development process (where it would be easier to implement). One disadvantage of this method is the fact that the programmers will have to produce a large number of stubs, this means extra work is needed on top of producing the actual system by producing units that will later be thrown away. 8
  9. 9. Figure 5: Top Down Integration Testing: Stage 2 Assuming the developer is using the decomposition-tree model the formula below can be used to calculate the number of stubs that must be produced. number of stubs = (number of nodes − 1) In our backup example, that means that the software engineers will have to spend time developing an additional 15 units that will be simply thrown- away as the project progress. In large scale projects the amount of time and therefore money spent developing huge numbers of stubs could easily become very large. 2.2 Bottom-up The bottom-up method begins by testing one of the leaves6 of the program with it’s parent7 node first. Any higher nodes that are directly connected to the nodes being tested are constructed as drivers.8 In our example (figure 6) the first nodes that we would test would be ‘Update Last USB Backup’ and ‘Backup to USB’ with a ‘Mount USB Drive’ created as a driver. The driver would be created with the minimum amount of code needed. 6 A type of node that has no child (directly connected) nodes. 7 A node that has a child is called the child’s parent node. 8 a piece of ‘throw-away’ code that emulates the unit in question. 9
  10. 10. Figure 6: Bottom Up Integration Testing: Stage 1 In exactly the same way as the top-down method, it is up to the user how to continue testing and how many nodes to create before performing another test (see figure 7). This has the exact same issue of number of tests vs ease of fault isolation. The major advantage of this method of integration testing is that the pro- gram itself is fully functional9 at every stage. This is in contrast from the top-down method, where as a a prototype can be created at an early stage, but it will have little or no functionality e.g. it may have a button for ‘USB Backup’ but that button will not do anything. However the obvious disad- vantage to this approach is the inability to create an early prototype for the client. This could easily lead to problems down the line, such as the client may question the progress of the project or when the prototype is finally produced the client might decide on changes that are difficult to implement at that late stage of the project. Although drivers are different from stubs (e.g. there are less drivers, but they are more complicated to produce) there is still the same problem of producing large quantities of ‘throw-away’ code. This problem will grow quickly in larger projects. 9 the units will be fully functional, however the entire program will not immediately have all the functionality in the specification. 10
  11. 11. Figure 7: Bottom Up Integration Testing: Stage 2 Assuming the developer is using the decomposition-tree model the formula below can be used to calculate the number of stubs that must be produced. number of drivers = (number of nodes − number of leaves) In our backup example this will mean that the software engineers will have to spend time developing an additional ten units of code that will simply be thrown away. 2.3 Sandwich The sandwich method attempts to solve some of the problems associated with the previous two methods by combining them into a single method. The initial leaf node and corresponding parent is created (along with any necessary drivers) plus the root node along with a corresponding child (and any stubs) are created. This way the program is tested from both the top of the tree moving down and the bottom of the tree moving up. In our example (figure 8), this would mean that initially the ‘Main’, ‘Backup’, ‘Update Last USB Backup’ and ‘Backup to USB’ would be created, with ‘Re- cent Backups’, ‘Manual Backup’, ‘USB Backup’ and ‘Server Backup’ created as stubs plus ‘Backup to USB’ created as a driver. These nodes would then be tested and if successful the tester will the start traversing the tree in both 11
  12. 12. directions (from the top and the bottom, see figure 9). Figure 8: Sandwich Integration Testing: Stage 1 This method combines the two major advantages of the previous two meth- ods (being able to produce an early prototype and producing a functional program at each stage of development). In addition to this, by using the sandwich method the number of stubs and drivers that have to be produced are reduced. However The significant disadvantage of the sandwich method is that fault isolation can be difficult, especially in larger projects. 2.4 Big bang “In its purest (and vilest) form, big-bang testing is no method at all - ‘Let’s fire it up and see if it works!’ It doesn’t of course”[9] The big bang method is the least systematic method of integration testing, the idea for this method is to simply go ahead and produce all of the indi- vidual nodes/units of the program and only once they have all been created do you perform the integration tests. This removes the problem of having to waste time producing ‘throw-away’ code, however quite obviously even moderately sized projects fault isolation is very difficult. This method has many problems associated with it and only one minor advantage such that it is considered to be “such transparently bad thinking”[10]. Despite all of this it is still “probably is the most commonly used method for the integration 12
  13. 13. Figure 9: Sandwich Integration Testing: Stage 2 purpose”[11], this popularity is likely to be because many project managers “just want to be able to report to the management that ‘Coding is complete’ as soon as possible, even if nothing works”[10]. 3 Call Graph Based The methods that have been discussed so far are based upon the functional decomposition trees rather than the actual structure of the program, also the major disadvantage in the previous10 testing techniques is the fact that it is necessary to produce large quantities of ‘throw-away’ code in the form of stubs and drivers. call graph based integration is a type of testing designed to solve these two problems. By using a call graph the “internal structure of the program is clearly rep- resented”[12] and similar incremental methods to the approaches discussed above (to-down, bottom-up etc) can be used. However the big difference be- tween this and the previous techniques is that no stubs or drivers are needed. All the testing will be performed on actual units of fully functional code i.e the program is constructed/produced before integration testing takes place. This sounds similar to the big bang approach of integration testing however by ensuring that the units that are tested are split up into smaller groups 10 excluding the ‘big bang’ method. 13
  14. 14. we can reduce the difficulty of fault isolation to a manageable level. The two main approaches to this form of testing will be discussed with use of an example door lock program which is detailed in table 2 and shown as a call graph in figure 10. Figure 10: Call Graph Example 14
  15. 15. Module Name Module Description Main This is the first module that will be run. Display Displays a list of users that have recently opened the door plus displays a message if a user’s card is valid or invalid. Override This is the emergency door override mod- ule, it can be activated by an admin to open the door without needing a card. Card Reader This module reads the card and checks to see if the user is valid or not by checking with the database, if it is valid it calls the lock controller. Database This module contains the list of all users on the system. Lock Controller When this method is called it unlocks the door (by calling the unlock module) then after a set amount of time it re-locks the door (by calling the lock module). Unlock Unlocks the door and calls the light mod- ule Lock Simply locks the door Lights This module turns on the lights (if they are not already on). Table 2:Door Lock Program Example 3.1 Pair-Wise Pair-Wise integration testing restricts the testing session to a pair of units in the call graph (see figure 11). This ensures that faults can be easily isolated while still working without stubs or drivers. A disadvantage to this method is that there needs to be as many test sessions are there are edges.11 However, because no time is wasted in producing ‘throw-away code’ (in the form of drivers or stubs) this method can be significantly quicker than the decomposition based methods. There is no specific traversal strategy that has to be followed when using this technique so long as in the end each edge has been successfully tested. 11 A representation of a relationship rendered as a straight or curved line. 15
  16. 16. Figure 11: Call Graph: Pair-Wise Example 3.2 Neighbourhood Neighbourhood Integration Testing attempts to reduce the number of testing sessions necessary in order to fully test the software system while still trying to keep the difficulty of fault isolation low. In order to do this the tech- nique restricts the testing session to the neighbourhood of the source node12 where the neighbourhood of a source node includes every other node that is directly connected to the that source node (see figure 12). Each source node (with it’s corresponding neighbourhood) is tested ensuring that every node (including sink nodes13 ) is tested. This greatly reduces the number of testing sessions there are, as the number of testing sessions is equal to the number of neighbourhoods that exists which can be calculated using the formula below. N umber of N eighbourhoods = (number of nodes − number of sink nodes) 12 A type of node that has outgoing relations to other nodes (similar to a parent node in a decomposition tree). 13 A type of node that has no outgoing relations (similar to a leaf in a decomposition tree). 16
  17. 17. In our example this will mean 5 tests sessions will have to be run which is a big improvement on the pair-based method which would have required 9 test sessions (one for each edge). This method attempts to balance the number of test sessions required (and therefore the amount of time required to test the system) and the difficulty of fault isolation (and therefore the amount of time wasted trying to find the fault). However in smaller software projects the size of the neighbourhood will be relatively low and therefore fault isolation would not be an issue, in larger projects the size of a neighbourhood could easily become large enough so that it would be difficult to isolate a single (or collection of) faults. Figure 12: Call Graph: Neighbourhood Example An additional problem that arises due to the fact that nodes overlap neigh- bourhoods. If fault is found and corrected in a node that is in several neigh- bourhoods, due to the fact that the node’s code will have been changed it 17
  18. 18. will be necessary to retest every neighbourhood that the node in question can be found in. This form of testing is called “Regression Testing” and is performed in every testing technique once a fault has been corrected to ensure that the corrected code has not caused faults elsewhere. 4 Software Testing Tools In general, Software testing tools exist to automate the testing process. Es- sentially testing tools create many “little programs that automatically check to ensure that your program behaves as expected”[13]. This greatly reduces the amount of time needed to test a system (be it a unit test, integration test or even system test). In general these tools have the ability to run many tests automatically and summerise the results into a readable format. There are two small disadvantages that are worth mentioning, first of all it is necessary to write more code that will not be used in the final program (similar to the problem with stubs and drivers) however the time saved by automating software tests greatly outweighs the time spent writing the necessary code.14 Secondly testing tools themselves like any other program are never totally bug free and while using one to test a system, there is the assumption that the results of the tests are accurate which may not be the case. 4.1 JUnit Junit is a generic unit testing framework for the Java programing language it was created by Kent Beck and Erich Gamma in 1998 and is used for regres- sion, unit and integration testing. As it is a generic framework it is supported by all the major java IDEs[13] (for example BlueJ, see figure 13). This allows the tester to automate the testing process, performing any number of tests with a click of a button. Below is an example JUnit test case15 , it is testing a simple item basket program to check to see if the totalCost method is working correctly. import junit.framework.TestCase; { public class BasketTest extends TestCase { 14 Obviously for very small programs this might not be true, but for any realistically sized project this statement holds. 15 The test case is to demonstrate the code used with JUnit, not to demonstrate inte- gration testing. 18
  19. 19. Figure 13: BlueJ: JUnit screenshot private Basket myBasket; private Item dvd1 private Item dvd2 } /** * This method runs before any test case method */ protected void setUp() { myBasket = new Basket(); dvd1 = new Item("Hot Fuzz", 9.99); dvd2 = new Item("Sean of the Dead", 3.99); myBasket.add(dvd1); myBasket.add(dvd2); } public void testTotalCost() { double totalCost = myBasket.totalCost(); double confirm = dvd1.price() + dvd2.price(); /* assertEquals checks to see if the two * variables are equal, if they are it does nothing * if however they are not it throws an unhandled exception */ 19
  20. 20. assertEquals(confirm, totalCost); } } Test Cases can be much larger and more complicated than the example above, also it is possible to create ‘Test Suites’ which are a collection on individual test cases which can all be run simply by running the test suite, this is very useful as you can create collections of test cases in different suites and run the group that is needed. Below is a small example of a test suite import junit.framework.Test; import junit.framework.TestSuite; public class ShoppingTestSuite { public static Test suite() { suite.addTestSuite(BasketTest.class); suite.addTestSuite(CheckoutTest.class;) return suite; } } JUnit is a testing tool designed to be easy to use, so that the average program- mer (who may not have much experience in testing) can write and perform tests while still allowing the user to create complicated test cases and suites. 4.2 Eggplant “Software integration testing; quicker, easier and more reliable with Eggplant”[14] Eggplant is software testing tool produced by Redstone. Rather than in- teracting with the underlying software code it attempts to recreate the the user experience and is designed to work complimentary with other code level testing tools. In essence this testing tool simulates a real user and can be set up to perform automated tests on the user interface of the program because it is based on performing functions on the user interface. It is not dependent on any one programing language or even operating system. Eggplant uses a scripting 20
  21. 21. Figure 14: Eggplant screenshot language called Sensetalk which is “a very high level, object oriented, inter- preted scripting language, based on a modular, extensible architecture.”[15] The language is designed so that it is very easy to use (hence the “very high level”) and Eggplant provides an interface for the script creation that identifies the elements in the interface that can be tested (for example buttons and text fields) which makes it a straight forward process to script actions for testing the elements. Below is the example16 sample script for testing a webpage form. LaunchApplication "Safari" WaitFor 5, "SafariMenu" TypeText "redstonesoftwaren" Click "NameField" TypeText "Tomt" TypeText "" TypeText "Very good post! I aggree totally with what you were sayingt" Click "SubmitButton" TypeCommand "Q" -- Quit Safari 16 The test case is to demonstrate the code used with Eggplant, not to demonstrate integration testing. 21
  22. 22. 5 Summary In summary, Software testing is a vital part of the software development pro- cess even though it is the most likely part that will be cut out (or at least significantly reduced) due to time and cost problems. Integration testing is a significant part of the overall testing activity and is very useful in discovering faults with the program that may otherwise go overlooked. The main goal of integration testing is to test how previously tested units (be they small modules or entire systems) interact with each other and un- cover problems that would not be apparent during the unit testing stages. As we have seen with the Mars Climate Orbiter example, not performing full integration testing on all systems can have disastrous consequences (in the example case, the consequences costed NASA over $300 million[7]) The two major types of integration testing that have been discussed were ‘de- composition tree based’ and ‘call graph based’. These two types of testing have their own advantages and disadvantages and both types have a number of different approaches to tackling the problem of integration testing. Decomposition Tree based integration testing primarily focused on the order that the individual units were developed and tested, the major disadvantage of this form of testing was the creation of throw-away code in the form of stubs and drivers. Call Graph based integration testing has the advantage of not requiring any additional throw-away code to be produced, however that does make fault- isolation a problem especially in larger software projects. There are a vast number of software testing tools available for use (both free and commercial tools) that are designed to help automate the testing process. The two tools that were discussed in this dissertation were JUnit, which is a generic unit testing framework for Java (therefore implemented in many different IDEs) and Eggplant which is a commercial tool produced by Redstone which is designed to recreate the user experience. These two tools are both very different from each other, JUnit aims specifically at testing Java code where as Eggplant aims at testing via the programs user interface. 22
  23. 23. References [1] BURNSTEIN, I. 2002. Practical Software Testing: A Process-Oriented Approach [2] MYERS, G. 2004. The Art of Software Testing [3] KAN, S. 2002. Metrics and Models in Software Quality Engineering [4] CRAIG, R; JACKIEL, C. 2002. Systematic Software Testing [5] Mars Climate Orbiter Mission Overview. Accessed 22/12/07 http: // mars. jpl. nasa. gov/ msp98/ orbiter/ mission. html [6] 1999. Mars Climate Orbiter Mishap Investigation Board Phase I Report. [7] Mars Climate Orbiter Fact Sheet. Accessed 22/12/07 http: // mars. jpl. nasa. gov/ msp98/ orbiter/ fact. html [8] JORGENSEN, P. 2002. Software Testing: A Craftman’s Approach [9] BEIZER, B. 1984. Software System Testing and Quality Assurance [10] KANER, C; FALK, J; NGUYEN H. 1993 Testing Computer Software [11] DHILLON, B. 1987. Reliability in Computer System Design [12] TSAO, H; Wu, Y. 2003 Testing and Quality Assurance for Component- Based Software [13] BECK, K. 2004. JUnit Pocket Guide [14] Redstone Software. Accessed 07/01/07, index.html [15] SenseTalk Reference Manual. Accessed 07/01/07, 20Reference.pdf 23
  24. 24. Figures 1 Mars Climate Orbiter. Accessed 07.01.07, jpg 2 Decomposition Tree Example, created by Thomas Bradley with Gliffy 2 Colour Code Key, created by Thomas Bradley with: 4 Top Down Integration Testing: Stage 1, created by Thomas Bradley with: 5 Top Down Integration Testing: Stage 2, created by Thomas Bradley with: 6 Bottom Up Integration Testing: Stage 1, created by Thomas Bradley with: 7 Bottom Up Integration Testing: Stage 2, created by Thomas Bradley with: 8 Sandwich Integration Testing: Stage 1, created by Thomas Bradley with: 9 Sandwich Integration Testing: Stage 2, created by Thomas Bradley with: 10 Call Graph Example, created by Thomas Bradley with: 11 Call Graph: Pair-Wise Example, created by Thomas Bradley with: 12 Call Graph: Neighbourhood Example, created by Thomas Bradley with: 13 BlueJ: Junit screenshot, taken of BlueJ version 2.2.1 with example code written by Thomas Bradley 14 Eggplant screenshot, taken of Eggplant Version 4.01 (1101) with example code written by Thomas Bradley 24