Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Making software testing greener


Published on

  • Be the first to comment

  • Be the first to like this

Making software testing greener

  1. 1. Making software testing greener Skill Level: Introductory Nageswararao V. Gokavarapu ( Software Test Specialist IBM 11 Aug 2009 Recently there has been an increased emphasis on the effective utilization of resources. Testing plays a key role in any software development lifecycle. Hence, effective utilization of resources in testing becomes the prerogative for any product team. Read about the effective utilization of resources available for testing on both hardware and software, thus leading to more efficient testing. Introduction There are two software testing methods, black-box testing and white-box testing, which are designed to cater to the Functional Requirement Specification (FRS) and System Requirement Specifications (SRS). SRS contains the complete description and behavior of all the functions of software to be developed. FRS contains the expected outputs for list of inputs for each function of the software. In white-box testing, the test engineer has access to the internal data structures and algorithms. Black-box testing treats the software as a "black box," without any knowledge of internal implementation and the test engineer will perform tests on the software as a customer would use the product. System testing is a method of black-box testing and is conducted on a complete, integrated system to judge the software compliance in the context of a FRS and SRS. The different types of testing that should be considered during product testing are: • Installation testing • Build verification testing Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 1 of 18
  2. 2. developerWorks® • Functionality verification testing • Usability testing • GUI software testing • Security testing • Load testing • Stress testing • Performance testing • Scalability testing • Regression testing Each of these testing methods ascertains the software product quality in different functional areas. Virtualization is a modern approach for enhancing a system's ability to complete tasks. Virtualization can be adopted in different areas in the computing world like hardware virtualization, operating system virtualization, network and storage virtualization, presentation virtualization, and application virtualization. Operating system-level virtualization is a server virtualization method where the kernel of an operating system allows for multiple isolated user-space instances. Each of these instances may look and feel like a real server from the point of view of its end user. Each of these instances are isolated and the kernel often provides resource management features to limit resources for every instance. This article address how to adopt operating system virtualization methods in different types of testing and thereby minimizing the utilization of resources in terms of hardware, software, and test engineers. Types of software testing The following are some examples of testing that will be performed during the testing phase of a software product. Installation test In this test, the test engineer installs and sets up the software. The testing process may involve full, partial, or upgrades to the installation/removal of the product. The simplest installation approach is to run an install program, sometimes called Package Software. The test engineer should perform every possible configuration with every mode of installation and should receive a better quality of testing so that the software can be released to customers with confidence. Making software testing greener Page 2 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  3. 3. developerWorks® Build Verification Test (BVT) This test involves the basic functional tests on the product to make sure the code changes by the development team in that particular build will not affect or regress any pre-existing features in the product. Once this test is passed successfully and the build is certified, then the functionality of the product is tested by the other methods of testing. As part of globalization, the BVT will be executed in supported locales of the software product that is available. Functional Verification Test (FVT) The purpose of this test is to check that a product conforms to its functional specification. As part of FVT, the functionality of the product is tested for all its supported features and then certifies the product readiness for the system test. In Globalization Verification Test (GVT), the FVT is also tested in all supported locales to make sure the product functionality is consistent. GUI software test The GUI test tests the software that supports the graphical user interface to ensure it meets the required specifications in FRS. Most products deliver system functionality to the clients using a GUI. The test designer must take care that the test suite covers all the functionality of the product with GUI, and also ensure that the suite fully exercises the additional functionality provided by the GUI. The GUI test also needs to be done in all supported locales and the test engineer needs to verify the standards defined for different objects in GUI, for different locales. Load test In load test, multiple users access the program concurrently to simulate the expected usage of a software program. To simulate the load, vendor-acquired software may be used. As many users access the application or product concurrently, the CPU usage, network or disk I/O operations, and disk space requirements are greater for these type of tests. Stress test Stress test refers to tests that put a greater emphasis on robustness, availability, and error handling under a heavy load on the product. The main goal of these tests is to ensure that the software does not crash in conditions of insufficient computational resources such as memory or disk space. It also determines the stability of the product by testing beyond normal operational capacity. As the product is tested in heavy load conditions, no other testing can be done on the server. Regression test Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 3 of 18
  4. 4. developerWorks® This test reveals the software regressions. Regressions are determined whenever a functionality that was previously working correctly stops working as intended with new test run. Typically, regressions occur as an unintended consequence of program changes during a bug fix. Regression testing include re-running previously run tests and checking whether the previously fixed issues have occurred. Regression testing is used to determine the correctness and quality of the product. The different types of tests discussed here require different environments to run the test. Some of the test types require less CPU and disk space, whereas some of the test types requires high CPU usage, I/O operations, and disk usage. So, the resource utilization of a particular test type depends on the sequence of tests performed in that test type. Adopting operating system virtualization for all test types is not possible because of dependencies. The following section describes how to choose the test area. Choosing the test area The following general questionnaire helps in identifying the test area to use operating system virtualization technologies to minimize system and human resources. General questionnaire for choosing test type: • How much CPU and memory allocation is required? • How many processes will run for executing the test? • How much virtual memory is required for running the test? • Does the test require any physical devices? • Will the test require modifying the kernel parameter at runtime? • Will the test access raw or routing sockets? • Does the test require mount/dismounting file system? Inside the operating system virtual instance, some facilities like mounting the file system, reading/writing the data to raw sockets, and changing the kernel parameters dynamically are not available. The file system mounting can be done from the global system environment to a particular OS instance. The virtual OS instance can access the mounted file systems without any issues, but direct mounting from OS instance is not possible. Similarly, accessing the physical devices is not possible from virtual instances. Even though there is a way to access these devices by exporting from the global environment, applications that do not support these exported devices will fail. So, based on the resources required for test types, the virtual opersting system techniques are adopted for product testing. Making software testing greener Page 4 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  5. 5. developerWorks® Example test areas for consideration As discussed previously, the product testing involves multiple test types and each of the test types needs different hardware, software, and network resources. For example, load test, stress test and performance tests, where the product is tested under high load and each of these tests uses the same system resources from the system. So, these tests should not be run on the same physical server at the same time. If these tests are running on different virtualized instances on a physical server, you need to be careful when configuring your system for allocating resources for these operating system-virtualized systems. The load tests require more resources, and all will share from the same physical server, so these benchmark tests should not be performed on operating system-virtualized systems. For installation test, build verification test, globalization verification test, and GUI software test, you can use the operating system virtual instances. In installation test, the product is tested in all supported modes of installation. Most of the time, the installation tests are executed in a freshly installed operating environment. Once the product is installed, it will add a set of configurations in the system environment. Even though the product is removed after testing, some of these environments will remain in the system. The subsequent installation on the same machine will use the existing configurations. Sometimes this will hide the installation issues that can be noticed in a freshly installed operating system environment. But practically speaking, it is not possible to perform each of the installation test scenarios in freshly installed operating system environments, because it includes additional operating system and third-party prerequisite software setup in the server. As part of globalization, the installation test is done in all supported locales. So, the number of machines required will be more. The operating system virtualization technology helps for installation tests. Once the operating system and other third-party prerequisites are installed in the physical server, you can make as many as virtual operating systems instances and all of these will have the same system operating environment as the global physical system. Creating these virtual instances will take less time. Table 1. Different test areas with effort in conventional methods Test type Number of servers Administration Test effort required effort Install tests 2 4x 12x BVT 2 4x 12x FVT 3 6x 24x GUI tests 2 4x 12x Table 1 shows different test areas, their server usage, and the administrator effort needed for the operating system, software setup,and testers effort in configuring and executing the test cases where x is the unit of time. Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 5 of 18
  6. 6. developerWorks® Most of the products support a minimum of three modes of installation modes like GUI, Console, and Silent. For installation test, if the product is tested in all modes with all supported locales (assume the product supports 10 locales), then installation must be tested in 30 various configurations. Similarly, BVT/FVT needs to be run in all locales to certify the product functionality. By using operating system virtualization technologies, the administration and test efforts are reduced, as shown in Table 2. The administration effort is reduced as the number of servers used is less and the test effort reduced, because operating system virtual instances contain all the software installed in a global system. Hence, there is no need to install vendor-acquired products inside each instance. It also saves power and contributes less heat, thus reducing the investment required to cool the systems. Table 2. Different test areas with efforts required using OS virtualization methods Test type No of servers Administration Test effort required effort Install tests 1 2x 8x BVT 1 2x 8x FVT 1 3x 12x GUI tests 1 2x 4x Another test area where virtualization methods is used is GVT. As part of the GVT, BVT and FVT execute in all supported locales. If the product is installed in a global physical system and the virtual instances are created inside, then there is no need to install the product inside each instance. Only some additional configuration needs to be done to execute these test suites. Even if the product image refreshes in the global system, it reflects immediately in all virtual instances. Hence, it reduces the test efforts. Operating system-level virtualization technologies Virtualization is done in different levels and each virtualization technique has a different advantage based on the usage. For product testing, hardware and operating system-level virtualization helps in reducing the number of physical servers required and time required to set up the servers. In operating system virtualization, different isolated instances of the operating system are created and all share the physical resources available in the system, as shown in Figure 1. The maximum resource usage for a each instance can be configured at the time of creation. This article discusses the different server virtualization methods to be used in product testing. Figure 1. Operating system virtualization Making software testing greener Page 6 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  7. 7. developerWorks® There are different operating system virtualization technologies available like AIX® V6.1 workload partitions (WPAR) and Solaris 10 containers. AIX 6.1 provides a WPAR facility that provides an application-isolated environment and the resource control for applications. WPAR monitors the processes running on a particular WPAR from the global environment. Different automated test scripts can be added in the startup profiles of the WPAR. Once the WPARs are started, all the automated tests are executed on different WPARs simultaneously. Manual tests can also be executed on each WPAR if needed. Using shared-system WPAR configuration, there is no need to install the product on each WPAR. The test suites will use the product installed in the global environment to run different test areas of product testing. Once the tests are started, Global environment can monitor the status logs for the initiated tests. Some of the test areas needs minimum system resources and these tests can be run at the same time. This saves the resources, mainly in terms of servers and cost. Solaris Container allows completely isolated virtual servers within a single operating system instance called zones. The zone facility allows sharing the system resources. Each zone behaves as a separate system. The similar virtualization technologies are also available on other operating systems like HP-UX, Linux and Windows. WPARs in AIX 6.1 AIX 6.1 supports two types of workload partitions, Application WPAR and System WPAR. Application WPAR creates and starts with the application in an isolated resource environment and automatically removes when the application terminates. System WPAR is a complete virtualized opersting system environment running Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 7 of 18
  8. 8. developerWorks® multiple services and applications. There are two different types of System WPAR: • Shared system WPAR • Non-shared (private) system WPAR. Each of these behaves as a completely isolated operating environment. In shared system WPAR, it uses the software installed in the global system and in private WPAR it allows for installation of the product inside the WPAR. 1. Create a WPAR You can create System WPAR by doing the following steps: 1. Log in to an AIX 6.1 system as a root user. Note that this is for a global system in a WPAR environment. 2. Select the WPAR name that you want to create. 3. If you want to mount the WPAR file systems on a specific mount point, create a directory; otherwise, by default WPAR uses the /wpars directory. 4. If you have created a directory, then change the directory permission to 700 (that is, drwx------). 5. Add the IP address of the WPAR in the /etc/hosts file. For example: 123.456.789.128 txwpar.domain_name txwpar # IP address of WPAR where: 123.456.789.128 is the IP address of the WPAR. txwpar is the WPAR name. 6. Run the mkwpar command as shown in Figure 2 to create a shared system WPAR. Figure 2. Creating shared system WPAR Making software testing greener Page 8 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  9. 9. developerWorks® In the example shown in Figure 2, above, txwpar is the WPAR name. -r is copy global network name resolution configuration into the workload. This copies the following: /etc/resolve.conf /etc/hosts /etc/netsvc.conf /etc/irs.conf /etc/networks -N is network settings ( [-N attr=value ...] ) address= 123.456.789.12 is the IP address interface = en0 To find the interface number, run the ifconfig -a command. For example: # ifconfig -a en0: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST, GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN> inet 123.456.789.126 netmask 0xffffff00 broadcast 123.456.789.255 tcp_sendspace 131072 tcp_recvspace 65536 netmask= is the netmask value For non-shared (private) WPARs only, you can use the -l option. This creates a private, writable version of /usr and /opt. Upon successful creation, it shows a successful message as shown in Figure 3. It also updates the network configurations to access the WPAR from outside. Figure 3. WPAR creation Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 9 of 18
  10. 10. developerWorks® 2. Start the WPAR: startwpar -v WPAR_name where -v is verbose and -m is maintenance mode. Once the WPAR is started successfully, as shown in Figure 4, it returns the status as a success. Figure 4. Start WPAR and check the WPAR status 3. Check the status of the WPAR: In Figure 4, the running status of WPAR is known by using lswpar command, where S means that txwpar is a system WPAR and A means that txwpar is active. Making software testing greener Page 10 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  11. 11. developerWorks® 4. Log in to the WPAR: clogin txwpar. 5. Now WPAR is ready to use for testing. As it is a shared system WPAR, it contains all the software installed in the global system. Assume that two WPARs are created and WPAR1 FVT and WPAR2 BVT are configured as shown in Figure 5. Both tests can start in parallel. Once it checks and makes sure that all test case execution is completed inside each WPAR, it can start the next test case. The results are written in a test log file inside the WPAR. The application running on AIX 6.1 system keeps on monitoring these logs and takes the necessary steps. In case of GVT, the same WPARs can be used to run the BVT and FVT in different locales by setting the locale one after the another. As a result, it saves lots of resources like servers, power consumption, installation, and configuration of different products on each server, test execution time, and server administration. Figure 5. Product testing with WPAR Zones in Solaris The Solaris Zones feature provides the facility to create separate operating system environments for running applications with dedicated resouce allocation like CPU and memory. There are two types of zones, global and non-global. A server on which the zones feature is enabled has one global zone and up to 8191 non-global zones. Non-global zones contains subset of the Solaris operating installed in the global zone and some additional packages. Each non-global zone is completely Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 11 of 18
  12. 12. developerWorks® isolated, has its own package database, and does not share package information with the global or other non-global zones. 1. To create zones, you need to have an IP address for each zone that you configure. • Connect to the Solaris 10 system as an administrator. • Choose the name for the zone. 1. Create a zone using the zonecfg command and follow the steps as shown in Figure 6. Figure 6. Create a Solaris zone 2. Set the IP address and physical network card interface ID for the zone created as shown in Figure 7. Get the additional information about the zone by issuing the info command. 2. Enter the ifconfig -a command to get the interface card name. Figure 7. Configure the zone with network interface Making software testing greener Page 12 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  13. 13. developerWorks® 3. Once the zone is created successfully, install the zone. This will copy the different softwares installed from the global zone to the newly created zone. The time taken for installing the zone depends on the number of packages installed in the global zone, as shown in Figure 8. Once the zone is installed, it shows the following output: Successfully installed zone <zone name> ". Figure 8. Install the zone once it is created 4. Now you are back to the normal login system prompt. To check the zone Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 13 of 18
  14. 14. developerWorks® status, run zoneadm list -cv . If the value in the ID column against the zone name is '-', it means the zone is created and installed but not running. Run the boot command, to start the zone and the ID column against zone name will show '1,' as shown in Figure 9. Figure 9. List all zones from the global zone 5. For allocating resources to zones, use zonecfg from the global zone and then specify the resources for the particular zone. If you are creating multiple zones, share the resources appropriately, as shown in Figure 10. In this example, zonecfg is allocating 40% CPU to the txzone. Figure 10. Allocating resources to zones 6. Reboot the zone to effect the resources allocated for a particular zone. Using prctl from the global zone, the CPU share for global zone can also be defined. The following example defines 60% CPU share to the global zone. prctl -r -n zone.cpu-shares -v 60 -i process $$. prctl -i zone global 7. Once the zone is created and resources are allocated, connect to the zone for setting up additional configurations, such as setting up the password, DNS name, IP address, Locale, and the like. Connect to the Making software testing greener Page 14 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  15. 15. developerWorks® zone using the zlogin command, as shown in Figure 11. Once you log in inside the zone, the global zone is not visible. To verify, run uname -a. To log out from the zone enterthe ~.(tilde dot) command. Figure 11. Login to the zone 8. Now the zone is ready with the required configuration. The newly created zone is completely isolated from other the zone as well as the global zone. To the end user, it looks like as a separate Solaris system. The Solaris zones are very useful for product installation tests, GVT tests, and resource manager tests. Figure 12. Solaris zones In Figure 12, three zones are configured inside the global zone and resources allocated as 15% CPU share for zone 1 and zone 3. For zone 2, a CPU share of 25% is allocated from the global zone. All of these Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 15 of 18
  16. 16. developerWorks® zones, including the global zone, use the same kernel to access the hardware available in the physical system. There are some test applications running inside zone 2 and the database servers are also configured inside this zone. The database clients installed in zone 1 and zone 3 will connect to zone 2 to execute the database-based test applications. Conclusion Adopting virtualization technologies saves resources like servers, installation and configuration of different products on each server, test execution time, and also manpower. It also saves the administration effort to maintain the servers. On AIX WPAR, even if one server goes down for maintenance, it is easy to move the WPARs from one server to another server with minimal effort, using the workload partition manager. With all these advantages, adopting virtualization techniques helps the testing environment to move closer towards green IT. Making software testing greener Page 16 of 18 © Copyright IBM Corporation 2009. All rights reserved.
  17. 17. developerWorks® Resources Learn • The AIX 6.1 Information provides technical details about AIX Operating system. • See Introduction to Workload Partition Management in IBM AIX Version 6.1 to understand AIX 6.1 WPAR operating system virtualization capability. • Workload Partition Management in IBM AIX Version 6.1 provides information of enhancements on WPAR with AIX 6.1 TL2. • The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration. • Stay current with developerWorks technical events and webcasts. • Browse the technology bookstore for books on these and other technical topics. Get products and technologies • Download IBM product evaluation versions or explore the online trials in the IBM SOA Sandbox and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®. Discuss • Check out developerWorks blogs and get involved in the developerWorks community. • Participate in the AIX and UNIX forums: • AIX Forum • AIX Forum for developers • Cluster Systems Management • IBM Support Assistant Forum • Performance Tools Forum • Virtualization Forum • More AIX and UNIX Forums About the author Nageswararao V. Gokavarapu Making software testing greener © Copyright IBM Corporation 2009. All rights reserved. Page 17 of 18
  18. 18. developerWorks® Nageswara Rao is a Post-graduate student from Andhra University and joined IBM in 2006 as Software Engineer. He is presently at the India Software Labs, IBM, working with the TXSeries test team and has expertise in designing and developing test applications for middleware like TXSeries on UNIX-like systems. His areas of interests are vVirtualization, transaction processing management, and UNIX systems. Making software testing greener Page 18 of 18 © Copyright IBM Corporation 2009. All rights reserved.