Optimize load handling for high-volume tests with IBM Rational Performance Tester

2,047 views

Published on

Summary: When using IBM® Rational® Performance Tester for testing diversified protocols and large volume load simulations, it is essential to optimize the performance of your testing machines and tools, as well as your network and infrastructure. In this article, you will discover best practices that you can adopt to enhance the load generation capability of Rational Performance Tester per machine by configuring both the testing tool and the operating system. You will also learn about techniques that you can use to alleviate trivial errors that occur during large volume load simulations.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,047
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
64
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Optimize load handling for high-volume tests with IBM Rational Performance Tester

  1. 1. Optimize load handling for high-volume tests with IBM Rational Performance Tester Best practices for your load testing environment Skill Level: Intermediate Bharath Raj Keshavamurthy (bharathrajbk@in.ibm.com) Enterprise Solutions Performance Analyst IBM 24 May 2011 When using IBM® Rational® Performance Tester for testing diversified protocols and large volume load simulations, it is essential to optimize the performance of your testing machines and tools, as well as your network and infrastructure. In this article, you will discover best practices that you can adopt to enhance the load generation capability of Rational Performance Tester per machine by configuring both the testing tool and the operating system. You will also learn about techniques that you can use to alleviate trivial errors that occur during large volume load simulations. This article is organized into three aspects of preparing your IBM® Rational® Performance Tester environment. In the first section, you will learn about methods for deploying the Rational Performance Tester load generation environment, and information about deployment patterns to avoid. In the second section, you will learn about tuning parameters that facilitate optimal performance test load generation and operating system performance. The last section includes information about what you can do before you launch a performance test run to run a successful performance test. Rational Performance Tester deployment practices to adopt and to avoid Consider the following suggestions for deploying your Rational Performance Tester load generation environment:Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 1 of 13
  2. 2. developerWorks® ibm.com/developerWorks • Use many agent machines with low-end configurations (in terms of number of CPUs and amount of memory) and have these machines generate load. If you use fewer agent machines with high-end configurations, then the operating system is limited by the total number of available sockets, processing parallelism, and limits on multi-threading. Therefore, use many agent machines with low-end configurations that are horizontally scaled out for performance testing, rather than vertically scaling agents on a few high-end machines. Note: Low-end configuration machines are those with a low number of CPUs and memory. However, the CPU speed should not be considered as low configuration. Recommended CPU speed for Rational Performance Tester agent machines is 2.33 GHz or greater. Figure 1. Comparison of different topologies in designing the load generation environment Larger view of Figure 1.Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 2 of 13
  3. 3. ibm.com/developerWorks developerWorks® • If the test agents are in a virtual environment, ensure that the logical partitions on the machines where the performance test agent software is installed have dedicated physical Ethernet controllers to support high volume network traffic. Important: Do not use logical host Ethernet adapters or virtualized Ethernet devices as networking interfaces for logical partitions that run performance test agent services on Rational Performance Tester agent logical partitions. • Ensure that all agents are in the same network as the controller and the performance test setup. Doing so ensures that there is minimal network latency and enables faster transfer of periodic statistics, test assets, and data between the host controller and agent machines. Good topology decisions The first two topologies are illustrated in figures 2 and 3. They are best practices that you can adopt for deploying Rational Performance Test environment Topology 1: Best mechanism for performance testing the solution During performance testing, you must host the load generation environment in the same network as that of the solution that is being tested so that there is minimal network latency between the load generation environment and the solution. After you minimize network latencies, it is not only easier to diagnose performance bottlenecks, but there is also a decrease in network errors and packet drops that can be difficult to troubleshoot, and might lead to transaction errors, despite the server operating correctly. Figure 2. First topology for the performance test environmentOptimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 3 of 13
  4. 4. developerWorks® ibm.com/developerWorks Larger view of Figure 2. Topology 2: Rational Performance Tester host controller is outside of the solution network, but the performance test agents are inside This topology is almost the same as the previous one. Here, there can be communication problems between the host controller and its agents due to varying networks issues, such as the bandwidth of the network pipe and the network cards of the servers that host Rational Performance Tester. With that, the shortcoming of this mechanism is primarily seen in issues that prevent smooth execution of performance tests in the following situations: a. General execution of performance test runs or execution of performance test runs for large durations / Endurance performance tests which run for days together and with quick sampling rates b. There are many test agents involved for load generation purposes, approximately 25 test agents or more. If the network connectivity and bandwidth between the host controller and its agents are not uniform, then this might cause communication errors (network packet losses) and further result in abnormal termination of performance tests. The following errors might occur: • "Workbench did not receive any message from Agent for more than xx milliseconds"Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 4 of 13
  5. 5. ibm.com/developerWorks developerWorks® • "Could not connect to Agent. Connection timed out" Figure 3. Second topology for performance testing Larger view of Figure 3. Deployment patterns to avoid Avoid the topology patterns illustrated in Figures 4 and 5 when you deploy your Rational Performance Tester environment. Pattern 1: Rational Performance Tester agents are deployed on the same hardware as the solution components Although this topology of performance testing guarantees minimal network latency between performance test agents and the solution, the operating system resources are shared between the solution and performance test environment. In this case, resource utilization analysis becomes very difficult due to extreme complexity involved in differentiating resource utilization of the same operating system between the solution environment and the performance test environment. Inaccurate analysis leads to inaccurate data for performance issues identification, analysis, and troubleshooting. Therefore, this topology is not suggested for performance test environment deployment. Consider the following operating system resources: • Processor use • Memory use • TCP/IP sockets for both the solution component and agent processesOptimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 5 of 13
  6. 6. developerWorks® ibm.com/developerWorks • Network bandwidth, including the number of bytes sent and received per second for both the solution component and agent process Note: Deploying performance test agents on solution components also has an impact on the use of available TCP/IP sockets at the operating system layer in addition to process, memory and network bandwidth utilization. Rational Performance Tester creates connections for the requests that it makes during performance test load generation. Every connection uses TCP/IP sockets. This number is limited to only 65,535 (as the TCP/IP protocol allows only 65,535 TCP/IP ports to be available at any operating system platform). If the performance test agents use a large number of TCP/IP sockets, then the solution components have a shortage of socket connections, which creates a performance bottleneck and leads to poor performance. Figure 4. First pattern to avoid for performance testing Larger view of Figure 4. Pattern 2: The entire load generation environment (host controller and test agents) is out of the solution network Avoid this topology for testing the solution because it results in network errors, packet drops, and so forth. Thus this topology not only leads to disruptions and errors during performance testing, but also becomes difficult to drive the desired throughput and load on solution servers. Figure 5. Second pattern to avoid for performance testingOptimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 6 of 13
  7. 7. ibm.com/developerWorks developerWorks® Larger view of Figure 5. Performance tuning for your test environment The following tuning parameters for Rational Performance Tester have been tried and tested successfully for generating a large user volume load of 35,000 virtual users with a user concurrency of 3,000 virtual user against a 3 tier web-based application that uses HTTP protocol. You can apply these parameters when you test a web application against a user load between 1000 – 10,000 concurrent virtual users. Disclaimer: These tuning parameters have been noted to provide 25 – 40% improvement in the network throughput behavior of the operating system and are subjected to the application workload characteristics as well. The performance improvement that was achieved varied by 5 – 10% on different platforms. The AIX platform had the most significant improvement. These parameters are a reference point. You can map them against the application workloads and volumes that you are testing so that you get an appropriate set of values for your tests. Table 1: Operating system tuning for optimal performance in load generation OS Parameter Value Implementation detailsOptimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 7 of 13
  8. 8. developerWorks® ibm.com/developerWorks Windows MaxUserPort 65534 Edit the registry under HKEY_LOCAL_MACHINE TcpTimedWaitDelay 30 > System > MaxFreeTcbs 72000 CurrentControlSet > Services > TCPIP > MaxHashTableSize 65535 Parameters. All the TcpWindowSize 65535 parameters have to added as a DWORD EnableDynamicBacklog 1 and the values set as Decimal MinimumDynamicBacklog20 MaximumDynamicBacklog000 1 DynamicBacklogGrowthDelta 10 TcpAckFrequency 1 Linux net.ipv4.ip_local_port_range 65000 2048 Edit /etc/sysctl.conf with the following net.core.rmem_default 262144 parameters and save net.core.wmem_default 262144 the file. Run the following command – net.core.wmem_max 33554432 sysctl –p net.core.rmem_max 33554432 net.core.netdev_max_backlog 5000 net.ipv4.tcp_no_metrics_save 1 net.ipv4.tcp_rmem 4096 16777216 33554432 net.ipv4.tcp_wmem 4096 16777216 33554432 net.core.optmem_max 40960 AIX sb_max 6192000 no –p –o <parameter> = clean_partial_conns 1 <value> tcp_timewait 1 chdev -l sys0 -a maxuproc=<value> tcp_keepidle 600 tcp_keepinit 40 tcp_nodelayack 1 tcp_keepintvl 10 Maxuproc 8192 tcp_sendspace 4096000 tcp_recvspace 4096000 tcp_ephemeral_low 1024 rfc1323 1Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 8 of 13
  9. 9. ibm.com/developerWorks developerWorks® Practices to consider before you launch a performance test • Ensure that the Rational Performance Test agent service has been started on the agent machines. • Ensure that there are no connections in CLOSE_WAIT or TIME_WAIT that use the netstat command before you start the tests from the agent. Note: TIME_WAIT connections happen when the server closes its connection, but the client waits for an acknowledgement signal to close the connection. Connections go to CLOSE_WAIT after exceeding the time duration under the TIME_WAIT state and the operating system must close this connection. Having a connection in any of these two states before you launch a performance test prevents this connection from being used by Rational Performance Tester during the test run, which suggests that no occurrences of these types of connections are there before the test. This ensures a clean network state before launch. • For agent machines on UNIX operating systems, look for a process named rpt_runner and stop any occurrence of it before you run a performance test. In Windows, look for Java in the Task Manager, and stop any occurrences of the same before you start your test. Note: Stopping the rpt_runner process in Linux, AIX, or Java in Windows does not stop the Rational Performance Tester agent service. The agent process is started at the beginning of every test launch. Hence, stopping these processes before launch of next test (if any exist after a test run completes) ensures that processes of the agent that is associated with previous tests are cleaned up from the operating system. These processes might continue to exist after test completion due to abrupt termination of performance test or errors during performance test. • On Microsoft Windows machines, enable the show heap status option on the host controller: Windows > Preferences > General > Show heap status. You can use this to keep track of memory utilization for Rational Performance Tester during tests. If the memory utilization increases, force the garbage collection manually by clicking on the garbage collection icon in the heap status bar. Knowing this helps you size the memory for the host controller machine. • Close the host controller workbench and launch it again to achieve minimal use of the JVM heap memory before you start your test. You must force garbage collection once to clean up any unreferenced objects in the Rational Performance Tester JVM heap memory before starting your test.Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 9 of 13
  10. 10. developerWorks® ibm.com/developerWorks • Delete all the contents in the following directories and restart the agent server before you run your test: • Temp directory of the operating system. (/tmp for Unix based operating systems, and C:Documents and Settings<User>temp directory for Windows) • Deployment_root directories of agents • Shared class resources that are present in C:Documents and Settingsuser nameLocal SettingsApplication Datajavasharedresources • In certain cases, you must restart the operating system of the performance test agent machines. • Ensure that adequate disk space is available in temp directory and the file system that contains the deployment_root folder. Recommended minimum value for the temp directory is 10 GB, for the deployment directory is 5 GB. • When generating high loads and considering sync points, use many small configuration agent machines rather than fewer large configuration machines to get better results • If the test duration is long or involves high throughput generation of requests, then avoid statistics sampling at a fast rate. A high statistics sampling rate is overhead for the performance test tool and must be kept at an optimal rate. For example, in a 2-hour test run, a 15-second sampling interval is an optimal statistics sampling rate. Figure 6. Statistics sampling rate in Rational Performance Tester • Set the error logs to minimum logging. Only errors and warning can be logged for primary test actions when running large volume tests to achieve optimal performance in load generation. Figure 7. Logging level options in Rational Performance TesterOptimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 10 of 13
  11. 11. ibm.com/developerWorks developerWorks®Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 11 of 13
  12. 12. developerWorks® ibm.com/developerWorks Resources Learn • Find out more on the Rational Performance Tester product overview page. Then explore the Rational Performance Tester page on IBM® developerWorks® for links to technical articles and browse the user assistance in the Rational Performance Tester Information Center. • Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products. • Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics. • Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends. • Watch developerWorks on-demand demos, ranging from product installation and setup demos for beginners to advanced functionality for experienced developers. • Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, any time, and many of the "Getting Started" ones are free. Get products and technologies • Download the trial version of IBM Rational Performance Tester. • Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently. Discuss • Join the Performance Testing forum, where you can share you questions and knowledge about IBM performance testing products. • Share your knowledge and help others who use Rational software by writing a developerWorks article. Youll get worldwide exposure, RSS syndication, a byline and a bio, and the benefit of professional editing and production on the developerWorks Rational website. • Follow Rational software on Facebook and Twitter (@ibmrational), and add your comments and requests. • Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis. • Connect with others who share your interests by joining the developerWorksOptimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 12 of 13
  13. 13. ibm.com/developerWorks developerWorks® community and responding to the developer-driven blogs. About the author Bharath Raj Keshavamurthy Bharath Raj works at IBM India Software Labs with the High Performance OnDemand Solutions (HiPODS) team as an Enterprise Solutions Performance Analyst. He started his career with IBM as a J2EE application developer using Web 2.0 technologies. He has worked on the performance of middleware, Java, and databases that are used for enterprise solutions in telecommunications, retail, and other industries. With an intensive background and experience in performance tuning, capacity planning, testing and engineering methodologies, Bharath Raj applies a holistic approach to the performance of an enterprise solution. He continues to use cutting edge technologies and imbibe them in enterprise web applications to achieve an optimized, faster and much more reliable solution.Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks© Copyright IBM Corporation 2011 Page 13 of 13

×