2012 10 23_3013_rational_integration_tester_fo

Performance Testing with IBM
Rational Integration Tester

Note
Before using this information and the product it supports, read the information in “Legal
Notices” on page 35.

© Copyright IBM Corporation 2001, 2012.

1 INTRODUCTION ............................................................................................................................. 3
2 BACKGROUND .............................................................................................................................. 4
3 PERFORMANCE TEST INFRASTRUCTURE ......................................................................................... 5
3.1 INTRODUCTION ....................................................................................................................... 5
3.2 ENGINES................................................................................................................................ 6
3.3 PROBES ................................................................................................................................ 7
3.4 AGENTS ................................................................................................................................. 7
4 ARCHITECTURE SCHOOL ............................................................................................................... 9
4.1 INTRODUCTION ....................................................................................................................... 9
4.2 BASIC SYSTEM SETUP ............................................................................................................ 9
4.3 AGENT & ENGINE SETUP....................................................................................................... 10
4.4 PROBE SETUP ...................................................................................................................... 10
5 CREATING THE LOAD GENERATING TEST ...................................................................................... 12
5.1 RE-USING FUNCTIONAL TEST RESOURCES ............................................................................. 12
5.2 BASIC SETUP ....................................................................................................................... 13
5.3 TIMED SECTIONS .................................................................................................................. 14
6 CREATING PERFORMANCE TESTS ................................................................................................ 16
6.1 INTRODUCTION ..................................................................................................................... 16
6.2 INITIAL SETUP ...................................................................................................................... 16
6.3 ADDING TESTS ..................................................................................................................... 17
6.4 ENGINE SETTINGS ................................................................................................................ 18
6.5 MANAGING PROBES .............................................................................................................. 18
7 RUNNING PERFORMANCE TESTS AND ANALYZING RESULTS........................................................... 20
7.1 RUNNING THE TEST .............................................................................................................. 20
7.2 VIEWING RESULTS ................................................................................................................ 21
7.3 MULTIPLE DATA SETS ........................................................................................................... 23
8 DATA DRIVING PERFORMANCE TESTS .......................................................................................... 25
8.1 DIFFERENCES FROM FUNCTIONAL TESTS ............................................................................... 25
8.2 DRIVING A LOAD GENERATING TEST WITH EXTERNAL DATA ..................................................... 25
9 LOAD PROFILES ......................................................................................................................... 27
9.1 PERFORMANCE TESTING SCENARIOS ..................................................................................... 27
9.2 CONSTANT GROWTH ............................................................................................................ 28
9.3 EXTERNALLY DEFINED LOAD PROFILES .................................................................................. 28
10 ADVANCED TOPICS .................................................................................................................. 30

Page 1 of 36 © IBM Corporation 2001, 2012

10.1 BACKGROUND TESTS ........................................................................................................ 30
10.2 LOG MEASUREMENT .......................................................................................................... 30
10.3 CREATING THE MEASUREMENT TEST ................................................................................... 31
10.4 ADDING THE MEASUREMENTS TO A PERFORMANCE TEST ..................................................... 34
11 LEGAL NOTICES ...................................................................................................................... 35


1 Introduction

This document serves as a training manual to help familiarize the user with the performance testing
capabilities available in IBM® Rational® Integration Tester. It is expected that the reader has already
been through the basic Rational Integration Tester training, and understands the workflow of Rational
Integration Tester.

In this course we will:

 Create performance tests
 Set up agents, probes, and engines to execute and monitor performance tests
 Analyze results of performance tests
 Manage the amount of load driven by a performance test over time
 Data drive performance tests


2 Background

When testing a service oriented architecture (SOA), there will be times when simply verifying
functional requirements will not be enough. Many systems will need to come with service level
agreements (SLAs) that will state a minimum level of performance that must be satisfied.
This level of performance may have a number of components. In particular, system uptimes and
message response times will be important. However, it will not be enough to test the system to check
that it can respond to a single message within a given amount of time – the system will need to hold up
under a certain amount of load as well. This load may take the form of a large number of messages,
extreme message rates, or large amounts of data. In addition, accurately modeling the load on the
system may require us to generate message requests from a number of different sources.
For experienced performance testers, this will all be fairly familiar. However, SOA environments bring
challenges on top of the traditional client‐server model. For example, services are often shared among
several applications and failure can occur anywhere along the transaction path. Considering both the
number of services in place and the many points at which they intersect—any one of which may not be
performant—how can we ensure that performance levels satisfy the nonfunctional requirements?
In addition, there is a fundamental difference between SOA performance testing and a traditional,
client‐server approach. Performance testers who are familiar with the traditional approach tend to talk
in terms of the number of users or “virtual users” that are required to generate the load. They also tend
to be concerned with end‐to‐end response times – the response time experienced by an end user. This
end‐to‐end performance testing is typically executed against a functionally proven, complete system.
SOA performance testers are still interested in response times but would be more interested in the
volume of messages sent between components – there is no requirement to wait until the system has
completed assembly or for a front end GUI interface to be created. Hence, the SOA performance tester
can begin testing much earlier.
When running performance tests, you will normally be faced with the following questions:
1. Does the system’s performance satisfy the requirements or SLAs?
2. At which point will the performance degrade?
3. Can the system handle sudden increases in traffic without compromising response time,
reliability, and accuracy?
4. Where are the system bottlenecks?
5. What is the system’s break point?
6. Will the system recover (and when)?
7. Does the system performance degrade if run for an extended period at relatively low levels of
load?
8. Are there any capacity issues that come from processing large amounts of data?


3 Performance Test Infrastructure

In this chapter, you will:
 Look at the distributed nature of a performance test
infrastructure.
 See how engines are used to execute actions within a
performance test
 Examine how data needs to be recorded from the system
under test, and how this can be done with probes
 Learn how performance test licensing is handled

3.1 Introduction
Before creating performance tests, we will need to revise how we create the infrastructure of Rational
Integration Tester. As in regular functional tests, we have the Rational Integration Tester GUI, and the
project database. However, these will work slightly differently in the context of a performance test.
While in a regular functional test, the GUI and the test are normally run from the same machine, a
performance test may be run from another machine, or may be distributed across a number of other
machines. This means that the Rational Integration Tester software, as presented by the GUI, also
provides a test controller, to manage any remote systems involved in the performance test.
In addition, the project database, which is optional for a functional test, becomes mandatory for
performance tests. This is due to the higher volume of data that is recorded during a performance test
– it cannot be easily presented in a simple console window, but will need to be summarized, and
possibly manipulated.
Besides the GUI, test controller, and project database, there are also three new items in our
infrastructure: engines, probes, and agents. They fit together in a framework to run tests and monitor
performance across a number of different systems.


3.2 Engines
An engine is the process that actually runs a test in Rational Integration Tester. When carrying out
functional testing using Rational Integration Tester, an engine exists underneath the surface, and runs
the tests on behalf on the controlling instance of Rational Integration Tester (ie, the instance of
Rational Integration Tester that is running the main performance test), on the user’s machine.
When performance testing, the engine is separated from the controlling instance of Rational
Integration Tester. The engine can be on the same machine as Rational Integration Tester, or it can be
on another machine. In fact, there may be more than one engine, spread across multiple machines.
If there is more than one engine, test iterations are spread across the available engines. For example, in
a performance test which is executing 40 tests per second with 2 engines, each engine would be
running 20 tests per second. The distribution of the tests is handled by the controlling instance of
Rational Integration Tester.
Using multiple engines lets us solve a number of problems. Most simply, if one machine is not capable
of generating a high enough load for a performance test, the load can be split across multiple machines.
Secondly, multiple engines give us the capability to distribute the load across multiple endpoints. For
example, if we need to simulate requests arriving from different parts of the world, or from different
networks, we can set up engines in such locations as to satisfy the demands of the performance test.


3.3 Probes
With such complex and heterogeneous platforms, it can difficult to understand what to measure apart
from transaction response times. It will be impossible to measure everything.
Probes are the tools used by Rational Integration Tester to gather statistics from the system under test.
There are a variety of probes available to the user:
 System Statistics Probe
 Windows Performance Monitor Probe
 TIBCO BusinessWorks Probe
 TIBCO Rendezvous Probes
 TIBCO EMS Probe
 Sonic MQ Probe
 webMethods Broker Probe
 webMethods Integration Server Probe
 JMX Probe
The probes will be deployed on the systems that you want to measure, and multiple probes can co‐
exist on the one system.
Recording statistics with Rational Integration Tester’s probes gives us access to much more
information than just the transaction response times. This can aid us to determine the cause of poor
performance – if response times are becoming too long after going past a certain number of requests
per second, we can use probes to see if it is due to load on the CPU, excess memory usage, message
queues growing larger, or another cause.
Whichever probes you choose, they will record statistics during the performance test, and send those
statistics to the controller, for writing to the project database. These writes are set up as a low‐priority
task, so that they cause as small an impact as possible on system performance.

3.4 Agents
Engines drive the tests, and probes monitor them. However, both need a host controlling them, talking
to the instance of Rational Integration Tester controlling the performance test. This role is played by
the Rational Integration Tester Agent.
The agent runs on each machine that has an engine or a probe, and handles the communications with
Rational Integration Tester. The agent can host an engine, a probe, or both at the same time. In fact, it
can also handle multiple probes or engines within the one agent.
The agent is installed with the Rational Test Performance Server (RTPS), or the Rational Test
Virtualization Server (RTVS). It can be run by hand, or set up as a service on the system it’s running on.
Due to this, each machine that requires an agent requires an installation of RTPS or RTVS. Following
the installation of RTPS or RTVS, the agent will need to be configured in the Library Manager. This
configuration follows the same procedure as that of Rational Integration Tester itself, and so will not
be covered in this training course.


Note: If you are running through this training material on a cloud instance or virtual machine, all
parts of the system will be on a single machine. This is purely for ease of configuration, and does
not reflect a real‐world scenario.


4 Architecture School

 Configure Rational Integration Tester to connect to the
system under test
 Set up an engine to run the performance test, with an
agent to host it
 Configure a probe to measure the performance of the
system during testing

4.1 Introduction
Creating a model of the system under test will be very similar for performance tests to functional tests.
However, in addition to modeling the system under test, the Architecture School perspective will also
be used to provide configuration data for the agents, engines, and probes in the system.
Adding this information to your Rational Integration Tester project should be done after the normal
process of modeling the system under test; configuration for the performance testing components is
then carried out in the Physical View of Architecture School. Note that as it is configured on a
physical basis, you may need to configure new components when setting up new environments.

4.2 Basic System Setup
In this example, we will be testing a web service – a simple Login service that takes a username and
password, and returns a login token. We will first start the service on our local machine, then
synchronize with the WSDL provided by the service.
1. On the Desktop of the cloud instance, there is a folder called WebServices. Open this folder, and
execute RunLoginService.bat.
2. This will pop up a window – keep it open, but minimize it.
3. Open up Rational Integration Tester, and start a new project. Note that you will need to use a
project database – one is already specified on the cloud instance by default, so you can keep
using this, but use the Test Connection option to make sure that it is working correctly. If you
are not using a cloud instance, please ask your instructor for the project database settings.
4. Once Rational Integration Tester is open, switch to the Synchronization view within
Architecture School.
5. Press the button in the toolbar at the top of the Synchronization view, and select
WSDL.
6. The Create a new External Resource dialog will appear. Press the New… button.
7. The New WSDL dialog will appear. Press the Select… button to select a new location.


8. Once the Select Location dialog appears, switch to the URL tab. In order to get the URL of the
WSDL, copy it from the window that popped up when you ran the login service.
9. Press OK to close the Select Location and New WSDL dialogs.
10. Click Next, and run through the rest of the synchronization process as normal.

4.3 Agent & Engine Setup
1. In some cases, the agent might be executed manually; however, in this example, the agent is
running as a Windows service on the localhost. This means that the agent can be found at
localhost:4476. However, in order to enter its details properly, we need to know the name of
the local host. Execute the command hostname at a command prompt.
2. Switch back to Rational Integration Tester, and go to the Physical View of Architecture
School.
3. Press the button at the left of the Physical View Toolbar, and select the Agent
option.
4. In the Host field, enter the hostname provided in step 1. For the Port number, make sure that
the default setting is 4476.
5. An engine called default is automatically attached to the engine – leave this as‐is, and press OK
to close the dialog and complete the agent configuration.

4.4 Probe Setup
We’re now going to setup the probe that we want to run on the same machine as the agent. Remember
that each probe will need to be running on an agent, or tests using that probe will fail. Also, probes can
be set up on individual hosts, or on services running on those hosts – for example, the System Statistics
probe will run on a particular host, but most of the technology‐specific probes will need to be attached
to a particular process on that host. If you need to use those particular probes in the future, they can be
configured by editing the properties of that physical component, similar to the way we will edit the
probe on the host machine in this exercise.
1. In the Physical View, each physical component will be visible in a tree underneath a Subnet
and a Host. Double click on the host (which should have the hostname we used in the previous
exercise) to bring up its properties. The screenshot below shows where to find the host, though
your hostname and IP will be different.

2. Once the properties dialog for your host has appeared, switch to the Probes tab.
3. For our tests, we’re going to use the Windows Performance Monitor probe. Select it so that it
can be configured.


4. The first, and most important, setting to note is the Hosting Agent at the very bottom of the
dialog – it tells us which agent is running this probe. Currently, we only have one agent to deal
with, but make sure that the agent for this probe is set to the agent created in the previous
exercise. If the agent is not set here, then any performance tests that attempt to use this probe
will fail.

5. As for the other settings, the probe should be set to Collect Statistics Every 1 second.
Following that, we need to specify what statistics we need to collect from the Windows
Performance Monitor.
6. Press the button to add a new counter. The Add Counters dialog will appear.
7. For our first counters, we’ll examine memory statistics. To do this, select Memory in the
Performance Object field. Under Counter, select Available Mbytes, then press the Add
button.
8. Repeat this for the Page Faults/sec counter, and any other data you are interested in.
9. Now select Processor in the Performance Object field, and add the % Processor Time
counter, along with any other counters that are of interest.
10. Press Done to return to the configuration of the probe.
11. Select all of the Counters, then press the button to validate that each one is working. The
dialog should now appear like so:

12. Press OK to close the dialog. Our system is now set up to gather statistics during performance
tests.


5 Creating the Load Generating Test

 Create a functional test that can be used as a load
generating test
 Encounter the test actions created for use within
performance tests
 Create a timed section within a test to capture timing and
status information

5.1 Re‐using Functional Test Resources
One of the advantages in using Rational Integration Tester for SOA testing is that functional tests can
easily be refactored to be run within a performance scenario.
This is important because when evaluating the performance of the system, it is insufficient to just send
a request and measure the time it takes for a response to arrive. For example, if a web service
operation rejects input and sends back a SOAP Fault message, the time it takes may be significantly
different from the time it takes to properly process a request and return a valid response. If a test does
not truly validate the outcome of an operation, it will provide an inaccurate view of the true system
performance.
In this case, we will create a simple functional test that we will use as the basis for our performance
tests, to illustrate how this works. This functional test will be used as a load generating test within our
main performance test.
When editing the load generating test, there are several new actions that may be used. These actions
will be ignored when running the test as a functional test – they will only be executed when it is run as
part of a performance test.
Performance Actions

Begin Timed Section: Mark the beginning of a timed section for a
performance test.

End Timed Section: Mark the end of a timed section for a performance test.

Log Measurement: Log data to the project database during a performance
test.


Note: The Initialise, Test Steps, and Tear Down sections become more important in performance
tests. When we are running multiple iterations of a load generating test, the Initialise part of the
test will only be run once at the beginning, and the Tear Down section once at the end. Only the
Test Steps will be run for each iteration of any load generating test used in the performance test.
This means we can, for example, set up and clean up a database within the Initialise and Tear
Down sections without impacting the data that we are actually interested in.

5.2 Basic Setup
1. Before we can create a performance test, we need to provide a load generating test that will
contain the actions carried out during each iteration. To do this, we’ll create a normal test, and
add a timed section to it. Go to Test Factory, and right click on the Login operation to bring up
the context menu. Select New > Tests > Test Using MEP.
2. The Create dialog will appear – press the Options button to bring up a Settings dialog.
3. On the Message Settings tab, make sure the option Include Optional Fields is selected, and
then press OK to return to the Create dialog.
4. Call the test loginBase, and press OK. A test will be created.
5. Open up the Send Request message, and fill in a username, password, and application in the
relevant fields. The contents don’t matter for this example – our login service will accept any
input for these fields.
6. We now need to set up the validation for this message. For our purposes, it will be enough to
check that a login token is returned, and that it contains hexadecimal digits broken up with
hyphens. Open up the Receive Reply action, and find the Token field. Double click on the
(Text) section below that to bring up the Field Editor.
7. In the top half of the Field Editor, make sure that the Equality validation is selected, as we will
be replacing this validation with a regular expression:

8. Change the Action Type option just below from Equality to Regex.
9. Enter the regular expression ^[a‐f0‐9‐]*$
10. To test that it is working, enter 44ef‐2ab7‐573d into the Document field, and press Test. The
Result field should update to say true (if it doesn’t, make sure that you haven’t accidentally
included a space character at the end of the string).


11. Now add –y8rr to the end of the Document field, giving 44ef‐2ab7‐573d–y8rr, and press Test
again. This should fail.
12. Press OK to close the Field Editor, and OK again to return to the test.
13. Save the test, and then run it in Test Lab to make sure that it passes. If there are any problems,
fix them before moving on.

5.3 Timed Sections
Timed sections, marked by the Begin Timed Section and End Timed Section actions, allow us to time
the execution of different parts of the test being executed. A single functional test can contain multiple
timed sections, which may overlap, or contain other timed sections.
For each timed section, Rational Integration Tester will log data into the project database while a
performance test is running. This will include not only the time taken for the timed section to execute,
but also the status of the section – whether it passed, failed, or timed out.
If no timed sections are added to the test, Rational Integration Tester will still record the length of time
taken to execute each iteration of the entire test, and the status at the end of that iteration.
1. Start by returning to the Test Factory.
2. Add two new actions to the test – a Begin Timed Section , and an End Timed Section .
The Begin Timed Section should go before the two messaging actions, while the End Timed
Section should go afterwards.
3. Open the Begin Timed Section action.
4. The timed section will need a name – call it S1.
5. Below the name of the timed section, there is an option to determine how this timed section will
be recorded (Pass/Fail/Timeout). We can take the status of the test at the end of the section, or
we can take the status of the test at the end of that iteration of the test. In this particular case,
since the timed section covers the entire test, it will not make any difference which of the two
options we choose.
6. Close the Begin Timed Section action, and open the End Timed Section action. This has only
one setting – the name of the timed section. Match it to S1, the section we started with the Begin
Timed Section action, and close the dialog.


Our load generating test is now complete. However, we need to state how many iterations we will
carry out, at what rate, and so on. We will do this separately, in a performance test.


6 Creating Performance Tests

 Create a simple performance test
 Add a load generating test to a performance test
 Configure a performance test to use selected engines and
probes

6.1 Introduction
Once we have a functional test that will create a load on the system, we can start putting together a
performance test. Performance tests are created in the Test Factory, and are contained within the same
tree structure as other test resources.
Our first performance test will be fairly basic, running a single load generating test at 1 iteration per
second. Later performance tests will look at changing the load on the system, and varying it over time.

6.2 Initial Setup
1. In the Test Factory Tree, right click on the Login operation, and select New > Tests >
Performance Test. Call the test simplePerformance.
2. The initial screen of the Performance Test Editor will appear.

3. Click on the text on the left hand side.
4. The right hand side will alter to show settings for the performance test. Make sure that you are
on the Execution tab.


5. Most settings can be left at their defaults, but change the length of the test phase to 30 seconds,
as in the screenshot below.

6.3 Adding Tests
1. A performance test on its own does nothing – it requires load generating or background tests in
order to test the performance of your system. We will now add the loginBase test as our load
generating test. Right click on the text on the left hand side – two options will
appear:

2. Click on Add Load Generating Test.
3. The load generating test will appear, and should be selected on the left hand side of the editor.
Configuration information for the load generating test will appear on the right hand side. The
first thing we need to do is to choose which test will be used for load generation. To do this,
make sure you are on the Execution tab, and find the Test Path field. Press the Browse button
next to that field, and select the loginBase test from the dialog that appears.
4. We’ll leave the other execution options at their default settings for the moment, as shown
below.


5. The load generating test is nearly ready to go – but first, we will need to say what engine (or
engines) will be executing this test.

6.4 Engine Settings
Our test can be run on one or more engines. For the purposes of this manual, we will only be using a
single engine running on a single agent, but in more complex tests, multiple engines can be set up in
different locations, splitting up the load between different machines. Regardless of how many engines
are being used, remember that the engines are all managed by a single controller ‐ the instance of
Rational Integration Tester that is running the performance test.
1. Switch from the Execution tab to the Engines tab.
2. Press the Add… button at the bottom of the screen.
3. A Select dialog should appear. In this case, there will be only one engine available – the default
engine attached to the agent we created in Architecture School. Select the default engine, and
press OK.
4. If multiple engines were available, we could select more of them by pressing the Add… button
again, and selecting other engines, but for this example, our test is now ready to be run – we just
need to state how we will be monitoring it.

6.5 Managing Probes
Each performance test can choose which probes it wants to use to gather data. Different tests may be
measuring different data. For example, one test may be gathering system data, while another may
gather statistics from the middleware layer.
Regardless of which probes are being requested here, they must still be set up in Architecture School.
For this example, we will use the Windows Performance Monitor probe, as set up in the previous
exercises.
1. On the left hand side of the Performance Test Editor, click on the Performance Test to switch
back to its settings.
2. Click on the Probes tab.


3. We can now select from the available probes. As we have only set up the Windows
Performance Monitor, check the box for that probe, and leave the others blank.

4. Save the simplePerformance test.


7 Running Performance Tests and Analyzing Results

 Execute a performance test and view the statistics shown
at runtime.
 View the results of a performance test in the Results
Gallery
 Compare results of multiple executions of a performance
test

7.1 Running the Test
The procedure for running a performance test is much the same as for a functional test – simply use
the Run button in the Test Lab, or double click on the test in the tree. While the performance test is
running, a summary of the data being gathered will be displayed in the console. For full reporting, we’ll
need to go to the Results Gallery, as we’ll see in the following exercise.

1. Switch to Test Lab.
2. Run the simplePerformance test.
3. Watch the console results – you will notice that the probes will be started 15 seconds before the
load generating tests are run, as per the settings in the performance test.

4. Once the load generating tests are being run, you will see counters for the numbers of tests
started, passed, failed, and the number of pending database writes. These are defined in the
table below:


Iterations started in the report interval (The setting is 'Collect statistics
every' in Performance Test Statistics tab). The default interval is 5
Started
seconds, so if the test is set for 10TPS this would show a total of 50
each time.
Passed Iterations passed so far during the performance test.
Iterations where message receivers did not get a response within their
Timed Out
configured timeout.
Failed Iterations failed so far during the performance test.
Database writes queued on the results database. Large numbers
Pending DB indicate that database access is slower than required and may be a
Writes result of a slow network connection. Note that the writes are buffered
and do not slow down the test rate.

Note: A performance test may run on longer than the specified time while remaining test instances
complete and database writes are flushed. In this case you will see the 'started' figure as zero for
those intervals since the given number of iterations has already been started.

7.2 Viewing Results
1. The Test Lab doesn’t show much in the way of results besides statistics for how many timed
sections passed/failed, etc. To get this information, we need to go to the Results Gallery.
Switch to that perspective now.
2. In the Results Gallery, you will see a single line describing basic information about your test –
start/end times, number of iterations, etc. Select this, and then click the Analyse Results button
at the bottom of the screen.
3. An empty chart will now appear. We’re going to populate this with some of the data we have
collected with our probes. This is available to the left of the chart, sorted into different
categories. Expand this out to navigate to Performance Test Sections (Based on Start
Times) > Average Pass Section Duration > simplePerformance / S1.
4. A checkbox should now be available – tick it.


5. A chart should now appear on the left hand side. Experiment with adding other information
gathered by your probes to the chart. In particular, look at the information recorded by the
Windows Performance Monitor Probe. Note that charts can be removed simply by unchecking
them again.
6. As you experiment, you may find a situation where the axes of charts do not match. For
example, if you look at the Windows Performance Monitor probe, and select both pieces of data
for the memory, you might see something like this:

The number of MB available is changing, but the scale of the chart for the number of page faults
is preventing us from seeing that information.
7. In order to fix this, we can edit the properties of one of the charts so that it is displayed on a
separate axis. Go to the left of the chart display, where you have selected your data, and double
click on one of the coloured lines that has appeared next to the checkbox.
8. A Choose Style dialog will appear. On the Style tab, you can change how the data is displayed
(colour of the line, type of chart, etc.). Change any settings here that are of interest.


9. Switch to the Data tab, and set the Axis to 2. Close the dialog, and the charts should update:

10. At this stage, we can edit the chart and give it a name and some notes, in the text fields below
the chart. We can also save the chart for later reference, or use the button to export data to a
CSV file.

7.3 Multiple Data Sets
So far, we’ve just looked at results for a single performance test. It is also possible to compare results of
multiple performance tests, run at different times. This allows us to see how changes made to our load
generating test, or changes made to the system, have affected the performance of the system.
1. Close the chart for the moment, and return to the Test Lab.
2. Run the simplePerformance test again.
3. Once it is complete, go back to the Results Gallery, and choose Analyse Results for the most
recent test run.
4. A chart will appear, as before. However, this time another option is open to us: we can compare
this test run to any previous test run. Switch to the Data Sets tab, and press the Add button.
5. Select your previous test run here.


6. Return to the Counters tab. For each counter, two charts will now be available, and you can use
this to compare the current results to the previous results.


8 Data Driving Performance Tests

 Learn the limitations of using the iterate actions within a
load generating test.
 Use the Input Mappings settings in the performance test
to data drive a load generating test.

8.1 Differences from Functional Tests
When creating functional tests in Rational Integration Tester, we can simply use the Iterate Test Data
action to run through a data set, and test the system with that particular data set. However, just
imagine that we had a test similar to the one we have created already, that sends a single message, and
receives a single message.
As an example, if we were to use the Iterate Test Data action (or any other Iterate test action) with that
test, to send 10 messages, and our performance test had also specified 10 iterations per second, we
could end up sending anywhere between 10 and 100 messages per second, with no precise control
over the load on the system. In addition, test duration and status data will be limited in their
usefulness – rather than measuring the messaging times, we would be measuring the time to execute
10 messages; similarly, we would be recording the pass/fail status of a group of 10 messages, rather
than a single message.
For these reasons, it is generally advised not to use the Iterate Test Data action within a performance
test. Instead, we can map a data source to a test using the Input Mappings tab for each Load Generating
or Background Test.

8.2 Driving a Load Generating Test with External Data
1. Create a copy of the loginBase test, and call it loginDataDriven.
2. Open the Send Request action of the loginDataDriven test, and go to the Config tab.
3. Quick Tag the UserName and Password fields.


4. Save the loginDataDriven test.
5. Now make a copy of the simplePerformance test, and call it dataDrivenPerformance.
6. Open the dataDrivenPerformance test, and go to the settings for the Load Generating Test.
7. Find the Test Path setting – it should currently be set to the loginBase test. Use the Browse
button to change this to the loginDataDriven test.
8. We now need a data source, so leave Rational Integration Tester to create one. Create a brand
new CSV file or Excel spreadsheet with the following data (or some of your own invention – just
remember to add a line with headings):
User,Pass
Jim,gr33nhat
Steve,ght3st3r
Monica,perf0rmance
Karen,eng1n3s
Ben,pr0b3s
9. Save your CSV/Excel file, and return to Rational Integration Tester. Create a File Data Source
or Excel Data Source to connect to your data. Remember to use the Refresh button to check
that the data has loaded properly, and Save the data source.
10. Go back to the dataDrivenPerformance test, and go to the Input Mappings tab. Underneath
this, you will see three new tabs appear – Config, Filter, and Store.
11. In the Config tab, use the Browse button to choose your test data set.
12. If we wanted to filter the incoming data, we could do that on the Filter tab. In this case, we’ll use
the entire data set supplied, so go to the Store tab.
13. Map the tags in the loginDataDriven test to the columns in your data source here, and save the
dataDrivenPerformance test.
14. Run the test and analyze the results.


9 Load Profiles

 Encounter standard performance testing scenarios,
including load, stress, and soak tests.
 Use the Constant Growth settings in a performance test
to provide an increase in the load on the system.
 Data drive the load on the system using the Externally
Driven settings in a performance test.

9.1 Performance Testing Scenarios
So far, our tests have been modeling a very simple scenario, running our load generating test at 1
iteration per second. However, there are a number of scenarios where we might like to use more
complex performance tests. We’ll look at a few example scenarios, and how Rational Integration Tester
can deal with them.
Load Testing
A load test attempts to represent a period in the working day, and test how the system will respond to
a similar load. For example, it may be anticipated that the greatest risk of non‐performance may be at
the start of the working day. The scenario will model the ramp up from the minimum number of users
to the peak login period during the first few hours of business.
Stress Testing
A stress test scenario is used to identify the break point of the system. The break point will thus be
identified as a specified load (and ramp‐up) and will be used to identify performance weaknesses in
the distributed system. This may be modeled with a simple linear increase in the load on the system.
Stress tests can also be useful in proving the recoverability of a system – how does the system break
and how gracefully can it recover under extreme loads? In this case, a bell curve could be used to view
how the system breaks as it approaches a point identified as a break point, and then to allow the
system some room to recover.
Soak Testing
A soak test scenario will be run against a constant low‐level load that may run for hours or days. This
scenario could be a modeling an iteration run once per minute, or less. Running the test for an
extended period of time will identify any issues that may manifest themselves over a longer period,
such as memory leaks.


High Intensity Scenarios
High intensity scenarios, whether they are load tests or stress tests, will require a lot of data. Some
basic math will be required to understand the overall data requirements in terms of the data to drive
the tests and the data that is required to be present in the system for reference and execution. If you
are expected to run at 100 transactions per second for over 3 hours then you will require 3 x 60 x 60 x
100 rows of data – over 1 million rows of data! Whether or not you can use repeating data in these
million rows of data will be determined by the nature of the system under test. Similar issues may
arise in particularly lengthy soak tests.
In addition, modeling the correct number of requests in a high intensity scenario may not be possible
with a single test engine. You may need to create multiple test engines in order to be able to create a
sufficient load on the system.

9.2 Constant Growth
Moving beyond a simple, constant load on the system, the simplest way to vary the load in a
performance test is to increase the load over time. To do this, we split the performance test up into
multiple phases of a given duration, increasing the number of iterations for each new phase. This will
give a simple demonstration of how the system handles an increasing amount of load.
1. Return to the Test Factory, and create a copy of the simplePerformance test.
2. Rename the new test, and call it threePhaseTest.
3. Open the threePhaseTest, and on the Execution tab, change the Number of test phases to 3.
4. Switch to the Load Generating Test on the left hand side, and go to the Execution tab.
5. Change the Initial target iterations setting to 5 per second.
6. Set the Increment per phase to 5. We will now have 3 phases, 5 iterations per second, 10
iterations per second, and 15 iterations per second.
7. Save the performance test, and run it from the Test Lab. You should see each phase execute in
the console; notice that each phase is running more and more tests per second.
8. Go to the Results Gallery to view the tests, and view the data in a chart.

9.3 Externally Defined Load Profiles
Using an external data source gives us much more control over the amount of load on the system. The
length of each phase can be varied as required. In addition, a constant level of growth is no longer
required – the exact number of iterations per second/minute/hour can be set for each individual
phase.
1. Minimize Rational Integration Tester, and create a new CSV or Excel file with the following data:

Period,Iterations
10,10
20,30
30,10


We will use the first column of the data to specify the length of each phase that we will run,
while the second supplies the number of iterations in each phase. As we will specify our units as
seconds in the performance test (minutes and hours are also possible), we will have 10 seconds
at the beginning of the test where we run 10 iterations per second, 20 seconds where we ramp
it up to 30 iterations per second, and then 30 seconds where we allow the system to go back to
the original 10 iterations per second.
2. Return to Rational Integration Tester, and create a File Data Source or Excel Data Source to
link to your data. Remember to use the Refresh button to check that the data loads correctly.
3. Save the new data source.
4. Create a copy of the simplePerformance Test, and call it externalPhaseTest.
5. Open the externalPhaseTest, and go to the Execution tab of the Performance Test settings.
6. Change the Load Profile to Externally Defined.
7. Next to Data set for load phases, press the Browse button to find and select the data source
containing the phase data.
8. Leave the Execute test phases field blank, and make sure that the Phase duration read from
column setting is set to Period.
9. Switch to the settings for the Load Generating Test, and go to the Execution tab.
10. Make sure that the number of iterations is read from the Iterations column.
11. Save the externalPhaseTest, and run it in the Test Lab.
12. Go to the Results Gallery, and view the test results. You may notice that the statistics for the
minimum, average, and maximum pass section durations spike during the test, indicating that
the load was slowing down the performance of the system.


10 Advanced Topics

 Encounter background tests, and discuss their uses.
 See how the Log Measurement action works, and how it
can be used.
 Use the Log Measurement action in a background test to
provide a custom probe.

10.1 Background Tests
So far, we have used a load generating test to provide a pre‐defined load upon the system. Multiple
load generating tests could be used, if required. However, in some cases, you may want to provide a
constant stimulus for the system while using your load generating test. There may also be situations
where you need to use stubs to simulate part of the system under test. Both of these situations can be
handled by adding a background test to your performance test.
A background test is a functional test (or stub) that will be run repeatedly for the duration of the
performance test (or, optionally, until the background test fails). This means that it will be run
concurrently with any load generating tests included in the performance test. Each background test
can have a single iteration running at a time, or may be run multiple times in parallel – unlike load
generating tests, this is not limited by the Rational Integration Tester license that is in use. However,
timing and status information will not be recorded for a background test.
Since background tests are run differently to load generating tests, several things should be kept in
mind. Firstly, the Initialise and Tear Down phases of the test will be run as normal. Second, while the
Begin Timed Section and End Timed Section actions can still be included in the functional test, they
will not have any effect on what is recorded into the project database during the performance test.

10.2 Log Measurement
The Log Measurement action can be used to log custom data into your database while running a
performance test. This may be useful in several situations.
Firstly, it may be used when recording data from the system under test, acting as a custom probe. This
may be necessary when information is required that is not covered by the standard probes – for
example, when querying proprietary systems for information.
An alternative use exists for systems where a message goes through several processes before a
response is received by Rational Integration Tester. In these cases, it may be desirable to measure the
time taken for a single process to provide a response, rather than measuring the entire round‐trip time
between the initial message sent from Rational Integration Tester, and that eventual response.


In the diagram below, Rational Integration Tester publishes a message to a queue, and waits for a
response. Using the data normally gathered by a performance test, we would be told how long it took
for the message to be processed by operations A, B, and C. However, if there were performance issues
as we increased the load on the system, we would not know if these could be narrowed down to a
single service.

For example, we might suspect that service B is where most of the delay is occurring. In order to
investigate this, we can add timestamps to fields in the message as it passes through the system.
Subtracting Time 2 from Time 3 would then give us the amount of time that was spent inside service B.
Using the log measurement action, this information could be recorded in the project database, and
analyzed later with respect to the load on the system.
When using the Log Measurement action, it is important to note that it cannot be used within a timed
section. This is because writing to the project database would alter the time taken during the execution
of the timed section, thereby skewing the timing information.

10.3 Creating the measurement test
In this example, we’ll use a background test and the log measurement action to act as a custom probe
for the system under test. This particular example will be gathering data about the bytes sent and
received by the system – note that this could also be gathered by the Windows Performance Monitor
probe.
We’ll be using a background test for two reasons: firstly, it means we don’t need to change our load
generating test; and second, we don’t want to have our probe constantly running – we’ll gather our
data every two seconds, rather than constantly polling the system and possibly adding extra
unintended load.
1. Create a new test for the Login operation, and call it byteMonitor.
2. Add a Run Command action to the test.


3. On the Config tab, enter the following command:
netstat ‐e | find "Bytes"
4. Make sure that the Wait for command execution to finish checkbox is ticked.
5. Press the Test button. You should see a single line of data for stdout, similar to the following:
Bytes 12008764 57368668
6. Switch to the Store tab, so we can store the data into tags.
7. We’ll need to store the two numbers into separate tags, which we’ll be calling bytesSent and
bytesReceived. To do this, right click on the stdout field, and choose Contents > Edit.
8. Make sure you’re looking at the Store tab within the window that appears, and then press the
New button.
9. Details for the data to store will appear below. The default action type should be set to Copy –
change it to Regular Expression.
10. Change the Tag to bytesReceived. You can also change the description field to match.
11. In the Expression section, type the regular expression d+ to match a number. Below that,
choose to Extract Instance 1, so that we’ll be extracting the first number found in the string. In
the example string given above, this would store 12008764 into the bytesReceived tag. You
can test this out with an example string to check that it is working correctly.
12. We still need to store the number of bytes sent. Press New again to generate a second store
action for the stdout field, and follow steps 9‐11 again, but this time set the Tag name to
bytesSent, and Extract Instance 2. Similar to the first action, if you were to test this out using
the example above, you should get a Result of 57368668.
13. Once you’re done, the two tags should be configured as seen below:


14. Press OK to close the Field Editor, and then OK again to close the test action.
15. Before we add the Log Measurement action, we’ll check that this is working as we expect. Add
a normal Log action, and log the values captured in the bytesReceived and bytesSent tags to
the console.
16. Run the test, and check that it works at the moment. If it doesn’t, check the preceding steps to
make sure that everything has been entered correctly.
17. Return to the Test Factory, and delete or disable the Log action.
18. Add a new Log Measurement action after the Run Command.
19. Set up the Log Measurement action as shown in the image below. This will graph the number
of bytes sent and received by looking up the values captured earlier. The attributes section
allows us to graph multiple sets of data – in this case we only have one, but at least one attribute
is required in order for the Log Measurement action to run.


20. Press OK to close the action.
21. Add a Sleep action to the end of the test. As we’ll be running this as a background test, and
background tests run continuously, we’ll want to pace this test so it doesn’t interfere with
system results. Set the Sleep action to have a fixed duration of 2000ms.
22. Save the byteMonitor Test.

10.4 Adding the Measurements to a Performance Test
1. Create a copy of the externalPhaseTest, and call it customLogging.
2. Edit the customLogging Test, and right click on the Performance Test label on the left hand
side of the editor – you’ll see the options for adding load generating and background tests. Add
a background test.
3. On the Execution tab for the Background Test, select byteMonitor for the Test Path field.
4. Make sure that Terminate on failure is not checked.
5. Switch to the Engines tab, and click Add. There will only be one engine available, as before –
select it.
6. Save the customLogging test, and run it in the Test Lab.
7. Once it has run, you should be able to view the results in the Results Gallery. The counter will
be found in the Log Values section.


11 Legal Notices
 The following paragraph does not apply to the United Kingdom or any other country where
such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES
CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF NON‐INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain
transactions, therefore, this statement may not apply to you.
 This information could include technical inaccuracies or typographical errors. Changes are
periodically made to the information herein; these changes will be incorporated in new editions
of the publication. IBM may make improvements and/or changes in the product(s) and/or the
program(s) described in this publication at any time without notice.
 If you are viewing this information in softcopy, the photographs and color illustrations may not
appear.
 Any references in this information to non‐IBM websites are provided for convenience only and
do not in any manner serve as an endorsement of those websites. The materials at those
websites are not part of the materials for this IBM product and use of those websites is at your
own risk.
 Any performance data contained herein was determined in a controlled environment.
Therefore, the results obtained in other operating environments may vary significantly. Some
measurements may have been made on development‐level systems and there is no guarantee
that these measurements will be the same on generally available systems. Furthermore, some
measurements may have been estimated through extrapolation. Actual results may vary. Users
of this document should verify the applicable data for their specific environment.
 Information concerning non‐IBM products was obtained from the suppliers of those products,
their published announcements or other publicly available sources. IBM has not tested those
products and cannot confirm the accuracy of performance, compatibility or any other claims
related to non‐IBM products. Questions on the capabilities of non‐IBM products should be
addressed to the suppliers of those products.
 All statements regarding IBM's future direction or intent are subject to change or withdrawal
without notice, and represent goals and objectives only.
 This information contains examples of data and reports used in daily business operations. To
illustrate them as completely as possible, the examples include the names of individuals,
companies, brands, and products. All of these names are fictitious and any similarity to the
names and addresses used by an actual business enterprise is entirely coincidental.
 This information contains sample application programs in source language, which illustrate
programming techniques on various operating platforms. You may copy, modify, and distribute
these sample programs in any form without payment to IBM, for the purposes of developing,
using, marketing or distributing application programs conforming to the application
programming interface for the operating platform for which the sample programs are written.
These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot
guarantee or imply reliability, serviceability, or function of these programs. The sample


programs are provided "AS IS", without warranty of any kind. IBM shall not be liable for any
damages arising out of your use of the sample programs.

Trademarks and service marks
 IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International
Business Machines Corp., registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the web at www.ibm.com/legal/copytrade.shtml.
 Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other
countries, or both.
 Java and all Java‐based trademarks and logos are trademarks or registered trademarks of
Oracle and/or its affiliates
 Other company, product, or service names may be trademarks or service marks of others.


2012 10 23_3013_rational_integration_tester_fo

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to 2012 10 23_3013_rational_integration_tester_fo

Similar to 2012 10 23_3013_rational_integration_tester_fo (20)

More from Darrel Rader

More from Darrel Rader (7)

2012 10 23_3013_rational_integration_tester_fo