Migration from legacy POTS (plain old telephone system) systems to VoIP technologies has not always proceeded smoothly. The migration must consider many complex issues, including security, reliability, quality of service, and interoperability. Testing is critical to making such migrations smooth and effective. This paper shows issues encountered during migrations along with testing results. It focuses on tests that show the capabilities of a VoIP (voice over Internet protocol) system.

1. The Association of System
Performance Professionals
The Computer Measurement Group, commonly called CMG, is a not for profit, worldwide organization of data processing
professionals committed to the measurement and management of computer systems. CMG members are primarily concerned
with performance evaluation of existing systems to maximize performance (eg. response time, throughput, etc.) and with capacity
management where planned enhancements to existing systems or the design of new systems are evaluated to find the necessary
resources required to provide adequate performance at a reasonable cost.
This paper was originally published in the Proceedings of the Computer Measurement Group’s 2006 International Conference.
For more information on CMG please visit http://www.cmg.org
Copyright 2006 by The Computer Measurement Group, Inc. All Rights Reserved
Published by The Computer Measurement Group, Inc., a non-profit Illinois membership corporation. Permission to reprint in whole
or in any part may be granted for educational and scientific purposes upon written application to the Editor, CMG Headquarters,
151 Fries Mill Road, Suite 104, Turnersville, NJ 08012. Permission is hereby granted to CMG members to reproduce this
publication in whole or in part solely for internal distribution with the member’s organization provided the copyright notice above is
set forth in full text on the title page of each item reproduced. The ideas and concepts set forth in this publication are solely those
of the respective authors, and not of CMG, and CMG does not endorse, guarantee or otherwise certify any such ideas or concepts
in any application or usage. Printed in the United States of America.

2. VoIP Capacity Testing
By Garland Kan
Member, Technical Staff
Spirent Communications
Sunnyvale, CA 94089-1003
Migration from legacy POTS (plain old telephone system) systems to VoIP technologies has not always
proceeded smoothly. The migration must consider many complex issues, including security, reliability,
quality of service, and interoperability. Testing is critical to making such migrations smooth and
effective. This paper shows issues encountered during migrations along with testing results. It focuses
on tests that show the capabilities of a VoIP (voice over Internet protocol) system.
Introduction jitter, delay, and packet loss can impact voice even at
Deploying a new system into production is quite low amounts. Jitter can cause voice gaps and
difficult. When it comes to voice, the burden of a packets may arrive too late to be assembled back into
reliable deployment is even greater—not only is voice the voice stream. If the delay on the voice packet is
critical to the operation of a company, users have too long, that packet will not be useful anymore: since
come to expect top quality and reliability from years of too long a period of time will have passed, that portion
using legacy POTS (plain old telephone system). of the voice stream will not be played to the listener.
Testing can help make new deployments safer and This will result in a missing section of the voice
more effective. stream, which may potentially affect quality and
Without testing a system, its key metrics cannot be intelligibility of the conversation. Jitter has the same
properly quantified and problems may occur during effect--if there is a large enough gap between the
live usage. If a problem is found after the system was arrival of a packet, usefulness of that packet will have
turned on and is in a production state, it becomes expired, and it will not be played into the voice stream.
much more difficult to deal with it. For example, would Without capacity testing, these problems might not be
you try to fix the problem in real time or switch the noticed until a large amount of users are on the
users off to another system and then troubleshoot it? network. This is why testing a softswitch or a voice
At times, there may not be another fully redundant network prior to live usage is important.
system to move all the users to, so the only option is
to perform troubleshooting and testing while live users Problems users and service providers might face
are on the system. One way to avoid this undesirable Most users don’t care about the underlining
scenario is to perform testing before the system goes technology they are using for an activity they want to
live. Most of the problems can be identified and perform, instead gravitating to the most effective and
flushed out during this time. most affordable method. For telephony, VoIP is
currently one of the cheaper methods to make phone
By performing capacity testing on a softswitch or voice calls while also offering some features that other
network prior to live usage, one can gain wealth of phone services currently do not.
useful information. It provides hard numbers on the
maximum rates the system can perform at and what Users who switch to VoIP through services such as
type of voice quality the users will experience at those Vonage or Skype-Out expect decent voice quality
rates. With this information, the system can be tuned since they are paying for this service. With a POTS
to provide better performance or voice quality prior to phone (a.k.a. landline), the call is on a dedicated line
customers actually using the system. versus a VoIP call, which uses a public network and
has to share bandwidth with others. Moving to this
What users are accustomed to type of VoIP service puts the user’s phone calls on the
Users experience and therefore expect high voice Internet, with no guarantee that it will work or have
quality and reliability not only from their current good voice quality.
landline telephones but even cellular phones.
Switching over to VoIP at the edge (VoIP phones) Going through the Internet for a VoIP call is not the
and/or in the core (VoIP transport) can lead to new best solution. At times the voice stream can
problems that the users don’t’ often experience with experience packet loss, high jitter time, or even port
their traditional phones. Problems such as packet blocking from unkind service providers. For a user to

3. get the best possible experience one would usually does not take into account traffic that was not
have to be on the service provider’s network offering generated by the test itself.
the VoIP service. When service providers offer VoIP
service, they have control of how data packets flow What kind of tests to perform and why?
through their network. QoS (quality of service) can What kind of voice quality are the users getting while
then be applied to the voice streams right from the at the maximum rates? Can the network or softswitch
user’s home, which will give priority to those packets handle the designed load? If it cannot handle the
over others such as web or mail traffic. This ensures designed load, the problem can be narrowed down to
that the phone call will have a certain amount of pinpoint which device in the network is the bottleneck
bandwidth even when it is sharing the same pipe with or configured incorrectly. It can also show if the
other traffic. logical layer 2 and layer 3 services are working
correctly and performing as expected. These
Voice networks questions can be answered with the following tests.
A softswitch (or voice network) deployment is a
complex project in many ways. The network may or Calls Per Second
may not need to handle other traffic simultaneously. This is a performance test to show how many calls a
Quality of service on the network for voice call setup network or softswitch can process in one second.
and media must be configured. Security needs to be
in place to protect users and core telephony servers. When setting up a call, at the minimum a few things
With all the different pieces in a voice network, need to happen. The call processing engine (this can
everything has to be configured correctly and work be a softswitch a.k.a. call manager) needs to look up
together to provide high voice quality and reliability. the call and authorize it. Then it has to keep records
Comprehensive capacity testing ensures that the for the state of this call. This can be a database entry
entire network runs harmoniously after VoIP traffic has or it can be as simple as a log file. If the provider
been added. wants to be paid for this call there has to be a CDR
(call detailed record) for this transaction, which may
New network technologies are being deployed to be as simple as using the log file in the state table or
accommodate various types of traffic on the same inserting a record into a database that is keeping track
medium. Technologies like MPLS (multiprotocol label of calls to bill.
switching) and traffic engineering can provide the
necessary overlay for the current network that will With all these activities going on for just a single call, it
allow voice to travel through the network efficiently can be quite overwhelming for a system to try and
and in a timely manner. With these new technologies process the load for a large amount of calls.
and methods, service providers have more control on Performance limitations of different parts of the
how data will traverse the network. If done correctly, it system will start to show up at this point. For
gives users a better experience when their VoIP example, the database might not be able to handle
phone calls traverse that network. more than X number of transaction per second or
memory consumption and CPU rates start to rise as
What is capacity testing? more and more calls are initiated.
Capacity testing allows you to load the network up to
a point where different mechanisms ensuring voice Softswitch manufacturers usually provide engineering
streams quality have to take effect and perform some specifications on how many calls per second their
action. This type of testing means saturating the voice system can perform. This figure can come from either
network or the softswitch with a large amount of calls theoretical numbers of what each piece of their
to initiate these mechanisms and, if necessary, system can do and then combining it to produce a
perform some corrective actions. projected CPS (calls per second) number or by
actually performing a CPS test. determining the
At low rates these mechanisms may be in use, but not method used to come up with their CPS figure
noticeable. Counters and logs on various devices can requires running a test or going through their data to
show that different layer 2 and layer 3 mechanisms of confirm how they derived that number. Besides, in
the OSI reference model are being utilized. The one most instances your deployment of their equipment is
thing it doesn’t show is the effectiveness of these probably utilizing best of the breed solution, which
mechanisms, when the system is fully utilized. often will not match what the equipment manufacturer
has or used. Different equipment will cause the
This type of testing is usually performed when there is limitations and performance of certain parts of the
no other traffic besides the test traffic on the system. network to either increase or decrease.
With other traffic on the system, such tests cannot
provide an accurate prediction, because it usually CPS Test Setup

4. Setting up a CPS test can happen in two ways. The was setup in a short amount of time. When the
first is to utilize just enough endpoints to achieve the number of calls exceeded 800, the blue and green
desired CPS rate. The second is to utilize the lines began to diverge from each other. This means
maximum number of endpoints the system is capable that the softswitch is taking longer to connect the call.
of to achieve the desired CPS rate. Without knowing As the test continues, the gap between the blue and
the internals of the softswitch or network functions, the green lines starts to widen until no more new
second method is preferable, as it stresses more parts endpoints are being started. If more endpoints start
of the system than the first method. and the gap widens even more, the new calls will
most likely fail.
Concurrent Call Test
The concurrent call test brings up the maximum
number of calls that the softswitch or network can
handle. Since this test is not trying to stress CPS
rates, the ramp up speed of the calls will be slow.
This test stresses the network differently than the CPS
test. It utilizes more of the network resources such as
bandwidth and quality of service (QoS) mechanisms
than call processing activities such as CPU cycles on
the softswitch, database queries, state table updates,
or CDRs for the call. This test shows how many
concurrent calls the system can handle and the voice
quality when the maximum number of concurrent calls
are active on the system/network.
Figure 1 – CPS Graph
Although there are many different voice quality
measurement standards, the most used currently is
The concurrent call test shows what the softswitch or
the ITU-T P.862 PESQ (perceptual evaluation of
network can do when fully-loaded with the maximum
speech quality) algorithm. This method utilizes an
number of calls possible. Under normal situations, this
industry set standard for voice quality measurements,
type of event rarely happens during live usage. There
providing a number between -0.5 to 4.5 with 4.5 being
can be times when it spikes up to that level, but in
the highest score possible. Usually scores over 4.0
normal operation it does not stay at that load. This test
are considered good voice quality.
gives insight to the network’s capabilities and the
voice quality at this load.
What type of metrics and results will come out of
the testing?
Figure 2 shows a concurrent call test run. It
The results from these tests will provide wealth of
represents a well-behaved voice network that is able
information, but the most important ones are
to establish the maximum number of calls, which is
described below. This information helps an engineer
just a little over 1500. This figure only shows the
and/or management make decisions about the
maximum number of calls that can be setup, but it
system.
does not show the voice quality during the maximum
load.
The CPS test gives the maximum rate at which the
softswitch or network can connect calls. Using that
number, one can plan an upgrade path for the voice
network. Or If the CPS rate is not close to what the
design requirement states, this test can be used to
find possible configuration problems or network
devices that cannot process calls at that load, or help
search for alternative solutions that hit the required
performance.
Figure 1 shows a CPS test detailing the amount of
originating (green line) and terminating calls (blue
line). The Y axis is the number of channels and the X
axis is the time. This test ran near the limits of a
particular softswitch. As you can see early on during
the test, with less than 800 calls, the blue and green
lines overlap. This means that each originating call Figure 2 – Concurrent Call Graph

5. More specific tests, like testing a softswitch or trying to
Figure 3 shows 60 seconds worth of the PESQ scores narrow down a bottleneck in the network, will connect
for the concurrent call test starting at 20 minutes into specific devices or areas of the network to stress
(1200 seconds) into the test. During the full load, all test them for performance and voice quality. This will
the PESQ scores stay above 4.39, which are good reduce the external variabilities at play during the test
scores. Users on a conversation that goes through to help diagnose the problem or get measurements for
this system at that time would not complain about that portion of the network.
voice quality.
Figure 4 below shows possible test points for a
4.47
softswitch or voice network test. The test device
4.46 (Abacus 5000) connects to an SS7 link, the IP
4.45 network emulating various VoIP protocols, and the
4.44
PSTN network. By testing from many different points
PESQ Values
4.43
4.42 on the network and utilizing different protocols and
4.41
4.4
technology, the test stresses many devices to find
4.39 their performance and quality levels.
4.38
4.37
4.36
00
02
04
06
08
10
12
14
16
18
20
22
24
26
28
30
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
Time (seconds)
Figure 3 – PESQ Over Time
Call setup time is another important metric because
users expect a call to establish relatively quickly. With
landlines, it usually takes only a couple of seconds
before a ring comes back; cell phones take a little
longer. Depending on how the users use a VoIP
connection, they may have different expectations.
Users making a call from their home with an ATA
(analog telephone adaptor) expect a landline type of Figure 4 – Network Diagram
response. If they use a soft-phone from a computer
or a PDA (personal digital assistant), they are more The test equipment used for this type of testing should
likely to accept longer call setup times. The second emulate a real world endpoint as close as possible
type of users are usually the early adopters familiar and utilize as many voice protocols as possible to
with new technologies and will tolerate some simulate a real voice environment. This means that
inconveniences such as longer call setup times and the endpoints generating the calls should mimic a real
maybe even lower voice quality. soft-phone.
Test Environment – Hardware and Software The software driving this test equipment should also
The test environment should encompass all the facilitate detailed testing by gathering and calculating
endpoints that the softswitch/voice network will detailed statistics on all phone calls generated by the
encounter in production state. In a service provider’s test system. This allows full monitoring of every call’s
network this might mean various VoIP protocols (SIP, performance to obtain accurate statistics on the
MGCP, H.248/MEGACO) and different PSTN system’s performance.
gateways (T1/E1, SS7) traversing the two different
types of voice networks. Conclusion
Performing capacity tests on a softswitch or network
For a good network wide test, the test equipment for voice deployment is critical. It ensures that
should connect into various endpoints to establish everything is configured correctly or that the current
calls. These endpoints are usually at the edge of the configuration supports a certain voice quality and call
provider’s network where their responsibility ends and capacity. Testing of this nature allows one to perform
where the handoff to the next service provider starts. the due diligence this kind of deployment requires.
Connecting and testing from these locations ensures
that the entire network is tested. The locations to When tests described in this paper are performed on
connect the test equipment to will vary from network a softswitch or network, it can provide a lot of
to network, but usually the same methodology applies. information on how voice traffic will be handled. This
information can be used to properly design and tune

6. the system from the start, preventing surprises and
unhappy users,