Moving from Despair to Repair to Prepare
Using Processes, Tools, and Measurement to Realize the Benefits of IP Telephony Applications
VoIP is here to stay, and nearly all enterprises are considering when and how to take advantage of this new technology. There are a variety of business motivations
driving VoIP adoption in the enterprise, and there are real costs savings, productivity increases, and image benefits to be gained. These benefits are typically enabled
by VoIP infrastructure on a converged network, but achieved through IP Telephony applications such as messaging, conferencing, contact center, and mobility.
Realizing these benefits can be a challenge, and organizations may experience frustration, stress and even despair as they work to deploy VoIP in their
environment. Simply put, VoIP is much more than “just another application on the network,” and most organizations have never managed an application with high
availability and performance requirements like those of VoIP. With the right tools and training, however, organizations may become more skilled and responsive at
troubleshooting and resolving issues — repair.
The best approach is to consider VoIP deployment as a lifecycle and prepare.
Careful planning throughout the entire lifecycle can help enterprises meet the challenges of VoIP deployment before they turn into quality of service issues.
The best way to prepare for and combat the quality issues inherent in a converged network is to test the network thoroughly before rollout, and the best way to
prepare for VoIP application rollout and troubleshooting is to baseline VoIP applications and then test across a range of conditions to anticipate and resolve problems
before they arise in production. When good tools and processes are used, VoIP deployments can go smoothly and provide great business benefit.
This paper is part one of a three-part series on enterprise VoIP, which was developed to provide information about the factors affecting both the infrastructure and
applications during VoIP deployments. It lays out a lifecycle approach for VoIP deployment and describes the multiple levels of measurement needed to manage
VoIP applications. Tools and techniques are offered for baselining, proactive problem identification, and troubleshooting. Testing is not an option, it is essential, so
specific examples of how testing addresses common VoIP issues are described to help enterprises confront these issues before they impact quality and erode the
business benefits of VoIP.
The State of Enterprise VoIP
Motivations and Challenges
VoIP is here. The proliferation of converged networks has opened the door to a new world of enhanced applications and services, including consumer and carrier
services, and a strong surge in enterprise VoIP adoption.
VoIP Adoption in Enterprises
When and How — Not If
For enterprises, VoIP adoption has become a matter of when and how — not if. With all major vendors focused on VoIP solutions, using VoIP is becoming the clear
choice for new “greenfield” installations. The year 2005 will be a “tipping point” when the number of new VoIP phones shipped to enterprises exceeds the number
of TDM phones. A migration to VoIP is in the future of nearly all enterprises: a recent Empirix survey reported that 52% are already deploying VoIP at some level,
46% have plans for deployment, and only 2% have no plans to deploy VoIP at all. See, Figure 1 below.
Figure 1: VoIP deployment plans among enterprises.
It’s the Applications
The benefits that are used to justify VoIP deployments have evolved along with the technology. Three years ago, for example, projects were typically justified on
the basis of reduced operating costs, either for toll bypass or the lower operating costs of a single network. Then two years ago, the single biggest justification
was virtualization, the ability to work anywhere and tie geographically dispersed and mobile employees into a single system. Both of these justifications remain
strong, but increasingly, enterprises have found that the strongest recurring value is in the applications themselves. These include messaging (voicemail and
unified messaging), contact centers, auto-attendants, CTI, and conferencing — not new applications but traditional ones that VoIP makes more manageable, more
cost-effective, and more uniformly available. The technology also supports new applications: presence-based contact centers, video streaming, dual-mode mobile
phones, etc. But business benefits from the traditional applications alone provide ample justification for the migration to VoIP. They provide reduced operating costs,
simpler network administration, improved team collaboration and productivity, and better communications with customers, partners and suppliers.
So, if VoIP delivers such value, then what’s the catch? Like any major new technology, VoIP brings with it a unique set of challenges as well as opportunities.
VoIP is not just another Application on the Network
The concept of VoIP is so simple that organizations often underestimate just how complex the reality of VoIP is. VoIP is still an emerging technology, so there are
new standards, moving feature sets, and competing approaches, which make interoperability a major challenge.
In addition, voice is a difficult medium for IP: it is very sensitive to the inherent characteristics of IP networks, including latency, jitter, and packet loss. Voice quality,
availability, and security are critical, but are difficult to measure and manage. (Quality also depends heavily on the characteristics of the phones, equipment, and
incoming signal, since echo, distortion, and noise matter, as do packet-level impairments). Yet expectations are high based on the service levels people have
become accustomed to from phone systems: five 9’s availability, toll quality, clean interoperability and applications, and dial tone even when the lights go out. Most
companies have never managed an application with such high availability and performance requirements. At the same time, VoIP deployment is nearly always a
migration, which means that legacy systems must be managed simultaneously and slowly upgraded while interoperating with new packet-based equipment.
And all of this occurs in newly converged IT teams, where neither the voice people nor the data people have worked with anything quite like it before.
Experiences of Despair
Pitfalls of IP Telephony
Since enterprise VoIP deployments first began in earnest around 2001, there have been a number of high profile disasters. Dow Chemical, Carnegie Mellon, Merrill
Lynch, and others began wide-scale deployments only to back out or change direction due to quality and acceptance issues. In cases like these, the cost of failed
deployment can be millions of dollars — and this doesn’t include the inevitable disruption to business operations. To call the feelings of those in this situation
despair may be putting it mildly.
Leaking Away the Promised Business Benefits
Less visible but more common are projects that run into trouble but persevere in quiet desperation, losing the business benefit along the way. Project managers
and vendors usually don’t want to talk about these cases, but they are far too frequent. Here are just three examples:
A major consumer product company.A major consumer product company deployed 1,000 endpoints over one and a half years. “A learning experience,” according
to the telecom director. “At first it was no different from any other PBX deployment. Then we learned you had to be careful about the network. Pushing new rules
to the firewall caused the phones to reboot. We learned we needed QoS from end to end and we had to upgrade our entire network to support QoS and PoE. To
top it off, 50 call center agents moved in. Traffic went up; we started having phantom problems that were hard to troubleshoot. Then we had lots of patches, with
each patch causing an outage. Our call center is online 24x7; there’s no time to do offline management or apply patches.”
After more than a year, the company realized it needed troubleshooting and testing tools. “The system is stable now, but the damage has been done; end users
still blame everything on the phones. Now we’re in a PR situation internally,” according to this manager. There are still many miles to go to reach full deployment,
and the promised cost savings have evaporated, at least for the next few years.
A major retailer. A major retailer deployed VoIP at one of its main outlets as the first part of a substantial multi-site rollout. During the planning phase, the project
team underestimated the amount of traffic that would traverse the network. Due to congestion, they had to redesign and upgrade the network mid-stream, incurring
substantial unanticipated costs and causing a flood of complaints from both voice and data users over a four-month period.
A major mortgage provider. A major mortgage provider has deployed VoIP in ten out of 19 offices, with 3,400 endpoints to date. The Director of Converged
Infrastructure has watched a one-year project turn into a three years (and counting) endeavor. “I’m holding off on our call centers, trading desks, and executives.
The way this project has gone, I’m afraid to put a VoIP phone on my CEO’s desk!” Although deployment was planned thoroughly in advance, the stability and feature
interactions of the IP Telephony server left much to be desired. “We had frequent upgrades and patches, and each one caused disruption to our users because it
changed how key features worked. Interoperability was a huge challenge, and key applications like voicemail and call center basically didn’t work. Our voice quality
is still troublesome, but we don’t really know how much of a problem it is. And the payback for this project is long gone.”
Common Pitfalls and Issues
For those who have successfully deployed VoIP and IP Telephony applications, the process is generally a learning experience. The combination of new technology
and new organizational structures, as well as operating at several layers simultaneously, makes each deployment project a challenge. Though each project will
have specific issues, there are a number of common pitfalls and successful practices to avoid each one:
Lack of Organizational Readiness. VoIP is new for data and voice teams in an IT organization; it requires skill and commitment from both groups and can provide
a growth opportunity for everyone. To be successful, however, good cooperation and clear lines of responsibility must be in place. One enterprise CIO said, “I walked
into the kickoff meeting and announced that we were doing a VoIP rollout and that it would provide significant business benefits. Within ten minutes, it was as if
a trench had opened in the middle of the floor, and grenades were being passed back and forth!” One successful large-scale deployment avoided this pitfall by
describing voice infrastructure and voice applications and setting up responsibilities for each.
Underestimating VoIP as just Another Application on the Network. Although awareness of the special place of voice and IP telephony is now widespread in IT
departments, it is still all too common to launch into a deployment under-prepared. Unlike most existing IP applications, voice and video applications are extremely
sensitive to delay, jitter, and packet loss. QoS must be implemented in most networks to achieve acceptable call quality. And the expectations for voice availability
across the enterprise are far higher than for most other applications. A useful technique is to provide cross-training, case studies, and the necessary tools for
the staff involved with a deployment.
Expecting Telephony-Grade Support. Many IP telephony vendors have a different business model and philosophy than those that IT departments are used to for
voice. (Tried to fix it, but that sentence is still a little clumsy and I had to read it several times to understand what you were saying because the reference to
telephony vendors sort of comes from left field — by the way, though in general your writing is excellent!) Telephony services are run on standard servers, making
the lines of responsibilities less clear than with refrigerator-scale legacy PBXs. Compounding the issue is the fact that many enterprises use multiple vendors for
different parts of their infrastructure. Preparing ahead of time for troubleshooting and system management can help to avoid the finger-pointing and delays
in problem resolution that often occur.
Depending on a Single-Shot Network Assessment. Though extremely useful, a network audit cannot discover the dynamic behavior of a complex network. As a
result, multiphase deployments often have trouble in later phases because conditions have changed since the initial assessment. Testing a network with actual
VoIP traffic on a regular basis can avoid ongoing headaches — many of which are caused by mis-configurations that are a side effect of some other project.
Lack of a Lifecycle View. One classic pitfall is to neglect troubleshooting until after deployment; another is to neglect the proper testing to extrapolate from a pilot
to a full-scale deployment. Viewing deployment as a set of concrete phases helps to ensure clean deployments and management.
Focusing Solely on Infrastructure and Ignoring Applications. In a recent survey, the majority of VoIP deployments were motivated by applications such as
messaging, contact centers, and telework rather than by a less expensive or more manageable infrastructure. There are three levels at play in every VoIP
deployment: packet-level, call-level, and application-level. The most successful deployments pay attention to all three layers throughout the lifecycle.
Although the picture painted above — with complex pitfalls and challenges, and interdependent planning and testing required — may seem daunting, it is possible
to have a clean deployment and realize significant business benefits. Moreover, every business must eventually go through this migration. While no complete
process is available today, the key is to seek the best practices available and build these into the plan and the budget.
Moving to Repair
For those enterprises that have reached the zone of despair in their deployment process, the way out is clear: fix current problems and plan the next phases with
more foresight and wisdom. In most cases, this means equipping the team with better knowledge and tools.
Multiple Layers of Measurement
Measurement is crucial for improvement; it’s an old adage that you can’t manage what you can’t measure. At any stage in a deployment it is important to capture
baselines and have the capability to measure at multiple levels. Table 1 shows a view of the layers involved in a VoIP deployment, some important measurements,
and some useful tools.
Table 1: VoIP Management and Tools
LAYER MEASUREMENTS TOOLS
• Packet Layer • Packet loss • Packet Sniffers and Testers
• Jitter • Element-Level Network Management
• Delay • Switch/router Consoles/statistics
• Call Layer • Call Completion Rates and Latency • Call and Protocol Analyzers
• Voice Quality • Infrastructure Testers
• Equipment Status • Equipment/Server Consoles
• Application Layer (format) • Application Completion Rates and Latency • Application Testers
• Cross-Platform Status • Transaction-Level Management
• Provider SLAs • SLA Management
By measuring the operations of currently deployed IP telephony applications regularly, IT staff can gain the information needed for troubleshooting, problem
resolution, growth, and further deployments. A positive side effect of a measurement program is that staff become exposed and “trained” in how the
infrastructure and applications work.
Troubleshooting Tools and Techniques
Resolving VoIP problems, especially intermittent problems, is very difficult. A skilled troubleshooter needs a variety of tools at his or her disposal. One indispensable
tool is an analyzer capable of tracing and troubleshooting VoIP at the call level. In part two of this series, we describe specific scenarios and techniques for tracing
and troubleshooting using a specific tool — the Hammer Call Analyzer — as an example.
Testing Tools as a Means to Repair
Testing tools are another invaluable addition to the VoIP deployer’s arsenal. While we will cover useful testing techniques below, it is important to remember that
while ideally testing should occur early in the process, it is never too late to test. Many deployment projects have come back to baselining or network testing mid-
project, after starting without it and running into problems.
The baseline scenarios that can be captured through automated testing of VoIP infrastructure and applications are invaluable. They create a captured record of normal
operation for a given environment, help find remaining issues before users do, and are one of the best ways to train IT staff in VoIP operations and troubleshooting.
Prepare and Avoid Despair
Being Ready for Deployment
Of course, avoiding issues during VoIP deployment is the goal, and the vast majority of issues are avoidable with proper planning, training, processes, and tools.
But, no amount of preparation will provide a rollout that is 100% trouble-free, so being ready to repair is important as well. By viewing deployment as a lifecycle
with distinct phases and by being ready for each phase ahead of time, deployments can be accomplished smoothly, achieving benefits more fully while saving
significant amounts of time and money.
The VoIP Deployment Lifecycle
Migration in Phases
Nearly every VoIP deployment is a migration from the existing TDM infrastructure and can be done in sub-projects. In turn, each sub-project goes through
distinct phases, as shown in Figure 2 below. Ideally, project staff should plan, design, implement, operate, and optimize each sub-project, and use the lessons
learned for the next sub-project.
Even when the situation does not allow a careful, measured rollout, it is very useful to think of a project in terms of these phases. By keeping the phases in mind
AND remembering to consider the applications, not just the infrastructure, successful deployment can be better assured.
Planning for Successful Deployment
Enterprise VoIP deployments, like any other complex IT project, involve careful planning and project management. Figure 3 below shows a typical timeline for a
single-site IPT deployment. Multi-site deployments often have similar timelines but rely on multiple cutover dates because sites are completed independently.
In any scenario, planning for quality assurance measures throughout the project is extremely beneficial. This includes baselining the current network during needs
assessment, testing the network infrastructure multiple times as the network is prepared for VoIP, testing vendor solutions during the procurement process, testing
applications and infrastructure during installation and through cutover, and ongoing testing, troubleshooting, and management after the cutover.
It is very important to prepare IT staff ahead of time by making the organizational responsibilities clear. Voice applications are poorly understood by most data
network specialists; the impact of infrastructure changes on these applications is typically unpredictable. One large financial institution created separate teams for
VoIP Infrastructure and VoIP Applications and reinforced the fact that VoIP deployment was a growth and learning opportunity for everyone involved. This covered
90% of those involved and greatly improved teamwork among groups that had been uncomfortable working together previously.
Design the Infrastructure
Getting Networks Ready for VoIP
Voice over IP is very sensitive to IP network problems; so it is highly desirable to create a clean network environment prior to deployment. This can be achieved
by well understood network design practices. However, it is important to remember that network problems, which may not cause issues for data applications, may
be very problematic for VoIP.
Network Infrastructure Testing
Pre-deployment testing is the systematic testing of the ability of the network to support Voice-over-IP traffic. Because this generally involves sampling network
performance, it is not a substitute for planning. It can, however, help provide a level of confidence and identify potential problems.
VoIP-specific testing is a critical, often overlooked, practice to check a variety of key factors, including:
Ability and expertise to configure the network properly
Part three of this series discusses testing techniques in more depth, and describes specific scenarios and techniques for benchmarking, assessment, and testing
using a specific tool — the Hammer VoIP Test Solution for Enterprises — as an example.
Implement: System and Application Testing
Voice applications — such as conferencing, messaging and voice mail, call routing and CTI, contact centers, voice self service/IVR, voice transaction systems,
trading turrets, teleworking, click-to-talk, etc. — are what typically provides value from IP Telephony. These applications benefit from another layer of testing, over
and above the infrastructure testing described above.
Applications tend to have a large number of potential paths and many configuration parameters. IP telephony applications are no exception, and they have the
additional twist that they interact with a new infrastructure that can change the effectiveness of the application — sometimes dramatically, sometimes subtly. Once
the VoIP infrastructure is tested and good baseline measurements are captured, application-level testing can be done very effectively. Typical areas to test are:
Signaling latency (speed of dial tone, speed of call transfer, etc.),
Reliability of application information delivery (screen pops, information elements use for routing, etc.),
Application performance (IVR responsiveness, Application performance under load, etc.)
Impact of VoIP on applications (speech recognition accuracy with packet loss, conference bridge
loudest-speaker detection, etc.)
All-paths testing (correct configuration of all forwarding, hunting, routing, voicemail and messaging configuration, etc.)