VoIP Perspective Competitive Intelligence Study Overview
Upcoming SlideShare
Loading in...5
×
 

VoIP Perspective Competitive Intelligence Study Overview

on

  • 329 views

 

Statistics

Views

Total Views
329
Views on SlideShare
329
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    VoIP Perspective Competitive Intelligence Study Overview VoIP Perspective Competitive Intelligence Study Overview Document Transcript

    • Competitive Intelligence Study for VoIP Providers Wave 2 1 VoIP Perspective Competitive widespread adoption of VoIP in consumer markets. Indeed, Keynote found in its first study that VoIP Intelligence Study Overview providers on average were able to complete calls In January 2006, Keynote Systems is reporting on only 96.9% of the time with a Mean Opinion Score the results of Wave 2 of its Competitive Intelligence (MOS) of less than 3.5 - considerably worse than study to evaluate how end-users perceive the core the standard for traditional phones of 4.2! It should reliability and audio clarity of leading VoIP Service not be surprising then that 52% of respondents in Providers. Keynote - an independent, trusted a recent survey by Harris Interactive indicated VoIP authority on Internet performance - conducts call quality as a key barrier to adoption. How can benchmarking studies for various industries on an VoIP providers in the residential and SMB market ongoing basis, to assess the experience of end- overcome consumer perception of poor reliability users with key applications over time. The VoIP and call-quality and live up to the dial-tone quality Competitive Intelligence study, first launched in July that consumers have come to expect over the years? 2005, evaluates critical performance factors that Keynote’s Competitive Intelligence study reveals the affect the consumer’s experience with VoIP service, in facts that help VoIP providers overcome the quality the New York and San Francisco markets. Keynote’s barrier relative to traditional phone service. Keynote first study, widely quoted in industry publications evaluates audio quality as well as underlying network and leading newsletters worldwide and based on an performance - whether DSL, Cable or Business Class industry-standard methodology, has already helped network - that affect the end-user experience and service providers enhance their end-user experience compares them across various VoIP service providers and reduce attrition rates dramatically. Wave 2 of the and traditional phone service. With this insight, study features several new service providers including Keynote provides actionable recommendations to AOL Total Talk, EarthLink trueVoice, and Time help improve end-user facing service quality and Warner Digital Phone, and provides trend analysis for capitalize on growing market adoption. providers included in the first study. Analysts estimate that residential adoption of VoIP Study Participants service is bound to grow exponentially from 6.5 Keynote measured and evaluated the performance million homes in 2004 to over 26 million homes of the following eleven consumer VoIP Service in 2008. The SMB and enterprise market forecasts Providers as part of this competitive intelligence are equally aggressive. Even so, VoIP reliability and study. These providers all offer VoIP service in both audio clarity remain important factors that limit the the San Francisco and New York metropolitan areas: (Time Warner Digital Phone was not available at all agent locations) Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.
    • Competitive Intelligence Study for VoIP Providers Wave 2 2 Keynote also measured and evaluated VoIP call performance across the following network carriers: The Comcast Cable, Time Warner Cable, SBC DSL, all contributed to the Most Reliable ranking. This and Verizon DSL connections used in this study high-level ranking illustrates which VoIP service are all standard residential service offerings. The providers are providing the most reliable VoIP Sprint Business Class network connections used service offering. The Best Audio Clarity ranking was in the study are standard data center or hosting computed as an aggregate of the performance factors facility service offerings. related to audio quality and audio delay. This high-level ranking illustrates which service providers Overall Best VoIP Service Providers are providing the best sounding and highest quality call audio in their VoIP service. Keynote ranked VoIP Service Providers in two categories: Most Reliable and Best Audio Clarity. The Keynote study found that the following VoIP The Service Availability, Average Number of Dial service providers ranked highest in the Most Reliable Attempts, and Dropped Calls Performance Factors and Best Audio Clarity rankings: Most Reliable VoIP Service Provider Best Audio Clarity Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.
    • Competitive Intelligence Study for VoIP Providers Wave 2 3 Overall Best Network Carriers of the performance factors related to audio quality and audio delay. This high-level ranking illustrates Keynote ranked Network Carriers in two categories: which network carriers are providing network Most Reliable and Best Audio Clarity. Service performance that results in the best sounding and Availability, Average Number of Dial Attempts, and highest quality call audio. Dropped Calls Performance Factors all contributed to the Most Reliable ranking. This high-level ranking The Keynote study found that the following network illustrates which network carriers are providing the carriers ranked highest in the Most Reliable and Best most reliable Internet service for VoIP traffic. The Best Audio Clarity rankings: Audio Clarity ranking was computed as an aggregate Most Reliable Network Carrier Best Audio Clarity Key Performance Factors for a single VoIP service provider or network carrier to excel in all ten aspects. Keynote’s Competitive To obtain the Most Reliable and Best Audio Clarity Intelligence Study Report presents detailed analysis rankings, Keynote measured, evaluated, and and results for each performance aspect separately. compared the performance of service providers and network carriers in ten Key Performance Factors. The The full report provides detailed answers to ten Key Performance Factors are: questions like: • Service Availability • Is VoIP service as reliable as PSTN (Public Switched Telephone Network) service? • Average Number of Dial Attempts • On which provider’s VoIP service does the call audio • Dropped Calls sound the best? • Outage Minutes • Which VoIP service provider had the most dropped • Average MOS (Mean Opinion Score) calls? • Percentage of Calls with Acceptable MOS • Which network carriers have the highest network- • MOS Geographic Uniformity level jitter? • Average Audio Delay • Are residential consumers using VoIP better off on • Percentage of Calls with Acceptable Audio Delay cable modems or DSL? • Audio Delay Geographic Uniformity The following sections offer highlights of the find- ings for three of the ten performance factors: Service Each Key Performance Factor represents a distinct Availability, Average MOS, and Average Audio Delay. aspect of VoIP call performance; as illustrated by the selection of results presented below. It is rare Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.
    • Competitive Intelligence Study for VoIP Providers Wave 2 4 Service Availability by VoIP Average Audio Delay by VoIP Service Provider Service Provider The Service Availability factor represents the The Average Audio Delay performance factor percentage of call attempts that are successful. Each measures the amount of time it takes audio to call attempt includes up to three dial attempts. travel from one end of the phone call to the other. An unsuccessful call is one in which no connection to Excessive delay can result in conversational disruption. the called number can be established. Unsuccessful ITU recommendations for the Publicly Switched calls can be the result of standard phone system errors Telephone Network encourage audio delays of 150 such as no dial tone, no ringing, ring no answer, ms or less. None of the VoIP service providers has or busy signal. VoIP service calls can also fail when achieved this goal. service is rejected for account problems. Average Audio Delay Service Availability Rank VoIP Service Provider Rank VoIP Service Provider 1 AT&T CallVantage 1 Time Warner Digital Phone 2 Time Warner Digital Phone 2 Vonage Study Results Average MOS by VoIP Service Provider The study results are available in two formats: A Mean Opinion Score (MOS) measures the voice audio quality of a phone call on a scale of 1 to 5. • An overview presentation, which reveals general Traditionally, this score was derived from a sample industry trends and highlights the performance of of subjective human opinion scores. Today, an only the best VoIP Service Providers and Network algorithm, ITU Standard P.862, known as PESQ Carriers. The insight gained from this competitive (Perceptual Evaluation of Speech Quality) replaces intelligence helps executives identify general areas the human panel and scores call audio by comparing of service improvement that will have the maximum a digital test audio file sent over a phone call with impact on VoIP customer experience. a reference copy of the same digital test audio • A detailed presentation that analyzes each of the file to see how much the audio degrades over the ten performance factors in depth. The detailed phone call. Comparing the two audio files, the presentation includes analysis of the data across PESQ algorithm can determine a MOS score for the VoIP Service Providers, Network Carriers, network call that has a very high correlation with human connection types, and different cities. The perception. The Average MOS for a VoIP service presentation also reports on the network-level provider is an indication of the audio quality that performance of each Network Carrier. Armed with each service provider delivers to end-users. this intelligence, IT Managers can tune specific aspects of their VoIP service to provide better Average MOS performance to customers than their competition. Rank VoIP Service Provider 1 Time Warner Digital Phone 2 Vonage Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.
    • Competitive Intelligence Study for VoIP Providers Wave 2 5 Study Methodology • Audio performance metrics include MOS (Mean Opinion Score) and values for audio delay, signal Keynote’s VoIP Perspective agent technology levels, noise levels, silence levels, and codec used. accurately measures the availability and performance • Network metric analysis includes packet counts, of VoIP phone systems. Unlike other product offerings latency, ordering, and jitter information. that estimate VoIP performance based on just network level performance metrics, VoIP Perspective • Standard call error codes such as no dial tone, no analyzes the received actual call audio for the highest ringing, ring no answer, busy signals, and operator measurement fidelity possible. intervention are captured and recorded. • Enhanced audio analysis goes beyond the PESQ • Call audio analysis is based on the ITU Standard standard to identify common characteristic flaws P.862, also known as PESQ (Perceptual Evaluation of like static, hiss, and hum in call audio. Speech Quality.) The figure below illustrates the overall study methodology: Overall Methodology 1 2 3 4 5 Investigation Deployment Data Collection Analysis Rankings • What VoIP • Accounts are created • Data is collected • Data sample • Rankings are providers are the at VoIP providers over a five week undergoes created and largest in two major using their normal period statistical analysis presented metropolitan areas: customer sign-in • Providers and • Best providers San Francisco and processes carriers are named New York? • Calls are defined investigated • What types of across all providers individually network carriers and ISP combinations • Perform first are being used • Each call uses the level problem to access these same reference audio diagnosis VoIP providers? to ensure consistency • Industry trends in measurement are noted from call to call • All calls are made to PSTN numbers to ensure comparable measurements • Measurement configurations are deployed to the Keynote VoIP perspective agents for measurement Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.
    • Competitive Intelligence Study for VoIP Providers Wave 2 6 How Calls Are Measured • Hang up— After a fixed amount of time, the agent hangs up the call. The VoIP hardware interface Keynote measures the performance of a VoIP service or software client sends the appropriate SIP or by making actual phone calls using the service and MGCP protocol commands to tear down the call measuring the performance of the call. Each call and hang up. involves a sequence of steps: • Audio Analysis—After the call is concluded, the • Dial the phone number—The agent uses a voice agent compares the audio received during the modem that can send the appropriate DTMF tones call with perfect quality reference audio, using an to the hardware adapter, acting just like a POTS implementation of ITU Standard P.862, also known telephone handset. For soft phones, the software as PESQ (Perceptual Evaluation of Speech Quality.) client is started and a “virtual user process” sends • Data Collection and Storage—Call measurement keystroke and mouse click events to the client data is pushed on a regular basis to central software, just as a live human user sitting at a databases at Keynote. personal computer would do. • Call setup—The VoIP phone adapter communicates Measurement Topology with the SIP Server or MGCP Call Agent to establish the call. Since the agent is calling a number on the and Frequency Publicly Switched Telephone Network (PSTN), the For this study, Keynote placed VoIP Perspective call is established with a SIP to PSTN Gateway or Agents in the New York and San Francisco MGCP to PSTN Media Gateway. metropolitan areas. Every thirty minutes, these Agents placed VoIP-to-PSTN calls across all • Responder answers the call—An automated combination of VoIP Service Provider and Network software process called a responder answers the call Carrier to PSTN phone numbers in both New York at the PSTN phone number. A separate responder and San Francisco. Data was collected over a six-week and phone line is used for each provider and city, to period from November 18, 2005 to December 31, avoid overlapping calls and preclude busy signals. 2005. A total of over 190,000 calls were placed, over • Send DTMF control signal— Once the call is 17,500 calls on each VoIP service provider, and over established with the PSTN phone number and 30,000 calls on each network carrier. the responder being called has answered, the agent sends a DTMF control code. This signals the The diagram on the next page illustrates the responder to play the digital recording of the test network topology of the study. In each city, there is audio message. a Responder with a dedicated PSTN phone line for each calling Agent. Each city has three calling Agents • Record received audio—Once the agent has sent connected to different network carriers: a Business the DTMF control signal, it immediately begins Class carrier, a residential Cable Modem carrier, and recording any received audio. a residential DSL carrier. Each Agent is equipped with • Record network performance measurements— the Analog Telephone Adapter hardware provided While the audio is being received, a packet analysis by each of the VoIP service providers and uses that process on the agent is also recording network- hardware to place calls on each VoIP service on a layer performance metrics including latency, packet regular schedule. loss, packet order, and jitter. SIP protocol packets are also examined and metrics regarding the various stages of the call (registration, setup, audio, and teardown) are also recorded. Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.
    • Competitive Intelligence Study for VoIP Providers Wave 2 7 Call Test Audio File Ranking Methodology To ensure a meaningful measurement of any voice The Most Reliable ranking was computed based on communications system, the test audio must be a the Service Availability, Average Number of Dial good representation of human speech. The Keynote Attempts, and Dropped Calls Performance Factors. VoIP Perspective Competitive Intelligence Study used Each provider earns points based on their relative a fifteen second test signal specifically designed for index ranking in each of the three performance voice audio quality testing. The test file includes the factors. Service Availability counts for 50% of the following characteristics: total, Average Number of Dial Attempts and Dropped Calls each count for 25% of the total. The final score • Includes male and female voices for each provider is out of a possible 1000 points. • Low noise floor The Best Audio Clarity ranking is computed • Correct speech audio levels by the formula: • Utterances of one to three seconds in duration 1000 x ((0.7 x MOS Component) + • Contains intervals of silence (0.3 x Audio Delay Component)) • Consistent volume Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.
    • Competitive Intelligence Study for VoIP Providers Wave 2 8 The MOS Component is a computed value between 0 and 1 for each provider that is based on the MOS measured for each call, factoring in acceptability and uniformity. The Delay component is a computed value between 0 and 1 that is based on the Audio Delay values measured for each call, factoring in acceptability and uniformity. About Keynote Systems Keynote Systems, The Internet Performance Authority®, is the worldwide leader in e-business performance management services. Over 2,100 corporate IT departments and 19,000 individual subscribers rely on Keynote’s growing range of measurement and monitoring, service level and customer experience management services to improve e-business performance by reducing costs, improving customer satisfaction and increasing profitability. Keynote Service Level Management (SLM) solutions provide enterprises with the tools to align IT and e-business goals. Keynote’s SLM solution suites offer comprehensive approach to the problem of managing e-business service levels effectively for IT, application deployment and support personnel and e-business executives. To learn more about Keynote’s Competitive Intelligence Study, visit: http://www.keynote.com/solutions/slm_service_level_ rankings.html, or contact us at sales@keynote.com. Keynote Competitive Intelligence Study for VoIP Providers Wave 2 / Syndicated Research Abstract / November-December 2005 / © 2006 Keynote Systems, Inc.