Agenda

• Spirent overview
• Key implementation issues
• What is Triple Play / Converged networks?
• Specifics on testing SIP
• Network Impairments and Parameters that Voice
and Video Affect Quality
• Metrics for Measuring Voice and Video Quality and
Performance
• Good test methodology


Spirent Communications

• Spirent is the test solution leader
– 1,800 employees in 14 countries
– More than 1,500 customers
– Sales and service capabilities in
30 countries


Implementation steps - Lab
Services
Deployment

Performance
Analysis

Industry Manufacturing
standards Quality Assurance

• Characterize system before
Functionality &
Performance
Testing
trial
Conformance Testing
• Validate system scalability
• Identify capacity limits
• Measure call performance
• Automate regression testing


Implementation steps - Network
Service
Assurance

Initial Deployment
Network Certification
Widespread
Network
Pilot Networks
Deployment

• Facilitate vendor selection
• Identify performance ceilings
• Enable accurate capacity planning
• End-to-end service assurance testing
• Improve operational performance
• Improve customer satisfaction

Key Implementation Issues

• Circuit to packet migration
• Scalability and Performance
• Voice quality
• Interoperability and conformance
• Budget pressures


Before you deploy!

• Network Equipment Manufacturers (Chips,
IP-PBX, Gateways, MSs & SSs)

– Characterize your system before trial
– Validate system scalability
– Identify capacity limits
– Measure call performance

• Service Providers
(NSPs, SPs, ITSPs)

– Define criteria for vendor selection
– Identify performance ceilings
– Accurately plan for your capacity needs
– End-to-end service assurance testing


Data Transmission
“Non-Real-Time” Applications

Telenet
Web Name
HTTP Resolving
DNS

Email and
Messaging
POP
Data Base SMPT
MS SQL Exchange
Oracle

File Transfer
FTP Music Downloading
Home control

Data Examples: Internet access, Email, File Transfer, Portals, Database Applications,
Gaming, Government Services, Online Commerce


Voice and Video
“Real-Time Applications”

IP Music/Audio/Radio Gaming
VoD VoIP, IP Telephony, Video Telephony Single / Multiplayer
G.711, G.729, G.728, G.726, G.723
H.261, H.263,
SIP, SIP-T, H323, Skinny, MGCP,
MEGACO/H.248

Real-Time Online
Communications
Instant Messenger
Webex IPTV Services
Multi-Media Netmeeting Broadcast, On-Demand,
RTP SIP Bi-directional / Interactive
H.264,Microsoft AVI, QuickTime (.mov) H.323 MPEG1, MPEG2,
Windows Media (.wmv, .asf), RealMedia (.rm), MPEG4, VC1, H264

Voice Applications: Phone service integrated with video
Video Applications: Broadcast TV, video on demand, distance learning

Converged Triple Play: Data, Voice and Video
With Network Impairments
Data
Video

Dialed
On
Ring Configure
Configure
Notify
Digits#
Hook
Off Hook
Dial
Good
Back
Hello
Bye

Connect Voice Good
On
Off Hook
Connect Ring Conversation
Disconnect
Hello
Bye
Hook

Impairments
Signaling Path can be heard in the voice conversation

Testing SIP Conformance

• Comprehensive and scriptable SIP call flows
• Complete configurable SIP signaling messages
• SIP protocol analysis
• Simplified flow diagrams with visual analysis
• Comprehensive conformance test suites


Testing SIP Conformance

• ETSI TS 102-027-1 v2.12,Tiphon:
– RFC 3261 user agent, proxy and redirect server
compliance
• Graphical SDL and TTCN tools
– Create, edit, compile and execute simulation scripts
and conformance tests
• Additional SIP messages beyond RFC 3261
– Included in torture tests
• Additional tests as defined by the SIP Forum


Testing with Configurable SIP

Configurable SIP call setup and call teardown
• Configurable call flows and messages
• Incoming message filter
– Adaptive signaling syntax for SIP
– Improves interoperability with new drafts and
non-conformant proprietary implementations


Testing with Configurable SIP

• Configurable messages:
 Invite, ACK, bye, register
 Responses: 1xx, 2xx, 3xx, 4xx, 5xx, 6xx
• Configurable timers, message intervals
• Enable and disable optional messages:
 Re-invite, cancel, options, message, info, notify, subscribe,
unsubscribe, update, refer, Prack
 Fix erroneous incoming messages “on the fly” with the
“search and replace” method
 Allows interoperability with SIP devices (including drafts,
non conformant, prototype)


SIP Message Registration Screen
Shots


SIP Message Origination


SIP Message Termination


Incoming Message Filters


TOS for SIP Signaling


Diffserv for SIP Signaling


Testing SIP Robustness

• Robustness testing
– Passed: does not crash, stable, or acceptable results
– Failed: crashes, unstable, or unacceptable results
• Security testing
– It is crucial to identify SIP security holes
• SIP testing tool
– Tests SIP robustness and security
– Comprehensive negative test suites for SIP


Real Signaling with real RTP

• Capability to do signaling with audio
• Capability to perform real time measurements
• Capability of using signaling without audio
• Problems of not using real signaling
• Problems of not using real RTP streams
• Real time objective metrics


Testing SIP-T
VoIP Network
MGC MGC
SIP-T

SS7 SS7
SIP-T
H.248/Megaco SIP-T H.248/Megaco
PSTN SIP Proxy PSTN
RTP/RTCP
Trunking
Trunking Gateway
SS7 Gateway
GR303
POTS 1000Base-SX/LX POTS
ISDN
10/100/1000Base-T
CAS
V5


SIP-T Performance Testing Suites

Performance testing
– Validate and stress-test SS7 ISUP and SIP
interworking with optional media, over thousands of
emulated user agents
• SIT-T testing
– Configurable SIP-T calls with intelligent protocols
• QoS and CoS testing
– Optional TOS/Diffserv and VLAN options in SIP-T
media calls, used to measure QoS with PESQ and e-
model
• Feature testing
– Automated and configurable SIP call set-up,
teardown, flows, messages


Network Impairments and Parameters
that Affect Voice and Video Quality

• Network Architecture • Timing Drift
• Types of Access Links • Route Flapping
• QoS controlled Edge Routing • Signaling protocol mismatches
• MTU Size • Network faults
• Packet Loss (Frame Loss) • Link Failures
• Out of order packets • Voice Only Impairments
• One Way Delay (Latency) – Echo

• Variable Delays (Jitter)
– Voice coding algorithms
– A/D and D/A Conversion
• Background Traffic
(Congestion, Bandwidth, – Noise – Circuit and External
Utilization, Network Load,
Load Sharing) • Video Only Impairments
– Video coding algorithms
– Fixed vs Variable Frame Rate


IP Network Architecture

Local Access Local Access
Source A B Destination
Device A
LAN A Core IP Network LAN B Device B
64 kbit/s 64 kbit/s
*128 kbit/s *128 kbit/s
256 kbit/s 256 kbit/s
1000BaseX *384 kbit/s *384 kbit/s 1000BaseX
* 100BaseT Switch 512 kbit/s Route flapping 512 kbit/s * 100BaseT Switch
100BaseT Hub *768 kbit/s One-way delay *768 kbit/s 100BaseT Hub
10BaseT *T1 (1.536 kbit/s) Jitter *T1 (1.536 kbit/s) 10BaseT
* WLAN (~4 Mbit/s) E1 (1.920 kbit/s) Packet loss E1 (1.920 kbit/s) * WLAN (~4 Mbit/s)
---------------------- E3 (34 Mbit/s) E3 (34 Mbit/s) ----------------------
Occupancy level *T3 (44 Mbit/s) *T3 (44 Mbit/s) Occupancy level
Packet loss ADSL (~256 kbit/s) ADSL (~2 Mbit/s) Packet loss
*Cable (~256 kbit/s) *Cable (~3 Mbit/s)
Fiber (1-10 Gbit/s) Fiber (1-10 Gbit/s)
-------------------- --------------------
Occupancy level Occupancy level
QoS edge router QoS edge router

* Case used in impairment tables

Affects
Data, Voice and Video Quality

Network Operating With Constant Delay

Affects
Voice and Video Quality


End to End Delay Sources
Core
Originating Network Terminating
LAN LAN
Originating Edge Edge Terminating
Gateway Router Core Router Gateway
Network
Routers

Fixed Fixed
• Serialization Fixed • Serialization
Fixed WAN WAN
Fixed • Switching Fixed Fixed
• Look ahead
• Switching •Propagation • Switching • Decoding
• Encoding
•Serialization
• Buffer Variable Variable Variable
• VAD
• Voice contention
Variable • Voice contention • De-jitter buffer
• Packetizing • Voice contention
• Data Contention • Data Contention • Packet loss
• Video Contention • Data Contention • Video Contention
Concealment
•Video Contention

• Algorithmic delay
Affects • Serialization delay
• Propagation delay
• Component delay

Echo Impairment on Converged network
Delay in IP Network
makes Echo sound worse Tail Circuit
TELR
IP
Phone MG
T1
Link 2 Wire
IP Network V

PBX POTS
Phone
ERLE ERL
IP
Phone Echo Canceller in MG
reduces Echo Level

ERLE – Echo Return Loss Enhancement Analog Analog
ERL – Echo Return Loss 4-Wire Link 2-Wire Link
TELR – Talker Echo Loudness Rating
RX RX

T1 Link E&M
Affects TX
TX Hybrid POTS
Voice Quality
Transformer Phone
Impedance
PBX Mismatch


Echo Impairment on Converged network
Electrical Coupling
Affects • Impedance Mismatch (Hybrid)
Voice Quality Acoustical Coupling
• Speakerphone

Converged
Network

Path A to B
Path B to A

Echo Path Side A B (250ms)
Echo Path Side (250ms)
Echo is caused by impedance mismatches in hybrid circuits (2w to
4w) and feedback between the telephone mouth piece and ear piece

Effect of Delay on Voice Quality
Voice Quality

> 25ms Echo Cancellation Required

<150 ms (with echo
cancellation): acceptable

150-400 ms:
acceptable if
delay expected

> 400 ms
unacceptable for
most applications


Effect of Echo Level on Voice Quality

Less Echo Less Echo

More Echo More Echo

Affects
TELR – Talker Echo Loudness Rating
(Signal to Echo Ratio) Voice Quality


Network with Variable Delays (Jitter)
• Variable processing delay
– A busy router or switch will take longer to look up the routing (address)
table
• Queuing delay
– Network congestion Affects
Delay (ms)

Time (s)

Jitter Characteristics
Delay (ms) Delay (ms)

Good

Bad
Delay (ms)

Severe

Time (s)

Affects


Packet Loss
Example: Queue Management

Threshold

Affects
Bit Bucket

RED (Random Early Discard)


Speech Compression Impairment

Voice Quality

G.711
Best Quality

Common Compression Types: G.711, G.729, G.728, G.726, G.723, AMR, EVRC


VAD – Voice Activity Detection
Timing may be different

No VAD
Affects
Voice Quality

VAD

Data is intentionally not sent during times of Silence


Impact Of Packet Size
Affects
Data, Voice and Video Quality
10 Bytes
= 10 ms Speech
20 Bytes Normal size for VoIP applications
= 20 ms Speech
40 Bytes
= 40 ms Speech
80 Bytes
= 80 ms Speech
• Typically Packets are kept small for best results
• Many equipment manufacturers use dynamic packet size
to optimize for network conditions

Mechanisms for Assuring QOS
QoS Class Applications (Examples) Node Mechanisms Network Techniques
Data (Y.1541)
0 Real-Time, loss sensitive, Constrained Routing and
Well Jitter sensitive, high Strict QoS. Guaranteed no Distance
Managed interaction (VoIP, VTC, over subscription on links.
IPTV
1 Real-Time, Jitter sensitive, Separate Queue with Less constrained Routing
Best interactive (VoIP, VTC). preferential servicing, and Distances
Voice Effort Traffic grooming
2 Transaction Data, Highly Constrained Routing and
Interactive, (Signaling) Separate Queue, Drop Distance
3 Transaction Data, priority Less constrained Routing
Interactive and Distances
4 Low Loss Only (Short Long Queue, Drop priority Any route/path
Video Transactions, Bulk
Data, Video Streaming)
5 Traditional Applications of Separate Queue (lowest Any route/path
Internet Default IP Networks priority)
Triple Play
• Class of Service (COS) ITU-T Y.1541 defines the 5 classes of service and their application
• Type of Services (TOS)
Affects
• TOS and COS are both elements with in an IP Packet Data, Voice and Video Quality
• DIFSER and RSVP provide mechanisms to improve QOS


TIA-921 and ITU-T G.NIMM
Test Profiles Based on QoS (Y.1541) Classes

Different test profiles for different Service Level Agreements (SLAs)

Impairment Type Units Range Impairment Type Units Range Impairment Type Units Range

Jitter ms +/- 50 Jitter ms +/- 75 Jitter ms 0 to +/-
250
One Way Latency ms 50 to 100 One Way Latency ms 50 to 200
One Way Latency ms 50 to 400
Sequential Packet Loss #sequential Random Sequential Packet Loss #sequential 2 to 5
packets loss only packets Sequential Packet Loss #sequential 2 to 500
packets
Rate of Sequential Loss sec-1 Rate of Sequential Loss sec-1 0 to 2
Rate of Sequential Loss sec-1 < 10-1
Random Packet Loss % 0 to 0.05 Random Packet Loss % 0 to 2 Random Packet Loss % 0 to 20
Out of Sequence Packets % 0 to 0.001 Out of Sequence Packets % 0 to 0.1 Out of Sequence Packets % 0 to 20

Profile A Profile B Profile C
Well Managed Network Best Effort Managed Network Un-Managed Network
Table 2 Table 3 Table 4


Early Voice Quality Testing


Voice Quality Testing

• Active (Intrusive) Testing
– Sends, Receives and compares Wave Files to measure voice quality
– MOS (Mean Opinion Score)
– PSQM, PSQM+ (Perceptual Speech Quality Measurement)
– PESQ (Perceptual Evaluation of Speech Quality)
– R-Value and J-MOS derived from PESQ
• Passive Testing
– R-Value – ITU-T P.VTQ
• Measures Voice Quality on RTP Packets
• Based on E-model
• Japan – J-MOS
• Similar Techniques can be used to measure Video Quality
– P.563 (ITU-T recommendation) 3SQM, P-Stream
• Measures Voice Quality of Voice traffic based on Audio Siginal
• Provides an estimate of PSQM


Active (Intrusive)
Voice Quality Testing
MOS, PSQM, PSQM+, PESQ, R-Factor (PESQ Derived)

Send Wave Files
DUT Receive Wave Files
Example: (ITU-T Female
Nice File with Pilot Tone)
Measures Voice Quality by Comparing Sent and Received Wave files

Sent (Green)
and Received
(Orange)
wave files

Expanded Sent
(Green) and
Received
(Orange) wave PESQ Score vs Number of PESQ Measurements
files
Values are different for Male, Female, different
Wave Files and different Languages

Passive Voice and Video Quality Testing
R-Factor/Emodel Measure:
Video Quality
MOS-LQ
MOS-CQ
MOS-PQ
J-MOS
Network R
User R
Burst statistics
Diagnostic data
IP
PSTN Network
E1/T1/E3/T3 RTP
/PRI/GR303,
V5,SLC96 Trunking
Gateway RTP
Measure:
Video Quality
ITU-T P.VTQ MOS-LQ
MOS-CQ
MOS-PQ
J-MOS
Network R
User R IP Telephone
Burst statistics
Diagnostic data

Passive Voice Quality Testing
P.563 (P-Stream, 3SQM)

RTP

DUT Receive Audio

Estimates Voice Quality based on 3
Characteristic of Received Audio


Voice Quality Measurements
Emodel P.861 P.862
PAMS PSQM/PSQM+ PESQ
5 0 4.5

~3.88

~3.65

~3.40

~3.13

~2.84

1 6.5 -0.5


Sample Voice Quality Test Results

MOS

PSQM

PESQ

Created by inducing
G.711 packets lost
G.723.1 (6300 bps)

Comparison of Scores for G.711 and G.723.1 (6300 bps)

Video Quality Measurement
• Video Compression
– Video Compress schemes affect the video quality
– H.261, H.263, H.264, VC1, MPEG-1, MPEG-2, MPEG-4,
Microsoft AVI, Windows Media (.wmv, .asf), RealMedia (.rm),
QuickTime (.mov)
• Interactive real-time applications (e.g., video
conferencing, voice over IP) are sensitive to latency and
Frame Rate
• Typical Video Quality Metrics
– Objective MOS – SNR
– Blockiness – Edge Noise
– Blur – Jerkiness
– PSNR – Error Blocks
– Spatial Resolution – Object Retention
– Temporal Resolution – Color Reproduction
Accuracy

Blockiness

Blockiness Original


Reference and Blocky Video

Blocky Original
(Reference)


Blur

Blur Original


Noise

Noise Original


Spatial (Pixel) Resolution

128X128 32X32 8X8

Spatial Resolution

Department of Computer Science
University of Canterbury
http://www.cosc.canterbury.ac.nz/people/mukundan/covn/Imgresl.htm

Temporal (Motion) Resolution
Vertical-Width (v)
Horizontal-
Width (h)

Temporal-Width (t)

Fk Fk+1 Fk+2 Fk+3 Fk+4 Fk+5

Video Frames


Models
• Video Quality Metrics (VQM)
– ITU-T SG9 and VQEG are working on standard
• TV Model - optimized for higher bit-rate digital television
systems with no frame dropping (e.g., MPEG-2)
• Videoconferencing Model - optimized for lower bit-rate
videoconferencing systems that drop frames (e.g., H.261,
H.263).
• General Model - optimized for a wide range of video
quality (videoconferencing, TV)
• Developer Model - optimized for a wide range of video
quality (videoconferencing, TV) with the added constraint
of fast computation.
• PSNR Model - based on the traditional peak signal-to-
noise-ratio (PSNR) calculation.

Techniques
• Full Reference (ITU-T J.144R and BT.1683)
– Video quality is calculated by comparing the received video with
the complete original video
• Reduced Reference (ANSI T1.801.03-2003 and ITU-T
J.143)
– Spatial and temporal information are calculated from original
video and transmitted to the receiving end
– Video quality is calculated by comparing the received video with
the reduced reference
• No Reference – Passive
– Video quality on based only on the received information (picture
content)
– Video quality is derived from RTP packet information similar to R-
Factor (E-Model) for voice quality


Full Reference

Original Video Received Video

Transmitted Video Signal
Processor Processor

Original Video

Video Quality Score
•MOS
Full Reference
•Blockiness
• Blur
• PSNR


Reduced Reference

Transmitted Video Received Video

Processor Processor

Reduced-Reference Signal
Low Rate

Video Quality Score
•MOS

Reduced-Reference •Blockiness
•Spatial (Pixel) and Temporal (Motion) Information • Blur
• PSNR


No-Reference

Transmitted Video Received Video

Processor Processor

No-Reference
Picture Content
or Video Quality Score
Passive monitoring of RTP •MOS
•Blockiness
• Blur
• PSNR


Types of Testing
Type of tests
Test these DUTs
• Video Quality Payload types
• IP PBX
• Voice quality • Video
• Gateways
• Functional • Voice
• IP Phone
• Scalability • Data
• Servers
• Troubleshooting • Fax
• Firewalls
• Conformance • Modem
• IAD
• Interoperability
• Triple Play (Data, With these interfaces
Voice and Video) • GigE 1000Base-SX,
With these protocols 1000Base-LX

• IP • PSTN • 10/100/1000Base-T

– SIP – CAS • Analog
– H.323 – PRI • T1/E1
– MGCP – SS7 • T3/E3
– Megaco/H.248 – NFAS
– Skinny – V5
Analyze Assure Accelerate – GR303

Types of Testing
•Call Establishment •Speech Quality Measurements
–Start Dial Signal Delay –PESQ
–Post Dial Delay –PSQM
–Call Duration –MOS
–Ring Duration –R Factor
•Call Disconnect –Echo Delay
–Connection Disconnect Delay –Round Trip Delay
–Release on Request –Echo Return Loss
•Call Statistics –Signal Pass Noise
–Connection set-up failures –Noise Level
–Connection premature disconnect •Video Quality Measurements
–Call completion percentages –MOS
–Blockiness
•Transport Layer Measurements
– Blur
–One-way Transmission Time – PSNR
–Roundtrip Transmission Time
–Jitter
–Packets out of order
–Packet Loss

Video Telephony Testing
Distributed Testing • Isolate Network Problems

Poor
Call Quality Summary • Results Over Time
2% Bad
Fair 1%
Call Quality Summary
24%

Nearly All Users
Dissatisfied Call Completion Rate by Day Fair
Not Recommended
Good • Results by Group
2% 1%
Many Users Poor
Dissatisfied Bad
10% Very Satisfied
One Way Delay by Call Group
20%
Some Users
Dissatisfied Good
180
14%
100 73%
Packet Loss by Call Group
160
95
% Complete

140
90 1.40
120
85 Satisfied
53%
Delay (ms)

80 100 1.20 Call Setup Time by Call Group
% Complete
75 80
Mon Tue Wed Thu Fri Sat Sun
1.00
% Complete 98 97 99 99 96 99 99 Jitter by Call Group
60 350

40 0.80
% Loss

300 80
20
0.60 Jitter by Hour
0
250 70 CHI-to-DAL
Call Setup Time (ms)

NYC - DAL NYC - CHI NYC - SJ DAL - CHI DAL - SJ CHI - SJ
Delay (ms) 122 98 106 173 132 112
0.40 200 60

150 Call Quality Summary by Hour
0.20 50
CHI-to-DAL
Jitter (ms)

100 40 100
0.00
80
% Loss 0.10 0.43
50 0.40 1.23 0.35 0.64
30
Jitter (ms)

60
0 5.00
20
NYC - DAL 40
NYC - CHI NYC - SJ DAL - CHI DAL - SJ CHI - SJ
Call Setup Time (ms) 150 175 130 313 110 105
20 4.00
10
Jitter (ms)
0
MOS

12 1 2 3 43.00 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11
0 A A A A A A A A A A A A P P P P P P P P P P P P
M M M M M M M M M M M M M M M M M M M M M M M M
Jitter (ms) 43 51 2.00 41 73 54 45
Jitter (ms) 77 69 67 65 68 68 69 70 73 75 76 93 92 100 82 83 81 80 79 79 79 79 79 78

1.00 MOS
12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11
AM AM AM AM AM AM AM AM AM AM AM AM PM PM PM PM PM PM PM PM PM PM PM PM

MOS 3.5 3.7 3.6 3.4 3.8 3.4 3.6 3.8 3.5 2.9 2.8 2.3 1.9 1.8 2.9 2.9 3.1 2.9 3.4 3.5 3.5 3.6 3.6 3.7


Good test methodology

Implementation, Validation & Observation
– Conformance testing
• IETF 3261 & new SIP RFC’s
– Stress testing
• Scriptable call flow
• Bulk signaling with real RTP
– Robustness testing
• SIPPING Torture Test
• PROTOS / ETSI TIPHON
– Visual protocol analysis
• Application & content decoding

Network Assurance and Testing During the Migration to VoIP