QoS and QoE Measurement and Monitoring Goals Lab, pre-deployment, Monitoring, failure analysis Measurement Methods Objective, Subjective, payload/codec based Monitoring Levels Transport, Transaction, content Monitoring Domains Head-end, network, Home

QoS and QoE

Measurement and Monitoring Goals

Lab, pre-deployment, Monitoring, failure analysis

Measurement Methods

Objective, Subjective, payload/codec based

Monitoring Levels

Transport, Transaction, content

Monitoring Domains

Head-end, network, Home

Name Challenge Answer network Overcome Packet loss, jitter, network congestion Improved protocols QoS mapping schemes Bandwidth No bandwidth in the network and at the home increased capacity inside the home and the network level Monitoring Measure hundreds of video streams in Gb Eth & IP environment Video over IP monitoring, new protocols & devices CRM costs Lower MTBF, call center calls and “truck rolls” Automatic maintenance, Automatic firmware update Home network monitoring

What does subscribers want? Content Content quality Content pricing Content availability V iewing experience Video Quality Channel Zapping delay Integration of services Usability One bill New services Customer services Technical Factors

What does subscribers want?

Content

Content quality

Content pricing

Content availability

V iewing experience

Video Quality

Channel Zapping delay

Integration of services

Usability

One bill

New services

Customer services

We can look at quality in different ways: Perceptual subjective Quality (MOS) Numerical Quality parameters (PSNR) How to connect (relation matrix between 1 and 2)

We can look at quality in different ways:

Perceptual subjective Quality (MOS)

Numerical Quality parameters (PSNR)

How to connect (relation matrix between 1 and 2)

In the Lab & In The Fields For Pre-Deployment/monitoring/Failure Analysis

In the Lab & In The Fields

For Pre-Deployment/monitoring/Failure Analysis

Analysis Problem Solving Tuning Pre Deployment Testing Lab Testing Design 24/7 Monitoring Deployment Phase Pre-Deployment Phase

Design & Lab testing Simulation and Emulation of the network Lab and testing tools Pre Deployment Stage Work on actual network Load testing Lab, testing, diagnosing and monitoring tools Deployment (production) Phase Mostly monitoring (probes) equipment, management systems, data filtering and diagnostics equipment

Design & Lab testing

Simulation and Emulation of the network

Lab and testing tools

Pre Deployment Stage

Work on actual network

Load testing

Lab, testing, diagnosing and monitoring tools

Deployment (production) Phase

Mostly monitoring (probes) equipment, management systems, data filtering and diagnostics equipment

Subjective Objective Payload based, codec aware, codec anaware

Subjective

Objective

Payload based, codec aware, codec anaware

S ubjective Accurate Expensive, not for monitoring Objective Repeatable For both testing and monitoring

S ubjective

Accurate

Expensive, not for monitoring

Objective

Repeatable

For both testing and monitoring

Objective Subjective MOS (Voice) BT500 (Video) Codec independent Packet based MDI Monitoring Testing Broadcast World HSI and Data World Network Monitoring Delay, Jitter Packet loss J.144 PSNR Payload Full Reference Reduced Reference No Reference Codec aware Packet based VQI V-Factor VQS Telchemy

Codec aware Packet based Payload Network Monitoring Codec independent Packet based Objective

J.144 PSNR Payload Full Reference Reduced Reference No Reference

ITU-T J.144 and ITU-R BT.1683 Full-reference perceptual models Digital TV Rec. 601 image resolution (PAL/NTSC) Bit rates: 768 kbps ~ 5 Mbps Compression errors

ITU-T J.144 and ITU-R BT.1683

Full-reference perceptual models

Digital TV

Rec. 601 image resolution (PAL/NTSC)

Bit rates: 768 kbps ~ 5 Mbps

Compression errors

ITU-T P.862 (Feb 2001) - Full Reference Full-reference perceptual model (PESQ) Signal-based measurement Narrow-band telephony and speech codecs P.862.1 provides output mapping for prediction on MOS scale ITU-T P.563 (May 2004) No-reference perceptual model Signal-based measurement Narrow-band telephony applications

ITU-T P.862 (Feb 2001) - Full Reference

Full-reference perceptual model (PESQ)

Signal-based measurement

Narrow-band telephony and speech codecs

P.862.1 provides output mapping for prediction on MOS scale

ITU-T P.563 (May 2004)

No-reference perceptual model

Signal-based measurement

Narrow-band telephony applications

ITU-T P.862.2 (Nov 2005): Extension of ITU-T P.862 Wide-band telephony and speech codecs (5 ~7Khz) ITU-T P.VTQ (on-going): Targeted at VoIP applications Minimum performance framework for no-reference packet-based measurement Models analyze packet statistics; speech payload is assumed Uses P.862 as a measurement reference

ITU-T P.862.2 (Nov 2005):

Extension of ITU-T P.862

Wide-band telephony and speech codecs (5 ~7Khz)

ITU-T P.VTQ (on-going):

Targeted at VoIP applications

Minimum performance framework for no-reference packet-based measurement

Models analyze packet statistics; speech payload is assumed

Uses P.862 as a measurement reference

Codec aware Packet based VQI V-Factor VQS Telchemy

Monitoring technique Codec dependent Incorporates network parameters data with codec behavior data Scales- could monitor thousands of channels Examples: VQS (Telchemy) VQI (Brix) V-Factor (QoSMetrics) The need a codec aware metrics Problem area Robust codec “ Raw” codec

Monitoring technique

Codec dependent

Incorporates network parameters data with codec behavior data

Scales- could monitor thousands of channels

Examples:

VQS (Telchemy)

VQI (Brix)

V-Factor (QoSMetrics)

Packet loss/discard typically occurs in high density periods Average can be misleading Impact of Burst of Packet Loss Base quality level depends on frame rate, codec type, bit rate Poor quality during burst of loss/discards 5-8 seconds 15-30 seconds Subjective compensation for variance between human and testing equipment view of loss

Based on MPQM (Moving Picture Quality Metrics) – high quality video measurement standard V = f(QER, PLR, R) QER – relative video codec quality PLR – Packet loss ratio (based on actual packet loss, jitter data and jitter buffer model) R – Image complexity factor (2-3) Adopted by Spirnet

Based on MPQM (Moving Picture Quality Metrics) – high quality video measurement standard

V = f(QER, PLR, R)

QER – relative video codec quality

PLR – Packet loss ratio (based on actual packet loss, jitter data and jitter buffer model)

R – Image complexity factor (2-3)

Adopted by Spirnet

Monitoring only Codec independent Based on network parameters data only Scales - could monitor thousands of channels Examples: MDI IneoQuest standardized by IETF

Monitoring only

Codec independent

Based on network parameters data only

Scales - could monitor thousands of channels

Examples:

MDI

IneoQuest

standardized by IETF

Transport Level Service (transaction) Level Media Quality Level Add Your Text Add Your Text Add Your Text Add Your Text Content Quality Video Stream Quality Transport Quality Transaction Quality

Transport Level

Service (transaction) Level

Media Quality Level

Example: Packet Loss Loss Patterns Jitter Delay Well understood Defined by ITU and IETF

Example:

Packet Loss

Loss Patterns

Jitter

Delay

Well understood

Defined by ITU and IETF

Examples Post-dial delay in PSTN/mobile networks Video start time Requires understanding in both network monitoring and signaling (IGMP, SIP) and in media coding (analysis of the media to discover dial tone or I frame)

Examples

Post-dial delay in PSTN/mobile networks

Video start time

Requires understanding in both network monitoring and signaling (IGMP, SIP) and in media coding (analysis of the media to discover dial tone or I frame)

Multicast saves bandwidth but creates signaling delays: Multicast Leave + Multicast Join + First I Frame + Up to 2 seconds buffering time Leave latency Join latency Signaling Latency First I Frame Media Latency Total Channel zapping Latency Buffering latency First frame viewed

Multicast saves bandwidth but creates signaling delays:

Multicast Leave +

Multicast Join +

First I Frame +

Up to 2 seconds buffering time

Unicast server is used to serve the stream till the second multicast stream has an “I Frame” Immediate I Frame S horter buffering time Leave latency Join latency Multicast stream Total Channel zapping Latency I Frame unicast unicast stream I Frame Multicast Shorter buffering time First frame viewed

Unicast server is used to serve the stream till the second multicast stream has an “I Frame”

Immediate I Frame

S horter buffering time

Unicast server is used to serve the stream till the second multicast stream has an “I Frame” Immediate I Frame Shorter buffering time Play to server Total VoD Signaling Latency Buffering latency First frame viewed

Unicast server is used to serve the stream till the second multicast stream has an “I Frame”

Immediate I Frame

Shorter buffering time

Content quality is a payload based measurement. Requires decoding of the video stream Understanding of the buffering and error concealment algorithms of the decoder CPU intensive – Does not scale Accurate Used mostly is Lab equipment and diagnostic equipment Examples: PSNR ITU-T J.144 Usually requires the reference (original) stream Tests: Source artifacts Source quality

Content quality is a payload based measurement.

Requires decoding of the video stream

Understanding of the buffering and error concealment algorithms of the decoder

CPU intensive – Does not scale

Accurate

Used mostly is Lab equipment and diagnostic equipment

Examples:

PSNR

ITU-T J.144

Usually requires the reference (original) stream

Tests:

Source artifacts

Source quality

Used in Diagnostics / Lab Used for Monitoring DSL DSL Forum TR-64, TR-69 LAN and WAN monitoring standards ITU Study Group 12 Algorithms for end-to-end transmission performance ITU VQEG – Video Quality Expert Group Video performance measurement based on Subjective tests Database ATIS IIF – IPTV Interoperability Forum QoS Metrics Standardization

VSTQ - Video Service Transmission Quality Transmission Quality - codec/ content independent Based on the rate and distribution of effective packet loss and discard VSPQ - Video Service Picture Quality Estimated viewing quality Considers the impact of VSTQ, video codec type and rate, resolution VSAQ - Video Service Audio Quality audio listening quality Considers the impact of VSTQ, audio codec type, sample rate, ….. VSMQ - Video Service Multimedia Quality overall user experience Combined effect of VSPQ, VSAQ, audio-video synchronization.. VSCQ - Video Service Control (Plane) Quality Considers responsiveness and reliability of control plane (trick play)

VSTQ - Video Service Transmission Quality

Transmission Quality - codec/ content independent

Based on the rate and distribution of effective packet loss and discard

VSPQ - Video Service Picture Quality

Estimated viewing quality

Considers the impact of VSTQ, video codec type and rate, resolution

VSAQ - Video Service Audio Quality

audio listening quality

Considers the impact of VSTQ, audio codec type, sample rate, …..

VSMQ - Video Service Multimedia Quality

overall user experience

Combined effect of VSPQ, VSAQ, audio-video synchronization..

VSCQ - Video Service Control (Plane) Quality

Considers responsiveness and reliability of control plane (trick play)

J.144 and PSNR examines the video content only (payload measurements) TR101290 examines only transport stream data and coherence without examining the video content V - Factor and VQS looks at packet loss, jitter and loss patterns data and incorporate it with codec information and video header information MDI – Examines only packet loss and packet loss patterns without considering the codec or video information TR101290 MPEG2TS Headers V-Factor, VQS MDI J.144, PSNR Video payload

J.144 and PSNR examines the video content only (payload measurements)

TR101290 examines only transport stream data and coherence without examining the video content

V - Factor and VQS looks at packet loss, jitter and loss patterns data and incorporate it with codec information and video header information

MDI – Examines only packet loss and packet loss patterns without considering the codec or video information

Headend Network VoD Home

Headend

Network

VoD

Home

DVB Transport stream monitoring standard Alerts and KPI: TS sync loss Sync byte error PAT error Continuity counter errors PMT error PID error Transport error CRC error PCR error PCR discontinuity PCR accuracy error PTS error

DVB Transport stream monitoring standard

Alerts and KPI:

TS sync loss

Sync byte error

PAT error

Continuity counter errors

PMT error

PID error

Transport error

CRC error

PCR error

PCR discontinuity

PCR accuracy error

PTS error

Head-end VoD Servers Encoders TS muxers Network core Routers Network edge ATM switches GBE Switches Access DSLAM / BRAS CPE STBs Home network Network Probe interface GBE, OC3/12, STM1/4 Raw Video input MPEG2 TS - DVB ASI, DVBoIP Digital and analog video Source: Brix-Video Quality Measurement Algorithms WP

Head-end

VoD Servers

Encoders

TS muxers

Network core

Routers

Network edge

ATM switches

GBE Switches

Access

DSLAM / BRAS

CPE

STBs

Home network

Network Probe interface

GBE, OC3/12, STM1/4

Raw Video input

MPEG2 TS - DVB ASI, DVBoIP

Digital and analog video

Video source quality Video encoding quality MPEG2 Transport stream parameters TR100290 1 st -3 rd level priority alerts Inputs Raw analog video in a variety of forms (SD/HD) DVB-ASI DVBoIP DVBoATM (AAL1, AAL5)

Video source quality

Video encoding quality

MPEG2 Transport stream parameters

TR100290

1 st -3 rd level priority alerts

Inputs

Raw analog video in a variety of forms (SD/HD)

DVB-ASI

DVBoIP

DVBoATM (AAL1, AAL5)

Pre-deployment RTSP signaling latency (delay) Performance under stress (user load) Trick Mode (FF, REW) Signaling delay Video quality Buffer state (overflow and underflow) Monitoring Loss of data (under stress) Interfaces: GBE 10GBE

Pre-deployment

RTSP signaling latency (delay)

Performance under stress (user load)

Trick Mode (FF, REW)

Signaling delay

Video quality

Buffer state (overflow and underflow)

Monitoring

Loss of data (under stress)

Interfaces:

GBE

10GBE

Scalability – Millions of STBs Cost – Monitoring hardware and/or software costs must not add to STB costs (multiply by number of STBs) Data – Large amounts of data which needs RT processing, filtering, logging and analysis Unclear network structure Non “standard” equipment I nstallation, support and debugging problems

Scalability – Millions of STBs

Cost – Monitoring hardware and/or software costs must not add to STB costs (multiply by number of STBs)

Data – Large amounts of data which needs RT processing, filtering, logging and analysis

Unclear network structure

Non “standard” equipment

I nstallation, support and debugging problems

Home network monitoring is currently a minor feature in IPTV “Support Automation” Support Automation handles: Installation Preventive maintenance STB and CPE remote management (TR69 and more) Service verification for field technician CRM support Interaction logging Home Network monitoring and repair

Home network monitoring is currently a minor feature in IPTV “Support Automation”

Support Automation handles:

Installation

Preventive maintenance

STB and CPE remote management (TR69 and more)

Service verification for field technician

CRM support

Interaction logging

Home Network monitoring and repair

2Wire solution -Three data views OSS /BSS Customer care Data Analysis Other companies in this field SupportSoft Enure Motive

2Wire solution -Three data views

OSS /BSS

Customer care

Data Analysis

Other companies in this field

SupportSoft

Enure

Motive

Lab Testing Monitoring Incumbent Telecom Agilent Ixia Spirnet Cable Rhode & Schwartz Telecom Agilent JDSU/ Acterna Spirnet Cable Thomson Snell&Wilcox New entrants Telecom Shenick Telecom Telchemy Bridge Technologies iBrix QoSMetrix