Video quality measurements can be performed using subjective, objective, and payload-based methods. Subjective methods involve human assessment while objective methods use measurement devices and are repeatable for testing and monitoring. Payload-based methods assess video quality by comparing the original and distorted video. Standardization bodies have defined various levels of measurement including transport, transaction, and content levels to analyze video quality from different perspectives.

1. Video Quality Measurements

2. Overview QoS and QoE Measurement Methods Objective, Subjective, payload/codec based Measurement and Monitoring Goals Lab, pre-deployment, Monitoring, failure analysis Monitoring Levels Transport, Transaction, content Monitoring Domains Head-end, network, Home

3. User Quality of Experience factors (QoE) What does subscribers want? Content Content quality Content pricing Content availability V iewing experience Video Quality Channel Zapping delay Integration of services Usability One bill New services Customer services Technical Factors

4. MEASUREMENT TAXONOMY Subjective Objective Payload based, codec aware, codec anaware

5. Measurement methods review S ubjective Human assessment of quality Expensive, not for monitoring Not repeatable Objective Measurement devices Repeatable For both testing and monitoring, failure analysis Objective Subjective Measurements

6. Multimedia monitoring methods Subjective measurement arte done by humans according to pre-defined protocols Voice – MOS Grade voice quality between 1-5 4+ - is very good quality Video – BT500 Subjective MOS (Voice) BT500 (Video) Measurements

7. Objective methods Objective methods are divided into: Payload based Packet based CODEC Aware CODEC Independent Network Monitoring Measurements

8. Payload based Measurements Payload base methods assess the video quality based on the video itself Reference based methods compares Original video quality to distorted video. Used mostly in lab equipment for codec performance analysis and comparison

9. Full Reference Methods PSNR Full Reference Objective Payload based MSE SSIM J.144 Full reference methods compares each frame of the original video to frame of the distorted video and provide distortion measurement MSE & PSNR are pixel based similarity measurements Most video quality measurements are done on Luma (Y)

10. PSNR & MSE PSNR and MSE is a pixel base video quality comparison tools

11. MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723 original Image PSNR/MSE Problems Quality degradation is not reflected by MSE/PSNR index

12. SSIM – “Solves” PSNR Problems SSIM is Structural Similarity index. Unlike PNR and MSE it does not compare images “pixel by pixel” but as small NxN “environments” are average variance and covariance the dynamic range of the pixel-values (typically )

13. ITU-T J.144 and ITU-R BT.1683 Full-reference perceptual models Digital TV Rec. 601 image resolution (PAL/NTSC) Bit rates: 768 kbps ~ 5 Mbps Compression errors Full Reference testing standards

14. Back to Objective measurement What happens when we don’t have the original (Reference video) or when we don’t have the processing power to do an extensive comparison? We could use network measurements and codec based degradation info to asses video quality Measurements

15. Packet – Codec Aware Monitoring technique Degrades video based on codec type by incorporating network parameters data with codec behavior data Scales- could monitor thousands of channels Examples: VQS (Telchemy) VQI (Brix) V-Factor (QoSMetrics) The need a codec aware metrics Problem area Robust codec “ Raw” codec

16. Codec Aware Methods Codec aware Packet based VQI V-Factor VQS Telchemy Objective methods

17. Example V - Factor Based on MPQM (Moving Picture Quality Metrics) – high quality video measurement standard V = f(QER, PLR, R) QER – relative video codec quality PLR – Packet loss ratio (based on actual packet loss, jitter data and jitter buffer model) R – Image complexity factor (2-3) Adopted by Spirnet

18. Packet – Codec Independent Monitoring only Codec independent Based on network parameters data only Scales - could monitor thousands of channels Examples: MDI IneoQuest standardized by IETF

19. MEASUREMENT & MONITORING In the Lab & In The Fields Pre-Deployment/monitoring/Failure Analysis

20. Measurement & Monitoring phases Analysis Problem Solving Tuning Pre Deployment Testing Lab Testing Design 24/7 Monitoring Deployment Phase Pre-Deployment Phase

21. Measurement & Monitoring phases Design & Lab testing Simulation and Emulation of the network Lab and testing tools Pre Deployment Stage Work on actual network Load testing Lab, testing, diagnosing and monitoring tools Deployment (production) Phase Mostly monitoring (probes) equipment, management systems, data filtering and diagnostics equipment

22. MONITORING LEVELS

23. Measurement Levels Transport Level Service (transaction) Level Media Quality Level Video Quality Transport Quality Transaction Quality

24. Transaction Level Examples Post-dial delay in PSTN/mobile networks Video start time for channel zapping & Video conf Requires understanding in both network monitoring and signaling (IGMP, SIP) and in media coding (analysis of the media to discover dial tone or I frame)

25. Channel zapping delay Multicast saves bandwidth but creates signaling delays: Multicast Leave + Multicast Join + First I Frame + Up to 2 seconds buffering time Leave latency Join latency Signaling Latency First I Frame Media Latency Total Channel zapping Latency Buffering latency First frame viewed

26. Transport Level Example: Packet Loss Loss Patterns Jitter Delay Well understood Defined by ITU and IETF

27. Content Level Content quality is a payload based measurement. Requires decoding of the video stream Understanding of the buffering and error concealment algorithms of the decoder CPU intensive – Does not scale Accurate Used mostly is Lab equipment and diagnostic equipment Examples: PSNR ITU-T J.144 Usually requires the reference (original) stream Tests: Source artifacts Source quality

28. Standardization landscape Used in Diagnostics / Lab Used for Monitoring DSL DSL Forum TR-64, TR-69 LAN and WAN monitoring standards ITU Study Group 12 Algorithms for end-to-end transmission performance ITU VQEG – Video Quality Expert Group Video performance measurement based on Subjective tests Database ATIS IIF – IPTV Interoperability Forum QoS Metrics Standardization

29. Example: ATIS IIF Quality Metrics VSTQ - Video Service Transmission Quality Transmission Quality - codec/ content independent Based on the rate and distribution of effective packet loss and discard VSPQ - Video Service Picture Quality Estimated viewing quality Considers the impact of VSTQ, video codec type and rate, resolution VSAQ - Video Service Audio Quality audio listening quality Considers the impact of VSTQ, audio codec type, sample rate, ….. VSMQ - Video Service Multimedia Quality overall user experience Combined effect of VSPQ, VSAQ, audio-video synchronization.. VSCQ - Video Service Control (Plane) Quality Considers responsiveness and reliability of control plane (trick play)

30. Monitoring levels J.144 and PSNR examines the video content only (payload measurements) TR101290 examines only transport stream data and coherence without examining the video content V - Factor and VQS looks at packet loss, jitter and loss patterns data and incorporate it with codec information and video header information MDI – Examines only packet loss and packet loss patterns without considering the codec or video information TR101290 MPEG2TS Headers V-Factor, VQS MDI J.144, PSNR Video payload