Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Video Quality Measurements


Published on

Review video quality measurement types and classification

Published in: Technology

Video Quality Measurements

  1. 1. Video Quality Measurements
  2. 2. Overview <ul><li>QoS and QoE </li></ul><ul><li>Measurement Methods </li></ul><ul><ul><li>Objective, Subjective, payload/codec based </li></ul></ul><ul><li>Measurement and Monitoring Goals </li></ul><ul><ul><li>Lab, pre-deployment, Monitoring, failure analysis </li></ul></ul><ul><li>Monitoring Levels </li></ul><ul><ul><li>Transport, Transaction, content </li></ul></ul><ul><li>Monitoring Domains </li></ul><ul><ul><li>Head-end, network, Home </li></ul></ul>
  3. 3. User Quality of Experience factors (QoE) <ul><li>What does subscribers want? </li></ul><ul><ul><li>Content </li></ul></ul><ul><ul><ul><li>Content quality </li></ul></ul></ul><ul><ul><ul><li>Content pricing </li></ul></ul></ul><ul><ul><ul><li>Content availability </li></ul></ul></ul><ul><ul><li>V iewing experience </li></ul></ul><ul><ul><ul><li>Video Quality </li></ul></ul></ul><ul><ul><ul><li>Channel Zapping delay </li></ul></ul></ul><ul><ul><li>Integration of services </li></ul></ul><ul><ul><ul><li>Usability </li></ul></ul></ul><ul><ul><ul><li>One bill </li></ul></ul></ul><ul><ul><ul><li>New services </li></ul></ul></ul><ul><ul><li>Customer services </li></ul></ul>Technical Factors
  4. 4. MEASUREMENT TAXONOMY <ul><li>Subjective </li></ul><ul><li>Objective </li></ul><ul><li>Payload based, codec aware, codec anaware </li></ul>
  5. 5. Measurement methods review <ul><li>S ubjective </li></ul><ul><ul><li>Human assessment of quality </li></ul></ul><ul><ul><li>Expensive, not for monitoring </li></ul></ul><ul><ul><li>Not repeatable </li></ul></ul><ul><li>Objective </li></ul><ul><ul><li>Measurement devices </li></ul></ul><ul><ul><li>Repeatable </li></ul></ul><ul><ul><li>For both testing and monitoring, failure analysis </li></ul></ul>Objective Subjective Measurements
  6. 6. Multimedia monitoring methods <ul><li>Subjective measurement arte done by humans according to pre-defined protocols </li></ul><ul><li>Voice – MOS </li></ul><ul><ul><li>Grade voice quality between 1-5 </li></ul></ul><ul><ul><li>4+ - is very good quality </li></ul></ul><ul><li>Video – BT500 </li></ul>Subjective MOS (Voice) BT500 (Video) Measurements
  7. 7. Objective methods <ul><li>Objective methods are divided into: </li></ul><ul><li>Payload based </li></ul><ul><li>Packet based </li></ul><ul><ul><li>CODEC Aware </li></ul></ul><ul><ul><li>CODEC Independent </li></ul></ul><ul><li>Network Monitoring </li></ul>Measurements
  8. 8. Payload based Measurements <ul><li>Payload base methods assess the video quality based on the video itself </li></ul><ul><li>Reference based methods compares Original video quality to distorted video. </li></ul><ul><li>Used mostly in lab equipment for codec performance analysis and comparison </li></ul>
  9. 9. Full Reference Methods PSNR Full Reference Objective Payload based MSE SSIM J.144 <ul><li>Full reference methods compares each frame of the original video to frame of the distorted video and provide distortion measurement </li></ul><ul><li>MSE & PSNR are pixel based similarity measurements </li></ul><ul><li>Most video quality measurements are done on Luma (Y) </li></ul>
  10. 10. PSNR & MSE <ul><li>PSNR and MSE is a pixel base video quality comparison tools </li></ul>
  11. 11. MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723 original Image PSNR/MSE Problems <ul><li>Quality degradation is not reflected by MSE/PSNR index </li></ul>
  12. 12. SSIM – “Solves” PSNR Problems <ul><li>SSIM is Structural Similarity index. Unlike PNR and MSE it does not compare images “pixel by pixel” but as small NxN “environments” </li></ul><ul><li>  are average variance and covariance </li></ul><ul><li>   the dynamic range of the pixel-values (typically ) </li></ul>
  13. 13. <ul><li>ITU-T J.144 and ITU-R BT.1683 </li></ul><ul><li>Full-reference perceptual models </li></ul><ul><li>Digital TV </li></ul><ul><li>Rec. 601 image resolution (PAL/NTSC) </li></ul><ul><li>Bit rates: 768 kbps ~ 5 Mbps </li></ul><ul><li>Compression errors </li></ul>Full Reference testing standards
  14. 14. Back to Objective measurement <ul><li>What happens when we don’t have the original (Reference video) or when we don’t have the processing power to do an extensive comparison? </li></ul><ul><li>We could use network measurements and codec based degradation info to asses video quality </li></ul>Measurements
  15. 15. Packet – Codec Aware <ul><li>Monitoring technique </li></ul><ul><li>Degrades video based on codec type by incorporating network parameters data with codec behavior data </li></ul><ul><li>Scales- could monitor thousands of channels </li></ul><ul><li>Examples: </li></ul><ul><ul><li>VQS (Telchemy) </li></ul></ul><ul><ul><li>VQI (Brix) </li></ul></ul><ul><ul><li>V-Factor (QoSMetrics) </li></ul></ul>The need a codec aware metrics Problem area Robust codec “ Raw” codec
  16. 16. Codec Aware Methods Codec aware Packet based VQI V-Factor VQS Telchemy Objective methods
  17. 17. Example V - Factor <ul><li>Based on MPQM (Moving Picture Quality Metrics) – high quality video measurement standard </li></ul><ul><li>V = f(QER, PLR, R) </li></ul><ul><ul><li>QER – relative video codec quality </li></ul></ul><ul><ul><li>PLR – Packet loss ratio (based on actual packet loss, jitter data and jitter buffer model) </li></ul></ul><ul><ul><li>R – Image complexity factor (2-3) </li></ul></ul><ul><li>Adopted by Spirnet </li></ul>
  18. 18. Packet – Codec Independent <ul><li>Monitoring only </li></ul><ul><li>Codec independent </li></ul><ul><li>Based on network parameters data only </li></ul><ul><li>Scales - could monitor thousands of channels </li></ul><ul><li>Examples: </li></ul><ul><ul><li>MDI </li></ul></ul><ul><ul><ul><li>IneoQuest </li></ul></ul></ul><ul><ul><ul><li>standardized by IETF </li></ul></ul></ul>
  19. 19. MEASUREMENT & MONITORING <ul><li>In the Lab & In The Fields </li></ul><ul><li>Pre-Deployment/monitoring/Failure Analysis </li></ul>
  20. 20. Measurement & Monitoring phases Analysis Problem Solving Tuning Pre Deployment Testing Lab Testing Design 24/7 Monitoring Deployment Phase Pre-Deployment Phase
  21. 21. Measurement & Monitoring phases <ul><li>Design & Lab testing </li></ul><ul><ul><li>Simulation and Emulation of the network </li></ul></ul><ul><ul><li>Lab and testing tools </li></ul></ul><ul><li>Pre Deployment Stage </li></ul><ul><ul><li>Work on actual network </li></ul></ul><ul><ul><li>Load testing </li></ul></ul><ul><ul><li>Lab, testing, diagnosing and monitoring tools </li></ul></ul><ul><li>Deployment (production) Phase </li></ul><ul><ul><li>Mostly monitoring (probes) equipment, management systems, data filtering and diagnostics equipment </li></ul></ul>
  23. 23. Measurement Levels <ul><li>Transport Level </li></ul><ul><li>Service (transaction) Level </li></ul><ul><li>Media Quality Level </li></ul>Video Quality Transport Quality Transaction Quality
  24. 24. Transaction Level <ul><li>Examples </li></ul><ul><ul><li>Post-dial delay in PSTN/mobile networks </li></ul></ul><ul><ul><li>Video start time for channel zapping & Video conf </li></ul></ul><ul><li>Requires understanding in both network monitoring and signaling (IGMP, SIP) and in media coding (analysis of the media to discover dial tone or I frame) </li></ul>
  25. 25. Channel zapping delay <ul><li>Multicast saves bandwidth but creates signaling delays: </li></ul><ul><ul><li>Multicast Leave + </li></ul></ul><ul><ul><li>Multicast Join + </li></ul></ul><ul><ul><li>First I Frame + </li></ul></ul><ul><ul><li>Up to 2 seconds buffering time </li></ul></ul>Leave latency Join latency Signaling Latency First I Frame Media Latency Total Channel zapping Latency Buffering latency First frame viewed
  26. 26. Transport Level <ul><li>Example: </li></ul><ul><ul><li>Packet Loss </li></ul></ul><ul><ul><li>Loss Patterns </li></ul></ul><ul><ul><li>Jitter </li></ul></ul><ul><ul><li>Delay </li></ul></ul><ul><li>Well understood </li></ul><ul><li>Defined by ITU and IETF </li></ul>
  27. 27. Content Level <ul><li>Content quality is a payload based measurement. </li></ul><ul><ul><li>Requires decoding of the video stream </li></ul></ul><ul><ul><li>Understanding of the buffering and error concealment algorithms of the decoder </li></ul></ul><ul><li>CPU intensive – Does not scale </li></ul><ul><li>Accurate </li></ul><ul><li>Used mostly is Lab equipment and diagnostic equipment </li></ul><ul><li>Examples: </li></ul><ul><ul><li>PSNR </li></ul></ul><ul><ul><li>ITU-T J.144 </li></ul></ul><ul><li>Usually requires the reference (original) stream </li></ul><ul><li>Tests: </li></ul><ul><ul><li>Source artifacts </li></ul></ul><ul><ul><li>Source quality </li></ul></ul>
  28. 28. Standardization landscape Used in Diagnostics / Lab Used for Monitoring DSL DSL Forum TR-64, TR-69 LAN and WAN monitoring standards ITU Study Group 12 Algorithms for end-to-end transmission performance ITU VQEG – Video Quality Expert Group Video performance measurement based on Subjective tests Database ATIS IIF – IPTV Interoperability Forum QoS Metrics Standardization
  29. 29. Example: ATIS IIF Quality Metrics <ul><li>VSTQ - Video Service Transmission Quality </li></ul><ul><ul><li>Transmission Quality - codec/ content independent </li></ul></ul><ul><ul><li>Based on the rate and distribution of effective packet loss and discard </li></ul></ul><ul><li>VSPQ - Video Service Picture Quality </li></ul><ul><ul><li>Estimated viewing quality </li></ul></ul><ul><ul><li>Considers the impact of VSTQ, video codec type and rate, resolution </li></ul></ul><ul><li>VSAQ - Video Service Audio Quality </li></ul><ul><ul><li>audio listening quality </li></ul></ul><ul><ul><li>Considers the impact of VSTQ, audio codec type, sample rate, ….. </li></ul></ul><ul><li>VSMQ - Video Service Multimedia Quality </li></ul><ul><ul><li>overall user experience </li></ul></ul><ul><ul><li>Combined effect of VSPQ, VSAQ, audio-video synchronization.. </li></ul></ul><ul><li>VSCQ - Video Service Control (Plane) Quality </li></ul><ul><ul><li>Considers responsiveness and reliability of control plane (trick play) </li></ul></ul>
  30. 30. Monitoring levels <ul><li>J.144 and PSNR examines the video content only (payload measurements) </li></ul><ul><li>TR101290 examines only transport stream data and coherence without examining the video content </li></ul><ul><li>V - Factor and VQS looks at packet loss, jitter and loss patterns data and incorporate it with codec information and video header information </li></ul><ul><li>MDI – Examines only packet loss and packet loss patterns without considering the codec or video information </li></ul>TR101290 MPEG2TS Headers V-Factor, VQS MDI J.144, PSNR Video payload