Video Quality Measurements

8,183 views
7,767 views

Published on

Review video quality measurement types and classification

Published in: Technology

Video Quality Measurements

  1. 1. Video Quality Measurements
  2. 2. Overview <ul><li>QoS and QoE </li></ul><ul><li>Measurement Methods </li></ul><ul><ul><li>Objective, Subjective, payload/codec based </li></ul></ul><ul><li>Measurement and Monitoring Goals </li></ul><ul><ul><li>Lab, pre-deployment, Monitoring, failure analysis </li></ul></ul><ul><li>Monitoring Levels </li></ul><ul><ul><li>Transport, Transaction, content </li></ul></ul><ul><li>Monitoring Domains </li></ul><ul><ul><li>Head-end, network, Home </li></ul></ul>
  3. 3. User Quality of Experience factors (QoE) <ul><li>What does subscribers want? </li></ul><ul><ul><li>Content </li></ul></ul><ul><ul><ul><li>Content quality </li></ul></ul></ul><ul><ul><ul><li>Content pricing </li></ul></ul></ul><ul><ul><ul><li>Content availability </li></ul></ul></ul><ul><ul><li>V iewing experience </li></ul></ul><ul><ul><ul><li>Video Quality </li></ul></ul></ul><ul><ul><ul><li>Channel Zapping delay </li></ul></ul></ul><ul><ul><li>Integration of services </li></ul></ul><ul><ul><ul><li>Usability </li></ul></ul></ul><ul><ul><ul><li>One bill </li></ul></ul></ul><ul><ul><ul><li>New services </li></ul></ul></ul><ul><ul><li>Customer services </li></ul></ul>Technical Factors
  4. 4. MEASUREMENT TAXONOMY <ul><li>Subjective </li></ul><ul><li>Objective </li></ul><ul><li>Payload based, codec aware, codec anaware </li></ul>
  5. 5. Measurement methods review <ul><li>S ubjective </li></ul><ul><ul><li>Human assessment of quality </li></ul></ul><ul><ul><li>Expensive, not for monitoring </li></ul></ul><ul><ul><li>Not repeatable </li></ul></ul><ul><li>Objective </li></ul><ul><ul><li>Measurement devices </li></ul></ul><ul><ul><li>Repeatable </li></ul></ul><ul><ul><li>For both testing and monitoring, failure analysis </li></ul></ul>Objective Subjective Measurements
  6. 6. Multimedia monitoring methods <ul><li>Subjective measurement arte done by humans according to pre-defined protocols </li></ul><ul><li>Voice – MOS </li></ul><ul><ul><li>Grade voice quality between 1-5 </li></ul></ul><ul><ul><li>4+ - is very good quality </li></ul></ul><ul><li>Video – BT500 </li></ul>Subjective MOS (Voice) BT500 (Video) Measurements
  7. 7. Objective methods <ul><li>Objective methods are divided into: </li></ul><ul><li>Payload based </li></ul><ul><li>Packet based </li></ul><ul><ul><li>CODEC Aware </li></ul></ul><ul><ul><li>CODEC Independent </li></ul></ul><ul><li>Network Monitoring </li></ul>Measurements
  8. 8. Payload based Measurements <ul><li>Payload base methods assess the video quality based on the video itself </li></ul><ul><li>Reference based methods compares Original video quality to distorted video. </li></ul><ul><li>Used mostly in lab equipment for codec performance analysis and comparison </li></ul>
  9. 9. Full Reference Methods PSNR Full Reference Objective Payload based MSE SSIM J.144 <ul><li>Full reference methods compares each frame of the original video to frame of the distorted video and provide distortion measurement </li></ul><ul><li>MSE & PSNR are pixel based similarity measurements </li></ul><ul><li>Most video quality measurements are done on Luma (Y) </li></ul>
  10. 10. PSNR & MSE <ul><li>PSNR and MSE is a pixel base video quality comparison tools </li></ul>
  11. 11. MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723 original Image PSNR/MSE Problems <ul><li>Quality degradation is not reflected by MSE/PSNR index </li></ul>
  12. 12. SSIM – “Solves” PSNR Problems <ul><li>SSIM is Structural Similarity index. Unlike PNR and MSE it does not compare images “pixel by pixel” but as small NxN “environments” </li></ul><ul><li>  are average variance and covariance </li></ul><ul><li>   the dynamic range of the pixel-values (typically ) </li></ul>
  13. 13. <ul><li>ITU-T J.144 and ITU-R BT.1683 </li></ul><ul><li>Full-reference perceptual models </li></ul><ul><li>Digital TV </li></ul><ul><li>Rec. 601 image resolution (PAL/NTSC) </li></ul><ul><li>Bit rates: 768 kbps ~ 5 Mbps </li></ul><ul><li>Compression errors </li></ul>Full Reference testing standards
  14. 14. Back to Objective measurement <ul><li>What happens when we don’t have the original (Reference video) or when we don’t have the processing power to do an extensive comparison? </li></ul><ul><li>We could use network measurements and codec based degradation info to asses video quality </li></ul>Measurements
  15. 15. Packet – Codec Aware <ul><li>Monitoring technique </li></ul><ul><li>Degrades video based on codec type by incorporating network parameters data with codec behavior data </li></ul><ul><li>Scales- could monitor thousands of channels </li></ul><ul><li>Examples: </li></ul><ul><ul><li>VQS (Telchemy) </li></ul></ul><ul><ul><li>VQI (Brix) </li></ul></ul><ul><ul><li>V-Factor (QoSMetrics) </li></ul></ul>The need a codec aware metrics Problem area Robust codec “ Raw” codec
  16. 16. Codec Aware Methods Codec aware Packet based VQI V-Factor VQS Telchemy Objective methods
  17. 17. Example V - Factor <ul><li>Based on MPQM (Moving Picture Quality Metrics) – high quality video measurement standard </li></ul><ul><li>V = f(QER, PLR, R) </li></ul><ul><ul><li>QER – relative video codec quality </li></ul></ul><ul><ul><li>PLR – Packet loss ratio (based on actual packet loss, jitter data and jitter buffer model) </li></ul></ul><ul><ul><li>R – Image complexity factor (2-3) </li></ul></ul><ul><li>Adopted by Spirnet </li></ul>
  18. 18. Packet – Codec Independent <ul><li>Monitoring only </li></ul><ul><li>Codec independent </li></ul><ul><li>Based on network parameters data only </li></ul><ul><li>Scales - could monitor thousands of channels </li></ul><ul><li>Examples: </li></ul><ul><ul><li>MDI </li></ul></ul><ul><ul><ul><li>IneoQuest </li></ul></ul></ul><ul><ul><ul><li>standardized by IETF </li></ul></ul></ul>
  19. 19. MEASUREMENT & MONITORING <ul><li>In the Lab & In The Fields </li></ul><ul><li>Pre-Deployment/monitoring/Failure Analysis </li></ul>
  20. 20. Measurement & Monitoring phases Analysis Problem Solving Tuning Pre Deployment Testing Lab Testing Design 24/7 Monitoring Deployment Phase Pre-Deployment Phase
  21. 21. Measurement & Monitoring phases <ul><li>Design & Lab testing </li></ul><ul><ul><li>Simulation and Emulation of the network </li></ul></ul><ul><ul><li>Lab and testing tools </li></ul></ul><ul><li>Pre Deployment Stage </li></ul><ul><ul><li>Work on actual network </li></ul></ul><ul><ul><li>Load testing </li></ul></ul><ul><ul><li>Lab, testing, diagnosing and monitoring tools </li></ul></ul><ul><li>Deployment (production) Phase </li></ul><ul><ul><li>Mostly monitoring (probes) equipment, management systems, data filtering and diagnostics equipment </li></ul></ul>
  22. 22. MONITORING LEVELS
  23. 23. Measurement Levels <ul><li>Transport Level </li></ul><ul><li>Service (transaction) Level </li></ul><ul><li>Media Quality Level </li></ul>Video Quality Transport Quality Transaction Quality
  24. 24. Transaction Level <ul><li>Examples </li></ul><ul><ul><li>Post-dial delay in PSTN/mobile networks </li></ul></ul><ul><ul><li>Video start time for channel zapping & Video conf </li></ul></ul><ul><li>Requires understanding in both network monitoring and signaling (IGMP, SIP) and in media coding (analysis of the media to discover dial tone or I frame) </li></ul>
  25. 25. Channel zapping delay <ul><li>Multicast saves bandwidth but creates signaling delays: </li></ul><ul><ul><li>Multicast Leave + </li></ul></ul><ul><ul><li>Multicast Join + </li></ul></ul><ul><ul><li>First I Frame + </li></ul></ul><ul><ul><li>Up to 2 seconds buffering time </li></ul></ul>Leave latency Join latency Signaling Latency First I Frame Media Latency Total Channel zapping Latency Buffering latency First frame viewed
  26. 26. Transport Level <ul><li>Example: </li></ul><ul><ul><li>Packet Loss </li></ul></ul><ul><ul><li>Loss Patterns </li></ul></ul><ul><ul><li>Jitter </li></ul></ul><ul><ul><li>Delay </li></ul></ul><ul><li>Well understood </li></ul><ul><li>Defined by ITU and IETF </li></ul>
  27. 27. Content Level <ul><li>Content quality is a payload based measurement. </li></ul><ul><ul><li>Requires decoding of the video stream </li></ul></ul><ul><ul><li>Understanding of the buffering and error concealment algorithms of the decoder </li></ul></ul><ul><li>CPU intensive – Does not scale </li></ul><ul><li>Accurate </li></ul><ul><li>Used mostly is Lab equipment and diagnostic equipment </li></ul><ul><li>Examples: </li></ul><ul><ul><li>PSNR </li></ul></ul><ul><ul><li>ITU-T J.144 </li></ul></ul><ul><li>Usually requires the reference (original) stream </li></ul><ul><li>Tests: </li></ul><ul><ul><li>Source artifacts </li></ul></ul><ul><ul><li>Source quality </li></ul></ul>
  28. 28. Standardization landscape Used in Diagnostics / Lab Used for Monitoring DSL DSL Forum TR-64, TR-69 LAN and WAN monitoring standards ITU Study Group 12 Algorithms for end-to-end transmission performance ITU VQEG – Video Quality Expert Group Video performance measurement based on Subjective tests Database ATIS IIF – IPTV Interoperability Forum QoS Metrics Standardization
  29. 29. Example: ATIS IIF Quality Metrics <ul><li>VSTQ - Video Service Transmission Quality </li></ul><ul><ul><li>Transmission Quality - codec/ content independent </li></ul></ul><ul><ul><li>Based on the rate and distribution of effective packet loss and discard </li></ul></ul><ul><li>VSPQ - Video Service Picture Quality </li></ul><ul><ul><li>Estimated viewing quality </li></ul></ul><ul><ul><li>Considers the impact of VSTQ, video codec type and rate, resolution </li></ul></ul><ul><li>VSAQ - Video Service Audio Quality </li></ul><ul><ul><li>audio listening quality </li></ul></ul><ul><ul><li>Considers the impact of VSTQ, audio codec type, sample rate, ….. </li></ul></ul><ul><li>VSMQ - Video Service Multimedia Quality </li></ul><ul><ul><li>overall user experience </li></ul></ul><ul><ul><li>Combined effect of VSPQ, VSAQ, audio-video synchronization.. </li></ul></ul><ul><li>VSCQ - Video Service Control (Plane) Quality </li></ul><ul><ul><li>Considers responsiveness and reliability of control plane (trick play) </li></ul></ul>
  30. 30. Monitoring levels <ul><li>J.144 and PSNR examines the video content only (payload measurements) </li></ul><ul><li>TR101290 examines only transport stream data and coherence without examining the video content </li></ul><ul><li>V - Factor and VQS looks at packet loss, jitter and loss patterns data and incorporate it with codec information and video header information </li></ul><ul><li>MDI – Examines only packet loss and packet loss patterns without considering the codec or video information </li></ul>TR101290 MPEG2TS Headers V-Factor, VQS MDI J.144, PSNR Video payload

×