Your SlideShare is downloading. ×
0
IPTV Performance Measurement

          Alan Clark
          Telchemy
Outline

•   IPTV Performance
•   Performance Measurement Approaches
•   Analyzing Encrypted Video Streams
•   Telchemy’s ...
IPTV Performance

• Transmission Impairments
   – Packet loss
• Codec configuration
   –   Quantization level
   –   GoP L...
IP Video basics – 1 – I Frames



     I             B             B              P             B




GoP Structure
- I fr...
IP Video basics – 2 – I Frames


                                 Macroblocks
IP Video basics – 3 – I Frames


                                 Slices
IP Video basics – 4 – P/B Frames



                               Motion encoded
Impact of lost packet



      I             B   B              P             B



Error in                               ...
Example – I frame vs B frame loss

• I frame example – 161 packets lost
  – MOS-V = 3.3
  – Loss Rate = 0.27%


• B frame ...
IPTV Performance Measurement Approaches

• Full Reference
  – ITU J.144, PEVQ, PSNR…


• Zero Reference
  – “Bitstream” al...
Industry Standards Activities

• ATIS IPTV Interoperability Forum (IIF)
  – QoS Metrics Task Force/ ATIS 0800008 Metrics
 ...
IP Based “Bitstream” algorithms

• Efficiently measure the performance of high
  bandwidth IP Video streams
• In IPTV serv...
VQmon/HD



                      Video
                   Assessment




        UDP
     RTP/ MP2T        Audio
       S...
VQmon/HD – Video Assessment


                                  Content
                                 Problem
         ...
VQmon/HD – Frame Detection



         Unscrambled     Picture       I/B/P
                         Header       Frame
   ...
Frame detection in encrypted streams

• Detect frame boundaries within the packet
  stream
• Measure frame size
• Analyze ...
VQmon/HD – Perceptual Quality



      Frame Quality

                              Perceptual Quality
  Impact       Impa...
Example 2 – Hero_D02

•   Detected GoP size 9, IBBPBB..
•   Measured bit rate 5542 kbits/sec
•   Image size 720x480, 29.97...
Example 3 – Hero-05

•   Detected GoP size 9, IBBPBB..
•   Measured bit rate 5541 kbits/sec
•   Image size 720x480, 29.97 ...
Example 4 - Scrambled vs Unscrambled Video – Chan20_1

• Unscrambled                 • Scrambled
   –   MOS-V = 3.1       ...
Telchemy’s IPTV Test Lab


  SA                       SA STB
Encoder

                             TI
                    ...
Example test - Noticeability

• Introduced impairment
   – Small number of macroblock errors (1, 4, 8)
   – Frame freezes
...
Frame Freeze

                            Noticeability of frame freeze


                  100
                  90
     ...
Macroblock Errors


          Noticeability of Macroblock Errors (high contrast)


    100
     90
     80
               ...
Current test project

• Impaired capture files, replayed through ADB
  Set Top Box to HD monitor
• MPEG-2 and H.264 encode...
Summary

•   IPTV Performance
•   Performance Measurement Approaches
•   Analyzing Encrypted Video Streams
•   Telchemy’s ...
Upcoming SlideShare
Loading in...5
×

Microsoft PowerPoint - IIT - IPTV Performance Measurement

1,630

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,630
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
67
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Microsoft PowerPoint - IIT - IPTV Performance Measurement"

  1. 1. IPTV Performance Measurement Alan Clark Telchemy
  2. 2. Outline • IPTV Performance • Performance Measurement Approaches • Analyzing Encrypted Video Streams • Telchemy’s IPTV Test Lab
  3. 3. IPTV Performance • Transmission Impairments – Packet loss • Codec configuration – Quantization level – GoP Length – Slice Structure – Bandwidth limitation • Interaction of loss with video frame structure
  4. 4. IP Video basics – 1 – I Frames I B B P B GoP Structure - I frames are typically large ( 00’s to 000’s of IP packets per frame) - P and B frames are typically much smaller (0’s to 00’s of packets per frame) - GoP length – typically 15-250 frames in length
  5. 5. IP Video basics – 2 – I Frames Macroblocks
  6. 6. IP Video basics – 3 – I Frames Slices
  7. 7. IP Video basics – 4 – P/B Frames Motion encoded
  8. 8. Impact of lost packet I B B P B Error in Error in I frame will P frame will propagate propagate through following through following P and B frames P and B frames Error in B frame will only impact this frame
  9. 9. Example – I frame vs B frame loss • I frame example – 161 packets lost – MOS-V = 3.3 – Loss Rate = 0.27% • B frame example – 161 packets lost – MOS-V = 3.7 – Loss Rate = 0.27%
  10. 10. IPTV Performance Measurement Approaches • Full Reference – ITU J.144, PEVQ, PSNR… • Zero Reference – “Bitstream” algorithms (e.g. IP based) – Decoded video analysis (pixel based) • Partial Reference – Feature extraction/ comparison – Augmented zero reference algorithms
  11. 11. Industry Standards Activities • ATIS IPTV Interoperability Forum (IIF) – QoS Metrics Task Force/ ATIS 0800008 Metrics – Test & Interoperability Task Force • ITU – SG9 – Standards for objective video test (J.nnn) – IPTV Focus Group – SG12 – Lead SG on QoS • Video Quality Experts Group (VQEG) – Testing of objective video quality algorithms • DSL Forum – WT135 – IPTV STB management
  12. 12. IP Based “Bitstream” algorithms • Efficiently measure the performance of high bandwidth IP Video streams • In IPTV services - content of RTP/MPEG-TS is typically encrypted • Ideally – report – Video MOS – Audio MOS – I/B/P frame statistics – RTP/MPEG TS statistics – IP Statistics
  13. 13. VQmon/HD Video Assessment UDP RTP/ MP2T Audio Stream Assessment Demultiplex IP Statistics
  14. 14. VQmon/HD – Video Assessment Content Problem Detection Content Alerts Content Analysis Video Per-Frame Perceptual MOS Frame Quality Quality Detection Computation Model TR101290 Metrics
  15. 15. VQmon/HD – Frame Detection Unscrambled Picture I/B/P Header Frame Decoding Metrics Per-Frame Transport Loss Packet Pattern Processing Metrics Heuristic Stream Frame Descriptors Scrambled/ Detection (BW..) Encrypted
  16. 16. Frame detection in encrypted streams • Detect frame boundaries within the packet stream • Measure frame size • Analyze sequence of frame sizes – Low motion/ high detail – obvious GoP structure – High motion/ low detail – GoP size less obvious – I frames can be inserted when scene changes • Assess content type
  17. 17. VQmon/HD – Perceptual Quality Frame Quality Perceptual Quality Impact Impact of MOS-V Of Codec & MOS-A Loss Bit rate Absolute Content Integration Sensitivity Model vs Relative Content Analysis Temporal Aggregate Masking Statistics Scene Content Change Type Detection Detect
  18. 18. Example 2 – Hero_D02 • Detected GoP size 9, IBBPBB.. • Measured bit rate 5542 kbits/sec • Image size 720x480, 29.97 frames/sec • MOS-V = 3.7 • MOS-A = 3.7 • EPSNR = 31.0 dB • Packet Loss Rate 0.11% • I frame packets = 16108 rcvd, 28 lost • P frame packets = 27551 rcvd, 42 lost • B frame packets = 29984 rcvd, 14 lost • MPEG Transport stats – Continuity Error count 12 – PCR Repetition Error count 285
  19. 19. Example 3 – Hero-05 • Detected GoP size 9, IBBPBB.. • Measured bit rate 5541 kbits/sec • Image size 720x480, 29.97 frames/sec • MOS-V = 1.8 • MOS-A = 1.8 • EPSNR = 21.0 dB • Packet Loss Rate = 1.55% • I frame packets = 15518 rcvd, 602 lost • P frame packets = 26886 rcvd, 483 lost • B frame packets = 30166 rcvd, 56 lost • MPEG Transport stats – Continuity Error count 163 – PCR Repetition Error count 282 – PCR Discontinuity Error count 3
  20. 20. Example 4 - Scrambled vs Unscrambled Video – Chan20_1 • Unscrambled • Scrambled – MOS-V = 3.1 – MOS-V = 3.0 – EPSNR = 27.5 – EPSNR = 27.0 – VSTQ = 41.5 – VSTQ = 41.5 – I frame packets – I frame packets • Received 6225 • Received 6367 • Lost 42 • Lost 42 – P/B frame packets – P/B frame packets • Received 20779 • Received 20637 • Lost 40 • Lost 21 – IP Statistics – IP Statistics • Loss rate = 0.23% • Loss rate = 0.23%
  21. 21. Telchemy’s IPTV Test Lab SA SA STB Encoder TI DaVinci TI Subjective test DaVinci - SS DCR/ACR Sigma Apple IP Black Darwin Emulator ST Micro Magic Capture Phillips Objective test VLC IP - PEVQ X - VQM (J.144) Multicast - PSNR Media Player Trace TCP Library Replay VLC
  22. 22. Example test - Noticeability • Introduced impairment – Small number of macroblock errors (1, 4, 8) – Frame freezes • Varied duration of impairment • Subject test – Was impairment • Not noticeable • Just noticeable • Definitely noticeable
  23. 23. Frame Freeze Noticeability of frame freeze 100 90 80 All 70 Noticeability 60 Flowers 50 Mobile 40 News 30 Tempete 20 10 0 0 50 100 150 200 Duration (ms)
  24. 24. Macroblock Errors Noticeability of Macroblock Errors (high contrast) 100 90 80 All 70 60 Flowers 50 Mobile 40 News 30 Tempete 20 10 0 0 50 100 150 D ura t io n o f e v e nt ( ms )
  25. 25. Current test project • Impaired capture files, replayed through ADB Set Top Box to HD monitor • MPEG-2 and H.264 encoder • H.264 – wide range of encoder configurations • Single Stimulus, Degradation Category Rating test • Series of tests – each with 60 x 10 second video clips, wide range of content • Presentation order randomized for each viewing panel • At least 4 different viewing panels
  26. 26. Summary • IPTV Performance • Performance Measurement Approaches • Analyzing Encrypted Video Streams • Telchemy’s IPTV Test Lab
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×