Video Quality Assessment and
      Comparative Evaluation of
Peer-to-Peer Video Streaming Systems

   Aditya Mavlankar    ...
Outline

       Introduction to P2P live video streaming
       Prior work on system performance assessment
       Test...
P2P Live Video Streaming

       Extension of P2P file-sharing
       Low-cost and scalable delivery mechanism
       S...
Related Work on Performance Assessment

       Networking related metrics, e.g. bandwidth usage, packet
        loss, con...
Test-Bed Setup
              576 X 9
              1024 X 5                                           Test center
        ...
Encoded Video Stream

       La Dolce Vita (Fellini, 1960)
       24 fps, 352x240 pixels
       H.264/AVC video codec, ...
Peer Churn Model

       30-minute simulation run
       During each 6-minute time-slot
         – Peer on with probabil...
Representative Results

       Tested systems
         – System A: Tree-based, push approach
         – System B: Mesh-ba...
Pre-Roll Delay
                                                45

                                                40
    ...
PSNR Drop (w/ traffic shaping)
                               System A (tree-based)                             System B (...
PSNR Drop (w/o traffic shaping)
                               System A (tree-based)                             System B ...
Statistics of Frame Freezes

      Frames frozen (as percentage of total frames to be displayed)

                       ...
Statistics of Frame Freezes (cont.)
  System A (tree-
                1
  based) employs
              0.9
  content-aware...
No. of Peers Failing to Decode a Frame
                                               System A (tree-based), Run 1        ...
Redundancy, Server Load and Parent-Peer Analysis


       Redundancy (bytes received in excess of required video
        ...
Summary

       Proposed methodology allows measuring video PSNR, buffering
        time, frame-freeze statistics, peers ...
Thank you!
http://www.stanford.edu/~maditya/publication.html
                    Related:
       [Agarwal, et al., TRIDENT...
Upcoming SlideShare
Loading in …5
×

presentation

353 views
318 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
353
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • In all the prior work that we came across, they did not control one or more of the following conditions: . . . Because of this they failed to propose a platform for fair head-to-head comparison of different systems.
  • a) Numbers are all for uplink b) how much overprovisioning? C) no downlink constraint
  • Some frame-freeze events are more than 100 frames long. Some freeze events are couple of frames long . . . Hence we look at the cdf of the length of the frame-freeze events.
  • System A is tree-based and hence pushes data along established paths. Hence less number of duplicates. System B is mesh-based. A peer advertises the chunks that is has and then complies with requests to relay those chunks to other peers. In general, it is hard to co-ordinate to eliminate the duplicates. A lot more diversity among the parent peers for System B.
  • The nature of our test-bed allows us to include more video quality metrics in our study.
  • Before concluding, I would like to say that apart from the results in this paper, we also reported some results from the same test-runs in another paper at TRIDENTCOM. That paper nicely complements this paper and I urge you to take a look! Thank you!
  • presentation

    1. 1. Video Quality Assessment and Comparative Evaluation of Peer-to-Peer Video Streaming Systems Aditya Mavlankar Sachin Agarwal Pierpaolo Baccichet Jatinder Pal Singh Bernd Girod Deutsche Telekom A.G., Stanford University Laboratories Stanford CA, USA Berlin, Germany
    2. 2. Outline  Introduction to P2P live video streaming  Prior work on system performance assessment  Test-bed setup  Performance of tested systems Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 2
    3. 3. P2P Live Video Streaming  Extension of P2P file-sharing  Low-cost and scalable delivery mechanism  Several deployed commercial implementations today  Increasing content / channels available Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 3
    4. 4. Related Work on Performance Assessment  Networking related metrics, e.g. bandwidth usage, packet loss, continuity index, etc. – CoolStreaming [Zhang et al., 2005]: PlanetLab – PPLive [Hei et al., 2006]: packet sniffing and crawling – SopCast [Sentinelli et al., 2007]: “watching”, PlanetLab – ...  No video PSNR results  No repeatable test conditions – Network conditions – Encoded video characteristics – Peer behavior  No fair head-to-head comparison of different systems Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 4
    5. 5. Test-Bed Setup 576 X 9 1024 X 5 Test center 2048 X 1 Berlin, Germany (15) Berlin, Germany [Emulated HS Broadband] Server 1, 2 PLR, delay, jitter and bandwidth measured for 576 X 16 representative real connections and emulated using 1024 X 5 2048 X 1 NISTNet Stanford, shaper traffic CA (22) Internet [Emulated HS Broadband] ISP Datacenter Erfurt, Germany 128 X 2 Berlin, Germany (8) 192 X 2 TU Munich, Germany (3) 576 X 2 [Emulated HS Broadband] [Real HS Broadband] 1024 X 2 3072 X 3 Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 5
    6. 6. Encoded Video Stream  La Dolce Vita (Fellini, 1960)  24 fps, 352x240 pixels  H.264/AVC video codec, 400 kbit/sec CBR bitstream, 42 dB PSNR  I B B P B B P B B P . . . (I frame every second)  H.264 bitstream wrapped in Microsoft ASF container, if required by tested system  Last frame error concealment Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 6
    7. 7. Peer Churn Model  30-minute simulation run  During each 6-minute time-slot – Peer on with probability 0.9 – Peer off with probability 0.1 – Peer can switch off for the rest of the run with probability 0.05  During last 5 minutes, peer off with probability 0.5 Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 7
    8. 8. Representative Results  Tested systems – System A: Tree-based, push approach – System B: Mesh-based, data-driven or pull approach  Emulation runs – Run 1: with traffic shaping (using NISTNet) – Run 2: without traffic shaping  Same realization of peer On-Off model for all runs Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 8
    9. 9. Pre-Roll Delay 45 40 Avg. PSNR across all clients [dB] 35 30 about 30 sec about 60 sec enough for enough for 25 System A System B (tree-based) (mesh-based) 20 Tree, w/ traffic shaping 15 Tree, w/o traffic shaping Mesh, w/ traffic shaping Mesh, w/o traffic shaping 10 0 10 20 30 40 50 60 Pre-roll delay [sec] Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 9
    10. 10. PSNR Drop (w/ traffic shaping) System A (tree-based) System B (mesh-based) 6 32 6 Avg. drop in PSNR [dB] Avg. drop in PSNR [dB] 5 5 4 4 3 3 2 2 1 1 0 0 0 20 40 0 20 40 Client ID Client ID Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 10
    11. 11. PSNR Drop (w/o traffic shaping) System A (tree-based) System B (mesh-based) 6 32 6 Avg. drop in PSNR [dB] Avg. drop in PSNR [dB] 5 5 4 4 3 3 2 2 1 1 0 0 0 20 40 0 20 40 Client ID Client ID Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 11
    12. 12. Statistics of Frame Freezes  Frames frozen (as percentage of total frames to be displayed) System A (tree-based) System B (mesh-based) Run 1 4.2% 2.0% Run 2 3.0% 0.2%  Average no. of distinct frame-freeze events per client in 30 min. System A (tree-based) System B (mesh-based) Run 1 64 23 Run 2 40 2 Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 12
    13. 13. Statistics of Frame Freezes (cont.) System A (tree- 1 based) employs 0.9 content-aware prioritization 0.8 Long frame freezes more likely with 0.7 System B (mesh- 0.6 based) cdf 0.5 0.4 0.3 System A, Run 1 System A, Run 2 0.2 System B, Run 1 System B, Run 2 0.1 0 20 40 60 80 100 120 Length of frame freeze [number of frames] Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 13
    14. 14. No. of Peers Failing to Decode a Frame System A (tree-based), Run 1 System B (mesh-based), Run 1 No. of peers failing to decode the frame No. of peers failing to decode the frame 40 40 30 30 20 20 10 10 0 0 0 10 20 30 0 10 20 30 Time [minutes] Time [minutes] Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 14
    15. 15. Redundancy, Server Load and Parent-Peer Analysis  Redundancy (bytes received in excess of required video stream bytes) – System A (tree-based): 6% in both runs – System B (mesh-based): 35% and 20% in Runs 1 (w/ traffic shaping) and 2 (w/o traffic shaping) respectively  For both Systems, peer receives on average less than 10% of its data directly from the server; slightly more for Run 2 of System B  System A (tree-based): Sustained downloads from lower number of parent peers Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 15
    16. 16. Summary  Proposed methodology allows measuring video PSNR, buffering time, frame-freeze statistics, peers failing to decode a frame, etc. beyond network usage, packet loss, etc.  Test conditions chosen by analyzing real-world conditions and experiments are repeatable  Tested three commercial-grade P2P video streaming systems  Room for improvement in current systems: – Long buffering time (10s of seconds) – Display freezes for more than 100 frames  Tested tree-based system outperforms mesh-based system: – Redundancy – Buffering time Mavlankar et al.: Comparative Evaluation of Peer-to-Peer Video Streaming Systems Jun. 25, 2008 16
    17. 17. Thank you! http://www.stanford.edu/~maditya/publication.html Related: [Agarwal, et al., TRIDENTCOM 2008]

    ×