Advances in Network-adaptive Video Streaming


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Advances in Network-adaptive Video Streaming

  1. 1. Advances in Network-adaptive Video Streaming Bernd Girod J. Chakareski, M. Kalman, Y. J. Liang, E. Setton, R. Zhang Information Systems Laboratory Department of Electrical Engineering Stanford University
  2. 2. Streaming media: a huge success <ul><li>Hundreds of thousands of streaming media servers deployed </li></ul><ul><li>> 1 million hours of streaming media content produced per month </li></ul><ul><li>Hundreds or millions streaming media players </li></ul><ul><li>RealPlayer </li></ul><ul><ul><li>Most popular Internet application second only to Internet Explorer [Media Metrix] </li></ul></ul><ul><ul><li>More than 400 million unique registered users </li></ul></ul><ul><ul><li>More than 200,000 new users per day </li></ul></ul><ul><ul><li>Open source code </li></ul></ul>
  3. 3. Internet Media Streaming <ul><li>Best-effort packet network </li></ul><ul><ul><li>low bit-rate </li></ul></ul><ul><ul><li>variable throughput </li></ul></ul><ul><ul><li>variable loss </li></ul></ul><ul><ul><li>variable delay </li></ul></ul><ul><li>Challenges </li></ul><ul><ul><li>compression </li></ul></ul><ul><ul><li>rate scalability </li></ul></ul><ul><ul><li>error resiliency </li></ul></ul><ul><ul><li>low latency </li></ul></ul>Streaming client DSL 56K modem Media Server Internet wireless
  4. 4. Outline <ul><li>What is network-adaptive video streaming? </li></ul><ul><ul><li>Better delivery of video packets by considering source coding, signal processing, and packet transport jointly </li></ul></ul><ul><ul><li>Application-layer joint source-channel coding techniques for the Internet </li></ul></ul><ul><li>This talk: review recent advances in </li></ul><ul><ul><li>Adaptive media playout </li></ul></ul><ul><ul><li>Rate-distortion optimal packet scheduling </li></ul></ul><ul><ul><li>Network-adaptive packet dependency management </li></ul></ul>
  5. 5. Adaptive Media Playout Fixed deadline Flexible deadline Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions 5% packet loss 2 sec average receiver buffer [Steinbach, F ärber, Girod, ICIP 2001]
  6. 6. Modification of Playout Speed <ul><li>Video: adaptation of display rate </li></ul><ul><li>Audio and speech: Stretching based on time-domain interpolation algorithm WSOLA [Verhelst et al., 1993, Liang 2001] </li></ul>Original packet Pitch- period 0 2 1 3 4 Template Output packet 1/2 0/1 2/3 3 4
  7. 7. Audio Speed Adjustment <ul><li>Waveform Similarity Overlap Add (WSOLA) method allows speed adjustment without changing pitch </li></ul><ul><li>Speech demo </li></ul><ul><li>Music demo </li></ul>original slower 30% faster 30% original slower 30% faster 30%
  8. 8. Reduced Pre-roll Time for Stored Streams  G =1.092,  B =0.42, T G =20 sec, T B =2 sec, T RTT =220 ms Probability of buffer underflow < 1% [ Kalman, Steinbach , Girod, ISCAS 200 2 ]
  9. 9. Rate Scalability by Playout Speed Adjustment Server 20 kbps 50 kbps 95 kbps 25 kbps 55 kbps 100 kbps Channel (mean throughput) 85 kbps (29.2 dB, 53.3 kbps) (33.1 dB, 93.6*0.9=84.2 kbps)
  10. 10. First Things First: Smart Prefetching Idea: Send more important packets earlier to allow for more retransmissions Server Client Internet Request stream Rate-distortion preamble Prefetch times Video data Repeat request Repeat request Repeat request
  11. 11. Streaming as a Packet Scheduling Problem <ul><li>Which media units should be selected for transmission, and when? </li></ul><ul><li>Requirements </li></ul><ul><ul><li>Meet rate constraint </li></ul></ul><ul><ul><li>Meet latency constraint </li></ul></ul><ul><ul><li>Maximize reconstruction quality </li></ul></ul><ul><li>Rate-distortion framework proposed, e.g., in [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001] </li></ul>time pre-encoded media units transmission opportunities server
  12. 12. Markov Decision Tree for One Packet ... N transmission opportunities before deadline send: 1 ack: 1 0 0 0 send: 1 0 send: 1 0 ack: 1 0 1 0 1 0 0 1 1 1 0 0 0 0 t current t current +  t t current +2  t Action Observation “ Policy“ minimizing J = D +  R
  13. 13. Packet Delay Jitter and Loss delay     pdf loss lead-time loss probability lead-time loss probability
  14. 14. Source Description <ul><li>Each media packet n is labeled by </li></ul><ul><ul><li>B n — size [in bits] of data unit n </li></ul></ul><ul><ul><li> d n —distortion reduction if n is decoded </li></ul></ul><ul><ul><li>t n — decoding deadline for n </li></ul></ul>P P I I B B B P P P I I B B B P A … … … A …
  15. 15. Source Description <ul><li>Each media packet n is labeled by </li></ul><ul><ul><li>B n — size [in bits] of data unit n </li></ul></ul><ul><ul><li> d n —distortion reduction if n is decoded </li></ul></ul><ul><ul><li>t n — decoding deadline for n </li></ul></ul>P B P P I I B B P P I I B B B P A … … … A … For video:  d n must be made “ state-dependent” to accurately capture concealment
  16. 16. R-D Optimized Streaming Performance <ul><li>Foreman </li></ul><ul><li>120 frames </li></ul><ul><li>10 fps, I-P-P-… </li></ul><ul><li>H.263+ 2 Layer SNR scalable </li></ul><ul><li>20 frame GOP </li></ul><ul><li>Copy Concealment </li></ul><ul><li>20 % loss forward and back </li></ul><ul><li>Γ -distributed delay </li></ul><ul><ul><li>κ = 10 ms </li></ul></ul><ul><ul><li>μ = 50 ms </li></ul></ul><ul><ul><li>σ = 23 ms </li></ul></ul><ul><li>Pre-roll 400ms </li></ul>PSNR [dB] Bit-Rate [kbps]
  17. 17. R-D Optimized Streaming with a Proxy Server Last hop [Chakareski, Chou, Girod, Asilomar 2002, MMSP 2002] Media Server Switch/ Router Proxy Server Client Backbone Network Packet Receiver-driven RaDiO streaming Sender-driven RaDiO streaming Buffer packets
  18. 18. RaDiO Edge: Experiment <ul><li>Video: Foreman, QCIF, 130 frames </li></ul><ul><li>Compression: H.264 </li></ul><ul><li>3-layer temporal scalability 72…144 kbps [Liang, 2002] </li></ul><ul><li>Backbone </li></ul><ul><ul><li>Packet loss rate: 10% </li></ul></ul><ul><ul><li>Delay: shifted  -distribution </li></ul></ul><ul><li>Last hop </li></ul><ul><ul><li>Packet loss rate: 1% </li></ul></ul><ul><ul><li>Delay: shifted  -distribution </li></ul></ul>
  19. 19. Streaming with Diversity Client Channel 1 Channel 2 Channel N Packet Path Diversity Server Diversity Media Server Client Channel 1 Channel 2 Channel N Media Server 1 Media Server 2 Media Server N
  20. 20. R-D Optimized Streaming over 2 Channels <ul><li>Video: Foreman, QCIF, 130 frames </li></ul><ul><li>Compression: H.263+ </li></ul><ul><li>2-layer SNR scalability 32/64 kbps </li></ul><ul><li>2 identical, independent 2-state Markov channels </li></ul><ul><li>Good/bad packet loss rates: 3%/15% </li></ul><ul><li> -distributed delays short/long </li></ul>[Chakareski, Girod, DCC 2003]
  21. 21. R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD) Latency: 100 ms RTT: 100 ms PSNR, in dB Bit-Rate, in kbps 3 dB 47 %
  22. 22. R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD) Latency: 100ms R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD) Multiple Deadlines Rate: 68.8 kbps Mean PSNR: 27.0 dB Single Deadline Rate: 89.0 kbps Mean PSNR: 23.9 dB
  23. 23. Network-adaptive Packet Dependency Management <ul><li>Very low latency: no retransmissions </li></ul><ul><li>Highly robust compressed representation by network-adaptive management of packet dependencies </li></ul><ul><li>Utilize ACK/NACK in source coder </li></ul><ul><li>H.263 RPS, MPEG-4 NEWPRED, H.264 multiframe prediction </li></ul>Time Transmission error NACK
  24. 24. Error Resilience vs. Coding Efficiency I P5 230 frames of Foreman coded using H.26L TML8.5. Average PSNR=33.4dB P1 P2
  25. 25. Rate-Distortion Optimal Reference Picture Selection [Wiegand, Färber, Girod, 2000] [Liang, Girod, 2002]
  26. 26. RD Performance of Optimal Reference Picture Selection <ul><li>LTM buffer: V=5 frames </li></ul><ul><li>Feedback round-trip time: d fb =7 frames </li></ul><ul><li>Packet loss rate: p= 10% </li></ul><ul><li>Comparison with P-I scheme, where each NACK triggers insertion of I-frame </li></ul>[Liang, Girod, 2002]
  27. 27. ORPS Performance over Time Axis
  28. 28. MaD Sequence at 10% packet loss No Retransmissions Optimal Reference Picture Selection Rate: 320 kbps Mean PSNR: 39.2 dB Adaptive I-Frame Insertion Rate: 320 kbps Mean PSNR: 38.2 dB
  29. 29. Conclusions <ul><li>Network-adaptive video streaming: jointly optimize compression, error control, packet transport, and decoding </li></ul><ul><li>Adaptive media playout: “real-time” more flexible than we thought </li></ul><ul><li>RaDiO streaming can provide virtual priority mechanisms </li></ul><ul><li>Proxy servers with RaDiO can improve streaming performance </li></ul><ul><li>Path Diversity can improve streaming performance </li></ul><ul><li>RaDiO streaming with multiple deadlines for accelerated retroactive decodingof late retransmissions (“catch-up decoding”) </li></ul><ul><li>Packet dependency management allows robust representations for streaming w/o retransmissions </li></ul>