Barcelona keynote web

1,027 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,027
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Barcelona keynote web

  1. 1. Video Coding For Compression . . . and Beyond Bernd Girod I nformation Systems Laboratory Department of Electrical Engineering Stanford University Compression
  2. 2. Bit Consumption of US Households Bit equivalent, assuming state-of-the-art compression, year 2000 [Source: UC Berkeley: How much Information] 0.0003% Internet 0.6% Video games 3.3% Home video 0.0002% Magazines 0.0002% Books 0.0003% Newspaper 0.4% Recorded Music 1.7% Radio 94% Television ~230 Exabyte/year Total for 70M households
  3. 3. Desirable Compression Ratios DSL ~200 kbps ~ 1,000 : 1 Dial-up modem, wireless link ~ 20 kbps ~ 10,000 : 1 ITU-R 601 166 Mbps SDTV broadcasting ~2 Mbps ~ 100 : 1 CIF QCIF
  4. 4. Outline <ul><li>Video compression – state-of-the-art </li></ul><ul><li>Beyond compression </li></ul><ul><ul><li>Rate-scalable video </li></ul></ul><ul><ul><li>Wavelet video coding </li></ul></ul><ul><ul><li>Error-resilient video transmission </li></ul></ul><ul><ul><li>Unequal error protection </li></ul></ul><ul><ul><li>Optimal scheduling for packet networks </li></ul></ul><ul><ul><li>Distributed video coding </li></ul></ul>
  5. 5. Outline <ul><li>Video compression – state-of-the-art </li></ul><ul><li>Beyond compression </li></ul><ul><ul><li>Rate-scalable video </li></ul></ul><ul><ul><li>Wavelet video coding </li></ul></ul><ul><ul><li>Error-resilient video transmission </li></ul></ul><ul><ul><li>Unequal error protection </li></ul></ul><ul><ul><li>Optimal scheduling for packet networks </li></ul></ul><ul><ul><li>Distributed video coding </li></ul></ul>
  6. 6. “ It has been customary in the past to transmit successive complete images of the transmitted picture.” [...] “ In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object.”
  7. 7. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer -
  8. 8. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in ¼-pixel accuracy Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer -
  9. 9. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Adaptive block sizes . . .
  10. 10. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer - Multiple Past Reference Frames
  11. 11. Motion-Compensated Hybrid Coding Standards: H.261 , MPEG-1, MPEG-2, H.263 , MPEG-4, H.264/AVC Video in Generalized B-Frames Entropy Coding Deq./Inv. Transform Motion- Compensated Predictor Control Data Quant. Transf. coeffs Motion Data 0 Intra/Inter Coder Control Decoder Motion Estimator Transform/ Quantizer -
  12. 12. Rate-Distortion Optimized Coder Control <ul><li>Minimize Lagrangian cost function </li></ul><ul><li>Strategy: minimize J i for each block i separately, using a common Lagrange multiplier  </li></ul>Total distortion Total bit-rate Distortion for block i Rate for block i Lagrangian cost for block i
  13. 13. Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) 0 1 2 3 4 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~15%
  14. 14. Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) 0 1 2 3 4 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference >25%
  15. 15. Multiple Reference Frames in H.264/AVC Mobile & Calendar (CIF, 30 fps) 0 1 2 3 4 26 27 28 29 30 31 32 33 34 35 36 37 38 R [Mbit/s] PSNR Y [dB] PBB... with generalized B pictures PBB... with classic B pictures PPP... with 5 previous references PPP... with 1 previous reference ~40%
  16. 16. Outline <ul><li>Video compression – state-of-the-art </li></ul><ul><li>Beyond compression </li></ul><ul><ul><li>Rate-scalable video </li></ul></ul><ul><ul><li>Wavelet video coding </li></ul></ul><ul><ul><li>Error-resilient video transmission </li></ul></ul><ul><ul><li>Unequal error protection </li></ul></ul><ul><ul><li>Optimal scheduling for packet networks </li></ul></ul><ul><ul><li>Distributed video coding </li></ul></ul>
  17. 17. Surprising Success of ITU-T Rec. H.263 ?? What H.263 was developed for . . . Analog videophone . . . and what is was used for. Internet video streaming
  18. 18. Internet Video Streaming <ul><li>How to accommodate heterogeneous bit-rates? </li></ul><ul><li>How to react to network congestion? </li></ul><ul><li>How to mitigate late or lost packets? </li></ul>Streaming client DSL dial-up modem Media Server Internet wireless
  19. 19. Fine Granular Scalability (FGS) <ul><li>H.264 with/without FGS option </li></ul><ul><li>Foreman sequence (5fps) </li></ul>~2dB gap Base layer 20 kbps Enhancement layer variable bit-rate Efficiency gap
  20. 20. Wavelet Video Coder Temporal Wavelet Transform Spatial Wavelet Transform LLL LLH LH LH Original video frames Embedded Quantization & Entropy Coding <ul><ul><li>[Taubman & Zakhor, 1994] [Ohm, 1994] [Choi & Woods, 1999] [Hsiang & Woods, VCIP ’99] . . . and others </li></ul></ul>7 6 5 4 3 2 1 0 H H H H H H H H H H H H H H H H H H
  21. 21. Lifting P U Even Frames Synthesis: Odd Frames Low Band High Band P U Even Frames Analysis: Odd Frames Low Band High Band Motion Compensation [Secker & Taubman, 2001] [Popescu & Bottreau, 2001]
  22. 22. MC Wavelet Coding vs. H.264/AVC 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 36 34 32 30 28 26 24 22 20 38 Luminance PSNR (dB) bit-rate (Mbps) Scalable MC 5/3 Wavelet Non-scalable H.264/AVC <ul><li>Sequence: Mobile CIF </li></ul><ul><li>H.264/AVC </li></ul><ul><li>high complexity RD control </li></ul><ul><li>CABAC </li></ul><ul><li>PBBPBBP . . . </li></ul><ul><li>5 prev/3 future reference frames </li></ul><ul><li>data courtesy of M. Flierl </li></ul>[Taubman & Secker, VCIP 2003] courtesy D. Taubman
  23. 23. Wavelet Synthesis with Lossy Motion Vector MC Wavelet Transform Motion Estimator Embedded Encoding Embedded Encoding Decoder Decoder Inverse Wavelet Transform Video in Video out [Taubman & Secker, ICIP03] Minimize J=D+  R Minimize J=D+  R
  24. 24. R-D Performance with Lossy Motion Vector [Taubman & Secker, VCIP 2003] courtesy D. Taubman Bit - Rate (kbps) Video PSNR (dB) 0 200 400 600 800 1000 1200 24 26 28 30 32 34 36 38 40 Embedded wavelet coefficients Lossless motion Non-embedded single-rate Embedded wavelet coefficients Lossy motion CIF Foreman
  25. 25. Outline <ul><li>Video compression – state-of-the-art </li></ul><ul><li>Beyond compression </li></ul><ul><ul><li>Rate-scalable video </li></ul></ul><ul><ul><li>Wavelet video coding </li></ul></ul><ul><ul><li>Error-resilient video transmission </li></ul></ul><ul><ul><li>Unequal error protection </li></ul></ul><ul><ul><li>Optimal scheduling for packet networks </li></ul></ul><ul><ul><li>Distributed video coding </li></ul></ul>
  26. 26. Priority Encoding Transmission (PET) information symbols block of packets [Albanese, Blömer, Edmonds, Luby, Sudan, 19 96] [Davis & Danskin, 1996] [Horn, Stuhlmuller, Link, Girod, 1999] [Puri, Ramchandran, 1999] [Mohr, Riskin, Ladner, 2000] [Stankovic, Hamzaoui, Xiong, 2002] [Chou, Wang, Padmanabhan, 2003] . . . and many more . . . … redundancy symbols enhancement layer base layer Reed-Solomon codeword K N-K packet network
  27. 27. Packet Delay Jitter and Loss delay     pdf loss lead-time loss probability lead-time loss probability
  28. 28. Smart Prefetching Idea: Send more important packets earlier to allow for more retransmissions Server Client Internet [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001] Request stream Rate-distortion preamble Packet Schedule Video packets Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule Updated Packet Schedule
  29. 29. Rate-Distortion Preamble <ul><li>Each media packet n is labeled by </li></ul><ul><ul><li>B n — size [in bits] of data unit n </li></ul></ul><ul><ul><li> d n —distortion reduction if n is decoded </li></ul></ul><ul><ul><li>t n — decoding deadline for n </li></ul></ul>P P I I B B B P P P I I B B B P … … …
  30. 30. Rate-Distortion Preamble <ul><li>Each media packet n is labeled by </li></ul><ul><ul><li>B n — size [in bits] of data unit n </li></ul></ul><ul><ul><li> d n —distortion reduction if n is decoded </li></ul></ul><ul><ul><li>t n — decoding deadline for n </li></ul></ul>P B P P I I B B P P I I B B B P … … … For video:  d n must be made “ state-dependent” to accurately capture concealment
  31. 31. Markov Decision Tree for One Packet ... N transmission opportunities before deadline send: 1 ack: 1 0 0 0 send: 1 0 send: 1 0 ack: 1 0 1 0 1 0 0 1 1 1 0 0 0 0 t current t current +  t t current +2  t Action Observation “ Policy“ minimizing J = D +  R
  32. 32. R-D Optimized Streaming Performance <ul><li>Foreman </li></ul><ul><li>120 frames </li></ul><ul><li>10 fps, I-P-P-… </li></ul><ul><li>H.263+ 2 Layer SNR scalable </li></ul><ul><li>20 frame GOP </li></ul><ul><li>Copy Concealment </li></ul><ul><li>20 % loss forward and back </li></ul><ul><li>Γ -distributed delay </li></ul><ul><ul><li>κ = 10 ms </li></ul></ul><ul><ul><li>μ = 50 ms </li></ul></ul><ul><ul><li>σ = 23 ms </li></ul></ul><ul><li>Pre-roll 400ms </li></ul>PSNR [dB] Bit-Rate [kbps] ~50 %
  33. 33. Naive Coding Questions <ul><li>To achieve graceful degradation in case of channel error for a digitally encoded signal, is an embedded signal representation (aka layers, aka data partitioning) always needed? </li></ul><ul><li>Can one, in general, send refinement information for an analog (i.e. uncoded) signal transmission over a noisy channel? </li></ul>
  34. 34. Digitally Enhanced Analog Transmission <ul><li>Forward error protection of the signal waveform </li></ul><ul><li>Information-theoretic bounds [Shamai, Verdu, Zamir,1998] </li></ul><ul><li>“ Systematic lossy source-channel coding” </li></ul>Analog Channel (uncoded) Wyner- Ziv Encoder Digital Channel Wyner- Ziv Decoder Side info
  35. 35. Forward Error Protection of Compressed Video <ul><li>Graceful degradation without a layered signal representation </li></ul>Any Old Video Encoder Video Decoder with Error Concealment Error-Prone channel S S’ Analog channel (uncoded) [Aaron, Rane, Girod, ICIP 2003] Wyner-Ziv Decoder A S * Wyner-Ziv Encoder A Wyner-Ziv Decoder B S ** Wyner-Ziv Encoder B
  36. 36. Wyner-Ziv MPEG Codec [Rane, Aaron, Girod, VCIP 2004] Channel Slepian-Wolf Encoder Wyner-Ziv Encoder ED T -1 Q -1 + MC S * MPEG Encoder main S Side information MPEG Encoder coarse T -1 q -1 ED + MC S’ R-S Decoder Reconstructed Frame at Encoder MPEG Encoder coarse R-S Encoder
  37. 37. Graceful Degradation with Forward Error Protection Main Stream @ 1.092 Mbps FEC (n,k) = (40,36) FEC bitrate = 120 Kbps Total = 1.2 Mbps WZ Stream @ 270 Kbps FEP (n,k) = (52,36) WZ bitrate = 120 Kbps Total = 1.2 Mbps
  38. 38. Visual Comparison of Degradation at Same PSNR With FEC 1 Mbps + 120 kbps (38.32 db) Foreman 50 CIF frames @ symbol error rate = 4 x 10 -4 With FEP 1 Mbps + 120 kbps (38.78 db)
  39. 39. Superior Robustness of FEP With FEC 1 Mbps + 120 kbps (33.03 db) Foreman 50 CIF frames @ symbol error rate = 10 -3 With FEP 1 Mbps + 120 kbps (38.40 db)
  40. 40. Lossy Compression with Side Information Source Encoder Decoder Source Encoder Decoder [Wyner, Ziv, 1976] For mse distortion and Gaussian statistics, rate-distortion functions of the two systems are the same .
  41. 41. Ultra-Low-Complexity Video Coding Interframe Decoder Intraframe Encoder K’ Interpolation/ Extrapolation Key frames K Conventional Intraframe coding Conventional Intraframe decoding X’ Scalar Quantizer Turbo Encoder Buffer WZ frames X Turbo Decoder Request bits Slepian-Wolf Codec Reconstruction Y [Aaron, Zhang, Girod, Asilomar 2002] [Aaron, Rane, Zhang, Girod, DCC 2003]
  42. 42. R-D Performance Ultra-Low-Complexity Video Coder <ul><li>Sequence: Foreman </li></ul><ul><li>WZ frames - even frames </li></ul><ul><li>Key frames - odd frames </li></ul><ul><li>Side information - motion compensated interpolation of key frames </li></ul>8 dB 3 dB
  43. 43. Ultra-Low-Complexity Video Coder H263+ Intraframe Coding 330 kbps, 32.9 dB Wyner-Ziv Codec 274 kbps, 39.0 dB
  44. 44. Ultra-Low-Complexity Video Coder H263+ I-B-I-B 276 kbps, 41.8 dB Wyner-Ziv Codec 274 kbps, 39.0 dB
  45. 45. Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab
  46. 46. Stanford Camera Array Courtesy Marc Levoy, Stanford Computer Graphics Lab
  47. 47. Light Field Compression Rate: 0.11 bpp PSNR 39.9 dB Rate: 0.11 bpp PSNR 37.4 dB Wyner-Ziv, Pixel-Domain JPEG-2000
  48. 48. Conclusions <ul><li>Video compression is very important . . . but there is more to video coding than compression </li></ul><ul><li>Rate-scalable video representations: mc lifting break-through </li></ul><ul><li>Robust video transmission </li></ul><ul><ul><li>Virtual priority mechanisms by packet scheduling </li></ul></ul><ul><ul><li>RD gains easily larger than from super-clever compression </li></ul></ul><ul><li>Distributed video coding: radically different approach </li></ul><ul><ul><li>Graceful degradation w/o layers </li></ul></ul><ul><ul><li>Ultra-low-complexity coders </li></ul></ul><ul><li>Ubiquitous J=D+  R </li></ul>
  49. 49. Acknowledgments Anne M. Aaron Jacob Chakareski Philip A. Chou J=D+  R Markus Flierl Sang-eun Han Mark Kalman Marc Levoy Yi Liang Shantanu Rane David Rebollo-Monedero Andrew Secker David Taubman Thomas Wiegand Xiaoqing Zhu Rui Zhang
  50. 50. Progress is a wonderful thing, if only it would stop . . . Robert Musil

×