3D Video: From Stereo to Multi-View

5,609 views

Published on

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,609
On SlideShare
0
From Embeds
0
Number of Embeds
128
Actions
Shares
0
Downloads
264
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

3D Video: From Stereo to Multi-View

  1. 1. 3D Video: From Stereo to Multi-View TEWI - Kolloquium, Universität Klagenfurt Karsten Müller Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institut Berlin Einsteinufer 37 10587 Berlin +49 30 310 02 – 0 info@hhi.fraunhofer.de Germany http://www.hhi.fraunhofer.de
  2. 2. Overview • Introduction, scenarios, requirements • 3D Video: Stereo Video + MVC • 3D Video: 3D Video Formats & Coding • International 3D media research and development • Summary Karsten Müller Slide 2 18.03.2010 Universität Klagenfurt
  3. 3. 3D Video Introduction • Provision of left and right eye view • Added depth sensation • Resolved 2D viewing ambiguity • Additional features (e.g. free viewpoint, depth-controlled object insertion) 3D Video is more than adding 1 dimension to 2D! Karsten Müller Slide 3 18.03.2010 Universität Klagenfurt
  4. 4. 3D Video Scenarios 3D cinema • Increasing number of 3D productions • First studios start to release all productions also in 3D • Technology: stereoscopic 3D, glasses based 3D home entertainment • Different types of displays available: stereoscopic, auto-stereoscopic with 2 … N views • Various technologies, input formats and display sizes • Glasses based systems may not be acceptable 3D mobile • auto-stereoscopic 2 view display with fixed viewing position • Good 3D viewing in spite of small display sizes Karsten Müller Slide 4 18.03.2010 Universität Klagenfurt
  5. 5. Stereo Video Solution Stereo- • Original Cameras = Views for Display Capture Production Format • Production format specification (e.g. SMPTE) Post- • Rectification, color correction, format conversion processing Transport • Conventional Stereo Format, e.g. Side-by-side Format MVC- Coding Transport • Multi-view video coding (MPEG-4 MVC/H.264) Format MVC- Decoding Display Format Karsten Müller FolieStereo- 5 • High-resolution glasses-based stereo display Display 18.03.2010 © Fraunhofer HHI
  6. 6. 3D format: Conventional Stereo video (CSV) Original Image Source: Interactive Visual Media Group Karsten Müller of Microsoft Research Slide 6 18.03.2010 Universität Klagenfurt
  7. 7. Current Coding Standards for Stereo Video • Advanced Video Coding (MPEG-4 AVC / H.264) – Simulcast – Stereo SEI (Supplementary Enhancement Information) – Auxiliary Picture Syntax • MPEG-C part 3 (container format for V+D, e.g. with AVC coding) • Multi-view Video Coding (MPEG-4 MVC / H.264) Karsten Müller Slide 7 18.03.2010 Universität Klagenfurt
  8. 8. Multi-view Video Coding • Joint Coding of camera views with similar content Karsten Müller Slide 8 18.03.2010 Universität Klagenfurt
  9. 9. MVC Coding Method • Coding approach for exploiting temporal and inter-view dependencies using MPEG4-AVC/H.264 • Usage of hierarchical B pictures in temporal direction • Usage of P or hierarchical B pictures in inter-view direction • Frame reordering to optimize memory usage No H.264 changes required for multi-view coding • Standardized in July 2008 • Adopted for 3D Blu Ray in Dec. 2009 Karsten Müller Slide 9 18.03.2010 Universität Klagenfurt
  10. 10. MVC Coding Structure- Picture Reordering Karsten Müller Slide 10 18.03.2010 Universität Klagenfurt
  11. 11. MVC Coding Structure Karsten Müller Slide 11 18.03.2010 Universität Klagenfurt
  12. 12. MVC Camera Setting Adjustment - Star Karsten Müller Slide 12 18.03.2010 Universität Klagenfurt
  13. 13. MVC Camera Setting Adjustment – 2D Array Karsten Müller Slide 13 18.03.2010 Universität Klagenfurt
  14. 14. MVC Coding Results • Coding structure allows AVC coder, to select the best inter-view-temporal neighbors within the MVC sequence • Coding gain obtained by usage of hierarchical B-frames and by exploitation of inter-view dependencies Karsten Müller Slide 14 18.03.2010 Universität Klagenfurt
  15. 15. MVC Coding Results – delta_PSNR Karsten Müller Slide 15 18.03.2010 Universität Klagenfurt
  16. 16. Short Term 3DTV will be Stereo TV anaglyph polarized shutter Broadcasters currently concentrate on stereo and will not invest in new infrastructures They will adopt a system, which they can feed through existing pipes: e.g. anaglyph or multiplexed stereo: checkerboard side-by-side above-below etc. Will Stereo TV be the long term solution? Karsten Müller Slide 16 18.03.2010 Universität Klagenfurt
  17. 17. The Answer is Given by Our Chancellor Karsten Müller Slide 17 18.03.2010 Universität Klagenfurt
  18. 18. Special Requirements for 3DTV 3DTV viewing conditions are more complex than in 3D cinema: • Glasses are generally not accepted in the living room • There are many different 3D display technologies • Varying viewing conditions (distance, display size) • User preferences have to be considered • Not all users can see stereo (≈10%) • Not all users want to see stereo • Users might not want to see all programmes in stereo Karsten Müller Slide 18 18.03.2010 Universität Klagenfurt
  19. 19. Multi-view Displays • The most common form of multi-view displays utilise either lenticular screens or parallax barriers. • Lenticular screens consist of a series of vertically aligned or slanted cylindrical lenses and parallax barriers vertically aligned apertures. • Light is guided in the appropriate directions by either focussing it or by blocking unwanted rays. VIEW 1 VIEW 2 VIEW 3 VIEW 1 VIEW 2 VIEW 3 1 41 41 4 32 1 LENTICULAR PARALLAX BARRIER Karsten Müller SCREEN Slide 19 18.03.2010 Image Source: De Montfort University, UK
  20. 20. 3D Video Color only: Assumption 1 • Multi-view Displays require N views (e.g. 5,9,22,50) • Why not transmitting N color views with MVC? Karsten Müller Slide 20 18.03.2010 Universität Klagenfurt
  21. 21. MVV and MVC Restrictions • Pure coding of multi-view video • Still linear dependency of bit rate from number of views • Number and position of views fixed over the whole 3D chain • No baseline adaptation for different displays • Number of Views fixed in MVV/MVC Karsten Müller Slide 21 18.03.2010 Universität Klagenfurt
  22. 22. MVC Restrictions • Coding experiments on camera density Original camera distance Karsten Müller Slide 22 18.03.2010 Universität Klagenfurt
  23. 23. MVC Restrictions (2) • Coding experiments on camera density with Rena test sequence average rate average rate per camera Karsten Müller Slide 23 18.03.2010 Universität Klagenfurt
  24. 24. 3D Video Color only: Assumption 2 • Multi-view Displays require N views (e.g. 5,9,22,50) • Why not transmitting 2 or 3 color views with MVC and reconstructing N views at the receiver? Karsten Müller Slide 24 18.03.2010 Universität Klagenfurt
  25. 25. Color Only Intermediate View Karsten Müller Original Image Source: Interactive Slide 25 Visual Media Group of Microsoft Research 18.03.2010
  26. 26. Depth map Color information 256 depth layers Original Image Source: Interactive Visual Media Group Karsten Müller of Microsoft Research Slide 26 18.03.2010 Universität Klagenfurt
  27. 27. Example Reconstruction Pure image blending Depth image based rendering Original Image Source: Interactive Visual Media Group Karsten Müller of Microsoft Research Slide 27 18.03.2010 Universität Klagenfurt
  28. 28. Advanced 3D Video Solutions Required • Color-only multi-view video too restricted – High number of views cannot be efficiently compressed and transmitted – View synthesis of new views at the receiver side from color-only data gives bad reconstruction quality • Additional Scene Geometry information required, e.g. per-pixel depth/disparity data Karsten Müller Slide 28 18.03.2010 Universität Klagenfurt
  29. 29. Depth-enhanced 3D Video Capture • Stereo / multiview capturing, depth provision Production Format • Production format specification (e.g. SMPTE) Post- • Rectification, color correction, format conversion processing Transport • 3D format specification Format • 3D video coding (e.g. MPEG) Coding Transport Format View Synthesis • High quality intermediate view synthesis (e.g. 2 view + 2 depth N view) Display Format Karsten Müller Folie 29 Display 18.03.2010 © Fraunhofer HHI • High-resolution stereo/multiview display
  30. 30. 3D Video Formats • Conventional stereo video (CSV) • Mixed resolution stereo (MRS) • Video plus depth (V+D) • Multiview video (MVV) • Multiview video + depth (MVD) • Layered depth video (LDV) • Depth-enhanced stereo (DES) Karsten Müller Slide 30 18.03.2010 Universität Klagenfurt
  31. 31. assuming multiple views with color & depth… Karsten Müller Slide 31 18.03.2010 Universität Klagenfurt
  32. 32. Stereo video (CSV) CSV Karsten Müller Slide 32 18.03.2010 Universität Klagenfurt
  33. 33. Mixed resolution stereo (MRS) MRS Karsten Müller Slide 33 18.03.2010 Universität Klagenfurt
  34. 34. Video plus depth (V+D) V+D Karsten Müller Slide 34 18.03.2010 Universität Klagenfurt
  35. 35. Multiview video (MVV) MVV Karsten Müller Slide 35 18.03.2010 Universität Klagenfurt
  36. 36. Multiview video plus depth (MVD) MVD Karsten Müller Slide 36 18.03.2010 Universität Klagenfurt
  37. 37. Layered depth video (LDV) LDV Karsten Müller Slide 37 18.03.2010 Universität Klagenfurt
  38. 38. Depth-enhance Stereo (DES) DES Karsten Müller Slide 38 18.03.2010 Universität Klagenfurt
  39. 39. Towards a new 3D Video Coding Standard • MPEG develops a new 3D video coding standard • Motivation for new standard: – Decouple production from coding format – MPEG-4 AVC/H.264 only optimized for 2D color video, but not for depth information Karsten Müller Slide 39 18.03.2010 Universität Klagenfurt
  40. 40. Advanced 3DTV concept based on MVD Karsten Müller Slide 40 18.03.2010 Universität Klagenfurt
  41. 41. Coverage of 3D Video Coding Capture Production Format Post- processing Transport Format Coding 3D components under consideration: – Transport format Transport Format – 3D video coding methods – Intermediate view synthesis View Synthesis Display Format Karsten Müller Folie 41 Display 18.03.2010 © Fraunhofer HHI
  42. 42. Challenges for 3DVC • Consider capturing technology, i.e. maximal 2-3 recorded views • Break linear dependency of coding bit rate from number of target views (e.g. MVC) • Provide scene geometry data in general form, i.e. pixel-wise depth data • Consider statistical properties of depth (and supplementary) data • Consider new quality evaluation methods for intermediate views • Provide high-quality view synthesis for continuous viewing range • Decrease depth and coding errors Karsten Müller Slide 42 18.03.2010 Universität Klagenfurt
  43. 43. Coding Experiments MVD with MVC • Joint color and depth coding for optimal intermediate view quality • Evaluation of total bit rate (e.g. of 2 color and 2 depth maps) vs. quality of decoded synthesized views • View synthesis is part of coding optimization • Reduction of coding and interpolation artifacts through high-quality view synthesis Karsten Müller Slide 43 18.03.2010 Universität Klagenfurt
  44. 44. Coding Results Intermediate View • Ballet (JMVM 7.0.1, GOP16, κ=½) Karsten Müller Slide 44 18.03.2010 Universität Klagenfurt
  45. 45. Coding Results Intermediate View • Breakdancers (JMVM 7.0.1, GOP16, κ=½) Karsten Müller Slide 45 18.03.2010 Universität Klagenfurt
  46. 46. Color Depth Bit Rate Distribution • Objective and subjective examples for optimal and non-optimal bit rate distribution • Variation of color and depth quality • Tradeoff between synthesized views and views at original positions Karsten Müller Slide 46 18.03.2010 Universität Klagenfurt
  47. 47. Coding Results Intermediate View • Ballet (JMVM 7.0.1, GOP16, κ=½) C30D30 C24D40 Karsten Müller Slide 47 18.03.2010 Universität Klagenfurt
  48. 48. Coding Results across Viewing Range • Ballet (JMVM 7.0.1, GOP16) Karsten Müller Slide 48 18.03.2010 Universität Klagenfurt
  49. 49. Coding Results Intermediate View • Breakdancers (JMVM 7.0.1, GOP16, κ=½) C30D36 C36D30 Karsten Müller Slide 49 18.03.2010 Universität Klagenfurt
  50. 50. Coding Results across Viewing Range • Breakdancers (JMVM 7.0.1, GOP16) Karsten Müller Slide 50 18.03.2010 Universität Klagenfurt
  51. 51. View Synthesis for MVD – High quality view interpolation is essential for 3D video applications with MVD data. – Processing steps: - Layer Extraction - Layer Projection - Hole Filling + Filtering Karsten Müller Slide 51 18.03.2010 Universität Klagenfurt
  52. 52. Synthesis Improvements (Ballet) simple merging layer-based view synthesis compressed uncompressed Karsten Müller Slide 52 18.03.2010 Universität Klagenfurt
  53. 53. International R&D in 3D Media • ISO/IEC MPEG 3DVC (3D video coding) • SMPTE Task Force of 3D to the home (3D master format) • ITU-R SG6 • DVB – TM - ES Stereoscopic (3D) TV • 3D@Home Consortium • Association in Korea, Japan,… • EU 3D, Immersive, Interactive Media Cluster • International Conferences – 3DTV-CON – 3DPVT Karsten Müller Slide 53 18.03.2010 Universität Klagenfurt
  54. 54. 3D Video Coding Group in MPEG • Nagoya Univ. • Samsung • Zhejiang Univ. • NTT • LG Electronics • ASTRI • UPM • Nokia • Qualcomm • Philips • NICT • GIST • Kwangwoon Univ. • ETRI • Sharp • JVC • Fraunhofer • Poznan Univ. of • NCTU/ITRI Technology • Motorola • Thomson • NXP • MERL • Ericsson • Peking Universtity • Logitech • Tsinghua Univ. • Orange/France • Coordination: • Huawei Telecom • Karsten Müller (HHI) • Xidian University • FUB Anthony Vetro (MERL) • Sony Karsten Müller • Telefonica Slide 54 18.03.2010 Universität Klagenfurt
  55. 55. 3D@Home Consortium • Steering Team 1: Content Creation & Production Objective: To focus on projects which improve the quality of 3D content for the home audience. • Steering Team 2: Content Storage, Transmission & Distribution Objective: To develop useful definitions and guidelines for the successful storage, transmission and distribution of 3D content. • Steering Team 3: 3D Promotion Objective: To promote the adoption of 3D into the home by promoting 3D format and the 3D@Home Consortium. • Steering Team 4: 3D Displays Karsten Müller Objective: To identify issues around 3D Displays and associated hardware. Slide 55 18.03.2010 Universität Klagenfurt
  56. 56. 3D@Home Consortium Karsten Müller Slide 56 18.03.2010 Universität Klagenfurt
  57. 57. EU 3D, Immersive, Interactive Media Cluster • Currently, a number of EU projects develop parts and products of the 3D media chain for certain application areas, e.g. 3D home entertainment or mobile services • Enhanced success of these EC funded projects and subsequent associated commercialization efforts if simultaneous operations of individual projects can be further orchestrated towards a strong mutual presence • Better highlight of Europe‘s good position in worldwide R&D • 3D, Immersive, Interactive Media Cluster is an umbrella structure for the projects and provides international contact and information about project activities • 3D Media, Immersive, Interactive Cluster is one of the main drivers towards "3D Media" in the context of the "Future Internet" activities supported by EU Commission (DG INFSO) Karsten Müller Slide 57 18.03.2010 Universität Klagenfurt
  58. 58. 3DII Media Cluster – Current Projects • Leadership: Prof. Levent Onural and Dr. Karsten Müller • Projects: Status: Framework Program • 3DTV ended 2008 FP6 • 3DPHONE ongoing FP7 3D for Mobile • MOBILE3DTV ongoing FP7 Systems • 3D4YOU ongoing FP7 3D for Broadcast • 2020 3D Media ongoing FP7 Systems • 3DPresence ongoing FP7 3D Videoconference • MUTED ended 2008 FP6 • HELIUM3D ongoing FP7 3D Display • Real3D ongoing FP7 Technology • i3DPost ongoing FP7 3D Content • VICTORY ended 2009 FP7 Description Karsten Müller Slide 58 18.03.2010 Universität Klagenfurt
  59. 59. 3DII Media Cluster – New Projects • Projects: Status: Framework Program • DIOMEDES new 2010 FP7 • MUSCADE new 2010 FP7 • 3D VIVANT new 2010 FP7 • BEAMING new 2010 FP7 • FINE new 2010 FP7 • SKYMEDIA new 2010 FP7 • SALA3D new 2010 FP7 • 3DLife new 2010 FP7 • FascinatE new 2010 FP7 Karsten Müller Slide 59 18.03.2010 Universität Klagenfurt
  60. 60. Summary • 3D technology is maturing due to world wide development from capturing to display • First systems are in use, mainly using stereoscopic displays and MVC coding • New challenging research topics in all areas of 3D video: – Reliable capturing devices with synchronized views – Improved time-consistent depth estimation – Geometry-enhanced multi-view coding (e.g. 3DVC) – Robust view synthesis – New high-resolution multi-view displays (e.g. 50 views, each with HDTV resolution!) Karsten Müller Slide 60 18.03.2010 Universität Klagenfurt

×