Compressive Light Field Displays

3,422 views

Published on

Overview of a new generation of glasses-free 3D (light field) displays.

Published in: Technology
  • Be the first to comment

Compressive Light Field Displays

  1. 1. CompressiveThis slide has a 16:9 media window Light Field Displays Gordon Wetzstein - MIT Media Lab Collaborators: Doug Lanman, Matt Hirsch, Ramesh Raskar, Wolfgang Heidrich
  2. 2. This slide has a 16:9 media window
  3. 3. viewer moves rightThis slide has a 16:9 media window viewer moves down 4D Light Field
  4. 4. Compressive Displays ComputationalDisplay Optics Processing
  5. 5. This slide has a 16:9 media window Display-adaptive Compression Computed Tomography Nonnegative Tensor4D Light Field Factorization Compressive Optics Uniform or Directional Backlight Stacked Layers (LCDs or Transparencies)
  6. 6. Prototype – Layered 3D, SIGGRAPH 2011 Attenuation Layers with Spacers Backlight
  7. 7. Video clip
  8. 8. Prototype – Tensor Display, SIGGRAPH 2012
  9. 9. Video clip
  10. 10. What do we mean by “glasses-free 3D”?binocular disparity convergence motion parallax accommodation/blur current glasses-based (stereoscopic) displays near-term glasses-free (light field) displays longer-term holographic displays
  11. 11. Is glasses-free 3D Technology ready? Nintendo 3DS MasterImage 3D Asus Eee Pad MeMO 3D LG Optimus 3D E3 2010 Computex 2011 Computex 2011 Mobile World Congress 2011 Toshiba 3DTV Prototype Sony 3DTV Prototype LG 3DTV Prototype CES 2011 CES 2011 CES 2011
  12. 12. Parallax Barriers – Ives 1903barrier2D display  Low resolution & very dim  Switchable 2D/3D with LCDs
  13. 13. Parallax Barriers – Ives 1903barrier Nintendo 3DS2D display  Low resolution & very dim  Switchable 2D/3D with LCDs
  14. 14. lenslets Integral Imaging – Lippmann 1908 2D display  Brighter than parallax barriers  Always low resolution, even for 2D
  15. 15. lenslets Integral Imaging – Lippmann 1908 2D display Alioscopy 3DHD 42 (1920x1200, 1x8 views)  Brighter than parallax barriers  Always low resolution, even for 2D
  16. 16. Directional Backlighting – 3M & MS Wedge 3M Directional Backlight Film Nelson and Brott, 2010 US Patent 7,847,869LED thin light guide LED  Requires 120 Hz for stereo  Not practical for multiview Microsoft Wedge
  17. 17. Directional Backlighting Lenslet array
  18. 18. Glasses-Free 3D Display LightSpace Sony Jones et al. 2007 Zebra Imaging MIT HolovideoHolograms DisplaysVolumetric  3Ddepth cues inside enclosure all objects only  opticallymechanically moving parts  mostly & computationally expensive  3D objects outside enclosure inexpensive off-the-shelf partsCompressive LF Displays  no moving parts efficient computationally
  19. 19. From Conventional to Compressive 3D Displays mask 2 mask 1 Conventional Parallax Barriers t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  20. 20. From Conventional to Compressive 3D Displays mask 2 mask 1 Time-shifted Parallax Barriers [Kim et al. 2007] High Resolution through High Speed t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  21. 21. From Conventional to Compressive 3D Displays Perceptual Integration time Time-shifted Parallax Barriers [Kim et al. 2007] High Resolution through High Speed t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  22. 22. From Conventional to Compressive 3D Displays mask 2 mask 1 High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010] Compression in Time – Nonnegative Matrix Factorization t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  23. 23. From Conventional to Compressive 3D Displays Perceptual Integration time High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010] Compression in Time – Nonnegative Matrix Factorization t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  24. 24. From Conventional to Compressive 3D Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth – Computed Tomography t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  25. 25. From Conventional to Compressive 3D Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth – Computed Tomography t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  26. 26. From Conventional to Compressive 3D Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels – Computed Tomography t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  27. 27. From Conventional to Compressive 3D Displays Perceptual Integration … … … time Tensor Displays Compression in Time & Pixels –Tensor Factorization t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  28. 28. From Conventional to Compressive 3D Displays Perceptual Integration … … … time Tensor Displays – Multilayer & Directional Backlighting t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  29. 29. From Conventional to Compressive 3D Displays Perceptual Integration thin! time Tensor Displays – Directional Backlighting t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  30. 30. Prototype – Layered 3D, SIGGRAPH 2011 Attenuation Layers with Spacers Backlight
  31. 31. Computed Tomography (CT) x-ray sensorsource: wikipedia 3D Reconstruction x-ray source Reconstructed 2D Slices 35
  32. 32. Tomographic Light Field Synthesis Image Formation Virtual Planes - ò c m (r )dr x L(x, q ) = e Attenuation Volume log L x, (r )dr cBacklight Tomographic Synthesis 2D Light Field log( L ) P 2 argmin log( L) P 2 0 x 36
  33. 33. CT vs. Layered 3D Computed Tomography Layered 3D reconstruct physical volume  thin stack of optimized layers sensor noise  no noise 37
  34. 34. Multi-Layer Decomposition viewer moves rightviewer moves down Input 4D Light Field 1 2 3 1 2 4 3 5 4 5 Photographs of Prototype Optimized Attenuation Layers
  35. 35. Depth of Field for 3D Displays Integral Imaging Parallax Barriers Cutoff (cycles/cm) Maximum Resolution Display Thickness Zwicker et al. 2006 Antialiasing + Display Prefilter Distance of Virtual Plane from Middle of Display (cm)
  36. 36. How Do Layers Increase Depth of Field? Integral Imaging Parallax Barriers Layered 3D Cutoff (cycles/cm) ? Maximum Resolution Display Thickness Distance of Virtual Plane from Middle of Display (cm)
  37. 37. Optimization: Number of Layers Two Layers Three Layers Five Layers
  38. 38. Optimization: Display Thickness Average Reconstruction PSNR for All Scenes PSNR in dB Number of Layers
  39. 39. Application to HDR Display “Square Root” Layers
  40. 40. Application to HDR Display “Square Root” Layers
  41. 41. Application to HDR Display Optimized Layers
  42. 42. Video clip
  43. 43. Limitations: Field of ViewFOV 10º FOV 20º FOV 45º
  44. 44. Personal Glasses-Free 3D Display Challenges for dynamic display:  Real-time computation  Engineering issues, moiré
  45. 45. Multi-Layer LCD – SIGGRAPH ASIA 2011
  46. 46. Barco E-2320 PAGrayscale IPS LCD1600x1200 @ 60 Hz
  47. 47. Four Stacked Liquid Crystal Panels Two Crossed Polarizers
  48. 48. Overview of LCDs I vertical polarizer color filter array liquid crystal cells horizontal polarizer I0 backlight Malus’ Law Intensity Modulation with Liquid Crystal Cells I = I0 sin2 (q )
  49. 49. Extending Layered 3D to Multi-layer LCDs Virtual Planes Design Optimization LCD • Eliminate redundant polarizers 3  Sequentially-crossed design LCD x 2 LCD 1 backlight 2D Light Field x
  50. 50. Extending Layered 3D to Multi-layer LCDs Virtual Planes Design Optimization LCD • Eliminate redundant polarizers 3  Use sequentially-crossed LCD x • Exploit field-sequential color 2  0.33 = 2.7% brightness LCD 1 backlight 2D Light Field x
  51. 51. Polarization Field Displays Virtual Planes Design Optimization LCD • Eliminate redundant polarizers 3  Use sequentially-crossed LCD x • Exploit field-sequential color 2  0.33 = 2.7% brightness LCD • Further optimize polarizers 1  Minimum is a crossed pair backlight 2D Light Field x
  52. 52. Modeling and Synthesizing Polarization Fields Virtual Planes LCD f3 3 Image Formation f2 K LCD x Q(x, q ) = åfk (x, q ) 2 k=1 f1 L(x, q ) = sin 2 (Q(x,q )) LCD 1 backlight Tomographic Synthesis Q(x,q ) = ±sin-1 ( ) L(x, q ) mod p 2D Light Field Q = Pf argmin Q - Pf 2 2 fmin £f £fmax x
  53. 53. Decompositions & Reconstructions viewer moves right viewer moves down Stacked Polarization Input 4D Light Field Rotating Layers 90° Optimized Rotation Angles for Each Layer 0°
  54. 54. Decompositions & Reconstructions viewer moves right viewer moves down Input 4D Light Field Reconstruction Results
  55. 55. Multi-layer LCD Attenuation Layers Polarization-Rotating Layers
  56. 56. Video clip
  57. 57. Tomographic Image Synthesis Target Light Field Projection Matrix LCD Pixel Values = * b=Ax
  58. 58. SART - Simultaneous Algebraic Reconstruction Technique b=Ax pre-compute some weights initial guess ATv Ax update clamp
  59. 59. Implementing Ax as Multiview Rendering
  60. 60. Implementing ATv as Projective Texture Mapping
  61. 61. Benefits & Limitations
  62. 62. Light Field “Slice” Representation Light Field moving to the right
  63. 63. Light Field “Slice” Representation Light Field Light Field Slice moving to the left
  64. 64. Light Field “Slice” Representation Light Field View from Above Light Field Slice moving to the left
  65. 65. Light Field “Slice” Representation Light FieldMultilayer Light Field Display Light Field Slice moving to the leftFront LayerMiddle LayerRear LayerBacklight
  66. 66. Light Field “Slice” Representation Light FieldMultilayer Light Field Display Light Field Slice L( moving to the leftFront Layer fm(3)(Middle Layer fm(2)( L(Rear Layer fm(1)(Backlight
  67. 67. Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field Tensor L(Front Layer Rear Layer fm(3)(Middle Layer L( fm(2)(Rear Layer fm(1)(Backlight
  68. 68. Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field Tensor L(Front Layer Rear Layer fm(3)(Middle Layer L( fm(2)(Rear Layer fm(1)(Backlight
  69. 69. Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field TensorFront Layer Rear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight
  70. 70. Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field TensorFront Layer Rear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight
  71. 71. Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field TensorFront Layer Rear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight
  72. 72. Light Field Tensor DecompositionTarget Light Field Tensor Rank-M Approximation Nonnegative Tensor Perceptual Factorization (NTF) Integration + + ... + Frame 1 Frame 2 Frame M
  73. 73. Light Field Tensor DecompositionTarget Light Field Tensor Rank-M Approximation Nonnegative Tensor Perceptual Factorization (NTF) Integration + + ... + Frame 1 Frame 2 Frame M
  74. 74. Light Field Tensor Decomposition Nonlinear (Multilinear) Optimization Problem Iterative Update Rules (see paper for details) Efficient GPU ImplementationForward Projection (Multiview Rendering) Back Projection (Projective Texture Mapping)
  75. 75. Design Tradespace: Layers vs. Frames PSNR without Directional Backlight# layers # frames PSNR with Directional Backlight# layers # frames
  76. 76. Design Tradespace: Layers vs. FramesPSNR without Directional Backlight # layers # frames PSNR with Directional Backlight # layers # frames 2 Layers, Layers,Directional–Backlight (Tensor Display) 3 Layers, 1 Frame Frames 3D – SIGGRAPH 2010) 1 Layer, 3 3 3 Frames(Layered (Tensor Display) 2011) Frames, 3 (HR3D SIGGRAPH Asia Original Original 2 Layers, 3 Frames 3 Layers, 1 Frame 3 Layers, 3 Frames 1 L, 3 F, Directional BL
  77. 77. Tensor Display Prototypes Reconfigurable Directional Backlight Three Layer
  78. 78. Tensor Display Prototypes 3 Layer LCD Directional BacklightHardware
  79. 79. Three Layer Prototype Video clip
  80. 80. Video clip
  81. 81. Directional Backlight Prototype Video clip
  82. 82. LCD + Directional BLView from above LCD with Directional Backlight, Rank 6 Directional BL
  83. 83. LCD with Directional Backlight, Rank 6 (as seen by obserer) Video clip
  84. 84. LCD with Directional Backlight, Rank 6 (as seen by obserer) Video clip
  85. 85. Filmed with High-speed Camera Directional Backlight Video clip Front LCD
  86. 86. Video clipLenslets only LCD+DBL Rank 1 LCD+DBL Rank 6
  87. 87. Limitation: Compressible Light Fields “Natural” 4D Light Field Random 4D Light Field
  88. 88. Next-generation Technology What about Content? Computational PhotographyConsumer Light Field CamerasRendered Footage Computational Displays Camera Rigs
  89. 89. SIGGRAPH 2012 Course on Computational Displays Code & Datasets onlineUse Layered 3D in your class! media.mit.edu/~gordonw cameraculture.media.mit.edu

×