Compressive Light Field Displays
Upcoming SlideShare
Loading in...5
×
 

Compressive Light Field Displays

on

  • 1,324 views

Overview of a new generation of glasses-free 3D (light field) displays.

Overview of a new generation of glasses-free 3D (light field) displays.

Statistics

Views

Total Views
1,324
Views on SlideShare
1,324
Embed Views
0

Actions

Likes
2
Downloads
36
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • [Gordon resumes at this slide.]
  • Multi-layer 3D display is a computed tomography problem. (Multi-Layer is 3D)Natural light fields are compressible, so a thin display with few layers can achieve the illusion of a thick virtual scene. (thin display creates thick scenes)So this leads to a new type of computational/compressive display.[probably same slide as personal device]
  • Basically, the tomographic light field synthesis then boils down to solving a linear equation system of the form Ax=b. b is the target light field, A the projection matrix, and the unknowns x are the LCD pixel values
  • This equation system can be solved with SART – a technique developed, tested, and refined over decades in the medical imaging community.Algorithm is very simple:Initialize some data and find an initial guess of the pixel valuesIteratively update and clamp the solutionUpdate rules require two important operations: a function computing the matrix-vector multiplication Ax, and a function computing the transpose matrix-vector multiplication ATv
  • Interpreting AX – is basically a multiview rendering step. In OpenGL just set the projection matrix to all different views of the light field and render the LCD layers textured with the given patterns.
  • Interpreting AX – is basically a multiview rendering step. In OpenGL just set the projection matrix to all different views of the light field and render the LCD layers textured with the given patterns.
  • Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
  • Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
  • Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
  • Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as wellPossibly use static framesEmphasize

Compressive Light Field Displays Compressive Light Field Displays Presentation Transcript

  • CompressiveThis slide has a 16:9 media window Light Field Displays Gordon Wetzstein - MIT Media Lab Collaborators: Doug Lanman, Matt Hirsch, Ramesh Raskar, Wolfgang Heidrich
  • This slide has a 16:9 media window
  • viewer moves rightThis slide has a 16:9 media window viewer moves down 4D Light Field
  • Compressive Displays ComputationalDisplay Optics Processing
  • This slide has a 16:9 media window Display-adaptive Compression Computed Tomography Nonnegative Tensor4D Light Field Factorization Compressive Optics Uniform or Directional Backlight Stacked Layers (LCDs or Transparencies)
  • Prototype – Layered 3D, SIGGRAPH 2011 Attenuation Layers with Spacers Backlight
  • Video clip
  • Prototype – Tensor Display, SIGGRAPH 2012
  • Video clip
  • What do we mean by “glasses-free 3D”?binocular disparity convergence motion parallax accommodation/blur current glasses-based (stereoscopic) displays near-term glasses-free (light field) displays longer-term holographic displays
  • Is glasses-free 3D Technology ready? Nintendo 3DS MasterImage 3D Asus Eee Pad MeMO 3D LG Optimus 3D E3 2010 Computex 2011 Computex 2011 Mobile World Congress 2011 Toshiba 3DTV Prototype Sony 3DTV Prototype LG 3DTV Prototype CES 2011 CES 2011 CES 2011
  • Parallax Barriers – Ives 1903barrier2D display  Low resolution & very dim  Switchable 2D/3D with LCDs
  • Parallax Barriers – Ives 1903barrier Nintendo 3DS2D display  Low resolution & very dim  Switchable 2D/3D with LCDs
  • lenslets Integral Imaging – Lippmann 1908 2D display  Brighter than parallax barriers  Always low resolution, even for 2D
  • lenslets Integral Imaging – Lippmann 1908 2D display Alioscopy 3DHD 42 (1920x1200, 1x8 views)  Brighter than parallax barriers  Always low resolution, even for 2D
  • Directional Backlighting – 3M & MS Wedge 3M Directional Backlight Film Nelson and Brott, 2010 US Patent 7,847,869LED thin light guide LED  Requires 120 Hz for stereo  Not practical for multiview Microsoft Wedge
  • Directional Backlighting Lenslet array
  • Glasses-Free 3D Display LightSpace Sony Jones et al. 2007 Zebra Imaging MIT HolovideoHolograms DisplaysVolumetric  3Ddepth cues inside enclosure all objects only  opticallymechanically moving parts  mostly & computationally expensive  3D objects outside enclosure inexpensive off-the-shelf partsCompressive LF Displays  no moving parts efficient computationally
  • From Conventional to Compressive 3D Displays mask 2 mask 1 Conventional Parallax Barriers t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays mask 2 mask 1 Time-shifted Parallax Barriers [Kim et al. 2007] High Resolution through High Speed t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays Perceptual Integration time Time-shifted Parallax Barriers [Kim et al. 2007] High Resolution through High Speed t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays mask 2 mask 1 High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010] Compression in Time – Nonnegative Matrix Factorization t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays Perceptual Integration time High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010] Compression in Time – Nonnegative Matrix Factorization t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth – Computed Tomography t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth – Computed Tomography t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels – Computed Tomography t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays Perceptual Integration … … … time Tensor Displays Compression in Time & Pixels –Tensor Factorization t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays Perceptual Integration … … … time Tensor Displays – Multilayer & Directional Backlighting t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • From Conventional to Compressive 3D Displays Perceptual Integration thin! time Tensor Displays – Directional Backlighting t t tParallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays 1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
  • Prototype – Layered 3D, SIGGRAPH 2011 Attenuation Layers with Spacers Backlight
  • Computed Tomography (CT) x-ray sensorsource: wikipedia 3D Reconstruction x-ray source Reconstructed 2D Slices 35
  • Tomographic Light Field Synthesis Image Formation Virtual Planes - ò c m (r )dr x L(x, q ) = e Attenuation Volume log L x, (r )dr cBacklight Tomographic Synthesis 2D Light Field log( L ) P 2 argmin log( L) P 2 0 x 36
  • CT vs. Layered 3D Computed Tomography Layered 3D reconstruct physical volume  thin stack of optimized layers sensor noise  no noise 37
  • Multi-Layer Decomposition viewer moves rightviewer moves down Input 4D Light Field 1 2 3 1 2 4 3 5 4 5 Photographs of Prototype Optimized Attenuation Layers
  • Depth of Field for 3D Displays Integral Imaging Parallax Barriers Cutoff (cycles/cm) Maximum Resolution Display Thickness Zwicker et al. 2006 Antialiasing + Display Prefilter Distance of Virtual Plane from Middle of Display (cm)
  • How Do Layers Increase Depth of Field? Integral Imaging Parallax Barriers Layered 3D Cutoff (cycles/cm) ? Maximum Resolution Display Thickness Distance of Virtual Plane from Middle of Display (cm)
  • Optimization: Number of Layers Two Layers Three Layers Five Layers
  • Optimization: Display Thickness Average Reconstruction PSNR for All Scenes PSNR in dB Number of Layers
  • Application to HDR Display “Square Root” Layers
  • Application to HDR Display “Square Root” Layers
  • Application to HDR Display Optimized Layers
  • Video clip
  • Limitations: Field of ViewFOV 10º FOV 20º FOV 45º
  • Personal Glasses-Free 3D Display Challenges for dynamic display:  Real-time computation  Engineering issues, moiré
  • Multi-Layer LCD – SIGGRAPH ASIA 2011
  • Barco E-2320 PAGrayscale IPS LCD1600x1200 @ 60 Hz
  • Four Stacked Liquid Crystal Panels Two Crossed Polarizers
  • Overview of LCDs I vertical polarizer color filter array liquid crystal cells horizontal polarizer I0 backlight Malus’ Law Intensity Modulation with Liquid Crystal Cells I = I0 sin2 (q )
  • Extending Layered 3D to Multi-layer LCDs Virtual Planes Design Optimization LCD • Eliminate redundant polarizers 3  Sequentially-crossed design LCD x 2 LCD 1 backlight 2D Light Field x
  • Extending Layered 3D to Multi-layer LCDs Virtual Planes Design Optimization LCD • Eliminate redundant polarizers 3  Use sequentially-crossed LCD x • Exploit field-sequential color 2  0.33 = 2.7% brightness LCD 1 backlight 2D Light Field x
  • Polarization Field Displays Virtual Planes Design Optimization LCD • Eliminate redundant polarizers 3  Use sequentially-crossed LCD x • Exploit field-sequential color 2  0.33 = 2.7% brightness LCD • Further optimize polarizers 1  Minimum is a crossed pair backlight 2D Light Field x
  • Modeling and Synthesizing Polarization Fields Virtual Planes LCD f3 3 Image Formation f2 K LCD x Q(x, q ) = åfk (x, q ) 2 k=1 f1 L(x, q ) = sin 2 (Q(x,q )) LCD 1 backlight Tomographic Synthesis Q(x,q ) = ±sin-1 ( ) L(x, q ) mod p 2D Light Field Q = Pf argmin Q - Pf 2 2 fmin £f £fmax x
  • Decompositions & Reconstructions viewer moves right viewer moves down Stacked Polarization Input 4D Light Field Rotating Layers 90° Optimized Rotation Angles for Each Layer 0°
  • Decompositions & Reconstructions viewer moves right viewer moves down Input 4D Light Field Reconstruction Results
  • Multi-layer LCD Attenuation Layers Polarization-Rotating Layers
  • Video clip
  • Tomographic Image Synthesis Target Light Field Projection Matrix LCD Pixel Values = * b=Ax
  • SART - Simultaneous Algebraic Reconstruction Technique b=Ax pre-compute some weights initial guess ATv Ax update clamp
  • Implementing Ax as Multiview Rendering
  • Implementing ATv as Projective Texture Mapping
  • Benefits & Limitations
  • Light Field “Slice” Representation Light Field moving to the right
  • Light Field “Slice” Representation Light Field Light Field Slice moving to the left
  • Light Field “Slice” Representation Light Field View from Above Light Field Slice moving to the left
  • Light Field “Slice” Representation Light FieldMultilayer Light Field Display Light Field Slice moving to the leftFront LayerMiddle LayerRear LayerBacklight
  • Light Field “Slice” Representation Light FieldMultilayer Light Field Display Light Field Slice L( moving to the leftFront Layer fm(3)(Middle Layer fm(2)( L(Rear Layer fm(1)(Backlight
  • Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field Tensor L(Front Layer Rear Layer fm(3)(Middle Layer L( fm(2)(Rear Layer fm(1)(Backlight
  • Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field Tensor L(Front Layer Rear Layer fm(3)(Middle Layer L( fm(2)(Rear Layer fm(1)(Backlight
  • Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field TensorFront Layer Rear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight
  • Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field TensorFront Layer Rear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight
  • Light Field Tensor Representation Light FieldMultilayer Light Field Display Light Field TensorFront Layer Rear Layer fm(3)(Middle Layer fm(2)(Rear Layer fm(1)(Backlight
  • Light Field Tensor DecompositionTarget Light Field Tensor Rank-M Approximation Nonnegative Tensor Perceptual Factorization (NTF) Integration + + ... + Frame 1 Frame 2 Frame M
  • Light Field Tensor DecompositionTarget Light Field Tensor Rank-M Approximation Nonnegative Tensor Perceptual Factorization (NTF) Integration + + ... + Frame 1 Frame 2 Frame M
  • Light Field Tensor Decomposition Nonlinear (Multilinear) Optimization Problem Iterative Update Rules (see paper for details) Efficient GPU ImplementationForward Projection (Multiview Rendering) Back Projection (Projective Texture Mapping)
  • Design Tradespace: Layers vs. Frames PSNR without Directional Backlight# layers # frames PSNR with Directional Backlight# layers # frames
  • Design Tradespace: Layers vs. FramesPSNR without Directional Backlight # layers # frames PSNR with Directional Backlight # layers # frames 2 Layers, Layers,Directional–Backlight (Tensor Display) 3 Layers, 1 Frame Frames 3D – SIGGRAPH 2010) 1 Layer, 3 3 3 Frames(Layered (Tensor Display) 2011) Frames, 3 (HR3D SIGGRAPH Asia Original Original 2 Layers, 3 Frames 3 Layers, 1 Frame 3 Layers, 3 Frames 1 L, 3 F, Directional BL
  • Tensor Display Prototypes Reconfigurable Directional Backlight Three Layer
  • Tensor Display Prototypes 3 Layer LCD Directional BacklightHardware
  • Three Layer Prototype Video clip
  • Video clip
  • Directional Backlight Prototype Video clip
  • LCD + Directional BLView from above LCD with Directional Backlight, Rank 6 Directional BL
  • LCD with Directional Backlight, Rank 6 (as seen by obserer) Video clip
  • LCD with Directional Backlight, Rank 6 (as seen by obserer) Video clip
  • Filmed with High-speed Camera Directional Backlight Video clip Front LCD
  • Video clipLenslets only LCD+DBL Rank 1 LCD+DBL Rank 6
  • Limitation: Compressible Light Fields “Natural” 4D Light Field Random 4D Light Field
  • Next-generation Technology What about Content? Computational PhotographyConsumer Light Field CamerasRendered Footage Computational Displays Camera Rigs
  • SIGGRAPH 2012 Course on Computational Displays Code & Datasets onlineUse Layered 3D in your class! media.mit.edu/~gordonw cameraculture.media.mit.edu