This document summarizes research on compressive light field displays. It begins by introducing the concept of light field displays and their collaborators. It then provides examples of prototypes from layered 3D displays to tensor displays. Finally, it outlines the evolution of display technologies from conventional parallax barriers to the most recent compressive displays that use techniques like nonnegative matrix factorization and computed tomography to achieve compression in time, pixels, and depth for glasses-free 3D viewing.
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
Compressive Light Field Displays
1. Compressive
This slide has a 16:9 media window
Light Field Displays
Gordon Wetzstein - MIT Media Lab
Collaborators: Doug Lanman,
Matt Hirsch, Ramesh Raskar, Wolfgang Heidrich
6. This slide has a 16:9 media window
Display-adaptive
Compression
Computed Tomography
Nonnegative Tensor
4D Light Field Factorization
Compressive Optics
Uniform or
Directional Backlight Stacked Layers
(LCDs or Transparencies)
11. What do we mean by “glasses-free 3D”?
binocular disparity convergence motion parallax accommodation/blur
current glasses-based (stereoscopic) displays
near-term glasses-free (light field) displays
longer-term holographic displays
12. Is glasses-free 3D Technology ready?
Nintendo 3DS MasterImage 3D Asus Eee Pad MeMO 3D LG Optimus 3D
E3 2010 Computex 2011 Computex 2011 Mobile World Congress 2011
Toshiba 3DTV Prototype Sony 3DTV Prototype LG 3DTV Prototype
CES 2011 CES 2011 CES 2011
13.
14. Parallax Barriers – Ives 1903
barrier
2D display
Low resolution & very dim
Switchable 2D/3D with LCDs
15. Parallax Barriers – Ives 1903
barrier
Nintendo 3DS
2D display
Low resolution & very dim
Switchable 2D/3D with LCDs
16. lenslets
Integral Imaging – Lippmann 1908
2D display
Brighter than parallax barriers
Always low resolution, even for 2D
17. lenslets
Integral Imaging – Lippmann 1908
2D display Alioscopy 3DHD 42''
(1920x1200, 1x8 views)
Brighter than parallax barriers
Always low resolution, even for 2D
18. Directional Backlighting – 3M & MS Wedge
3M Directional Backlight Film
Nelson and Brott, 2010
US Patent 7,847,869
LED thin light guide LED
Requires 120 Hz for stereo
Not practical for multiview Microsoft Wedge
20. Glasses-Free 3D Display
LightSpace Sony Jones et al. 2007 Zebra Imaging MIT Holovideo
Holograms Displays
Volumetric
3Ddepth cues inside enclosure
all objects only
opticallymechanically moving parts
mostly & computationally expensive
3D objects outside enclosure
inexpensive off-the-shelf parts
Compressive LF Displays
no moving parts efficient
computationally
21. From Conventional to Compressive 3D Displays
mask 2
mask 1
Conventional Parallax Barriers
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
22. From Conventional to Compressive 3D Displays
mask 2
mask 1
Time-shifted Parallax Barriers [Kim et al. 2007]
High Resolution through High Speed
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
23. From Conventional to Compressive 3D Displays
Perceptual Integration
time
Time-shifted Parallax Barriers [Kim et al. 2007]
High Resolution through High Speed
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
24. From Conventional to Compressive 3D Displays
mask 2
mask 1
High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010]
Compression in Time – Nonnegative Matrix Factorization
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
25. From Conventional to Compressive 3D Displays
Perceptual Integration
time
High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010]
Compression in Time – Nonnegative Matrix Factorization
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
26. From Conventional to Compressive 3D Displays
mask K
…
mask 2
mask 1
Layered 3D [Wetzstein et al., SIGGRAPH 2011]
Compression in Pixels & Depth – Computed Tomography
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
27. From Conventional to Compressive 3D Displays
mask K
…
mask 2
mask 1
Layered 3D [Wetzstein et al., SIGGRAPH 2011]
Compression in Pixels & Depth – Computed Tomography
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
28. From Conventional to Compressive 3D Displays
mask K
…
mask 2
mask 1
Layered 3D [Wetzstein et al., SIGGRAPH 2011]
Compression in Pixels – Computed Tomography
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
29. From Conventional to Compressive 3D Displays
Perceptual Integration
…
…
…
time
Tensor Displays
Compression in Time & Pixels –Tensor Factorization
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
30. From Conventional to Compressive 3D Displays
Perceptual Integration
…
…
…
time
Tensor Displays – Multilayer & Directional Backlighting
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
31. From Conventional to Compressive 3D Displays
Perceptual Integration
thin!
time
Tensor Displays – Directional Backlighting
t
t t
Parallax Barriers Time-Shifted HR3D Layered 3D Tensor Displays
1903 Parallax Barriers 2007 SIG Asia 2010 SIGGRAPH 2011 SIGGRAPH 2012
35. Computed Tomography (CT)
x-ray sensor
source: wikipedia
3D Reconstruction
x-ray source
Reconstructed 2D Slices
35
36. Tomographic Light Field Synthesis
Image Formation
Virtual Planes
- ò c m (r )dr
x
L(x, q ) = e
Attenuation Volume log L x, (r )dr
c
Backlight
Tomographic Synthesis
2D Light Field
log( L ) P
2
argmin log( L) P 2
0
x 36
37. CT vs. Layered 3D
Computed Tomography Layered 3D
reconstruct physical volume thin stack of optimized layers
sensor noise no noise
37
38. Multi-Layer Decomposition
viewer moves right
viewer moves down
Input 4D Light Field
1
2
3 1
2
4 3
5
4
5 Photographs of Prototype
Optimized Attenuation Layers
39. Depth of Field for 3D Displays
Integral Imaging Parallax Barriers
Cutoff (cycles/cm)
Maximum Resolution
Display Thickness
Zwicker et al. 2006 Antialiasing + Display Prefilter
Distance of Virtual Plane from Middle of Display (cm)
40. How Do Layers Increase Depth of Field?
Integral Imaging Parallax Barriers Layered 3D
Cutoff (cycles/cm)
?
Maximum Resolution
Display Thickness
Distance of Virtual Plane from Middle of Display (cm)
69. Light Field “Slice” Representation
Light Field
Light Field Slice
moving to the left
70. Light Field “Slice” Representation
Light Field
View from Above Light Field Slice
moving to the left
71. Light Field “Slice” Representation
Light Field
Multilayer Light Field Display Light Field Slice
moving to the left
Front Layer
Middle Layer
Rear Layer
Backlight
72. Light Field “Slice” Representation
Light Field
Multilayer Light Field Display Light Field Slice
L(
moving to the left
Front Layer
fm(3)(
Middle Layer
fm(2)(
L(
Rear Layer
fm(1)(
Backlight
73. Light Field Tensor Representation
Light Field
Multilayer Light Field Display Light Field Tensor
L(
Front Layer
Rear Layer
fm(3)(
Middle Layer
L(
fm(2)(
Rear Layer
fm(1)(
Backlight
74. Light Field Tensor Representation
Light Field
Multilayer Light Field Display Light Field Tensor
L(
Front Layer
Rear Layer
fm(3)(
Middle Layer L(
fm(2)(
Rear Layer
fm(1)(
Backlight
75. Light Field Tensor Representation
Light Field
Multilayer Light Field Display Light Field Tensor
Front Layer
Rear Layer
fm(3)(
Middle Layer
fm(2)(
Rear Layer
fm(1)(
Backlight
76. Light Field Tensor Representation
Light Field
Multilayer Light Field Display Light Field Tensor
Front Layer
Rear Layer
fm(3)(
Middle Layer
fm(2)(
Rear Layer
fm(1)(
Backlight
77. Light Field Tensor Representation
Light Field
Multilayer Light Field Display Light Field Tensor
Front Layer
Rear Layer
fm(3)(
Middle Layer
fm(2)(
Rear Layer
fm(1)(
Backlight
78. Light Field Tensor Decomposition
Target Light Field Tensor Rank-M Approximation
Nonnegative Tensor Perceptual
Factorization (NTF) Integration
+ + ... +
Frame 1 Frame 2 Frame M
79. Light Field Tensor Decomposition
Target Light Field Tensor Rank-M Approximation
Nonnegative Tensor Perceptual
Factorization (NTF) Integration
+ + ... +
Frame 1 Frame 2 Frame M
80. Light Field Tensor Decomposition
Nonlinear (Multilinear)
Optimization Problem
Iterative Update Rules
(see paper for details)
Efficient GPU Implementation
Forward Projection (Multiview Rendering) Back Projection (Projective Texture Mapping)
81.
82. Design Tradespace: Layers vs. Frames
PSNR without Directional Backlight
# layers
# frames
PSNR with Directional Backlight
# layers
# frames
83. Design Tradespace: Layers vs. Frames
PSNR without Directional Backlight
# layers
# frames
PSNR with Directional Backlight
# layers
# frames 2 Layers, Layers,Directional–Backlight (Tensor Display)
3 Layers, 1 Frame Frames 3D – SIGGRAPH 2010)
1 Layer, 3 3 3 Frames(Layered (Tensor Display) 2011)
Frames, 3 (HR3D SIGGRAPH Asia
Original
Original 2 Layers, 3 Frames 3 Layers, 1 Frame 3 Layers, 3 Frames 1 L, 3 F, Directional BL
95. Next-generation Technology
What about Content?
Computational Photography
Consumer Light Field Cameras
Rendered Footage Computational Displays
Camera Rigs
96. SIGGRAPH 2012 Course on
Computational Displays
Code & Datasets online
Use Layered 3D in your class!
media.mit.edu/~gordonw
cameraculture.media.mit.edu
Editor's Notes
[Gordon resumes at this slide.]
Multi-layer 3D display is a computed tomography problem. (Multi-Layer is 3D)Natural light fields are compressible, so a thin display with few layers can achieve the illusion of a thick virtual scene. (thin display creates thick scenes)So this leads to a new type of computational/compressive display.[probably same slide as personal device]
Basically, the tomographic light field synthesis then boils down to solving a linear equation system of the form Ax=b. b is the target light field, A the projection matrix, and the unknowns x are the LCD pixel values
This equation system can be solved with SART – a technique developed, tested, and refined over decades in the medical imaging community.Algorithm is very simple:Initialize some data and find an initial guess of the pixel valuesIteratively update and clamp the solutionUpdate rules require two important operations: a function computing the matrix-vector multiplication Ax, and a function computing the transpose matrix-vector multiplication ATv
Interpreting AX – is basically a multiview rendering step. In OpenGL just set the projection matrix to all different views of the light field and render the LCD layers textured with the given patterns.
Interpreting AX – is basically a multiview rendering step. In OpenGL just set the projection matrix to all different views of the light field and render the LCD layers textured with the given patterns.
Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as wellPossibly use static framesEmphasize