Keynote - SPIE Stereoscopic Displays & Applications 2014

0 views
946 views

Published on

Keynote sludes

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
0
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
55
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
  • Use the original video clips in higher res and the same layout but better quality and not this!Get rid of the gray background as well
  • Keynote - SPIE Stereoscopic Displays & Applications 2014

    1. 1. Compressive Displays Combining Optical Fabrication, Computation, and Perceptual Tricks to Build the Displays of the Future Gordon Wetzstein MIT Media Lab media.mit.edu/~gordonw displayblocks.org
    2. 2. Evolution? This slide has a 16:9 media window 1928 2013
    3. 3. displayblocks.org
    4. 4. Images: Wikipedia, Shinya Yoshioka
    5. 5. Nature This slide has a 16:9 media window Image: Desafio Monteverde and Arenal Volcano Tours Costa Rica
    6. 6. Nature Image: National Geographic
    7. 7. Three-layer Tensor Display
    8. 8. This slide has a 16:9 media window
    9. 9. This slide has a 16:9 media window Compressive Light Field Displays
    10. 10. This slide has a 16:9 media window
    11. 11. viewer moves right viewer moves down This slide has a 16:9 media window 4D Light Field
    12. 12. Compressive Displays Display Optics Computational Processing
    13. 13. Some Depth Cues of the HVS binocular disparity convergence current glasses-based (stereoscopic) displays near-term: compressive light field displays longer-term: holographic displays motion parallax accommodation/blur
    14. 14. Parallax Barriers – Ives 1903 barrier 2D display • low resolution & very dim • switchable 2D/3D with LCDs
    15. 15. lenslets Integral Imaging – Lippmann 1908 2D display • brighter than parallax barriers • always low resolution, even for 2D
    16. 16. Next-generation Displays computation optics & electronics interaction human visual system
    17. 17. Structural Formula for Compressive Display Innovation M N C N U C O C multiplexing nonlinear compression optimization user experience content community next
    18. 18. I. Multiplexing
    19. 19. Multiplexing Most displays are 2D! Spatial Multiplexing
    20. 20. Space vs Time visual acuity / retina display: 20/20 = 1 arc minute @ 10” need >300 dpi multiplexing requires 300++ dpi flicker fusion threshold: 20-60 Hz (depends on light) LCDs: 120-240 Hz DMDs, ferroelectric LCDs: KHz much easier with currently available hardware!
    21. 21. Combine Space and Time mask 2 mask 1 Conventional Parallax Barriers
    22. 22. Combine Space and Time mask 2 mask 1 Time-shifted Parallax Barriers [Kim et al. 2007] High Resolution through High Speed
    23. 23. I. Multiplexing Use temporal multiplexing if you can!
    24. 24. II. Nonlinear Pixel Interaction & Coupling
    25. 25. mask 2 mask 1 Conventional Parallax Barriers
    26. 26. nonlinear l = p1*p2 p2 mask 2 mask 1 blocked! p1 Conventional Parallax Barriers
    27. 27. linear l = p1+p2 p2 mask 2 mask 1 not blocked! p1 Additive Layers / Most Volumetric Displays e.g., LEDs or transparent OLEDs
    28. 28. Holography – Nonlinear Interaction plane wave emitted wavefront: U(x) received intensity: I(x) = T {U(x)} 2 = Re {T {U(x)}} + Im {T {U(x)}} 2 hologram 2 screen or retina
    29. 29. Holography – Coupling plane wave æ x'ö æ x'ö W (x, u = l sin(q )) = ò t ç x + ÷ t ç x - ÷ e2 p ix'u dx ' è 2ø è 2ø Fourier transform of all points interacting with each other! hologram nonlinear interaction & pixel coupling!
    30. 30. Layered 3D – SIGGRAPH 2011 attenuation layers with spacers backlight
    31. 31. Layered 3D – SIGGRAPH 2011 mask K … mask 2 mask 1
    32. 32. Layered 3D – SIGGRAPH 2011 l = p1*p2*…*pK mask K pK … mask 2 mask 1 p2 p1
    33. 33. Layered 3D – SIGGRAPH 2011 mask K … mask 2 mask 1
    34. 34. Tensor Displays – SIGGRAPH 2012 Reconfigurable directional backlight three layer
    35. 35. Three Layer Prototype
    36. 36. Directional Backlight Prototype
    37. 37. Tensor Displays – Directional Backlight Perceptual Integration mask 2 microlens array mask 1 time
    38. 38. view from above LCD with directional backlight, rank 6 LCD + directional BL conventional lenslets
    39. 39. LCD with directional backlight, rank 6 (as seen by obserer)
    40. 40. Filmed with High-speed Camera directional backlight front LCD
    41. 41. Polarization Fields – SIGGRAPH ASIA 2011
    42. 42. four stacked liquid crystal panels two crossed polarizers
    43. 43. Polarization Fields – SIGGRAPH ASIA 2011 design optimization LCD 3 LCD 2 LCD 1 backlight • eliminate redundant polarizers  sequentially-crossed design
    44. 44. Polarization Fields – SIGGRAPH ASIA 2011 design optimization LCD 3 • eliminate redundant polarizers  use sequentially-crossed LCD 2 • exploit field-sequential color  0.33 = 2.7% brightness LCD 1 backlight
    45. 45. Polarization Fields – SIGGRAPH ASIA 2011 design optimization LCD 3 • eliminate redundant polarizers  use sequentially-crossed LCD 2 • exploit field-sequential color  0.33 = 2.7% brightness LCD 1 • further optimize polarizers  minimum is a crossed pair backlight
    46. 46. Polarization Fields – SIGGRAPH ASIA 2011 f3 LCD 3 f2 LCD 2 f1 LCD 1 backlight image formation K Q(x, q ) = åfk (x, q ) k=1 L(x, q ) = sin 2 (Q(x,q ))
    47. 47. II. Nonlinear Pixel Interaction & Coupling Design optical systems that provide nonlinear pixel coupling!
    48. 48. III. Compression
    49. 49. Which Light Field is More Compressible? a b
    50. 50. Which Light Field is More Compressible? a b
    51. 51. Compressive Optics 16:9 media window This slide has a Display-adaptive Compression Computed Tomography 4D Light Field Nonnegative Tensor Factorization Uniform or Directional Backlight Stacked Layers (LCDs or Transparencies) Observer = Decoder
    52. 52. Give those Pixels a Break! Applied Mathematics Benefits for Optics & Electronics • sparse optimization • fewer pixels • low-rank factorization • relaxation on refresh rate • … • thinner form factors • …
    53. 53. From Conventional to Compressive Displays mask 2 mask 1 Conventional Parallax Barriers t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    54. 54. From Conventional to Compressive Displays mask 2 mask 1 Time-shifted Parallax Barriers [Kim et al. 2007] High Resolution through High Speed t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    55. 55. From Conventional to Compressive Displays Perceptual Integration time Time-shifted Parallax Barriers [Kim et al. 2007] High Resolution through High Speed t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    56. 56. From Conventional to Compressive Displays mask 2 mask 1 High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010] Compression in Time – Nonnegative Matrix Factorization t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    57. 57. From Conventional to Compressive Displays Perceptual Integration time High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010] Compression in Time – Nonnegative Matrix Factorization t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    58. 58. From Conventional to Compressive Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth – Computed Tomography t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    59. 59. From Conventional to Compressive Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels & Depth – Computed Tomography t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    60. 60. From Conventional to Compressive Displays mask K … mask 2 mask 1 Layered 3D [Wetzstein et al., SIGGRAPH 2011] Compression in Pixels – Computed Tomography t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    61. 61. From Conventional to Compressive Displays … Perceptual Integrat … … time Tensor Displays Compression in Time & Pixels –Tensor Factorization t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    62. 62. From Conventional to Compressive Displays … Perceptual Integrat … … time Tensor Displays – Multilayer & Directional Backlighting t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    63. 63. From Conventional to Compressive Displays thin! Perceptual Integrat time Tensor Displays – Directional Backlighting t Parallax Barriers 1903 Time-Shifted Parallax Barriers 2007 HR3D SIG Asia 2010 t Layered 3D SIGGRAPH 2011 Tensor Displays SIGGRAPH 2012 t
    64. 64. III. Compression Design new optical architectures that directly exploit data compression!
    65. 65. IV. Optimization via High-performance Computing
    66. 66. Layered 3D – SIGGRAPH 2011 attenuation layers with spacers backlight
    67. 67. Computed Tomography (CT) This slide has a 16:9 media window Image: Wikipedia x-ray sensor 3D Reconstruction x-ray source Reconstructed 2D Slices
    68. 68. Computed Tomography (CT) This slide has a 16:9 media window
    69. 69. Tomographic Light Field Synthesis image formation virtual planes x L(x, q ) = e - ò c m (r )dr log ( L ( x, q )) = - ò m (r)dr attenuation volume c backlight tomographic synthesis 2D light field argmin log( L) P 0 x 2 2
    70. 70. Tomographic Polarization Field Synthesis virtual planes f3 LCD 3 f2 LCD 2 image formation K x f1 Q(x, q ) = åfk (x, q ) k=1 L(x, q ) = sin 2 (Q(x,q )) LCD 1 tomographic synthesis backlight Q(x,q ) = ±sin-1 2D light field ( argmin Q - Pf fmin £f £fmax x ) L(x, q ) mod p 2 2
    71. 71. Low-rank Light Field Factorization light field two-layer light field display L( front layer fm(2)( rear layer backlight fm(1)(
    72. 72. Low-rank Light Field Factorization two-layer light field display front layer front layer fm(2)( rear layer backlight Lanman et al. – SIGGRAPH Asia 2010 fm(1)( rear layer L( ` rank-1
    73. 73. Low-rank Light Field Factorization high-speed LCDs = perceptual average two-layer light field display L( front layer fm(2)( rear layer backlight Lanman et al. – SIGGRAPH Asia 2010 fm(1)( ` rank-4
    74. 74. Low-rank Light Field Factorization high-speed LCDs = perceptual integration G ~ L FG Lanman et al. – SIGGRAPH Asia 2010 F ~ ` L rank-4
    75. 75. Low-rank Light Field Factorization arg min L - FG Wfield F,G ³ 0 , for emitted light 2 F,G ` L = G F
    76. 76. Low-rank Light Field Factorization light field light field tensor multi-layer light field display L( fm(3)( middle layer fm(2)( rear layer backlight fm(1)( rear layer front layer L(
    77. 77. Low-rank Light Field Factorization Target Light Field Tensor Rank-M Approximation nonnegative tensor factorization (NTF) + ... + + frame 1 perceptual integration frame 2 frame M
    78. 78. Low-rank Light Field Factorization nonlinear (multilinear) optimization problem iterative update rules (see paper for details) Efficient GPU Implementation forward projection (multiview rendering) back projection (projective texture mapping)
    79. 79. IV. Optimization via High-performance Computing Find optimal pixel states via optimization!
    80. 80. V. User Experience
    81. 81. medical? Nvidia University of Louisiana entertainment? Gregg Favalora’s Perspecta James Cameron’s Avatar What’s the Killer App of 3D Displays? scientific visualization? games?
    82. 82. (Interactive) Experiences! leap motion ms kinect
    83. 83. Vision-correcting Display
    84. 84. unfortunately, this content is not public at the moment
    85. 85. V. User Experience Design user experiences, not devices!
    86. 86. VI. Community
    87. 87. displayblocks.org Code, data & instructions online!
    88. 88. VI. Community Let’s share our knowledge!
    89. 89. VII. Content
    90. 90. Content Generation Live Action consumer light field cameras camera rigs Rendered Footage
    91. 91. Lytro Light Field Camera
    92. 92. sensor Lytro Light Field Camera lenslets
    93. 93. Compressive Light Field Photography – SIG 2013 Exploit redundancy Sparsify Preserve information Compressive Reconstruction Light Field Atoms Mask-based Light Field Coding
    94. 94. Prototype Setup with a Variable Mask LCoS Virtual sensor Polarizing Imaging Beamsplitter Lens Camera Image sensor
    95. 95. Display-Adaptive Image Synthesis – SIG 2013 “conventional pipeline” 1. render trillions of rays 2. large-scale optimization 3. display pixel states
    96. 96. Display-Adaptive Image Synthesis – SIG 2013
    97. 97. Display-Adaptive Image Synthesis – SIG 2013 samples simulation prototype 1-3% of rendered & optimized samples = same quality
    98. 98. VII. Content Content is crucial! How you capture / render it too!
    99. 99. VIII. What’s Next?
    100. 100. Accommodative Depth Cues
    101. 101. Transactions on Graphics 2013
    102. 102. Compressive Superresolution Display
    103. 103. unfortunately, this content is not public at the moment
    104. 104. Compressive Light Field Projection
    105. 105. unfortunately, this content is not public at the moment
    106. 106. Head-mounted Displays Google Glass Oculus Rift Avegant Glyph Nvidia Near Eye Display
    107. 107. Structural Formula for Compressive Display Innovation M N C N U C O C multiplexing nonlinear compression optimization user experience content community next
    108. 108. Gordon Wetzstein MIT Media Lab media.mit.edu/~gordonw displayblocks.org collaborators: Matt Hirsch (MIT) Doug Lanman (NVIDIA) Andrew Maimone (UNC) Felix Heide (UBC) Fu-Chung Huang (UC Berkeley) Vincent Lee (MIT) James Gregson (UBC) Brian Barsky (UC Berkeley) Henry Fuchs (UNC) Ramesh Raskar (MIT) Wolfgang Heidrich (UBC) code, data & instructions online use Layered 3D in your class!

    ×