This document summarizes Gordon Wetzstein's presentation on compressive display systems. It discusses the evolution of displays from parallax barriers in 1903 to modern tensor displays. Computational techniques like low-rank light field factorization and tensor factorization can be used to compress light fields and reduce the number of pixels needed in multi-layer displays. These compressive displays integrate optimization, sensing, and human perception to provide capabilities like high dynamic range, super resolution, and 3D displays using a single display system. Wetzstein envisions these compressive displays enabling new form factors and applications in mobile devices, projections systems, and head mounted displays.

1. Compressive Display Systems
Gordon Wetzstein
MIT / Stanford University
media.mit.edu/~gordonw
displayblocks.org

3. This slide has a 16:9 media window
Evolution?
1928 2014

4. Nature
Image: National Geographic
Evolution!

5. uberpixel

6. Next-generation Devices
computation optics & electronics human visual
system
interaction
optics (compressive) computationsensing
Computational &
Compressive Displays
Computational
Imaging

7. displayblocks.org

8. Images: Wikipedia, Shinya Yoshioka

9. This slide has a 16:9 media window
Nature
Image: Desafio Monteverde and Arenal Volcano Tours Costa Rica

10. Three-layer Tensor Display

11. This slide has a 16:9 media window
Compressive Light
Field Displays

12. This slide has a 16:9 media window

13. This slide has a 16:9 media window
viewer moves right
viewermovesdown
4D Light Field

14. Display Optics
Computational
Processing
Compressive Displays

16. 2D display
barrier
Parallax Barriers – Ives 1903
• low resolution & very dim

17. Integral Imaging – Lippmann 1908
• low-res, but brighter than parallax barriers
2D display
lenslets

18. patent drawings - early 20th century

19. multiplexing
nonlinear
compression
optimization
user experience
content
community
next
M
N
O
C
U C
C
N
Structural Formula for Compressive Displays

20. multiplexing
nonlinear
compression
optimization
user experience
content
community
next
M
N
O
C
U C
C
N
Nonlinear Pixel Interaction & Coupling

21. layer 2
layer 1
Conventional Parallax Barriers

22. layer 2
layer 1
Conventional Parallax Barriers
blocked!
p1
p2
l = p1*p2
nonlinear

23. layer 2
layer 1
Most Volumetric Displays / Additive Layers
e.g., LEDs or transparent OLEDs
not blocked!
p1
p2
l = p1+p2
linear

24. Holography – Nonlinear Interaction
plane wave
hologram
emitted wavefront:
screen or retina
received intensity:
I(x) = T U(x){ }
2
= Re T U(x){ }{ }
2
+Im T U(x){ }{ }
2
U(x)

25. Holography – Coupling
plane wave
hologram
W(x,u = lsin(q)) = t x +
x'
2
æ
è
ç
ö
ø
÷t x -
x'
2
æ
è
ç
ö
ø
÷e2pix'u
dx'ò
Fourier transform of all points interacting
with each other!
nonlinear interaction & pixel coupling!

26. attenuation layers with
spacers
backlight
Layered 3D – SIGGRAPH 2011

27. Layered 3D – SIGGRAPH 2011
layer 2
layer K
layer 1
…

28. Layered 3D – SIGGRAPH 2011
layer 2
layer K
layer 1
…
p1
p2
pK
l = p1*p2*…*pK

29. Layered 3D – SIGGRAPH 2011
…
layer 2
layer K
layer 1

30. Tensor Displays – SIGGRAPH 2012
directional backlight three layer
Reconfigurable

32. time
Tensor Displays – Directional Backlight
Perceptual Integration
layer 2
microlens array
layer 1

33. LCD with directional backlight, rank 6
LCD + directional BL
conventional lenslets
view from above

34. LCD with directional backlight, rank 6 (as seen by observer)

35. front LCD
directional backlight
Filmed with High-speed Camera

36. four stacked liquid crystal panels
two crossed polarizers

37. LCD 1
LCD 2
LCD 3
backlight
Polarization Fields – SIGGRAPH ASIA 2011
polarizer
polarizer
color filter array
polarizer
color filter array
polarizer
color filter array

38. backlight
Polarization Fields – SIGGRAPH ASIA 2011
LCD 1
LCD 2
LCD 3
polarizer
polarizer

39. image formation
f1
f2
f3
L(x,q)=sin2
Q(x,q)( )
backlight
Polarization Fields – SIGGRAPH ASIA 2011
LCD 1
LCD 2
LCD 3

40. multiplexing
nonlinear
compression
optimization
user experience
content
community
next
M
N
O
C
U C
C
N
Compression

41. Which Light Field is More Compressible?
a b

42. Which Light Field is More Compressible?
a b

43. This slide has a 16:9 media window
4D Light Field
Uniform or
Directional Backlight Stacked Layers
Nonnegative Tensor
Factorization
Display-adaptive
Compression
Compressive
Computed Tomography
(LCDs or Transparencies)
Optics
Observer = Decoder

44. Give those Pixels a Break!
Applied Mathematics
• sparse optimization
• low-rank factorization
• computed tomography
• …
Benefits for Optics & Electronics
• fewer pixels
• relaxation on refresh rate
• thinner form factors
• …

45. Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
Conventional Parallax Barriers
layer 2
layer 1
From Conventional to Compressive Displays

46. …
…
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
…
time
Perceptual Integration
Tensor Displays – Multilayer & Directional Backlighting
From Conventional to Compressive Displays

47. Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
Perceptual Integration
Tensor Displays – Directional Backlighting
time
thin!
From Conventional to Compressive Displays

48. multiplexing
nonlinear
compression
optimization
user experience
content
community
next
M
N
O
C
U C
C
N
Real-time Optimization

49. This slide has a 16:9 media window
4D Light Field
Display-adaptive
Compression
Compressive
Optics
via optimization

50. attenuation layers with
spacers
backlight
Layered 3D – SIGGRAPH 2011

51. ?
• limited baseline tomography
• use algebraic approaches!
Layered 3D – SIGGRAPH 2011

52. This slide has a 16:9 media window
Computed Tomography (CT)
Image:Wikipedia
x-ray source
x-ray sensor
3D Reconstruction
Reconstructed 2D Slices

53. Tomographic Light Field Synthesis
q
2D light field
x
x
q
backlight
attenuation volume
virtual planes
image formation
L(x,q) = e
- m(r)dr
c
ò
log L x,q( )( )=- m(r)dr
c
ò
2
2
0
P)log(argmin 



L
tomographic synthesis

54. Limits of Tomographic Light Field Synthesis
image formation
L(x,q) = e
- m(r)dr
c
ò
log L x,q( )( )=- m(r)dr
c
ò
2
2
0
P)log(argmin 



L
tomographic synthesis
log space
…
p1
p2
pK
…
p1
p2
pK
time
l = p1*…*pK
…
p1
p2
pK
log(l) = log(p1)+…+log(pK)
l = (p1*p2*…*pK)1+…+(p1*p2*…*pK)N
loglin
???

55. backlight
rear layer
front layer
two-layer light field display
Low-rank Light Field Factorization
light field
fm
(1)(x1)
fm
(2)(x2)
x1
x2
L(x1, x2)

56. backlight
rear layer
front layer
two-layer light field display
fm
(1)(x1)
fm
(2)(x2)
x1
x2
L(x1, x2)
`
front layer
rearlayer
rank-1
Lanman et al. – SIGGRAPH Asia 2010
Low-rank Light Field Factorization

57. backlight
rear layer
front layer
two-layer light field display
fm
(1)(x1)
fm
(2)(x2)
x1
x2
L(x1, x2)
` rank-4
Lanman et al. – SIGGRAPH Asia 2010
high-speed LCDs = perceptual average
Low-rank Light Field Factorization

58. ` rank-4
F
G
L
~
Lanman et al. – SIGGRAPH Asia 2010
Low-rank Light Field Factorization
high-speed LCDs = perceptual average
arg min
F,G
L - FG W
2
, for F,G ³ 0
objective function:

59. fm
(1)(x1)
fm
(2)(x2)
fm
(3)(x3)
x1
x2
x3
L(x2, x3)
multi-layer light field display
backlight
rear layer
middle layer
front layer
light field
Light Field Slice Representation

60. fm
(1)(x1)
fm
(2)(x2)
fm
(3)(x3)
x1
x2
x3
L(x1,x2,x3)
RearLayer
x3
x2
x1
L(x1,x2,x3)
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation

61. fm
(1)(x1)
fm
(2)(x2)
fm
(3)(x3)
x1
x2
x3
RearLayer
x3
x2
x1
L(x1,x2,x3)
L(x1,x2,x3)
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation

62. fm
(1)(x1)
fm
(2)(x2)
fm
(3)(x3)
RearLayer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation

63. fm
(1)(x1)
fm
(2)(x2)
fm
(3)(x3)
RearLayer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation

64. fm
(1)(x1)
fm
(2)(x2)
fm
(3)(x3)
RearLayer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation

65. frame M
+ ... +
Target Light Field Tensor
frame 1
nonnegative tensor
factorization (NTF)
Rank-M Approximation
perceptual
integration
frame 2
+
Low-rank Tensor Factorization

66. iterative update rules
nonlinear (multilinear)
optimization problem
Low-rank Tensor Factorization
• standard form – Tensor Compendium
• multiplicative update keep factors positive
• basically steepest descent with fixed step length

67. “standard” formulationalternate formulation
Low-rank Tensor Factorization

68. alternate formulation
forward projection (multiview rendering) back projection (projective texture mapping)
Efficient GPU Implementation
Tensor Factorization - Implementation

70. multiplexing
nonlinear
compression
optimization
user experience
content
community
next
M
N
O
C
U C
C
N
Who cares?

71. Vision-correcting Display
perceived image
displayed image
SIGGRAPH 2014 - Display Session, Tue morning

72. Vision-correcting Display
iPod Touch prototypeprinted transparency
SIGGRAPH 2014 - Display Session, Tue morning

73. prototype construction
300 dpi or higher

74. vision-correcting displayconventional display

75. multiplexing
nonlinear
compression
optimization
user experience
content
community
next
M
N
O
C
U C
C
N
What’s next?

76. resolution
contrast
3D display capabilities

77. Compressive Multi-mode Display
Optics Express 2014

78. This slide has a 16:9 media window
High-speed LCD
High-speed LCD
Electronically-switchable
Diffuser

79. High-speed LCD
High-speed LCD
Electronically-switchable
Diffuser OFF
3D Display Mode

80. Light Field Factorization – LCD Patterns
Front Layer Rear Layer

81. Light Field Factorization – Results

82. High Dynamic Range Display Mode
High-speed LCD
High-speed LCD
Electronically-switchable
Diffuser OFF

83. Target HDR Image

84. Solver = Light Field without Parallax
Front Layer Rear Layer

85. ConventionalHigh Dynamic Range

86. Superresolution Display Mode
High-speed LCD
High-speed LCD
Electronically-switchable
Diffuser ON

87. Results from Prototype

88. Results from Prototype

89. Light Field Projection
SIGGRAPH 2014 – ETech & Paper

90. Compressive Light Field Projection

93. multiplexing
nonlinear
compression
optimization
user experience
content
community
next
N
O
C
U C
C
N
Structural Formula for Compressive Displays
M
Mobile Displays
Optics Express 2014
Projection Displays
SIGGRAPH 2014
Coded Illuminationfor
Microscopy & Lithography
Head Mounted Displays
CGF 2010, SIGGRAPH 2014
Monitors / TVs
SIG2011,2012,2013, SIGAsia 2009,2011

94. Cool Displays at SIGGRAPH 2014
Technical Papers Sessions
Emerging Technologies
• Displays, Tuesday 10:45-12:15, Hall A
• Computational Sensing and Display, Tue 3:45-5:15, Hall A
• AR & VR Displays
• Light Field Projection, HDR Projection
• much more!

95. Gordon Wetzstein
Massachusetts Institute of Technology
media.mit.edu/~gordonw
displayblocks.org
Matt Hirsch (MIT)
Doug Lanman (MIT/NVIDIA/Oculus VR)
Andrew Maimone (UNC)
Felix Heide (UBC)
Fu-Chung Huang (UC Berkeley)
Belen Masia (University of Zaragoza)
collaborators
sources of funding & hardware
Wolfgang Heidrich (UBC/KAUST)
Ramesh Raskar (MIT)
Diego Gutierrez (University of Zaragoza)
Brian Barsky (UC Berkeley)
Henry Fuchs (UNC)

96. Gordon Wetzstein
Massachusetts Institute of Technology
media.mit.edu/~gordonw
displayblocks.org
Stanford looking for:
• students, postdocs, interns
• (industry) collaborators