Primal-dual coding photography is a new photographic technique that uses coded illumination and exposure to selectively record user-defined subsets of light paths, generalizing conventional photography. It modulates the contribution of specific light paths using a "probing matrix" of primal codes for illumination and dual codes for exposure over multiple frames. This allows effects like enhancing direct light, capturing indirect light of different ranges, separating light transport effects, and making 3D regions invisible or color-coded in the photo. The technique provides guarantees of optimality and convergence for reconstructing images from the coded photo measurements.

1. Primal-Dual Coding
to Probe Light Transport
Matthew O’Toole
University of Toronto
Ramesh Raskar
MIT Media Lab
Kyros Kutulakos
University of Toronto

2. conventional photography
photo records all light paths
that reach camera sensor

3. primal-dual coding photography
photo only records a user-defined
subset of light paths

4. conventional photography
primal-dual coding photography

5. new photography regime: primal-dual coding
modulate contribution of specific light path
contributions
transport probing equation
generalizes conventional photography & confocal imaging [Wilson et al. 96]
combine illumination coding [Schechner et al. 07] & exposure coding [Hitomi et al. 11]
efficient primal-dual codes by stochastic matrix probing
provides convergence & optimality guarantees [Bekas et al. 07]

6. 1. the result of light transport from
sources to sensor pixels
2. the image recorded by the camera’s
sensor
what is a photo?

7. transport matrix [Ng et al. 03]photo
light
conventional photography
degrees of freedom

8. primal-dual coding photography
“probing matrix”
degrees of freedom

9. multiply
primal-dual coding photography
“probing matrix”
degrees of freedom

10. primal-dual coding photography
“probing matrix”
degrees of freedom

11. primal-dual coding photography
degrees of freedom

12. mathematical formulation of
primal-dual coding photography

16. one-shot primal-dual coding
photography

17. experimental setup
SLR camera LCD mask
projector
scene
beamsplitter

18. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

19. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

20. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

21. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

22. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

23. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

24. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

25. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

26. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

27. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

28. step 2
illuminate scene
with vector .
(primal code)
step 3
attenuate image
with vector .
(dual code)
step 4
repeat times
step 1
open shutter
step 5
close shutter

30. Rademacher primal-dual codes
stochastic diagonal estimator [Bekas et al. 07]
primal codes are Rademacher random vectors: random vector in
dual codes derive from primal code:
codes converge to identity probing matrix:
variance of pixel for primal-dual codes
aperture correlation (microscopy) is a diagonal estimator [Wilson et al. 96, Levoy et al. 04]
stochastic estimator for general probing
dual codes for general probing matrix :

31. designing probing matrices

32. designing probing matrices
transport matrix

33. designing probing matrices
transport matrix

34. designing probing matrices
camera
projector
transport matrix

35. designing probing matrices
camera
projector
camera
projector
transport matrix

36. designing probing matrices
camera
projector
camera
projector
transport matrix

37. designing probing matrices
camera
projector
camera
projector
transport matrix

38. coaxial example: contrast-enhancing direct light
conventional
photo
all light paths

39. coaxial example: contrast-enhancing direct light
direct +
½ indirect
conventional
photo
direct + 1/2 indirect
light paths

40. coaxial example: contrast-enhancing direct light
direct +
½ indirect
conventional
photo
direct +
¼ indirect
direct +
1/16 indirect
direct + 1/16 indirect
light paths

41. coaxial example: capturing short to long range paths
conventional
all light paths

42. coaxial example: capturing short to long range paths
conventional
indirect
indirect light paths

43. coaxial example: capturing short to long range paths
conventional
indirect mid-range indirect
medium to long range
indirect light paths

44. coaxial example: capturing short to long range paths
conventional
indirect mid-range indirect
long-range indirect
long range indirect
light paths

45. coaxial example: separating light transport effects
conventional
all light paths

46. coaxial example: separating light transport effects
conventional
indirect
indirect light paths

47. coaxial example: separating light transport effects
conventional
indirect direct + backscatter
direct + back-scatter
light paths

48. coaxial example: separating light transport effects
indirect [Nayar et al. 06]
indirect direct + backscatter
direct [Nayar et al. 06]

49. coaxial example: separating light transport effects
indirect direct + backscatter
low-freq. indirect high-freq. indirect

50. conventional
all light paths
coaxial example: de-scattering [Fuchs et al. 08]

51. conventional de-scattered image
de-scattered diagonal
coaxial example: de-scattering [Fuchs et al. 08]

52. coaxial example: de-scattering [Fuchs et al. 08]
conventional
direct + 1/16 indirect
direct + 1/16 indirect
light paths
de-scattered image

53. conventional
direct + 1/16 indirect shifted indirect
de-scattered image
coaxial example: de-scattering [Fuchs et al. 08]
shifted indirect +
1/16 light paths

54. non-coaxial example: making 3D regions invisible
all light paths

55. “privacy zone”
non-coaxial example: making 3D regions invisible

56. non-coaxial example: making 3D regions invisible

57. non-coaxial example: making 3D regions invisible

58. non-coaxial example: making 3D regions invisible

59. privacy zone
non-coaxial example: making 3D regions invisible

60. non-coaxial example: color coding regions of space
color coded zones

61. non-coaxial example: color coding regions of space

62. non-coaxial example: color coding regions of space

63. real-time PDC photography
matching 1440Hz DLP kits for illumination and attenuation

64. real-time PDC photography
projector
matching 1440Hz DLP kits for illumination and attenuation

65. real-time PDC photography
mask
camera
matching 1440Hz DLP kits for illumination and attenuation
DMD

67. additional details on primal-dual coding
non-negative primal-dual codes
many cases realized with strictly non-negative codes
for all others, subtract two photos computationally
codes robust to optical misalignments
account for both spatial and temporal misalignments
one-shot versus multi-shot
one-shot limited by projector display rate
multi-shot limited by camera read noise

68. concluding remarks
primal-dual coding photography
produces new types of photos governed by probing equation
combines coded illumination and coded exposure
guarantees optimality and convergence rate of codes
can be a see-through device
future work
new possibilities for optical applications
combine with existing computational photography techniques (image-based relighting,
geometry acquisition, …)

69. Matthew O’Toole
University of Toronto
Ramesh Raskar
MIT Media Lab
Kyros Kutulakos
University of Toronto
Primal-Dual Coding
to Probe Light Transport
http://www.dgp.toronto.edu/~motoole/primaldualcoding.html

70. 64 codes
16 codes 32 codes
8 codes
128 codes 256 codes