This document describes the concept of dual photography, which uses Helmholtz reciprocity to interchange lights and cameras in a scene. It discusses how the transposed transport matrix can be used to generate virtual captured images from virtual projected patterns. It also describes different methods used to capture the transport matrix, including fixed pattern scanning and adaptive multiplexed illumination. Limitations discussed include scenes with significant global illumination effects and situations where the camera and projector are at a large angle.

1. Dual Photography
Pradeep Sen | Billy Chen | Gaurav Garg | Stephen Marschner | Mark
Horowitz | Marc Levoy | Hendrik Lensch
Prashanth PM

2. Introduction
 Dual Photography uses the concept of Helmholtz Reciprocity to
interchange the lights and cameras in a scene.

3. Helmholtz Reciprocity
Bidirectional Reflectance Distribution
Function (BRDF)
fr(ωi, ωo)
(outgoing radiance / Incoming radiance)
Helmholtz Reciprocity
fr(ωi, ωo) = fr(ωo ,ωi)

4. First Relighting Work to make use of
Helmholtz Reciprocity.

5. Radiance transfer between
incoming and outgoing directions is
symmetric

6. So, we can generate an image from
the point of view of the projector.

7. Primal Configuration
 c’ : Captured Image (nm x 1)
 p’ : Projected Pattern (pq x 1)
 T : Transport Matrix (mn x pq)

9. Dual Configuration
 c”: Virtual Projected Pattern (nm x 1)
 p”: Virtual Captured Image (pq x 1)
 TT: Transposed Transport Matrix (pq x nm)

10. Transport Matrix
T’’ = TT due to Helmholtz Reciprocity

11. Dual Photography is the act of
multiplying the transposed matrix by a
desired lighting image vector.

12. Photography without a camera.
Image at right acquire using a 3x3 pixel
square scan across the projector.
Two photo resistors are used instead of
a camera.

13. Comparison with Debevec et. al. [2000]
In Debevec’s “Acquiring the Reflectance Field of a Human Face“, distant point
light sources are assumed, and there is no spatial variation within the light
source.
Hence, sharp shadows cannot be cast onto the scene.
Besides, a fewer number of light sources are required.

14. Optimizations in Capturing the T matrix.
Brute Force Method
Scan a single pixel per captured camera frame
 Projector and Camera each have O(n6) pixels
 Full T matrix would have O(n12) elements
 HDR imagery required for scenes containing both specular and diffuse inter-
reflections.

15. Even at a rate of 25 HDR images per
minute, the capture process could take
weeks !

16. Fixed Pattern Scanning
 Assume each projector pixel affects a
small, localized region of the camera. Divide
the region into blocks.
 Repeat exposures and encode each block’s
illuminated pixels with a unique binary
encoding.

17. T
Fixed Pattern Scanning
1 17
….. …..
Projector Pixels
1
1 17
17

18.  We can determine the spot being lit using a truth table (bit encoding) for 3
locations.
4 2 1 Spot
0 0 0 0
0 0 1 1
0 1 0 2
0 1 1 3
1 0 0 4
1 0 1 5
1 1 0 6
1 1 1 7
Hamming error correcting codes are added to account for bit errors due to noisy
measurement.

19. Limitations of Fixed Pattern Scanning
Requires one-to-one correspondence between camera
and projector pixels.
 This only supports direct illumination properly.
Diffuse Illumination can map many projector pixels to
the same camera pixel.
 This violates the initial assumption.

20. Adaptive Multiplexed Illumination
LEVEL 1
 An 8x8 pixel project is used.
 All Pixels are Illuminated.

21. Adaptive Multiplexed Illumination
LEVEL 2
 The single block from Level 1 is subdivided
into four blocks.
 Conflict is detected between regions 2 and 4
 Additional subdivision is required

22. Adaptive Multiplexed Illumination
 Conflicting blocks from Level 2 are not
co-scheduled in Level 3.
 Some blocks can be scheduled in parallel.
 Two new conflicts detected (6 & 12, 8 & 10).
 The lower-leftmost block causes no illumination
and is culled

23. Adaptive Multiplexed Illumination
 Final subdivision is applied in Level 4.
 Many blocks are acquired in parallel.

24. Drawbacks of Adaptive Multiplexed Illumination
 Does not do well for a scene with several diffuse inter-
reflections.
 A single projector pixel can affect the entire scene.

26. Hierarchical Assembly of the Transport Matrix

27. Dual from Adaptive Multiplexing Dual from Hierarchical Assembly
Inflections between red wall and box
is almost lost.

28. Results
Primal Dual

29. Scene Relighting
Primal and Dual Images can be relit
by multiplying T or TT by the
desired matte as p’ or c.”

30. Scene Relighting
Primal Dual Dual with relit pattern

31. Dual – hand Dual – moving light beam

32. Scene Relighting
Projectors are not parallelizable, only one projector can be used at a time. But
cameras can be parallelized to capture a 6D reflectance field.

33. Scene Relighting

34. Results
Dual – left lit Dual – right lit

35. Results
Dual – soft shadows Dual – hi res matte

36. Results
Dual – Soft Shadows Dual – Shadows cast by synthetic model

37. Limitations
Scenes containing significant global illumination
effects reduce parallelism of the adaptive method.

38. Limitations
 Cameras have greater depth-of-field, better focus
control, and more imaging controls
 Limited depth-of-field can result in out-of-focus dual
images

39. Limitations
 Suppose the camera and projector are at a large angle
apart from each other.
 There may be many regions in the scene with no
direct light transport between the camera and
projector.

40. How to read your opponent’s card ?

41. Thank You
 Questions ?