3D Shape and Indirect Appearance by Structured Light Transport Matthew O'Toole, John Mather, and Kiriakos N. Kutulakos. CVPR, 2014. Abstract: We consider the problem of deliberately manipulating the direct and indirect light flowing through a time-varying, fully-general scene in order to simplify its visual analysis. Our approach rests on a crucial link between stereo geometry and light transport: while direct light always obeys the epipolar geometry of a projector-camera pair, indirect light overwhelmingly does not. We show that it is possible to turn this observation into an imaging method that analyzes light transport in real time in the optical domain, prior to acquisition. This yields three key abilities that we demonstrate in an experimental camera prototype: (1) producing a live indirect-only video stream for any scene, regardless of geometric or photometric complexity; (2) capturing images that make existing structured-light shape recovery algorithms robust to indirect transport; and (3) turning them into one-shot methods for dynamic 3D shape capture.

1. 3D Shape and Indirect Appearance
by Structured Light Transport
Matthew O’Toole John Mather Kiriakos N. Kutulakos
Department of Computer Science
University of Toronto
http://www.dgp.toronto.edu/~motoole/slt

2. 1. indirect-only imaging

3. 1. indirect-only imaging
2. indirect-invariant imaging

4. 1. indirect-only imaging
2. indirect-invariant imaging 3. one-shot, multi-pattern imaging

5. contributions

6. using epipolar geometry to analyze light transport
contributions

7. using epipolar geometry to analyze light transport
new live video channels for computer vision
• indirect-only and direct-only imaging
• indirect-invariant imaging of structured patterns
• one-shot, multi-pattern imaging
contributions

8. using epipolar geometry to analyze light transport
new live video channels for computer vision
• indirect-only and direct-only imaging
• indirect-invariant imaging of structured patterns
• one-shot, multi-pattern imaging
structured light transport for 3D shape acquisition
contributions

9. basic light paths

10. projector
basic light paths

11. projectorcamera
basic light paths

12. camera projector
basic light paths
direct light

13. camera projector
basic light paths
indirect light
(scattered)

14. camera projector
mirror
basic light paths
indirect light
(specular)

15. camera projector
mirror
basic light paths

16. 1. indirect-only imaging

17. camera projector
mirror
basic light paths

18. camera projector
mirror
epipolar constraint & light transport
direct paths satisfy
epipolar constraints

19. camera projector
mirror
epipolar constraint & light transport
indirect paths almost
never satisfy constraints

20. camera projector
mirror
epipolar constraint & light transport
indirect paths almost
never satisfy constraints

21. projectorcamera
blocking epipolar paths with patterns & masks
mirror

22. camera projector
mirror
projection
pattern
mask
pattern
blocking epipolar paths with patterns & masks

23. camera projector
mirror
blocking epipolar paths with patterns & masks
projection
pattern
mask
pattern

24. camera projector
mirror
blocking epipolar paths with patterns & masks
complementary random
epipolar patterns

25. camera projector
mirror
blocking epipolar paths with patterns & masks
complementary random
epipolar patterns

26. camera projector
mirror
blocking epipolar paths with patterns & masks
complementary random
epipolar patterns

27. camera projector
mirror
blocking epipolar paths with patterns & masks
complementary random
epipolar patterns

28. camera projector
mirror
1. open electronic shutter
2. for i = 1 to N
use random epipolar mask &
project complementary pattern
3. close electronic shutter
blocking epipolar paths with patterns & masks
complementary random
epipolar patterns

29. live indirect-only and direct-only video stream
conventional imaging

30. live indirect-only and direct-only video stream
conventional imaging indirect-only imaging

31. live indirect-only and direct-only video stream
conventional imaging indirect-only imaging
direct-only imaging

32. live indirect-only and direct-only video stream
conventional imaging indirect-only imaging
direct-only imaging

33. live indirect-only and direct-only video stream
conventional imaging indirect-only imaging
direct-only imaging

34. live indirect-only and direct-only video stream
conventional imaging indirect-only imaging
direct-only imaging

35. live indirect-only and direct-only video stream
conventional imaging indirect-only imaging
direct-only imaging

36. separating direct & indirect components [e.g. Achar et al. 2013]
• multiple shots, motion processing
• only low-frequency indirect light (i.e. diffuse)
transport-robust shape acquisition [e.g. Gupta et al. 2012]
• assumption: only high-freq patterns can be robust to indirect light
• no specular transport
structured light transport
• live video, fully-general transport, no motion processing
• any projection pattern can be made robust to indirect light
• new forms of one-shot 3D imaging
related work

37. structured light transport: theory

38. structured light transport: theory
dominance of non-
epipolar transport

39. transport matrixmask
illumin.
deriving mask and
illumination patterns
• exact derivation of patterns
is NP-hard [Zhong 2012]
• instead, use random codes,
and approximate imaging in
expectation
structured light transport: theory
dominance of non-
epipolar transport

40. 2. indirect-invariant imaging

41. live indirect-invariant video stream
conventional structured light
indirect-invariant structured light

42. mirror
shape acquisition by structured light

43. mirror
cameraprojector
shape acquisition by structured light

44. mirror
cameraprojector
shape acquisition by structured light

45. conventional structured light

46. reconstructed 3D shape

47. indirect-invariant structured light

48. reconstructed 3D shape (same algorithm)

49. conventional structured light

50. reconstructed 3D shape

51. indirect-invariant structured light

52. reconstructed 3D shape (same algorithm)

53. 3. one-shot, multi-pattern,
indirect-invariant imaging

54. one-shot, multi-pattern, indirect-invariant imaging

55. one-shot, multi-pattern, indirect-invariant imaging

56. reconstructed 3D shape

57. • a new general imaging technique for computer vision
• combines epipolar geometry and light transport analysis
• theoretical analysis & a physical device for live video
generation
• SIGGRAPH 2014: extension to time-of-flight imaging
structured light transport

58. 3D Shape and Indirect Appearance
by Structured Light Transport
Matthew O’Toole John Mather Kiriakos N. Kutulakos
Department of Computer Science
University of Toronto
http://www.dgp.toronto.edu/~motoole/slt
Poster ID: O-4A-4