The document discusses using coded masks and modulation techniques to capture light field information and enable digital refocusing and 6D displays with a single 2D sensor. It proposes placing a coded mask in front of the sensor to heterodyne the light field and extract its 4D information. Several applications are mentioned, including coded illumination for motion capture, a 6D display using spatial and illumination variation, and a light field camera that can digitally refocus using a single photograph.

1. Media Lab, MIT Cambridge, MA From 4D Capture to 6D Display: A mask-based approach Ramesh Raskar

2. Discussion Topics What is the info content of a 3D scene? Encoding appearance and geometric complexity What are the dimensions beyond viewpt? Lighting? What other optical sensors we can use?

3. R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns Siggraph 2007 Prakash: Lighting-Aware Motion Capture Using Photosensing Markers and Multiplexed Illuminators

4. Vicon Motion Capture High-speed IR Camera Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis

5. Imperceptible Tags under clothing, tracked under ambient light Hidden Marker Tags Outdoors Unique Id

6. Labeling Space (Indoor GPS) Each location receives a unique temporal code But 60Hz video projector is too slow Projector Tags Pos=0 Pos=25 5 Time

7. Pattern MSB Pattern MSB-1 Pattern LSB For each tag From light sequence, decode x and y coordinate Transmit back to RF reader ( Id , x, y ) 0 1 1 0 0 X=12

8. Inside of Multi-LED Emitter

9. Tag

10. Analog Space Labeling Multi-LED Beacon 1 Beacon 2 Beacon 3 Tag N ?

11. Imperceptible Tags Location

12. Location Orientation

13. 3D Overlay Orientation

14. Imperceptible Tags Incident Illumination

15. Inverse Optical Mo-Cap High Speed Camera Detect blobs in each frame Reflective/Emitting Marker Disambiguate in camera Only Location High Speed Projector Label the 3D space Photosensing Marker Find ego-position Location, Orientation, Illum

16. On-set MoCap: Location + Orientation + Incident Illumination

17. Coded Illumination Sensor Skin 500 Hz with Id for each Marker Tag Capture in Natural Environment Visually imperceptible tags Photosensing Tag can be hidden under clothes Ambient lighting is ok Unlimited Number of Tags Light sensitive fabric for dense sampling Non-imaging, complete privacy Base station and tags only a few 10’s $ Body scan + bio Elderly, patients, athletes, performers

18. Project Topics Structured Light Scanning Fast Stripping Can you build a scanner using very low cost hardware? Without full 2D cameras or video projectors? Global-direct Separation Can you scan difficult (global effect) using direct/global separation?

19. Towards a 6D Display Passive Reflectance Field Display Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008 1 2 1 1 1 MPI Informatik, Germany 2 MIT

20. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

21. [Lippman 1908] [Nakajima et al. 2001] ... Martin Fuchs <mfuchs@mpi-inf.mpg.de>

22. electronic: [Nayar et al. 2004] slit based / different patterns: [Scharstein 1996] Martin Fuchs <mfuchs@mpi-inf.mpg.de>

23. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

24. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

25. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

26. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

27. Improved Design Martin Fuchs <mfuchs@mpi-inf.mpg.de>

28. Variance with Observer Martin Fuchs <mfuchs@mpi-inf.mpg.de> recall:

29. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

30. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

31. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

32. Observer-Variance Martin Fuchs <mfuchs@mpi-inf.mpg.de>

33. 6D Construction Martin Fuchs <mfuchs@mpi-inf.mpg.de>

34. Illumination + Spatial Variation Martin Fuchs <mfuchs@mpi-inf.mpg.de>

35. Variance with Observation Angle Martin Fuchs <mfuchs@mpi-inf.mpg.de>

36. Towards 6D Martin Fuchs <mfuchs@mpi-inf.mpg.de>

37. 6D Results Martin Fuchs <mfuchs@mpi-inf.mpg.de>

38. Future Work Efficient manufacturing scale precision How fine can we get our structures? is 6D really practical? Extensions for local illumination ? Martin Fuchs <mfuchs@mpi-inf.mpg.de>

39. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern

40. In Focus Photo LED

41. Out of Focus Photo: Open Aperture

42. Out of Focus Photo: Coded Aperture

43. Captured Blurred Photo

44. Refocused on Person

45. Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera Mask? Sensor Mask Sensor Mask? Sensor Mask Sensor

46. Light Field Inside a Camera

47. Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a Camera

48. Stanford Plenoptic Camera [Ng et al 2005] 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125 μ square-sided microlenses

49. Digital Refocusing [Ng et al 2005] Can we achieve this with a Mask alone?

50. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor

51. Heterodyne Light Field Camera Scanner sensor Mask [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor

52. How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?

53. Radio Frequency Heterodyning Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz Baseband Audio Signal

54. Optical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main Lens Object Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz

55. Captured 2D Photo Encoding due to Mask

56. 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT

57. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9

58. A Theory of Mask-Enhanced Camera Mask == Light Field Modulator Intensity of ray gets multiplied by Mask Convolution in Frequency domain Main Lens Object Mask Sensor

59. Related Work Light Field Capture Gortler et al., Levoy & Hanrahan, SIG’96, Isaksen et al.‘SIG00 Light Field Microscopy: Levoy et al. SIG’06 Integral Photography Lippman’08, Ives’30, Georgeiv et al. EGSR’06, Okano et.al’97 Camera arrays: Wilburn et al. SIG’05 Flatbed Scanner + Lenslet array: Yang, 2000 Light Field Video Camera: Wilburn et.al'02 Programmable Aperture: Liang et. al ICIP 2007 Plenoptic Camera Wang and Adelson’92 Ng et al.’05

60. f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice – Fourier Slice Theorem Photo = Slice of Light Field in Fourier Domain [Ren Ng, SIGGRAPH 2005]

61. How to Capture 2D Light Field with 1D Sensor ? f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice Fourier Light Field Space

62. Extra sensor bandwidth cannot capture extra dimension of the light field f θ f x f θ 0 f x0 Sensor Slice Extra sensor bandwidth

63. f θ f x ??? ??? ??? ???

64. Solution: Modulation Theorem Make spectral copies of 2D light field f θ f x f θ 0 f x0 Modulation Function

65. f θ Modulated Light Field f x f θ 0 f x0 Modulation Function Sensor Slice captures entire Light Field

66. Demodulation to recover Light Field f θ f x Reshape 1D Fourier Transform into 2D 1D Fourier Transform of Sensor Signal

67. f θ f x f θ 0 f x0 Modulation Function == Sum of Impulses Physical Mask = Sum of Cosines

68. 1/f 0 Mask Tile Cosine Mask Used

69. Where to place the Mask? Mask Sensor Mask Modulation Function Mask Modulation Function f x f θ Mask Sensor

70. Mask Sensor Where to place the Mask? Mask Modulation Function f x f θ

71. Mask Sensor d v α α = (d/v) ( π /2) Where to place the Mask? Mask Modulation Function

72. Captured 2D Photo Encoding due to Cosine Mask

73. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9

74. Digital Refocusing Only cone in focus Captured Photo

75. Full resolution 2D image of Focused Scene Parts Captured 2D Photo Image of White Lambertian Plane divide

76. Coding and Modulation in Camera Using Masks Coded Aperture for Full Resolution Digital Refocusing Heterodyne Light Field Camera Mask? Sensor Mask Sensor Mask Sensor

77. Discussion Topics What is the info content of a 3D scene? Encoding appearance and geometric complexity Two approaches for multi-view capture or display, Lenslet (multiscopic), pin-hole array (parallax barrier) Third choice Multiplexing coding: can we build a display on this principle Mask can go anywhere, what else can we achieve? Should we think about multi-camera arr like this

78. Mask-based Approaches Coded Illumination Motion Capture [2007] 6D Display Lighting aware [2008] Optical Heterodyning Light Field Capture [2007] http://raskar.info

79. Discussion Topics What is the info content of a 3D scene? Encoding appearance and geometric complexity What are the dimensions beyond viewpt? Lighting? What other optical sensors we can use? What are other display technologies? Materials, configuration