Raskar Banff

1,148 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,148
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Raskar Banff

    1. 1. Media Lab, MIT Cambridge, MA From 4D Capture to 6D Display: A mask-based approach Ramesh Raskar
    2. 2. Discussion Topics <ul><li>What is the info content of a 3D scene? </li></ul><ul><ul><li>Encoding appearance and geometric complexity </li></ul></ul><ul><li>What are the dimensions beyond viewpt? </li></ul><ul><ul><li>Lighting? </li></ul></ul><ul><li>What other optical sensors we can use? </li></ul>
    3. 3. R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns Siggraph 2007 Prakash: Lighting-Aware Motion Capture Using Photosensing Markers and Multiplexed Illuminators
    4. 4. Vicon Motion Capture High-speed IR Camera Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis
    5. 5. Imperceptible Tags under clothing, tracked under ambient light Hidden Marker Tags Outdoors Unique Id
    6. 6. Labeling Space (Indoor GPS) Each location receives a unique temporal code But 60Hz video projector is too slow Projector Tags Pos=0 Pos=25 5 Time
    7. 7. Pattern MSB Pattern MSB-1 Pattern LSB <ul><li>For each tag </li></ul><ul><li>From light sequence, decode x and y coordinate </li></ul><ul><li>Transmit back to RF reader ( Id , x, y ) </li></ul>0 1 1 0 0 X=12
    8. 8. Inside of Multi-LED Emitter
    9. 9. Tag
    10. 10. Analog Space Labeling Multi-LED Beacon 1 Beacon 2 Beacon 3 Tag N ?
    11. 11. Imperceptible Tags Location
    12. 12. Location Orientation
    13. 13. 3D Overlay Orientation
    14. 14. Imperceptible Tags Incident Illumination
    15. 15. Inverse Optical Mo-Cap <ul><li>High Speed Camera </li></ul><ul><li>Detect blobs in each frame </li></ul><ul><li>Reflective/Emitting Marker </li></ul><ul><li> Disambiguate in camera </li></ul><ul><li> Only Location </li></ul><ul><li>High Speed Projector </li></ul><ul><li>Label the 3D space </li></ul><ul><li>Photosensing Marker </li></ul><ul><li> Find ego-position </li></ul><ul><li> Location, Orientation, Illum </li></ul>
    16. 16. On-set MoCap: Location + Orientation + Incident Illumination
    17. 17. Coded Illumination Sensor Skin <ul><li>500 Hz with Id for each Marker Tag </li></ul><ul><li>Capture in Natural Environment </li></ul><ul><ul><li>Visually imperceptible tags </li></ul></ul><ul><ul><li>Photosensing Tag can be hidden under clothes </li></ul></ul><ul><ul><li>Ambient lighting is ok </li></ul></ul><ul><li>Unlimited Number of Tags </li></ul><ul><ul><li>Light sensitive fabric for dense sampling </li></ul></ul><ul><li>Non-imaging, complete privacy </li></ul><ul><li>Base station and tags only a few 10’s $ </li></ul><ul><li>Body scan + bio </li></ul><ul><ul><li>Elderly, patients, athletes, performers </li></ul></ul>
    18. 18. Project Topics <ul><li>Structured Light Scanning </li></ul><ul><ul><li>Fast Stripping </li></ul></ul><ul><ul><ul><li>Can you build a scanner using very low cost hardware? </li></ul></ul></ul><ul><ul><ul><li>Without full 2D cameras or video projectors? </li></ul></ul></ul><ul><ul><li>Global-direct Separation </li></ul></ul><ul><ul><ul><li>Can you scan difficult (global effect) using direct/global separation? </li></ul></ul></ul>
    19. 19. Towards a 6D Display Passive Reflectance Field Display Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008 1 2 1 1 1 MPI Informatik, Germany 2 MIT
    20. 20. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    21. 21. <ul><li>[Lippman 1908] [Nakajima et al. 2001] ... </li></ul>Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    22. 22. electronic: [Nayar et al. 2004] slit based / different patterns: [Scharstein 1996] Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    23. 23. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    24. 24. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    25. 25. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    26. 26. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    27. 27. Improved Design Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    28. 28. Variance with Observer Martin Fuchs <mfuchs@mpi-inf.mpg.de> <ul><li>recall: </li></ul>
    29. 29. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    30. 30. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    31. 31. Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    32. 32. Observer-Variance Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    33. 33. 6D Construction Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    34. 34. Illumination + Spatial Variation Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    35. 35. Variance with Observation Angle Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    36. 36. Towards 6D Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    37. 37. 6D Results Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    38. 38. Future Work <ul><li>Efficient manufacturing </li></ul><ul><li>scale </li></ul><ul><li>precision </li></ul><ul><li>How fine can we get our structures? </li></ul><ul><ul><li>is 6D really practical? </li></ul></ul><ul><li>Extensions for local illumination ? </li></ul>Martin Fuchs <mfuchs@mpi-inf.mpg.de>
    39. 39. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern
    40. 40. In Focus Photo LED
    41. 41. Out of Focus Photo: Open Aperture
    42. 42. Out of Focus Photo: Coded Aperture
    43. 43. Captured Blurred Photo
    44. 44. Refocused on Person
    45. 45. Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera Mask? Sensor Mask Sensor Mask? Sensor Mask Sensor
    46. 46. Light Field Inside a Camera
    47. 47. Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a Camera
    48. 48. Stanford Plenoptic Camera [Ng et al 2005] <ul><li>4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens </li></ul>Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125 μ square-sided microlenses
    49. 49. Digital Refocusing [Ng et al 2005] Can we achieve this with a Mask alone?
    50. 50. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor
    51. 51. Heterodyne Light Field Camera Scanner sensor Mask [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor
    52. 52. How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?
    53. 53. Radio Frequency Heterodyning Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz Baseband Audio Signal
    54. 54. Optical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main Lens Object Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz
    55. 55. Captured 2D Photo Encoding due to Mask
    56. 56. 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT
    57. 57. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9
    58. 58. A Theory of Mask-Enhanced Camera <ul><li>Mask == Light Field Modulator </li></ul><ul><li>Intensity of ray gets multiplied by Mask </li></ul><ul><li>Convolution in Frequency domain </li></ul>Main Lens Object Mask Sensor
    59. 59. Related Work <ul><li>Light Field Capture </li></ul><ul><ul><li>Gortler et al., Levoy & Hanrahan, SIG’96, Isaksen et al.‘SIG00 </li></ul></ul><ul><ul><li>Light Field Microscopy: Levoy et al. SIG’06 </li></ul></ul><ul><ul><li>Integral Photography </li></ul></ul><ul><ul><ul><li>Lippman’08, Ives’30, Georgeiv et al. EGSR’06, Okano et.al’97 </li></ul></ul></ul><ul><ul><li>Camera arrays: Wilburn et al. SIG’05 </li></ul></ul><ul><ul><li>Flatbed Scanner + Lenslet array: Yang, 2000 </li></ul></ul><ul><ul><li>Light Field Video Camera: Wilburn et.al'02 </li></ul></ul><ul><ul><li>Programmable Aperture: Liang et. al ICIP 2007 </li></ul></ul><ul><ul><li>Plenoptic Camera </li></ul></ul><ul><ul><ul><li>Wang and Adelson’92 </li></ul></ul></ul><ul><ul><ul><li>Ng et al.’05 </li></ul></ul></ul>
    60. 60. f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice – Fourier Slice Theorem <ul><li>Photo = Slice of Light Field in Fourier Domain </li></ul><ul><ul><ul><li>[Ren Ng, SIGGRAPH 2005] </li></ul></ul></ul>
    61. 61. How to Capture 2D Light Field with 1D Sensor ? f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice Fourier Light Field Space
    62. 62. Extra sensor bandwidth cannot capture extra dimension of the light field f θ f x f θ 0 f x0 Sensor Slice Extra sensor bandwidth
    63. 63. f θ f x ??? ??? ??? ???
    64. 64. Solution: Modulation Theorem Make spectral copies of 2D light field f θ f x f θ 0 f x0 Modulation Function
    65. 65. f θ Modulated Light Field f x f θ 0 f x0 Modulation Function Sensor Slice captures entire Light Field
    66. 66. Demodulation to recover Light Field f θ f x Reshape 1D Fourier Transform into 2D 1D Fourier Transform of Sensor Signal
    67. 67. f θ f x f θ 0 f x0 Modulation Function == Sum of Impulses Physical Mask = Sum of Cosines
    68. 68. 1/f 0 Mask Tile Cosine Mask Used
    69. 69. Where to place the Mask? Mask Sensor Mask Modulation Function Mask Modulation Function f x f θ Mask Sensor
    70. 70. Mask Sensor Where to place the Mask? Mask Modulation Function f x f θ
    71. 71. Mask Sensor d v α α = (d/v) ( π /2) Where to place the Mask? Mask Modulation Function
    72. 72. Captured 2D Photo Encoding due to Cosine Mask
    73. 73. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9
    74. 74. Digital Refocusing Only cone in focus Captured Photo
    75. 75. Full resolution 2D image of Focused Scene Parts Captured 2D Photo Image of White Lambertian Plane divide
    76. 76. Coding and Modulation in Camera Using Masks Coded Aperture for Full Resolution Digital Refocusing Heterodyne Light Field Camera Mask? Sensor Mask Sensor Mask Sensor
    77. 77. Discussion Topics <ul><li>What is the info content of a 3D scene? </li></ul><ul><ul><li>Encoding appearance and geometric complexity </li></ul></ul><ul><li>Two approaches for multi-view capture or display, </li></ul><ul><ul><li>Lenslet (multiscopic), pin-hole array (parallax barrier) </li></ul></ul><ul><li>Third choice </li></ul><ul><ul><li>Multiplexing </li></ul></ul><ul><ul><li>coding: can we build a display on this principle </li></ul></ul><ul><li>Mask can go anywhere, what else can we achieve? </li></ul><ul><li>Should we think about multi-camera arr like this </li></ul>
    78. 78. Mask-based Approaches <ul><li>Coded Illumination </li></ul><ul><ul><li>Motion Capture [2007] </li></ul></ul><ul><li>6D Display </li></ul><ul><ul><li>Lighting aware [2008] </li></ul></ul><ul><li>Optical Heterodyning </li></ul><ul><ul><li>Light Field Capture [2007] </li></ul></ul>http://raskar.info
    79. 79. Discussion Topics <ul><li>What is the info content of a 3D scene? </li></ul><ul><ul><li>Encoding appearance and geometric complexity </li></ul></ul><ul><li>What are the dimensions beyond viewpt? </li></ul><ul><ul><li>Lighting? </li></ul></ul><ul><li>What other optical sensors we can use? </li></ul><ul><li>What are other display technologies? </li></ul><ul><ul><li>Materials, configuration </li></ul></ul>

    ×