Upcoming SlideShare
×

# Introduction to Camera Challenges - Ramesh Raskar

1,223 views

Published on

Published in: Art & Photos, Technology
1 Like
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
1,223
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
73
0
Likes
1
Embeds 0
No embeds

No notes for slide
• Precursor to Google Streetview Maps
• Show HDR image here using HDRshop
• Talk about limitations: Colocated artifacts, color coherency, ref can’t be obtain by subtraction
• &lt;Algorithm&gt; The algorithm consists of the following steps: First, we compute the gradient fields of the daytime and nighttime input images by using simple forward differencing in the x and y direction. Thresholding the gradient field images allows us to compute the locally-important areas. These are areas of high variance in the nighttime image, shown here in white. The pixels of the importance image are white in the locally-important areas that are taken from the nighttime image and black in the context areas that are taken from the daytime image. We use prost-processing (eroding, fattening and feathering) to consolidate the selected areas and ensure smooth transitions. A mixed gradient field is computed as a weighted mean of the input gradient fields, using the pixel values in the importance image as weights. The final result is obtained by integrating the mixed gradient field.
• &lt;Gradient field integration&gt; Image reconstruction from gradients fields is an approximate invertibility problem, and still a very active research area. We are trying to obtain image I from a gradient field G composed of two images that represent the differences in the x and y direction. In 2D, a modified gradient vector field G may not be integrable. We use one of the direct methods recently proposed to minimize the error nabla I - G. The estimate of the desired intensity function I’ , so that G = nabla I’ , can be obtained by solving the Poisson differential equation nabla 2 I’ = divG , involving a Laplace and a divergence operator. We use the full multigrid method to solve the Laplace equation.
• When we take a photograph of a group of people, such as this image on the left, what we get is a frozen moment of time that is often less natural, and less attractive than the scene we remember. This is because the cognitive processes that form our visual memories integrate over a range of time to form a subjective impression. This memory will likely look a lot more like the image on the right, where everyone is smiling naturally. The goal of our photomontage system is to help us create photographs that better match the image we see in our mind’s eye. To do so, we begin with a stack of images, and combine the best parts of each to form an image that is better than any of the originals.
• The tradeoffs in the CAMERA ADJUSTMENTS dont match the tradeoffs in APPEARANCE of what we want to photograph
• Better than any one photo : keep the best from each of them.
• Precursor to Google Streetview Maps
• Check Steve Seitz and U of Washington Phototourism Page
• Full-Scale Schlieren Image Reveals The Heat Coming off of a Space Heater, Lamp and Person
• We call our tool NETRA: near eye tool for refractive assessment such as nearsightedness/far/astigmatism Basic idea is to create a unique interactive lightfield display near the eye and is possible due to the highresolution of modern LCDs.
• In a confocal laser scanning microscope, a laser beam passes through a light source aperture and then is focused by an objective lens into a small (ideally diffraction limited ) focal volume within a fluorescent specimen. A mixture of emitted fluorescent light as well as reflected laser light from the illuminated spot is then recollected by the objective lens. A beam splitter separates the light mixture by allowing only the laser light to pass through and reflecting the fluorescent light into the detection apparatus. After passing a pinhole , the fluorescent light is detected by a photodetection device (a photomultiplier tube (PMT) or avalanche photodiode ), transforming the light signal into an electrical one that is recorded by a computer.
• http://www.flickr.com/photos/pgoyette/107849943/in/photostream/
• But if the photographic signal is RAY CHANGES rather than absolute pixel values, it re-opens some long-settled questions in image processing; namely ‘what are the best ways to DEPICT visually significant changes? For example, everyone here knows the visually correct way to convert colors to their equivalent gray value . *BUT NOBODY HERE* (including me) can tell me the one true correct way to convert CHANGES in COLOR to CHANGES in LUMINANCE. There are visually significant CHANGES in color that get LOST when we simply remove the chrominance …
• ### Introduction to Camera Challenges - Ramesh Raskar

1. 1. © 2004 Marc Levoy The CityBlock Project Precursor to Google Streetview Maps
2. 2. Image Fusion & ReconstructionImage Fusion & Reconstruction • Single photo:Single photo: forces narrow tradeoffs:forces narrow tradeoffs: – Focus, Exposure, aperture, time, sensitivity, noise,Focus, Exposure, aperture, time, sensitivity, noise, – Usual result: Incomplete visual appearance.Usual result: Incomplete visual appearance. Multiple photosMultiple photos, assorted settings, assorted settings for Optics, Sensor, Lighting, Processingfor Optics, Sensor, Lighting, Processing • Fusion:Fusion: ‘Merge the best parts’‘Merge the best parts’ • Reconstruction:Reconstruction: Detect photo changes;Detect photo changes; compute scene invariantscompute scene invariants
3. 3. High Dynamic Range ImagingHigh Dynamic Range Imaging • Cameras have limited dynamic range Small Exposure image, dark inside 1/500 sec Large exposure image, saturated outside ¼ sec Images from Raanan Fattal
4. 4. High Dynamic Range ImagingHigh Dynamic Range Imaging • Combine images at different exposures • Exposure Bracketing • [Mann and Picard 95, Debevec et al 96] Images from Raanan Fattal
5. 5. How could we put all this information into one image ?
6. 6. Tone Map 20 bit image for 8 bit DisplayTone Map 20 bit image for 8 bit Display
7. 7. input smoothed (structure, large scale) residual (texture, small scale) Gaussian Convolution BLUR HALOS Naïve Approach: Gaussian Blur
8. 8. Impact of Blur and Halos • If the decomposition introduces blur and halos, the final result is corrupted. Sample manipulation: increasing texture (residual × 3)
9. 9. input smoothed (structure, large scale) residual (texture, small scale) edge-preserving: Bilateral Filter Bilateral Filter: no Blur, no Halos
10. 10. input
11. 11. increasing texture with Gaussian convolution H A L O S
12. 12. increasing texture with bilateral filter N O H A L O S
13. 13. Bilateral Filter on 1D Signal BF
14. 14. p Our Strategy Reformulate the bilateral filter – More complex space:  Homogeneous intensity  Higher-dimensional space – Simpler expression: mainly a convolution  Leads to a fast algorithm weights applied to pixels
15. 15. Attenuate High GradientsAttenuate High Gradients I(x) 1 105 1 Intensity I(x) 1 105 Intensity Maintain local detail at the cost of global range Fattal et al Siggraph 2002
16. 16. Attenuate High GradientsAttenuate High Gradients I(x) 1 105 G(x) 1 105 Intensity Gradient I(x) 1 105 Intensity Maintain local detail at the cost of global range Fattal et al Siggraph 2002
17. 17. Attenuate High GradientsAttenuate High Gradients I(x) 1 105 G(x) 1 105 Intensity Gradient I(x) 1 105 Intensity Keep low gradients Fattal et al Siggraph 2002
22. 22. Local Illumination ChangeLocal Illumination Change Original gradient field: Original Image: f * f∇ Modified gradient field: v Perez et al. Poisson Image editing, SIGGRAPH 2003
23. 23. Ambient Flash Self-Reflections and Flash HotspotSelf-Reflections and Flash Hotspot Hands Face Tripod
24. 24. ResultAmbient Flash Reflection LayerReflection Layer Hands Face Tripod
26. 26. Intensity Gradient Vectors in Flash and Ambient ImagesIntensity Gradient Vectors in Flash and Ambient Images Same gradient vector direction Flash Gradient Vector Ambient Gradient Vector Ambient Flash No reflections
27. 27. Reflection Ambient Gradient Vector Different gradient vector direction With reflections Ambient Flash Flash Gradient Vector
29. 29. Flash Projection = Result Residual = Reflection Layer Co-located Artifacts Ambient
30. 30. Recovering foreground layerRecovering foreground layer – Find tensor based on background image – Transform gradient field of foreground image Foreground maskImage Difference
31. 31. Dark Bldgs Reflections on bldgs Unknown shapes
32. 32. ‘Well-lit’ Bldgs Reflections in bldgs windows Tree, Street shapes
33. 33. Background is captured from day-time scene using the same fixed camera Night Image Day Image Context Enhanced Image
34. 34. Mask is automatically computed from scene contrast
35. 35. But, Simple Pixel Blending Creates Ugly Artifacts
36. 36. Pixel Blending
37. 37. Pixel Blending Our Method: Integration of blended Gradients
38. 38. Nighttime imageNighttime image Daytime imageDaytime image Gradient fieldGradient field ImportanceImportance image Wimage W FinalresultFinalresult Gradient fieldGradient field Mixed gradient fieldMixed gradient field GG11 GG11 GG22 GG22 xx YY xx YY II11 I2 GG GG xx YY
39. 39. Reconstruction from Gradient FieldReconstruction from Gradient Field • Problem: minimize error |∇ I’ – G| • Estimate I’ so that G = ∇ I’ • Poisson equation ∇ 2 I’ = div G • Full multigrid solver I’I’ GGXX GGYY
40. 40. Rene Magritte, ‘Empire of the Light’ Surrealism
41. 41. actual photomontageset of originals perceived
42. 42. Source images Brush strokes Computed labeling Composite
43. 43. Brush strokes Computed labeling
44. 44. • No Flash:No Flash: Candle warmth, but high noiseCandle warmth, but high noise • Flash:Flash: low noise, but no candle warmthlow noise, but no candle warmth Photography: Full of Tradeoffs...Photography: Full of Tradeoffs... No-flash Flash
45. 45. Image A: Warm, shadows, but too Noisy (too dim for a good quick photo) No-flash
46. 46. Image B: Cold, Shadow-free, Clean (flash: simple light, ALMOST no shadows)
47. 47. MERGE BEST OF BOTH: apply ‘Cross Bilateral’ or ‘Joint Bilateral’
48. 48. (it really is much better!)
49. 49. Image Fusion & ReconstructionImage Fusion & Reconstruction • Single photo:Single photo: forces narrow tradeoffs:forces narrow tradeoffs: – Focus, Exposure, aperture, time, sensitivity, noise,Focus, Exposure, aperture, time, sensitivity, noise, – Usual result: Incomplete visual appearance.Usual result: Incomplete visual appearance. Multiple photosMultiple photos, assorted settings, assorted settings for Optics, Sensor, Lighting, Processingfor Optics, Sensor, Lighting, Processing • Fusion:Fusion: ‘Merge the best parts’‘Merge the best parts’ • Reconstruction:Reconstruction: Detect photo changes;Detect photo changes; compute scene invariantscompute scene invariants
50. 50. The Media Lab Camera Culture Epsilon Photography Capture multiple photos, each with slightly different camera parameters. • Exposure settings • Spectrum/color settings • Focus settings • Camera position • Scene illumination
51. 51. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004 NEARNEAR
52. 52. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004 FARFAR
53. 53. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
54. 54. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
55. 55. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
56. 56. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
57. 57. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
58. 58. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
59. 59. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
60. 60. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
61. 61. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
62. 62. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
63. 63. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
64. 64. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
65. 65. FUSION: Best-Focus DistanceFUSION: Best-Focus Distance Agrawala et al., Digital Photomontage SIGGRAPH 2004
66. 66. Source images ‘Graph Cuts’ Solution FUSION Agrawala et al., Digital Photomontage SIGGRAPH 2004
67. 67. What else can we extend?What else can we extend? Film-Like Camera Parameters:Film-Like Camera Parameters: • Field of View: image stitching for panoramasField of View: image stitching for panoramas • Dynamic Range:Dynamic Range: Radiance MapsRadiance Maps • Frame Rate: Interleaved VideoFrame Rate: Interleaved Video • Resolution: ‘Super-resolution’ methodsResolution: ‘Super-resolution’ methods Visual Appearance & Content:Visual Appearance & Content: • Tone Map:Tone Map: Detail in every shadow and highlightDetail in every shadow and highlight • Color2grey:Color2grey: KeepKeep allall color changes in grayscalecolor changes in grayscale • Temporal Continuity: Space-time fusionTemporal Continuity: Space-time fusion • Viewpoint Constraints:Viewpoint Constraints: Multiple COP imagesMultiple COP images and more…and more…
68. 68. The Media Lab Camera Culture Epsilon Photography Capture multiple photos, each with slightly different camera parameters. • Exposure settings • Spectrum/color settings • Focus settings • Camera position • Scene illumination
69. 69. The Media Lab Camera Culture Project Ideas
70. 70. © 2004 Marc Levoy The CityBlock Project Precursor to Google Streetview Maps
71. 71. What is ‘interesting’ here? Social voting in the real world = ‘popular’
72. 72. Vein ViewerVein Viewer (Luminetx)(Luminetx) Near-IR camera locates subcutaneous veins and projectNear-IR camera locates subcutaneous veins and project their location onto the surface of the skin.their location onto the surface of the skin. Coaxial IR cameraCoaxial IR camera + Projector+ Projector Coaxial IR cameraCoaxial IR camera + Projector+ Projector
73. 73. Focus Adjustment: Sum of Bundles
74. 74. http://www.mne.psu.edu/psgdl/FSSPhotoalbum/index1.htm
75. 75. Varying PolarizationVarying Polarization Yoav Y. Schechner, Nir Karpel 2005Yoav Y. Schechner, Nir Karpel 2005 Best polarization state Worst polarization state Best polarization state Recovered image [Left] The raw images taken through a polarizer. [Right] White-balanced results: The recovered image is much clearer, especially at distant objects, than the raw image
76. 76. Varying PolarizationVarying Polarization • Schechner, Narasimhan, NayarSchechner, Narasimhan, Nayar • Instant dehazingInstant dehazing of images usingof images using polarizationpolarization
77. 77. Spatial Augmented Reality | Raskar 2011 Pamplona , Mohan, Oliveira, Raskar, Siggraph 2010 NETRA: Near Eye Tool for Refractive Assessment EyeNetra.com
78. 78. 90 Confocal Microscopy Examples Slides by Doug Lanman
79. 79. Beyond Visible SpectrumBeyond Visible Spectrum CedipRedShift
80. 80. MIT Media LabMIT Media Lab Camera CultureCamera Culture Ramesh RaskarRamesh Raskar MIT Media LabMIT Media Lab http:// CameraCulture . info/http:// CameraCulture . info/ Computational Camera &Computational Camera & Photography:Photography: Computational Camera &Computational Camera & Photography:Photography:
81. 81. http://www.flickr.com/photos/pgoyette/107849943/in/photostream/
82. 82. ScheimpflugScheimpflug principleprinciple
83. 83. Ramesh Raskar, Computational Illumination Computational Illumination
84. 84. Edgerton 1930’sEdgerton 1930’s Multi-flash Sequential Photography Stroboscope (Electronic Flash) Shutter Open Flash Time
85. 85. Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk Mitsubishi Electric Research Labs (MERL), Cambridge, MA U of California at Santa Barbara U of North Carolina at Chapel Hill Non-photorealistic Camera:Non-photorealistic Camera: Depth Edge DetectionDepth Edge Detection andand StylizedStylized RenderingRendering usingusing Multi-Flash ImagingMulti-Flash Imaging
86. 86. Depth Edges
87. 87. Our MethodCanny
88. 88. Flash MattingFlash Matting Flash Matting, Jian Sun, Yin Li, Sing Bing Kang, and Heung-Yeung Shum, Siggraph 2006
89. 89. DARPA Grand ChallengeDARPA Grand Challenge
90. 90. The Media Lab Camera Culture Epsilon Photography Capture multiple photos, each with slightly different camera parameters. • Exposure settings • Spectrum/color settings • Focus settings • Camera position • Scene illumination
91. 91. The Media Lab Camera Culture Lens Sensor Camera Static Scene Image Destabilization [Mohan, Lanman et al. 2009]
92. 92. The Media Lab Camera Culture Static Scene Lens Motion Sensor Motion Camera Image Destabilization [Mohan, Lanman et al. 2009]
93. 93. MIT Media Lab Camera Culture Our Prototype
94. 94. MIT Media Lab Camera Culture Adjusting the Focus Plane all-in-focus pinhole image
95. 95. MIT Media Lab Camera Culture Defocus Exaggeration destabilization simulates a reduced f-number
96. 96. The Media Lab Camera Culture Capturing Gigapixel Images [Kopf et al, 2007] 3,600,000,000 Pixels Created from about 800 8 MegaPixel Images
97. 97. The Media Lab Camera Culture Capturing Gigapixel Images [Kopf et al, 2007]
98. 98. Color Original Grayscale New Method Color2Gray:Color2Gray: Salience-PreservingSalience-Preserving Color RemovalColor Removal SIGGRAPH 2005 Gooch, Olsen, Tumblin, Gooch