Ramesh Raskar presents his vision for computational photography and discusses several open research problems. He envisions cameras that provide ultimate post-capture control over focus and motion blur. He also discusses capturing the essence of a scene through techniques like multi-flash photography. Raskar outlines a wish list that includes understanding the world through analyzing photos, sharing visual experiences while preserving privacy, and looking around corners using techniques like transient imaging. The talk promotes co-designing optics and digital processing to change the rules of photography.

1. Computational Light Transport and Computational Photography: Inverse problemsCamera CultureRamesh RaskarRamesh Raskarhttp://raskar.infoMIT Media Labraskar@mit.edu

3. How to Invent?After X, what is neXtFull Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010Ramesh Raskar, MIT Media Lab

4. Ramesh Raskar, http://raskar.infoX+YXneXtXdXX++XFull Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010

5. Simple Exercise .. Image CompressionSave Bandwidth and storageWhat is neXt

6. Strategy #1: XdExtend it to next (or some other) dimension ..

7. X = Idea you just heardConceptPatentNew Product/Best project/invention awardProduct featureDesignArtAlgorithm

8. Ramesh Raskar, http://raskar.infoX+YXneXtXdXX++XFull Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010

9. Research .. http://raskar.infoHow to come up w ideas: Idea HexagonHow to write a paperHow to give a talkOpen research problemsHow to decide merit of a projectHow to attend a conference, brainstormFacebook.com/ rRaskarTipsGet on Seminar/Talks mailing lists worldwidehttp://www.cs.virginia.edu/~robins/YouAndYourResearch.htmlWhy do so few scientists make significant contributions and so many are forgotten in the long run?Highly recommended Hamming talk at Bell Labs

10. Is project worthwhile? Heilmeier's Questionshttp://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_CatechismWhatWhat are you trying to do? Articulate your objectives using absolutely no jargon.Related workHow is it done today, and what are the limits of current practice?ContributionWhat's new in your approach and why do you think it will be successful?MotivationWho cares?If you're successful, what difference will it make?ChallengesWhat are the risks and the payoffs?How much will it cost?How long will it take?EvaluationWhat are the midterm and final "exams" to check for success?Raskar additions: Why now? (why not before, what’s new that makes possible)Why us? (wrong answers: I am smart, I can work harder than others)

11. Great Research: Strive for FiveBefore Five teams Be first, often let others do detailsBeyond Five years What no one is thinking aboutWithin Five layers of ‘Human’ Impact RelevanceBeyond Five minutes of description Deep, iterative, participatoryFusing Five+ Expertise Multi-disciplinary, proactiveRamesh Raskar, http://raskar.info

12. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskarInverse ProblemsHow to do Research in ImagingInverse Problems, Reconstruction, Rank and SparsityCo-design of Optics and ComputationPhotons not just pixelsMid-level cuesComputational PhotographyOpen research problemsCompressive Sensing for High Speed EventsLimits of CS for general imagingComputational Light TransportLooking Around Corners, trillion fpsLightfields: 3D Displays and Holograms

13. Tools forVisual ComputingShadowRefractiveReflectiveFernald, Science [Sept 2006]

14. Computational PhotographyCamera CultureRamesh Raskar

15. Traditional PhotographyDetectorLensPixelsMimics Human Eye for a Single Snapshot: Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static worldImageCourtesy: Shree Nayar

16. PictureComputational Camera + Photography: Optics, Sensors and ComputationsGeneralizedSensorGeneralized OpticsComputationsRay Reconstruction4D Ray BenderUpto 4D Ray SamplerMerged Views, Programmable focus and dynamic range, Closed-loop Controlled Illumination, Coded exposure/apertures

17. Computational PhotographyNovel IlluminationLight SourcesModulatorsComputational CamerasGeneralized OpticsGeneralizedSensorGeneralizedOpticsProcessing4D Incident Lighting4D Ray BenderRay ReconstructionUpto 4D Ray Sampler4D Light FieldDisplayScene: 8D Ray ModulatorRecreate 4D Lightfield

18. Computational Photography [Raskar and Tumblin]captures a machine-readable representation of our world tohyper-realistically synthesize the essence of our visual experience. ResourcesICCP 2012, Seattle Apr 2012Papers due Dec 2nd, 2011http://wikipedia.org/computational_photographyhttp://raskar.info/photo

19. Computational PhotographyComputational Photography aims to make progress on both axisPhototourismComprehensiveEssenceScene completion from photosAugmented Human ExperienceLooking Around CornersPriorsCaptureHuman Stereo VisionMetadataCodedDepthfg/bgNon-visual Data, GPSVirtual Object InsertionSpectrumDecomposition problems8D reflectance fieldDirect/GlobalLightFieldsRelightingEpsilonAngle, spectrum awareCamera ArrayHDR, FoVFocal stackResolutionMaterial editing from single photoDigitalMotion MagnificationRawLow LevelMid LevelHighLevelHyper realismSynthesis/Analysis

20. Co-designing Optical and Digital ProcessingComputational Light TransportOpticsDisplaysSensorsComputational PhotographyPhoton HackingIlluminationSignal ProcessingComputer VisionMachine LearningBit Hacking

21. Take home pointsCo-design of hw/swAvoid computational or optical chauvinism in imaging (Camera flash/Kinect)Hardware cost going to zero, Parallel technology trendsComputer vision not just mimicking human vision/perceptionBorrow ideas from other fields: astronomy, scientific imaging, audio, communicationsPhotons not just PixelsChange the rules of the gameOptics, Sensors, Illum, Priors, Sparsity, TransformsMeta-data, Internet collection, Crowdsourcing

22. Computational PhotographyWish List: Open Research ProblemsCamera CultureRamesh Raskar

23. Wish #1Ultimate Post-capture ControlCamera CultureRamesh Raskar

24. Digital Refocusing using Light Field Camera125μ square-sided microlenses[Ng et al 2005]

25. Motion Blur in Low Light

26. TraditionalBlurred PhotoDeblurred Image

27. Fluttered Shutter CameraRaskar, Agrawal, Tumblin Siggraph2006Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence

28. Preserves High Spatial FrequenciesFourier TransformSharp PhotoBlurred PhotoPSF == Broadband FunctionFlutter Shutter: Shutter is OPEN and CLOSED

29. Coded ExposureTraditionalDeblurred ImageDeblurred ImageImage of Static Object

30. Motion Blur in Low Light

31. Fast periodic phenomenaVocal folds flapping at 40.4 HzBottling line4000 fps hi-speed camera500 fps hi-speed camera

32. Compressive Sensing Single Pixel Cameraimagecompressive imagemeasurement matrix

33. Periodic signals-fP-2fP-4fP3fP-3fP0fMax- fMax2fPfP=1/P4fPPeriodic signal x(t) with period PtP = 16msPeriodic signal with period P and band-limited to fMax = 500 Hz. Fourier transform is non-zero only at multiples of fP=1/P ~ 63Hz.

34. High speed cameraP = 16msTs = 1/(2 fMax)-fP-2fP-4fP-3fP4fP3fP2fP0fMax- fMaxfP=1/PNyquist Sampling of x(t) Periodic signal has regularly spaced, sparse Fourier coefficients. Is it necessary to use a high-speed video camera? Why waste bandwidth?

35. Traditional StrobingUse low frame-rate camera and generate beat frequencies.PtLow exposure to avoid blurring. Low light throughput.Period known apriori.Strobing animation credit Wikipedia

36. tPRandom Projections Per Frame of Camera using Coded Strobing PhotographyIn every exposure duration observe different linear combinations of the periodic signal.Advantage of the design Exposure coding independent of the frequency

37. On an average, light throughput is 50%Coded Strobing Photography. Reddy, D., Veeraraghavan, A., Raskar, R. IEEE PAMI 2011

38. Observation Modelx at 2000fpsy at 25fps

39. Signal Modelx at 2000fpsy at 25fps

40. Signal & Observation ModelAis M x N, M<<Nx at 2000fpsy at 25fps N / M = 2000 / 25 = 80

41. Recovery: SparsityVery few non-zero elementsy = A sObserved valuesMixing matrixStructured Sparse CoefficientsBasis Pursuit De-noising

42. Simulation on hi-speed toothbrush25fps normal camera25fps coded strobing cameraReconstructed frames2000fps hi-speed camera~100X speedup

43. Rotating mill toolMill tool rotating at 50HzReconstructed Video at 2000fpsNormal Video: 25fpsCoded Strobing Video: 25fpsBlur increases as rotational velocity increases rotating at 200Hzrotating at 150Hzrotating at 100Hzincreasing blur

44. Compressive Sensing for Images .. A good idea?Single Pixel Cameraimagecompressive imagemeasurement matrix

45. Is Randomized Projection-based Captureapt for Natural Images ? Periodic SignalsProgressive ProjectionsRandomized ProjectionsCompression Ratio[Pandharkar, Veeraraghavan, Raskar 2009]

46. Compact ProgrammableLights ?

47. Wish #1Ultimate Post-capture ControlDigital Refocus and Motion blur

48. Emulate studio light from compact flashCamera CultureRamesh Raskar

49. Wish #2Freedom from FormSize, Weight, Power, UI

50. Flat camera: Bidirectional screen (BiDi)Shallow DoF from tiny lensCamera CultureRamesh Raskar

51. Wish #3Understand the WorldCamera CultureRamesh Raskar

52. Convert single 2D photo into 3D ?Snavely, Seitz, SzeliskiU of Washington/Microsoft: Photosynth

53. Exploit Community Photo CollectionsU of Washington/Microsoft: Photosynth

54. Wish #3Understand the WorldIdentify/recognize Materials

55. 3D Awareness

56. Interact with informationCamera CultureRamesh Raskar

57. Wish #4Sharing Visual ExperienceLifeLog Auto-summary

58. Privacy in public and authentication

59. Hyper-real Photo Frames

60. Print ‘material’ Camera CultureRamesh Raskar

61. Wish #5Capturing EssenceCamera CultureRamesh Raskar

62. What are the problems with ‘real’ photo in conveying information ?Why do we hire artists to draw what can be photographed ?

63. ShadowsClutterMany ColorsHighlight Shape EdgesMark moving partsBasic colors

64. Depth Edges with MultiFlashRaskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004

69. Depth DiscontinuitiesInternal and externalShape boundaries, Occluding contour, Silhouettes

70. Depth Edges

71. Our MethodCanny

72. ResultPhotoCanny Intensity Edge DetectionOur Method

73. QuestionsWhat will a camera look like in 10,20 years?How will a billion networked and portable cameras change the social culture? How will online photo collections transform visual social computing?How will movie making/new reporting change?

74. Photos of tomorrow: computed not recordedhttp://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/

75. Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.infoSensorComputational Photography Wish ListPost-capture control

76. Emulate studio lights with compact flash

77. Focus and motion blur

78. New forms

79. Flat camera, large LCDs as cameras

80. Image destabilization for larger aperture

81. Understand the world

82. Real or fake

83. Place 2D photo into 3D

84. Look around corner

85. Bokode: long distance barcode

86. Sharing

87. Lifelogs auto summary

88. Privacy/Verification

89. 6D photoframes

90. Essence

91. New visual arts

92. Multi-flash camera

93. Delta-camera and Blind-cameraTake home pointsCo-design of hw/swAvoid computational or optical chauvinism in imaging (Camera flash/Kinect)Hardware cost going to zero, Parallel technology trendsComputer vision not just mimicking human vision/perceptionBorrow ideas from other fields: astronomy, scientific imaging, audio, communicationsPhotons not just PixelsChange the rules of the gameOptics, Sensors, Illum, Priors, Sparsity, TransformsMeta-data, Internet collection, Crowdsourcing

94. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskarInverse ProblemsHow to do Research in ImagingInverse Problems, Reconstruction, Rank and SparsityCo-design of Optics and ComputationPhotons not just pixelsMid-level cuesComputational PhotographyOpen research problemsCompressive Sensing for High Speed EventsLimits of CS for general imagingComputational Light TransportLooking Around Corners, trillion fpsLightfields: 3D Displays and Holograms

95. Every Photon has a Story

96. What isaround the corner ?

97. Can you look around the corner ?

98. Multi-path Analysis2nd Bounce1st Bounce3rd Bounce

99. Femto-Photography (Transient Imaging)FemtoFlashTrillion FPS cameraWith M Bawendi, MIT ChemistrySerious SyncComputational Optics2011: CVPR (Pandharkar, Velten, Bardagjy, Bawendi, Raskar)

100. 2009: Marr PrizeHonorable Mention (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)

101. 2008: Transient Light Transport (Raskar, Davis, March 2008)Inverting Light TransportDirect/GlobalMultiple Scattering[Seitz , Kutulakos, Matsushita 2005][Nayar, Raskar et al 2006][Atcheson et al 2008][Kutulakos, Steger 2005]Dual PhotographyLIDAR[Sen et al 2005]

102. Multi-Dimensional Light Transport5-D TransportGigapan

103. Collision avoidance, robot navigation, …

104. zxSLRsOccluderStreak-camera3rd bounceCLaser beamBEchoes of Light

105. Steady State 4DImpulse Response, 5D

106. Scene with Ultra fast illumination and camerahidden elementsRaw 5D CaptureTime profilesSignal Proc.Photo, geometry, reflectance beyond line of sight Novel light transport models and inference algorithms®t3D Time imagesFemto-PhotographyTime Resolved Multi-path Imaging

107. TeamMoungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Associate, MIT Media LabRohitPandharkar, RA, MIT Media LabOtkrist Gupta, RA, MIT Media LabAndrew Matthew Bardagjy, RA, MIT Media LabNikhil Naik, RA, MIT Media LabTyler Hutchison, RA, MIT Media LabEverett Lawson, MIT Media LabRamesh Raskar, Asso. Prof., MIT Media LabCamera CultureRamesh Raskar

108. Photos from Streak CameraCapture SetupHidden Scene

109. Photos from Streak CameraCapture SetupHidden SceneOverlayReconstruction

110. Motion beyond line of sightPandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011

111. …, bronchoscopies, …Participating Media

112. PhotoFirst BounceLater Bounces+DirectGlobal[Nayar, Krishnan, Grossberg, Raskar 2006]

114. Each frame = ~2ps = 0.6 mm of Light Travel

115. Ripples of Waves

118. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskarInverse ProblemsHow to do Research in ImagingInverse Problems, Reconstruction, Rank and SparsityCo-design of Optics and ComputationPhotons not just pixelsMid-level cuesComputational PhotographyOpen research problemsCompressive Sensing for High Speed EventsLimits of CS for general imagingComputational Light TransportLooking Around Corners, trillion fpsLightfields: 3D Displays and Holograms

120. View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale

121. Two Layer Displaysbarrierlensletsensor/displaysensor/displayPB = dim displaysLenslets = fixed spatial and angular resolutionDynamic Masks = Brighter, High spatial resolution

122. Limitations of 3D DisplayParallaxbarrierLCD displayFrontBackLanman, Hirsch, Kim, RaskarSiggraph Asia 2010

123. Light Field Analysis of BarrierskL[i,k]i`kg[k]iL[i,k]f[i]light box

124. Content-Adaptive Parallax BarriersL[i,k]`kg[k]if[i]light box

125. ImplementationComponents 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]Content-Adaptive Parallax BarriersL[i,k]`kg[k]if[i]light box

126. Content-Adaptive Parallax Barriers`=

127. Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010Rank-Constrained Displays and LF Adaptation`Content-Adaptive Parallax Barriers=All dual layer display = rank-1 constraint Light field display is a matrix approximation problemExploit content-adaptive parallax barriers

128. Optimization: Iteration 1rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

129. Optimization: Iteration 10rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

130. Optimization: Iteration 20rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

131. Optimization: Iteration 30rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

132. Optimization: Iteration 40rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

133. Optimization: Iteration 50rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

134. Optimization: Iteration 60rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

135. Optimization: Iteration 70rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

136. Optimization: Iteration 80rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

137. Optimization: Iteration 90rear mask: f1[i,j]front mask: g1[k,l]reconstruction (central view)Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.

138. Content-Adaptive Front Mask (1 of 9)

139. Content-Adaptive Rear Mask (1 of 9)

140. Emitted 4D Light Field

141. Conclusion`Content-Adaptive Parallax Barriers= Described a rank constraint for all dual-layer displays

142. With a fixed pair of masks, emitted light field is rank-1

143. Achieved higher-rank approximation using temporal multiplexing

144. With T time-multiplexed masks, emitted light field is rank-T

145. Constructed a prototype using off-the-shelf panels

146. Demonstrated light field display is a matrix approximation problem

147. Introduced content-adaptive parallax barriers

148. Applied weighted NMF to optimize weighted Euclidean distance to targetAdaptation increases brightness and refresh rate of dual-stacked LCDs

149. Parallax Barrier: Np=103 pix.Hologram: NH=105 pix.ϕP∝w/dϕH∝λ/tHθp=10 pixθH =1000 pixFourier PatchxH =100 patcheswxp=100slitsHorstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009

150. Augmented Light Field118wave optics basedrigorous but cumbersomeWigner Distribution FunctionWDFAugmented LFTraditional Light FieldTraditional Light Fieldray optics basedsimple and powerfulInterference & DiffractionInteraction w/ optical elementsOh, Raskar, Barbastathis 2009: Augmented Light Field

151. positionlight field transformerLFLFLFLF(diffractive)optical elementReferenceplaneLF propagationLF propagationLight FieldsGoal: Representing propagation, interaction and image formation of light using purely position and angle parametersangle

152. Augmented Lightfield for Wave Optics EffectsWDFWigner Distribution FunctionAugmented Light FieldLight FieldLFLF < WDFLacks phase propertiesIgnores diffraction, interferrenceRadiance = PositiveALF ~ WDFSupports coherent/incoherentRadiance = Positive/NegativeVirtual light sources

153. Free-space propagationLight field transformerVirtual light projector Possibly negative radiance121

154. Lightfieldvs Hologram Displays

155. Is hologram just another ray-based light field?Can a hologram create any intensity distribution in 3D?Why hologram creates a ‘wavefront’ but PB does not?Why hologram creates automatic accommodation cues?What is the effective resolution of HG vs PB?

156. Zooming into the Light FieldRays: No Bending1 Fresnel HG Patchp Wm* * p d(θ)q d(θ)* qqpp* q WmL(x,θ)W(x,u)Wm= sincd = deltauθ

157. Algebraic Rank ConstraintRank-3Rank-1Rank-1s1*s1m2s1*m2s1(a) Parallax Barrier(c) Hybrid(b) Holograms1s1

158. (a) Two Slits, CoherentInterferencexʹRank-1-1Transformu-TransformR45, Dx<t(x+xʹ/2)t*(x-xʹ/2)>t(x1)t*(x2)t(x+xʹ/2)t*(x-xʹ/2)W(x,u)

159. L1(x,θ)(a)L1L2(x,θ)L2L3(x,θ)s1m2L3hHϕ1ϕ1z2ϕ1ϕ1z1rdL3(x,θ)L1(x,θ)L2(x,θ)

160. Is hologram just another ray-based light field?Can a hologram create any intensity distribution in 3D?Why hologram creates a ‘wavefront’ but PB does not?Why hologram creates automatic accommodation cues?What is the effective resolution of HG vs PB?

161. Three QuestionsWhat are the benefits of higher dimensional imaging?Why is the algebraic rank of a Light Field not full?What makes looking around the corner possible?

162. How to do Research in Imaginghttp://raskar.infoHow to come up w ideas: Idea HexagonHow to write a paperHow to give a talkOpen research problemsHow to decide merit of a projectHow to attend a conference, brainstormFacebook.com/ rRaskarTipsGet on Seminar/Talks mailing lists worldwidehttp://www.cs.virginia.edu/~robins/YouAndYourResearch.htmlWhy do so few scientists make significant contributions and so many are forgotten in the long run?Highly recommended Hamming talk at Bell Labs

163. Take home pointsCo-design of hw/swAvoid computational or optical chauvinism in imaging (Camera flash/Kinect)Hardware cost going to zero, Parallel technology trendsComputer vision not just mimicking human vision/perceptionBorrow ideas from other fields: astronomy, scientific imaging, audio, communicationsPhotons not just PixelsChange the rules of the gameOptics, Sensors, Illum, Priors, Sparsity, TransformsMeta-data, Internet collection, Crowdsourcing

164. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskarInverse ProblemsHow to do Research in ImagingInverse Problems, Reconstruction, Rank and SparsityCo-design of Optics and ComputationPhotons not just pixelsMid-level cuesComputational PhotographyOpen research problemsCompressive Sensing for High Speed EventsLimits of CS for general imagingComputational Light TransportLooking Around Corners, trillion fpsLightfields: 3D Displays and HologramsApply for internships/post-docneXt