Computational Light Transport <br />and <br />Computational Photography: <br />Inverse problems<br />Camera Culture<br />R...
How to Invent?<br />After X, what is neXt<br />Full Presentation at <br />http://www.slideshare.net/cameraculture/raskar-i...
Ramesh Raskar, http://raskar.info<br />X+Y<br />X<br />neXt<br />Xd<br />X<br />X++<br />X<br />Full Presentation at <br /...
Simple Exercise .. <br />Image Compression<br />Save Bandwidth and storage<br />What is neXt<br />
Strategy #1:    Xd<br />Extend it to next (or some other) dimension ..<br />
X = <br />Idea you just heard<br />Concept<br />Patent<br />New Product/Best project/invention award<br />Product feature<...
Ramesh Raskar, http://raskar.info<br />X+Y<br />X<br />neXt<br />Xd<br />X<br />X++<br />X<br />Full Presentation at <br /...
Research .. <br />http://raskar.info<br />How to come up w ideas: Idea Hexagon<br />How to write a paper<br />How to give ...
Is project worthwhile? Heilmeier's Questions<br />http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism...
Great Research: Strive for Five<br />Before Five teams<br />	Be first,  often let others do details<br />Beyond Five years...
MIT Media Lab          raskar@mit.edu  http://cameraculture.info	fb.com/rraskar<br />Inverse Problems<br />How to do Resea...
Tools <br />for<br />Visual Computing<br />Shadow<br />Refractive<br />Reflective<br />Fernald, Science [Sept 2006]<br />
Computational Photography<br />Camera Culture<br />Ramesh  Raskar<br />
Traditional  Photography<br />Detector<br />Lens<br />Pixels<br />Mimics Human Eye for a Single Snapshot:<br />	Single Vie...
Picture<br />Computational  Camera + Photography: Optics, Sensors and Computations<br />GeneralizedSensor<br />Generalized...
Computational Photography<br />Novel Illumination<br />Light Sources<br />Modulators<br />Computational Cameras<br />Gener...
Computational Photography [Raskar and Tumblin]<br />captures a machine-readable representation of our world to<br />hyper-...
Computational Photography<br />Computational Photography aims to make progress on both axis<br />Phototourism<br />Compreh...
Co-designing Optical and Digital Processing<br />Computational Light Transport<br />Optics<br />Displays<br />Sensors<br /...
Take home points<br />Co-design of hw/sw<br />Avoid computational or optical chauvinism in imaging  <br />(Camera flash/Ki...
Computational Photography<br />Wish List: <br />Open Research Problems<br />Camera Culture<br />Ramesh  Raskar<br />
Wish #1<br />Ultimate Post-capture Control<br />Camera Culture<br />Ramesh  Raskar<br />
Digital Refocusing using Light Field Camera<br />125μ square-sided microlenses<br />[Ng et al 2005]<br />
Motion Blur in Low Light<br />
Traditional<br />Blurred Photo<br />Deblurred Image<br />
Fluttered Shutter Camera<br />Raskar, Agrawal, Tumblin Siggraph2006<br />Ferroelectric shutter in front of the lens is tur...
Preserves High Spatial Frequencies<br />Fourier Transform<br />Sharp Photo<br />Blurred Photo<br />PSF == Broadband Functi...
Coded Exposure<br />Traditional<br />Deblurred Image<br />Deblurred Image<br />Image of Static Object<br />
Motion Blur in Low Light<br />
Fast periodic phenomena<br />Vocal folds flapping at 40.4 Hz<br />Bottling line<br />4000 fps hi-speed camera<br />500 fps...
Compressive Sensing <br />Single Pixel Camera<br />image<br />compressive image<br />measurement matrix <br />
Periodic signals<br />-fP<br />-2fP<br />-4fP<br />3fP<br />-3fP<br />0<br />fMax<br />- fMax<br />2fP<br />fP=1/P<br />4f...
High speed camera<br />P = 16ms<br />Ts = 1/(2 fMax)<br />-fP<br />-2fP<br />-4fP<br />-3fP<br />4fP<br />3fP<br />2fP<br ...
Traditional Strobing<br />Use low frame-rate camera and generate beat frequencies.<br />P<br />t<br />Low exposure to avoi...
t<br />P<br />Random Projections Per Frame of Camera using Coded Strobing Photography<br />In every exposure duration obse...
 On an average, light throughput is 50%</li></ul>Coded Strobing Photography. Reddy, D., Veeraraghavan, A., Raskar, R. IEEE...
Observation Model<br />x at 2000fps<br />y at 25fps <br />
Signal Model<br />x at 2000fps<br />y at 25fps <br />
Signal & Observation Model<br />Ais M x N,  M<<N<br />x at 2000fps<br />y at 25fps <br />N / M = 2000 / 25 = 80<br />
Recovery: Sparsity<br />Very few non-zero elements<br />y    =                  A                s<br />Observed values<br...
Simulation on hi-speed toothbrush<br />25fps normal camera<br />25fps coded strobing camera<br />Reconstructed frames<br /...
Rotating mill tool<br />Mill tool rotating at 50Hz<br />Reconstructed Video at 2000fps<br />Normal Video: 25fps<br />Coded...
Compressive Sensing for Images .. A good idea?<br />Single Pixel Camera<br />image<br />compressive image<br />measurement...
Is Randomized Projection-based Captureapt for Natural Images ? <br />Periodic Signals<br />Progressive  Projections<br />R...
Compact ProgrammableLights ?<br />
Wish #1<br />Ultimate Post-capture Control<br /><ul><li>Digital Refocus and Motion blur
Emulate studio light from compact flash</li></ul>Camera Culture<br />Ramesh  Raskar<br />
Wish #2<br />Freedom  from  Form<br /><ul><li>Size, Weight, Power, UI
Flat camera: </li></ul>		Bidirectional screen (BiDi)<br /><ul><li>Shallow DoF from tiny lens</li></ul>Camera Culture<br />...
Wish #3<br />Understand the World<br />Camera Culture<br />Ramesh  Raskar<br />
Convert single 2D photo into 3D ?<br />Snavely, Seitz, Szeliski<br />U of Washington/Microsoft: Photosynth<br />
Exploit Community Photo Collections<br />U of Washington/Microsoft: Photosynth<br />
Wish #3<br />Understand the World<br /><ul><li>Identify/recognize Materials
3D Awareness
Interact with information</li></ul>Camera Culture<br />Ramesh  Raskar<br />
Wish #4<br />Sharing Visual Experience<br /><ul><li>LifeLog Auto-summary
Privacy in public and authentication
Hyper-real Photo Frames
Print ‘material’ </li></ul>Camera Culture<br />Ramesh  Raskar<br />
Wish #5<br />Capturing Essence<br />Camera Culture<br />Ramesh  Raskar<br />
What are the problems with ‘real’ photo in conveying information ?<br />Why do we hire artists to draw what can be photogr...
Shadows<br />Clutter<br />Many Colors<br />Highlight Shape Edges<br />Mark moving parts<br />Basic colors<br />
Depth Edges with MultiFlash<br />Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004<br />
Depth Discontinuities<br />Internal and externalShape boundaries, Occluding contour, Silhouettes<br />
Depth Edges<br />
Our Method<br />Canny<br />
Result<br />Photo<br />Canny Intensity Edge Detection<br />Our Method<br />
Questions<br />What will a camera look like in 10,20 years?<br />How will a billion networked and portable cameras change ...
Photos of tomorrow:  computed not recorded<br />http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/<br />
Camera Culture Group, MIT Media Lab                    Ramesh  Raskar    http://raskar.info<br />Sensor<br />Computational...
Emulate studio lights with compact flash
Focus and motion blur
New forms
Flat camera, large LCDs as cameras
Image destabilization for larger aperture
Understand the world
Real or fake
Place 2D photo into 3D
Look around corner
Bokode: long distance barcode
Sharing
Lifelogs auto summary
Privacy/Verification
6D photoframes
Essence
New visual arts
Multi-flash camera
Delta-camera and Blind-camera</li></li></ul><li>Take home points<br />Co-design of hw/sw<br />Avoid computational or optic...
MIT Media Lab          raskar@mit.edu  http://cameraculture.info	fb.com/rraskar<br />Inverse Problems<br />How to do Resea...
Every  Photon  has a Story<br />
What isaround the corner ?<br />
Can you look around the corner ?<br />
Multi-path Analysis<br />2nd Bounce<br />1st Bounce<br />3rd Bounce<br />
Femto-Photography (Transient Imaging)<br />FemtoFlash<br />Trillion FPS camera<br />With M Bawendi, MIT Chemistry<br />Ser...
2009:  Marr PrizeHonorable Mention (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)
2008: Transient Light Transport (Raskar, Davis, March 2008)</li></li></ul><li>Inverting Light Transport<br />Direct/Global...
Multi-Dimensional Light Transport<br />5-D Transport<br />Gigapan<br />
Collision avoidance, robot navigation, …<br />
z<br />x<br />S<br />L<br />R<br />s<br />Occluder<br />Streak-camera<br />3rd bounce<br />C<br />Laser beam<br />B<br />E...
Steady State 4D<br />Impulse Response, 5D<br />
Scene with <br />Ultra fast illumination and camera<br />hidden elements<br />Raw <br />5D Capture<br />Time profiles<br /...
Team<br />Moungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Assoc...
Photos from Streak Camera<br />Capture Setup<br />Hidden Scene<br />
Photos from Streak Camera<br />Capture Setup<br />Hidden Scene<br />Overlay<br />Reconstruction<br />
Motion beyond line of sight<br />Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar,  CVPR 2011 <br />
…, bronchoscopies, …<br />Participating Media<br />
Photo<br />First Bounce<br />Later Bounces<br />+<br />Direct<br />Global<br />[Nayar, Krishnan, Grossberg, Raskar   2006]...
Each frame = ~2ps = 0.6 mm of Light Travel<br />
Ripples of Waves<br />
MIT Media Lab          raskar@mit.edu  http://cameraculture.info	fb.com/rraskar<br />Inverse Problems<br />How to do Resea...
View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale<br />
Two Layer Displays<br />barrier<br />lenslet<br />sensor/display<br />sensor/display<br />PB = dim displays<br />Lenslets ...
 Limitations of 3D Display<br />Parallaxbarrier<br />LCD display<br />Front<br />Back<br />Lanman, Hirsch, Kim, RaskarSigg...
Light Field Analysis of Barriers<br />k<br />L[i,k]<br />i<br />`<br />k<br />g[k]<br />i<br />L[i,k]<br />f[i]<br />light...
Content-Adaptive Parallax Barriers<br />L[i,k]<br />`<br />k<br />g[k]<br />i<br />f[i]<br />light box<br />
Implementation<br />Components<br /><ul><li> 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]</li></li></ul><l...
Content-Adaptive Parallax Barriers<br />`<br />=<br />
Lanman, Hirsch, Kim, Raskar   Siggraph Asia 2010<br />Rank-Constrained Displays and LF Adaptation<br />`<br />Content-Adap...
Optimization: Iteration 1<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel ...
Optimization: Iteration 10<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel...
Optimization: Iteration 20<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel...
Upcoming SlideShare
Loading in...5
×

Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School

1,808

Published on

How to do research, Idea Hexagon, Rank and Sparsity in imaging problems, Looking around corners, compressive sensing of periodic phenomena, 3D displays, fast computation

Published in: Technology, Art & Photos
1 Comment
2 Likes
Statistics
Notes
  • See useful resources for students on http://raskar.info

    http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,808
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
80
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide
  • Six ways of coming up with new ideas based on an idea ‘X’.Ramesh RaskarAssociate ProfessorMIT Media Labhttp://raskar.infohttp://cameraculture.infoFull Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010http://raskar.infohttp://cameraculture.info
  • Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • X up: Airbags for car, for helicopter
  • Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism
  • Five on Five= If more than five teams in the world are doing the same research, don’t do it.= If you disappear for five years, will someone do it anyway? Then your idea is not that great anyway. = Can you explain your work in five sentences to your grandmahow it will impact human life?= If you can explain the idea in five minutes to a student and disappear for five years, will s/he be able to do it on her/his own without additional input from you/without iterations .. It is too obvious and lacks depth .. Don’t do it.= Strive to work on ideas that may require five+ disciplines .. Today’s research is highly team-driven and more diverse the required team composition, more fun you will have and also indicates a natural barrier to entry for others satisfying condition 1 and 2Much like the food pyramid, five servings are the goal and will make you stronger .. But ok if your research project does not satisfy all five conditions
  • But the world is 4D
  • See computationalphotography.orgMove away from obsession about SNR, space-bandwidth, diffraction limit and so on
  • My work involves creative new ways to play with light by co-designing optical and digital processing.My work lies at the INTERSECTION of processing of photons and processing of bits.At MERL, I transformed the field of computational photography, with key papers and impact on productsAt Media Lab, I invented a new field ‘computational light transport’
  • Compressive sensing via random projections not suitable for images and even videos
  • Rudy Burger, ‘don’t use flash and destroy the image’Can we use flash not just for improving scene brightness but for enhancing the mood? Like in studio lights?Main difference between professionals and consumers is lighting.
  • http://cameraculture.infohttp://raskar.info
  • My idea is to use the multiple bounces of light i.e. echoes of light.Echoes of sound -&gt; Echoes of lightWe all know about echoes of sound.But sounds travels slow and we can actually hear the echoesLight travels fast so we need specialized hardware to ‘listen’ to these echoes.So we end up using light sources and cameras that run at a trillion frames per second (not a million and not a billion, but trillion)
  • Trillion fps camera (which was previously used only for specialized biochemistry expt)This new form of imaging is possible by fusion of dissimilar .. A specialized camera previously used only in biochemistry labs and a new computational method that analyzes multiple bounces of light.I started the project just before I joined MIT in summer 2008.The hardware we use is in the lab of Prof Bawendi, MIT Chemistry, who is now a collaborator
  • Here is a road map for this ambitious research project based on time-resolved imaging .. Non line of sight Looking around corner (LaC) is just one example .. Such Time resolved imaging requires one to develop a completely new set of tool for understanding our world.This is a project I started just before coming to MIT via an NSF proposal.
  • The reconstruction is very low right now, about 80x80 pixels. So these are just baby steps.
  • The reconstruction is very low right now, about 80x80 pixels. So these are just baby steps.Data collected and reconstructions program by Andreas Velten, scientist in my group
  • Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
  • A cross section through a single M. rhetenor scale. Light reflected off each level of the “Christmas tree structure” gives the butterfly its iridescent color. Credit: Pete Vukusic, University of Exeter
  • Lanman, Hirsch, Kim, RaskarSiggraph Asia 2010
  • Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School

    1. 1. Computational Light Transport <br />and <br />Computational Photography: <br />Inverse problems<br />Camera Culture<br />Ramesh Raskar<br />Ramesh Raskar<br />http://raskar.info<br />MIT Media Lab<br />raskar@mit.edu<br />
    2. 2.
    3. 3. How to Invent?<br />After X, what is neXt<br />Full Presentation at <br />http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010<br />Ramesh Raskar, MIT Media Lab<br />
    4. 4. Ramesh Raskar, http://raskar.info<br />X+Y<br />X<br />neXt<br />Xd<br />X<br />X++<br />X<br />Full Presentation at <br />http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010<br />
    5. 5. Simple Exercise .. <br />Image Compression<br />Save Bandwidth and storage<br />What is neXt<br />
    6. 6. Strategy #1: Xd<br />Extend it to next (or some other) dimension ..<br />
    7. 7. X = <br />Idea you just heard<br />Concept<br />Patent<br />New Product/Best project/invention award<br />Product feature<br />Design<br />Art<br />Algorithm<br />
    8. 8. Ramesh Raskar, http://raskar.info<br />X+Y<br />X<br />neXt<br />Xd<br />X<br />X++<br />X<br />Full Presentation at <br />http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010<br />
    9. 9. Research .. <br />http://raskar.info<br />How to come up w ideas: Idea Hexagon<br />How to write a paper<br />How to give a talk<br />Open research problems<br />How to decide merit of a project<br />How to attend a conference, brainstorm<br />Facebook.com/ rRaskar<br />Tips<br />Get on Seminar/Talks mailing lists worldwide<br />http://www.cs.virginia.edu/~robins/YouAndYourResearch.html<br />Why do so few scientists make significant contributions and so many are forgotten in the long run?<br />Highly recommended Hamming talk at Bell Labs<br />
    10. 10. Is project worthwhile? Heilmeier's Questions<br />http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism<br />What<br />What are you trying to do? Articulate your objectives using absolutely no jargon.<br />Related work<br />How is it done today, and what are the limits of current practice?<br />Contribution<br />What's new in your approach and why do you think it will be successful?<br />Motivation<br />Who cares?<br />If you're successful, what difference will it make?<br />Challenges<br />What are the risks and the payoffs?<br />How much will it cost?<br />How long will it take?<br />Evaluation<br />What are the midterm and final "exams" to check for success?<br />Raskar additions: <br />Why now? (why not before, what’s new that makes possible)<br />Why us? (wrong answers: I am smart, I can work harder than others) <br />
    11. 11. Great Research: Strive for Five<br />Before Five teams<br /> Be first, often let others do details<br />Beyond Five years<br /> What no one is thinking about<br />Within Five layers of ‘Human’ Impact<br /> Relevance<br />Beyond Five minutes of description<br /> Deep, iterative, participatory<br />Fusing Five+ Expertise<br /> Multi-disciplinary, proactive<br />Ramesh Raskar, http://raskar.info<br />
    12. 12. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar<br />Inverse Problems<br />How to do Research in Imaging<br /><ul><li>Inverse Problems, Reconstruction, Rank and Sparsity</li></ul>Co-design of Optics and Computation<br />Photons not just pixels<br />Mid-level cues<br />Computational Photography<br />Open research problems<br />Compressive Sensing for High Speed Events<br />Limits of CS for general imaging<br />Computational Light Transport<br />Looking Around Corners, trillion fps<br />Lightfields: 3D Displays and Holograms<br />
    13. 13. Tools <br />for<br />Visual Computing<br />Shadow<br />Refractive<br />Reflective<br />Fernald, Science [Sept 2006]<br />
    14. 14. Computational Photography<br />Camera Culture<br />Ramesh Raskar<br />
    15. 15. Traditional Photography<br />Detector<br />Lens<br />Pixels<br />Mimics Human Eye for a Single Snapshot:<br /> Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world<br />Image<br />Courtesy: Shree Nayar<br />
    16. 16. Picture<br />Computational Camera + Photography: Optics, Sensors and Computations<br />GeneralizedSensor<br />Generalized<br /> Optics<br />Computations<br />Ray Reconstruction<br />4D Ray Bender<br />Upto 4D Ray Sampler<br />Merged Views, Programmable focus and dynamic range, Closed-loop Controlled Illumination, Coded exposure/apertures<br />
    17. 17. Computational Photography<br />Novel Illumination<br />Light Sources<br />Modulators<br />Computational Cameras<br />Generalized<br /> Optics<br />GeneralizedSensor<br />Generalized<br />Optics<br />Processing<br />4D Incident Lighting<br />4D Ray Bender<br />Ray Reconstruction<br />Upto 4D Ray Sampler<br />4D Light Field<br />Display<br />Scene: 8D Ray Modulator<br />Recreate 4D Lightfield<br />
    18. 18. Computational Photography [Raskar and Tumblin]<br />captures a machine-readable representation of our world to<br />hyper-realistically synthesize the essence of our visual experience. <br />Resources<br />ICCP 2012, Seattle Apr 2012<br />Papers due Dec 2nd, 2011<br />http://wikipedia.org/computational_photography<br />http://raskar.info/photo<br />
    19. 19. Computational Photography<br />Computational Photography aims to make progress on both axis<br />Phototourism<br />Comprehensive<br />Essence<br />Scene completion from photos<br />Augmented Human Experience<br />Looking Around Corners<br />Priors<br />Capture<br />Human Stereo Vision<br />Metadata<br />Coded<br />Depth<br />fg/bg<br />Non-visual Data, GPS<br />Virtual Object Insertion<br />Spectrum<br />Decomposition problems<br />8D reflectance field<br />Direct/Global<br />LightFields<br />Relighting<br />Epsilon<br />Angle, spectrum aware<br />Camera Array<br />HDR, FoV<br />Focal stack<br />Resolution<br />Material editing from single photo<br />Digital<br />Motion Magnification<br />Raw<br />Low Level<br />Mid Level<br />HighLevel<br />Hyper realism<br />Synthesis/Analysis<br />
    20. 20. Co-designing Optical and Digital Processing<br />Computational Light Transport<br />Optics<br />Displays<br />Sensors<br />Computational Photography<br />Photon Hacking<br />Illumination<br />Signal Processing<br />Computer Vision<br />Machine Learning<br />Bit Hacking<br />
    21. 21. Take home points<br />Co-design of hw/sw<br />Avoid computational or optical chauvinism in imaging <br />(Camera flash/Kinect)<br />Hardware cost going to zero, Parallel technology trends<br />Computer vision not just mimicking human vision/perception<br />Borrow ideas from other fields: astronomy, scientific imaging, audio, communications<br />Photons not just Pixels<br />Change the rules of the game<br />Optics, Sensors, Illum, <br />Priors, Sparsity, Transforms<br />Meta-data, Internet collection, Crowdsourcing<br />
    22. 22. Computational Photography<br />Wish List: <br />Open Research Problems<br />Camera Culture<br />Ramesh Raskar<br />
    23. 23. Wish #1<br />Ultimate Post-capture Control<br />Camera Culture<br />Ramesh Raskar<br />
    24. 24. Digital Refocusing using Light Field Camera<br />125μ square-sided microlenses<br />[Ng et al 2005]<br />
    25. 25. Motion Blur in Low Light<br />
    26. 26. Traditional<br />Blurred Photo<br />Deblurred Image<br />
    27. 27. Fluttered Shutter Camera<br />Raskar, Agrawal, Tumblin Siggraph2006<br />Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence<br />
    28. 28. Preserves High Spatial Frequencies<br />Fourier Transform<br />Sharp Photo<br />Blurred Photo<br />PSF == Broadband Function<br />Flutter Shutter: Shutter is OPEN and CLOSED<br />
    29. 29. Coded Exposure<br />Traditional<br />Deblurred Image<br />Deblurred Image<br />Image of Static Object<br />
    30. 30. Motion Blur in Low Light<br />
    31. 31. Fast periodic phenomena<br />Vocal folds flapping at 40.4 Hz<br />Bottling line<br />4000 fps hi-speed camera<br />500 fps hi-speed camera<br />
    32. 32. Compressive Sensing <br />Single Pixel Camera<br />image<br />compressive image<br />measurement matrix <br />
    33. 33. Periodic signals<br />-fP<br />-2fP<br />-4fP<br />3fP<br />-3fP<br />0<br />fMax<br />- fMax<br />2fP<br />fP=1/P<br />4fP<br />Periodic signal x(t) with period P<br />t<br />P = 16ms<br />Periodic signal with period P and band-limited to fMax = 500 Hz. <br />Fourier transform is non-zero only at multiples of fP=1/P ~ 63Hz.<br />
    34. 34. High speed camera<br />P = 16ms<br />Ts = 1/(2 fMax)<br />-fP<br />-2fP<br />-4fP<br />-3fP<br />4fP<br />3fP<br />2fP<br />0<br />fMax<br />- fMax<br />fP=1/P<br />Nyquist Sampling of x(t) <br />Periodic signal has regularly spaced, sparse Fourier coefficients. <br />Is it necessary to use a high-speed video camera? Why waste bandwidth?<br />
    35. 35. Traditional Strobing<br />Use low frame-rate camera and generate beat frequencies.<br />P<br />t<br />Low exposure to avoid blurring. Low light throughput.<br />Period known apriori.<br />Strobing animation credit Wikipedia<br />
    36. 36. t<br />P<br />Random Projections Per Frame of Camera using Coded Strobing Photography<br />In every exposure duration observe different linear combinations of the periodic signal.<br />Advantage of the design <br /><ul><li> Exposure coding independent of the frequency
    37. 37. On an average, light throughput is 50%</li></ul>Coded Strobing Photography. Reddy, D., Veeraraghavan, A., Raskar, R. IEEE PAMI 2011<br />
    38. 38. Observation Model<br />x at 2000fps<br />y at 25fps <br />
    39. 39. Signal Model<br />x at 2000fps<br />y at 25fps <br />
    40. 40. Signal & Observation Model<br />Ais M x N, M<<N<br />x at 2000fps<br />y at 25fps <br />N / M = 2000 / 25 = 80<br />
    41. 41. Recovery: Sparsity<br />Very few non-zero elements<br />y = A s<br />Observed values<br />Mixing matrix<br />Structured Sparse Coefficients<br />Basis Pursuit De-noising<br />
    42. 42. Simulation on hi-speed toothbrush<br />25fps normal camera<br />25fps coded strobing camera<br />Reconstructed frames<br />2000fps hi-speed camera<br />~100X speedup<br />
    43. 43. Rotating mill tool<br />Mill tool rotating at 50Hz<br />Reconstructed Video at 2000fps<br />Normal Video: 25fps<br />Coded Strobing Video: 25fps<br />Blur increases as rotational velocity increases <br />rotating at 200Hz<br />rotating at 150Hz<br />rotating at 100Hz<br />increasing blur<br />
    44. 44. Compressive Sensing for Images .. A good idea?<br />Single Pixel Camera<br />image<br />compressive image<br />measurement matrix <br />
    45. 45. Is Randomized Projection-based Captureapt for Natural Images ? <br />Periodic Signals<br />Progressive Projections<br />Randomized Projections<br />Compression Ratio<br />[Pandharkar, Veeraraghavan, Raskar 2009]<br />
    46. 46. Compact ProgrammableLights ?<br />
    47. 47. Wish #1<br />Ultimate Post-capture Control<br /><ul><li>Digital Refocus and Motion blur
    48. 48. Emulate studio light from compact flash</li></ul>Camera Culture<br />Ramesh Raskar<br />
    49. 49. Wish #2<br />Freedom from Form<br /><ul><li>Size, Weight, Power, UI
    50. 50. Flat camera: </li></ul> Bidirectional screen (BiDi)<br /><ul><li>Shallow DoF from tiny lens</li></ul>Camera Culture<br />Ramesh Raskar<br />
    51. 51. Wish #3<br />Understand the World<br />Camera Culture<br />Ramesh Raskar<br />
    52. 52. Convert single 2D photo into 3D ?<br />Snavely, Seitz, Szeliski<br />U of Washington/Microsoft: Photosynth<br />
    53. 53. Exploit Community Photo Collections<br />U of Washington/Microsoft: Photosynth<br />
    54. 54. Wish #3<br />Understand the World<br /><ul><li>Identify/recognize Materials
    55. 55. 3D Awareness
    56. 56. Interact with information</li></ul>Camera Culture<br />Ramesh Raskar<br />
    57. 57. Wish #4<br />Sharing Visual Experience<br /><ul><li>LifeLog Auto-summary
    58. 58. Privacy in public and authentication
    59. 59. Hyper-real Photo Frames
    60. 60. Print ‘material’ </li></ul>Camera Culture<br />Ramesh Raskar<br />
    61. 61. Wish #5<br />Capturing Essence<br />Camera Culture<br />Ramesh Raskar<br />
    62. 62. What are the problems with ‘real’ photo in conveying information ?<br />Why do we hire artists to draw what can be photographed ?<br />
    63. 63. Shadows<br />Clutter<br />Many Colors<br />Highlight Shape Edges<br />Mark moving parts<br />Basic colors<br />
    64. 64. Depth Edges with MultiFlash<br />Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004<br />
    65. 65.
    66. 66.
    67. 67.
    68. 68.
    69. 69. Depth Discontinuities<br />Internal and externalShape boundaries, Occluding contour, Silhouettes<br />
    70. 70. Depth Edges<br />
    71. 71. Our Method<br />Canny<br />
    72. 72. Result<br />Photo<br />Canny Intensity Edge Detection<br />Our Method<br />
    73. 73. Questions<br />What will a camera look like in 10,20 years?<br />How will a billion networked and portable cameras change the social culture? <br />How will online photo collections transform visual social computing?<br />How will movie making/new reporting change?<br />
    74. 74. Photos of tomorrow: computed not recorded<br />http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/<br />
    75. 75. Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info<br />Sensor<br />Computational Photography Wish List<br /><ul><li>Post-capture control
    76. 76. Emulate studio lights with compact flash
    77. 77. Focus and motion blur
    78. 78. New forms
    79. 79. Flat camera, large LCDs as cameras
    80. 80. Image destabilization for larger aperture
    81. 81. Understand the world
    82. 82. Real or fake
    83. 83. Place 2D photo into 3D
    84. 84. Look around corner
    85. 85. Bokode: long distance barcode
    86. 86. Sharing
    87. 87. Lifelogs auto summary
    88. 88. Privacy/Verification
    89. 89. 6D photoframes
    90. 90. Essence
    91. 91. New visual arts
    92. 92. Multi-flash camera
    93. 93. Delta-camera and Blind-camera</li></li></ul><li>Take home points<br />Co-design of hw/sw<br />Avoid computational or optical chauvinism in imaging <br />(Camera flash/Kinect)<br />Hardware cost going to zero, Parallel technology trends<br />Computer vision not just mimicking human vision/perception<br />Borrow ideas from other fields: astronomy, scientific imaging, audio, communications<br />Photons not just Pixels<br />Change the rules of the game<br />Optics, Sensors, Illum, <br />Priors, Sparsity, Transforms<br />Meta-data, Internet collection, Crowdsourcing<br />
    94. 94. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar<br />Inverse Problems<br />How to do Research in Imaging<br /><ul><li>Inverse Problems, Reconstruction, Rank and Sparsity</li></ul>Co-design of Optics and Computation<br />Photons not just pixels<br />Mid-level cues<br />Computational Photography<br />Open research problems<br />Compressive Sensing for High Speed Events<br />Limits of CS for general imaging<br />Computational Light Transport<br />Looking Around Corners, trillion fps<br />Lightfields: 3D Displays and Holograms<br />
    95. 95. Every Photon has a Story<br />
    96. 96. What isaround the corner ?<br />
    97. 97. Can you look around the corner ?<br />
    98. 98. Multi-path Analysis<br />2nd Bounce<br />1st Bounce<br />3rd Bounce<br />
    99. 99. Femto-Photography (Transient Imaging)<br />FemtoFlash<br />Trillion FPS camera<br />With M Bawendi, MIT Chemistry<br />Serious Sync<br />Computational Optics<br /><ul><li>2011: CVPR (Pandharkar, Velten, Bardagjy, Bawendi, Raskar)
    100. 100. 2009: Marr PrizeHonorable Mention (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)
    101. 101. 2008: Transient Light Transport (Raskar, Davis, March 2008)</li></li></ul><li>Inverting Light Transport<br />Direct/Global<br />Multiple Scattering<br />[Seitz , Kutulakos, Matsushita 2005]<br />[Nayar, Raskar et al 2006]<br />[Atcheson et al 2008]<br />[Kutulakos, Steger 2005]<br />Dual Photography<br />LIDAR<br />[Sen et al 2005]<br />
    102. 102. Multi-Dimensional Light Transport<br />5-D Transport<br />Gigapan<br />
    103. 103. Collision avoidance, robot navigation, …<br />
    104. 104. z<br />x<br />S<br />L<br />R<br />s<br />Occluder<br />Streak-camera<br />3rd bounce<br />C<br />Laser beam<br />B<br />Echoes of Light<br />
    105. 105. Steady State 4D<br />Impulse Response, 5D<br />
    106. 106. Scene with <br />Ultra fast illumination and camera<br />hidden elements<br />Raw <br />5D Capture<br />Time profiles<br />Signal <br />Proc.<br />Photo, geometry, reflectance beyond line of sight <br />Novel light transport models and inference <br />algorithms<br />®<br />t<br />3D Time images<br />Femto-PhotographyTime Resolved Multi-path Imaging<br />
    107. 107. Team<br />Moungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Associate, MIT Media LabRohitPandharkar, RA, MIT Media Lab<br />Otkrist Gupta, RA, MIT Media LabAndrew Matthew Bardagjy, RA, MIT Media Lab<br />Nikhil Naik, RA, MIT Media LabTyler Hutchison, RA, MIT Media LabEverett Lawson, MIT Media Lab<br />Ramesh Raskar, Asso. Prof., MIT Media Lab<br />Camera Culture<br />Ramesh Raskar<br />
    108. 108. Photos from Streak Camera<br />Capture Setup<br />Hidden Scene<br />
    109. 109. Photos from Streak Camera<br />Capture Setup<br />Hidden Scene<br />Overlay<br />Reconstruction<br />
    110. 110. Motion beyond line of sight<br />Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011 <br />
    111. 111. …, bronchoscopies, …<br />Participating Media<br />
    112. 112. Photo<br />First Bounce<br />Later Bounces<br />+<br />Direct<br />Global<br />[Nayar, Krishnan, Grossberg, Raskar 2006]<br />
    113. 113.
    114. 114. Each frame = ~2ps = 0.6 mm of Light Travel<br />
    115. 115. Ripples of Waves<br />
    116. 116.
    117. 117.
    118. 118. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar<br />Inverse Problems<br />How to do Research in Imaging<br /><ul><li>Inverse Problems, Reconstruction, Rank and Sparsity</li></ul>Co-design of Optics and Computation<br />Photons not just pixels<br />Mid-level cues<br />Computational Photography<br />Open research problems<br />Compressive Sensing for High Speed Events<br />Limits of CS for general imaging<br />Computational Light Transport<br />Looking Around Corners, trillion fps<br />Lightfields: 3D Displays and Holograms<br />
    119. 119.
    120. 120. View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale<br />
    121. 121. Two Layer Displays<br />barrier<br />lenslet<br />sensor/display<br />sensor/display<br />PB = dim displays<br />Lenslets = fixed spatial and angular resolution<br />Dynamic Masks = Brighter, High spatial resolution <br />
    122. 122. Limitations of 3D Display<br />Parallaxbarrier<br />LCD display<br />Front<br />Back<br />Lanman, Hirsch, Kim, RaskarSiggraph Asia 2010<br />
    123. 123. Light Field Analysis of Barriers<br />k<br />L[i,k]<br />i<br />`<br />k<br />g[k]<br />i<br />L[i,k]<br />f[i]<br />light box<br />
    124. 124. Content-Adaptive Parallax Barriers<br />L[i,k]<br />`<br />k<br />g[k]<br />i<br />f[i]<br />light box<br />
    125. 125. Implementation<br />Components<br /><ul><li> 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]</li></li></ul><li>Content-Adaptive Parallax Barriers<br />L[i,k]<br />`<br />k<br />g[k]<br />i<br />f[i]<br />light box<br />
    126. 126. Content-Adaptive Parallax Barriers<br />`<br />=<br />
    127. 127. Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010<br />Rank-Constrained Displays and LF Adaptation<br />`<br />Content-Adaptive Parallax Barriers<br />=<br />All dual layer display = rank-1 constraint<br /> Light field display is a matrix approximation problem<br />Exploit content-adaptive parallax barriers<br />
    128. 128. Optimization: Iteration 1<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    129. 129. Optimization: Iteration 10<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    130. 130. Optimization: Iteration 20<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    131. 131. Optimization: Iteration 30<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    132. 132. Optimization: Iteration 40<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    133. 133. Optimization: Iteration 50<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    134. 134. Optimization: Iteration 60<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    135. 135. Optimization: Iteration 70<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    136. 136. Optimization: Iteration 80<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    137. 137. Optimization: Iteration 90<br />rear mask: f1[i,j]<br />front mask: g1[k,l]<br />reconstruction (central view)<br />Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.<br />Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.<br />
    138. 138. Content-Adaptive Front Mask (1 of 9)<br />
    139. 139. Content-Adaptive Rear Mask (1 of 9)<br />
    140. 140. Emitted 4D Light Field<br />
    141. 141. Conclusion<br />`<br />Content-Adaptive Parallax Barriers<br />=<br /><ul><li> Described a rank constraint for all dual-layer displays
    142. 142. With a fixed pair of masks, emitted light field is rank-1
    143. 143. Achieved higher-rank approximation using temporal multiplexing
    144. 144. With T time-multiplexed masks, emitted light field is rank-T
    145. 145. Constructed a prototype using off-the-shelf panels
    146. 146. Demonstrated light field display is a matrix approximation problem
    147. 147. Introduced content-adaptive parallax barriers
    148. 148. Applied weighted NMF to optimize weighted Euclidean distance to target</li></ul>Adaptation increases brightness and refresh rate of dual-stacked LCDs<br />
    149. 149. Parallax Barrier: Np=103 pix.<br />Hologram: NH=105 pix.<br />ϕP∝w/d<br />ϕH∝λ/tH<br />θp=10 pix<br />θH =1000 pix<br />Fourier Patch<br />xH =100 patches<br />w<br />xp=100slits<br />Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009<br />
    150. 150. Augmented Light Field<br />118<br />wave optics based<br />rigorous but cumbersome<br />Wigner Distribution Function<br />WDF<br />Augmented LF<br />Traditional Light Field<br />Traditional Light Field<br />ray optics based<br />simple and powerful<br />Interference & Diffraction<br />Interaction w/ optical elements<br />Oh, Raskar, Barbastathis 2009: Augmented Light Field <br />
    151. 151. position<br />light field transformer<br />LF<br />LF<br />LF<br />LF<br />(diffractive)<br />optical element<br />Reference<br />plane<br />LF propagation<br />LF propagation<br />Light Fields<br />Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters<br />angle<br />
    152. 152. Augmented Lightfield for Wave Optics Effects<br />WDF<br />Wigner Distribution Function<br />Augmented Light Field<br />Light Field<br />LF<br />LF < WDF<br />Lacks phase properties<br />Ignores diffraction, interferrence<br />Radiance = Positive<br />ALF ~ WDF<br />Supports coherent/incoherent<br />Radiance = Positive/Negative<br />Virtual light sources<br />
    153. 153. Free-space propagation<br />Light field transformer<br />Virtual light projector<br /> Possibly negative radiance<br />121<br />
    154. 154. Lightfieldvs Hologram Displays<br />
    155. 155. Is hologram just another ray-based light field?<br />Can a hologram create any intensity distribution in 3D?<br />Why hologram creates a ‘wavefront’ but PB does not?<br />Why hologram creates automatic accommodation cues?<br />What is the effective resolution of HG vs PB?<br />
    156. 156. Zooming into the Light Field<br />Rays: No Bending<br />1 Fresnel HG Patch<br />p Wm<br />* <br />* <br />p d(θ)<br />q d(θ)<br />* <br />q<br />q<br />p<br />p<br />* <br />q Wm<br />L(x,θ)<br />W(x,u)<br />Wm= sinc<br />d = delta<br />u<br />θ<br />
    157. 157. Algebraic Rank Constraint<br />Rank-3<br />Rank-1<br />Rank-1<br />s1*<br />s1<br />m2<br />s1*<br />m2<br />s1<br />(a) Parallax Barrier<br />(c) Hybrid<br />(b) Hologram<br />s1<br />s1<br />
    158. 158. (a) Two Slits, Coherent<br />Interference<br />xʹ<br />Rank-1<br />-1<br />Transform<br />u<br />-Transform<br />R45, D<br />x<br /><t(x+xʹ/2)t*(x-xʹ/2)><br />t(x1)t*(x2)<br />t(x+xʹ/2)t*(x-xʹ/2)<br />W(x,u)<br />
    159. 159. L1(x,θ)<br />(a)<br />L1<br />L2(x,θ)<br />L2<br />L3(x,θ)<br />s1<br />m2<br />L3<br />hH<br />ϕ1<br />ϕ1<br />z2<br />ϕ1<br />ϕ1<br />z1<br />r<br />d<br />L3(x,θ)<br />L1(x,θ)<br />L2(x,θ)<br />
    160. 160. Is hologram just another ray-based light field?<br />Can a hologram create any intensity distribution in 3D?<br />Why hologram creates a ‘wavefront’ but PB does not?<br />Why hologram creates automatic accommodation cues?<br />What is the effective resolution of HG vs PB?<br />
    161. 161. Three Questions<br />What are the benefits of higher dimensional imaging?<br />Why is the algebraic rank of a Light Field not full?<br />What makes looking around the corner possible?<br />
    162. 162. How to do Research in Imaging<br />http://raskar.info<br />How to come up w ideas: Idea Hexagon<br />How to write a paper<br />How to give a talk<br />Open research problems<br />How to decide merit of a project<br />How to attend a conference, brainstorm<br />Facebook.com/ rRaskar<br />Tips<br />Get on Seminar/Talks mailing lists worldwide<br />http://www.cs.virginia.edu/~robins/YouAndYourResearch.html<br />Why do so few scientists make significant contributions and so many are forgotten in the long run?<br />Highly recommended Hamming talk at Bell Labs<br />
    163. 163. Take home points<br />Co-design of hw/sw<br />Avoid computational or optical chauvinism in imaging <br />(Camera flash/Kinect)<br />Hardware cost going to zero, Parallel technology trends<br />Computer vision not just mimicking human vision/perception<br />Borrow ideas from other fields: astronomy, scientific imaging, audio, communications<br />Photons not just Pixels<br />Change the rules of the game<br />Optics, Sensors, Illum, <br />Priors, Sparsity, Transforms<br />Meta-data, Internet collection, Crowdsourcing<br />
    164. 164. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar<br />Inverse Problems<br />How to do Research in Imaging<br /><ul><li>Inverse Problems, Reconstruction, Rank and Sparsity</li></ul>Co-design of Optics and Computation<br />Photons not just pixels<br />Mid-level cues<br />Computational Photography<br />Open research problems<br />Compressive Sensing for High Speed Events<br />Limits of CS for general imaging<br />Computational Light Transport<br />Looking Around Corners, trillion fps<br />Lightfields: 3D Displays and Holograms<br />Apply for internships/post-doc<br />neXt<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×