Your SlideShare is downloading. ×
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Raskar, Rank and Sparsity in Computational Photography and Computational Light Transport, Computer Vision Summer School

1,731

Published on

How to do research, Idea Hexagon, Rank and Sparsity in imaging problems, Looking around corners, compressive sensing of periodic phenomena, 3D displays, fast computation

How to do research, Idea Hexagon, Rank and Sparsity in imaging problems, Looking around corners, compressive sensing of periodic phenomena, 3D displays, fast computation

Published in: Technology, Art & Photos
1 Comment
1 Like
Statistics
Notes
  • See useful resources for students on http://raskar.info

    http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,731
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
79
Comments
1
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Six ways of coming up with new ideas based on an idea ‘X’.Ramesh RaskarAssociate ProfessorMIT Media Labhttp://raskar.infohttp://cameraculture.infoFull Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010http://raskar.infohttp://cameraculture.info
  • Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • X up: Airbags for car, for helicopter
  • Full Presentation at http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
  • http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism
  • Five on Five= If more than five teams in the world are doing the same research, don’t do it.= If you disappear for five years, will someone do it anyway? Then your idea is not that great anyway. = Can you explain your work in five sentences to your grandmahow it will impact human life?= If you can explain the idea in five minutes to a student and disappear for five years, will s/he be able to do it on her/his own without additional input from you/without iterations .. It is too obvious and lacks depth .. Don’t do it.= Strive to work on ideas that may require five+ disciplines .. Today’s research is highly team-driven and more diverse the required team composition, more fun you will have and also indicates a natural barrier to entry for others satisfying condition 1 and 2Much like the food pyramid, five servings are the goal and will make you stronger .. But ok if your research project does not satisfy all five conditions
  • But the world is 4D
  • See computationalphotography.orgMove away from obsession about SNR, space-bandwidth, diffraction limit and so on
  • My work involves creative new ways to play with light by co-designing optical and digital processing.My work lies at the INTERSECTION of processing of photons and processing of bits.At MERL, I transformed the field of computational photography, with key papers and impact on productsAt Media Lab, I invented a new field ‘computational light transport’
  • Compressive sensing via random projections not suitable for images and even videos
  • Rudy Burger, ‘don’t use flash and destroy the image’Can we use flash not just for improving scene brightness but for enhancing the mood? Like in studio lights?Main difference between professionals and consumers is lighting.
  • http://cameraculture.infohttp://raskar.info
  • My idea is to use the multiple bounces of light i.e. echoes of light.Echoes of sound -> Echoes of lightWe all know about echoes of sound.But sounds travels slow and we can actually hear the echoesLight travels fast so we need specialized hardware to ‘listen’ to these echoes.So we end up using light sources and cameras that run at a trillion frames per second (not a million and not a billion, but trillion)
  • Trillion fps camera (which was previously used only for specialized biochemistry expt)This new form of imaging is possible by fusion of dissimilar .. A specialized camera previously used only in biochemistry labs and a new computational method that analyzes multiple bounces of light.I started the project just before I joined MIT in summer 2008.The hardware we use is in the lab of Prof Bawendi, MIT Chemistry, who is now a collaborator
  • Here is a road map for this ambitious research project based on time-resolved imaging .. Non line of sight Looking around corner (LaC) is just one example .. Such Time resolved imaging requires one to develop a completely new set of tool for understanding our world.This is a project I started just before coming to MIT via an NSF proposal.
  • The reconstruction is very low right now, about 80x80 pixels. So these are just baby steps.
  • The reconstruction is very low right now, about 80x80 pixels. So these are just baby steps.Data collected and reconstructions program by Andreas Velten, scientist in my group
  • Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
  • A cross section through a single M. rhetenor scale. Light reflected off each level of the “Christmas tree structure” gives the butterfly its iridescent color. Credit: Pete Vukusic, University of Exeter
  • Lanman, Hirsch, Kim, RaskarSiggraph Asia 2010
  • Transcript

    • 1. Computational Light Transport
      and
      Computational Photography:
      Inverse problems
      Camera Culture
      Ramesh Raskar
      Ramesh Raskar
      http://raskar.info
      MIT Media Lab
      raskar@mit.edu
    • 2.
    • 3. How to Invent?
      After X, what is neXt
      Full Presentation at
      http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
      Ramesh Raskar, MIT Media Lab
    • 4. Ramesh Raskar, http://raskar.info
      X+Y
      X
      neXt
      Xd
      X
      X++
      X
      Full Presentation at
      http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
    • 5. Simple Exercise ..
      Image Compression
      Save Bandwidth and storage
      What is neXt
    • 6. Strategy #1: Xd
      Extend it to next (or some other) dimension ..
    • 7. X =
      Idea you just heard
      Concept
      Patent
      New Product/Best project/invention award
      Product feature
      Design
      Art
      Algorithm
    • 8. Ramesh Raskar, http://raskar.info
      X+Y
      X
      neXt
      Xd
      X
      X++
      X
      Full Presentation at
      http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010
    • 9. Research ..
      http://raskar.info
      How to come up w ideas: Idea Hexagon
      How to write a paper
      How to give a talk
      Open research problems
      How to decide merit of a project
      How to attend a conference, brainstorm
      Facebook.com/ rRaskar
      Tips
      Get on Seminar/Talks mailing lists worldwide
      http://www.cs.virginia.edu/~robins/YouAndYourResearch.html
      Why do so few scientists make significant contributions and so many are forgotten in the long run?
      Highly recommended Hamming talk at Bell Labs
    • 10. Is project worthwhile? Heilmeier's Questions
      http://en.wikipedia.org/wiki/George_H._Heilmeier#Heilmeier.27s_Catechism
      What
      What are you trying to do? Articulate your objectives using absolutely no jargon.
      Related work
      How is it done today, and what are the limits of current practice?
      Contribution
      What's new in your approach and why do you think it will be successful?
      Motivation
      Who cares?
      If you're successful, what difference will it make?
      Challenges
      What are the risks and the payoffs?
      How much will it cost?
      How long will it take?
      Evaluation
      What are the midterm and final "exams" to check for success?
      Raskar additions:
      Why now? (why not before, what’s new that makes possible)
      Why us? (wrong answers: I am smart, I can work harder than others)
    • 11. Great Research: Strive for Five
      Before Five teams
      Be first, often let others do details
      Beyond Five years
      What no one is thinking about
      Within Five layers of ‘Human’ Impact
      Relevance
      Beyond Five minutes of description
      Deep, iterative, participatory
      Fusing Five+ Expertise
      Multi-disciplinary, proactive
      Ramesh Raskar, http://raskar.info
    • 12. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
      Inverse Problems
      How to do Research in Imaging
      • Inverse Problems, Reconstruction, Rank and Sparsity
      Co-design of Optics and Computation
      Photons not just pixels
      Mid-level cues
      Computational Photography
      Open research problems
      Compressive Sensing for High Speed Events
      Limits of CS for general imaging
      Computational Light Transport
      Looking Around Corners, trillion fps
      Lightfields: 3D Displays and Holograms
    • 13. Tools
      for
      Visual Computing
      Shadow
      Refractive
      Reflective
      Fernald, Science [Sept 2006]
    • 14. Computational Photography
      Camera Culture
      Ramesh Raskar
    • 15. Traditional Photography
      Detector
      Lens
      Pixels
      Mimics Human Eye for a Single Snapshot:
      Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world
      Image
      Courtesy: Shree Nayar
    • 16. Picture
      Computational Camera + Photography: Optics, Sensors and Computations
      GeneralizedSensor
      Generalized
      Optics
      Computations
      Ray Reconstruction
      4D Ray Bender
      Upto 4D Ray Sampler
      Merged Views, Programmable focus and dynamic range, Closed-loop Controlled Illumination, Coded exposure/apertures
    • 17. Computational Photography
      Novel Illumination
      Light Sources
      Modulators
      Computational Cameras
      Generalized
      Optics
      GeneralizedSensor
      Generalized
      Optics
      Processing
      4D Incident Lighting
      4D Ray Bender
      Ray Reconstruction
      Upto 4D Ray Sampler
      4D Light Field
      Display
      Scene: 8D Ray Modulator
      Recreate 4D Lightfield
    • 18. Computational Photography [Raskar and Tumblin]
      captures a machine-readable representation of our world to
      hyper-realistically synthesize the essence of our visual experience.
      Resources
      ICCP 2012, Seattle Apr 2012
      Papers due Dec 2nd, 2011
      http://wikipedia.org/computational_photography
      http://raskar.info/photo
    • 19. Computational Photography
      Computational Photography aims to make progress on both axis
      Phototourism
      Comprehensive
      Essence
      Scene completion from photos
      Augmented Human Experience
      Looking Around Corners
      Priors
      Capture
      Human Stereo Vision
      Metadata
      Coded
      Depth
      fg/bg
      Non-visual Data, GPS
      Virtual Object Insertion
      Spectrum
      Decomposition problems
      8D reflectance field
      Direct/Global
      LightFields
      Relighting
      Epsilon
      Angle, spectrum aware
      Camera Array
      HDR, FoV
      Focal stack
      Resolution
      Material editing from single photo
      Digital
      Motion Magnification
      Raw
      Low Level
      Mid Level
      HighLevel
      Hyper realism
      Synthesis/Analysis
    • 20. Co-designing Optical and Digital Processing
      Computational Light Transport
      Optics
      Displays
      Sensors
      Computational Photography
      Photon Hacking
      Illumination
      Signal Processing
      Computer Vision
      Machine Learning
      Bit Hacking
    • 21. Take home points
      Co-design of hw/sw
      Avoid computational or optical chauvinism in imaging
      (Camera flash/Kinect)
      Hardware cost going to zero, Parallel technology trends
      Computer vision not just mimicking human vision/perception
      Borrow ideas from other fields: astronomy, scientific imaging, audio, communications
      Photons not just Pixels
      Change the rules of the game
      Optics, Sensors, Illum,
      Priors, Sparsity, Transforms
      Meta-data, Internet collection, Crowdsourcing
    • 22. Computational Photography
      Wish List:
      Open Research Problems
      Camera Culture
      Ramesh Raskar
    • 23. Wish #1
      Ultimate Post-capture Control
      Camera Culture
      Ramesh Raskar
    • 24. Digital Refocusing using Light Field Camera
      125μ square-sided microlenses
      [Ng et al 2005]
    • 25. Motion Blur in Low Light
    • 26. Traditional
      Blurred Photo
      Deblurred Image
    • 27. Fluttered Shutter Camera
      Raskar, Agrawal, Tumblin Siggraph2006
      Ferroelectric shutter in front of the lens is turnedopaque or transparent in a rapid binary sequence
    • 28. Preserves High Spatial Frequencies
      Fourier Transform
      Sharp Photo
      Blurred Photo
      PSF == Broadband Function
      Flutter Shutter: Shutter is OPEN and CLOSED
    • 29. Coded Exposure
      Traditional
      Deblurred Image
      Deblurred Image
      Image of Static Object
    • 30. Motion Blur in Low Light
    • 31. Fast periodic phenomena
      Vocal folds flapping at 40.4 Hz
      Bottling line
      4000 fps hi-speed camera
      500 fps hi-speed camera
    • 32. Compressive Sensing
      Single Pixel Camera
      image
      compressive image
      measurement matrix
    • 33. Periodic signals
      -fP
      -2fP
      -4fP
      3fP
      -3fP
      0
      fMax
      - fMax
      2fP
      fP=1/P
      4fP
      Periodic signal x(t) with period P
      t
      P = 16ms
      Periodic signal with period P and band-limited to fMax = 500 Hz.
      Fourier transform is non-zero only at multiples of fP=1/P ~ 63Hz.
    • 34. High speed camera
      P = 16ms
      Ts = 1/(2 fMax)
      -fP
      -2fP
      -4fP
      -3fP
      4fP
      3fP
      2fP
      0
      fMax
      - fMax
      fP=1/P
      Nyquist Sampling of x(t)
      Periodic signal has regularly spaced, sparse Fourier coefficients.
      Is it necessary to use a high-speed video camera? Why waste bandwidth?
    • 35. Traditional Strobing
      Use low frame-rate camera and generate beat frequencies.
      P
      t
      Low exposure to avoid blurring. Low light throughput.
      Period known apriori.
      Strobing animation credit Wikipedia
    • 36. t
      P
      Random Projections Per Frame of Camera using Coded Strobing Photography
      In every exposure duration observe different linear combinations of the periodic signal.
      Advantage of the design
      • Exposure coding independent of the frequency
      • 37. On an average, light throughput is 50%
      Coded Strobing Photography. Reddy, D., Veeraraghavan, A., Raskar, R. IEEE PAMI 2011
    • 38. Observation Model
      x at 2000fps
      y at 25fps
    • 39. Signal Model
      x at 2000fps
      y at 25fps
    • 40. Signal & Observation Model
      Ais M x N, M<<N
      x at 2000fps
      y at 25fps
      N / M = 2000 / 25 = 80
    • 41. Recovery: Sparsity
      Very few non-zero elements
      y = A s
      Observed values
      Mixing matrix
      Structured Sparse Coefficients
      Basis Pursuit De-noising
    • 42. Simulation on hi-speed toothbrush
      25fps normal camera
      25fps coded strobing camera
      Reconstructed frames
      2000fps hi-speed camera
      ~100X speedup
    • 43. Rotating mill tool
      Mill tool rotating at 50Hz
      Reconstructed Video at 2000fps
      Normal Video: 25fps
      Coded Strobing Video: 25fps
      Blur increases as rotational velocity increases
      rotating at 200Hz
      rotating at 150Hz
      rotating at 100Hz
      increasing blur
    • 44. Compressive Sensing for Images .. A good idea?
      Single Pixel Camera
      image
      compressive image
      measurement matrix
    • 45. Is Randomized Projection-based Captureapt for Natural Images ?
      Periodic Signals
      Progressive Projections
      Randomized Projections
      Compression Ratio
      [Pandharkar, Veeraraghavan, Raskar 2009]
    • 46. Compact ProgrammableLights ?
    • 47. Wish #1
      Ultimate Post-capture Control
      • Digital Refocus and Motion blur
      • 48. Emulate studio light from compact flash
      Camera Culture
      Ramesh Raskar
    • 49. Wish #2
      Freedom from Form
      • Size, Weight, Power, UI
      • 50. Flat camera:
      Bidirectional screen (BiDi)
      • Shallow DoF from tiny lens
      Camera Culture
      Ramesh Raskar
    • 51. Wish #3
      Understand the World
      Camera Culture
      Ramesh Raskar
    • 52. Convert single 2D photo into 3D ?
      Snavely, Seitz, Szeliski
      U of Washington/Microsoft: Photosynth
    • 53. Exploit Community Photo Collections
      U of Washington/Microsoft: Photosynth
    • 54. Wish #3
      Understand the World
      • Identify/recognize Materials
      • 55. 3D Awareness
      • 56. Interact with information
      Camera Culture
      Ramesh Raskar
    • 57. Wish #4
      Sharing Visual Experience
      • LifeLog Auto-summary
      • 58. Privacy in public and authentication
      • 59. Hyper-real Photo Frames
      • 60. Print ‘material’
      Camera Culture
      Ramesh Raskar
    • 61. Wish #5
      Capturing Essence
      Camera Culture
      Ramesh Raskar
    • 62. What are the problems with ‘real’ photo in conveying information ?
      Why do we hire artists to draw what can be photographed ?
    • 63. Shadows
      Clutter
      Many Colors
      Highlight Shape Edges
      Mark moving parts
      Basic colors
    • 64. Depth Edges with MultiFlash
      Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004
    • 65.
    • 66.
    • 67.
    • 68.
    • 69. Depth Discontinuities
      Internal and externalShape boundaries, Occluding contour, Silhouettes
    • 70. Depth Edges
    • 71. Our Method
      Canny
    • 72. Result
      Photo
      Canny Intensity Edge Detection
      Our Method
    • 73. Questions
      What will a camera look like in 10,20 years?
      How will a billion networked and portable cameras change the social culture?
      How will online photo collections transform visual social computing?
      How will movie making/new reporting change?
    • 74. Photos of tomorrow: computed not recorded
      http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/
    • 75. Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info
      Sensor
      Computational Photography Wish List
      • Post-capture control
      • 76. Emulate studio lights with compact flash
      • 77. Focus and motion blur
      • 78. New forms
      • 79. Flat camera, large LCDs as cameras
      • 80. Image destabilization for larger aperture
      • 81. Understand the world
      • 82. Real or fake
      • 83. Place 2D photo into 3D
      • 84. Look around corner
      • 85. Bokode: long distance barcode
      • 86. Sharing
      • 87. Lifelogs auto summary
      • 88. Privacy/Verification
      • 89. 6D photoframes
      • 90. Essence
      • 91. New visual arts
      • 92. Multi-flash camera
      • 93. Delta-camera and Blind-camera
    • Take home points
      Co-design of hw/sw
      Avoid computational or optical chauvinism in imaging
      (Camera flash/Kinect)
      Hardware cost going to zero, Parallel technology trends
      Computer vision not just mimicking human vision/perception
      Borrow ideas from other fields: astronomy, scientific imaging, audio, communications
      Photons not just Pixels
      Change the rules of the game
      Optics, Sensors, Illum,
      Priors, Sparsity, Transforms
      Meta-data, Internet collection, Crowdsourcing
    • 94. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
      Inverse Problems
      How to do Research in Imaging
      • Inverse Problems, Reconstruction, Rank and Sparsity
      Co-design of Optics and Computation
      Photons not just pixels
      Mid-level cues
      Computational Photography
      Open research problems
      Compressive Sensing for High Speed Events
      Limits of CS for general imaging
      Computational Light Transport
      Looking Around Corners, trillion fps
      Lightfields: 3D Displays and Holograms
    • 95. Every Photon has a Story
    • 96. What isaround the corner ?
    • 97. Can you look around the corner ?
    • 98. Multi-path Analysis
      2nd Bounce
      1st Bounce
      3rd Bounce
    • 99. Femto-Photography (Transient Imaging)
      FemtoFlash
      Trillion FPS camera
      With M Bawendi, MIT Chemistry
      Serious Sync
      Computational Optics
      • 2011: CVPR (Pandharkar, Velten, Bardagjy, Bawendi, Raskar)
      • 100. 2009: Marr PrizeHonorable Mention (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)
      • 101. 2008: Transient Light Transport (Raskar, Davis, March 2008)
    • Inverting Light Transport
      Direct/Global
      Multiple Scattering
      [Seitz , Kutulakos, Matsushita 2005]
      [Nayar, Raskar et al 2006]
      [Atcheson et al 2008]
      [Kutulakos, Steger 2005]
      Dual Photography
      LIDAR
      [Sen et al 2005]
    • 102. Multi-Dimensional Light Transport
      5-D Transport
      Gigapan
    • 103. Collision avoidance, robot navigation, …
    • 104. z
      x
      S
      L
      R
      s
      Occluder
      Streak-camera
      3rd bounce
      C
      Laser beam
      B
      Echoes of Light
    • 105. Steady State 4D
      Impulse Response, 5D
    • 106. Scene with
      Ultra fast illumination and camera
      hidden elements
      Raw
      5D Capture
      Time profiles
      Signal
      Proc.
      Photo, geometry, reflectance beyond line of sight
      Novel light transport models and inference
      algorithms
      ®
      t
      3D Time images
      Femto-PhotographyTime Resolved Multi-path Imaging
    • 107. Team
      Moungi G. Bawendi, Professor, Dept of Chemistry, MITJames Davis, UC Santa CruzAndreas Velten, Postdoctoral Associate, MIT Media LabRohitPandharkar, RA, MIT Media Lab
      Otkrist Gupta, RA, MIT Media LabAndrew Matthew Bardagjy, RA, MIT Media Lab
      Nikhil Naik, RA, MIT Media LabTyler Hutchison, RA, MIT Media LabEverett Lawson, MIT Media Lab
      Ramesh Raskar, Asso. Prof., MIT Media Lab
      Camera Culture
      Ramesh Raskar
    • 108. Photos from Streak Camera
      Capture Setup
      Hidden Scene
    • 109. Photos from Streak Camera
      Capture Setup
      Hidden Scene
      Overlay
      Reconstruction
    • 110. Motion beyond line of sight
      Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
    • 111. …, bronchoscopies, …
      Participating Media
    • 112. Photo
      First Bounce
      Later Bounces
      +
      Direct
      Global
      [Nayar, Krishnan, Grossberg, Raskar 2006]
    • 113.
    • 114. Each frame = ~2ps = 0.6 mm of Light Travel
    • 115. Ripples of Waves
    • 116.
    • 117.
    • 118. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
      Inverse Problems
      How to do Research in Imaging
      • Inverse Problems, Reconstruction, Rank and Sparsity
      Co-design of Optics and Computation
      Photons not just pixels
      Mid-level cues
      Computational Photography
      Open research problems
      Compressive Sensing for High Speed Events
      Limits of CS for general imaging
      Computational Light Transport
      Looking Around Corners, trillion fps
      Lightfields: 3D Displays and Holograms
    • 119.
    • 120. View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale
    • 121. Two Layer Displays
      barrier
      lenslet
      sensor/display
      sensor/display
      PB = dim displays
      Lenslets = fixed spatial and angular resolution
      Dynamic Masks = Brighter, High spatial resolution
    • 122. Limitations of 3D Display
      Parallaxbarrier
      LCD display
      Front
      Back
      Lanman, Hirsch, Kim, RaskarSiggraph Asia 2010
    • 123. Light Field Analysis of Barriers
      k
      L[i,k]
      i
      `
      k
      g[k]
      i
      L[i,k]
      f[i]
      light box
    • 124. Content-Adaptive Parallax Barriers
      L[i,k]
      `
      k
      g[k]
      i
      f[i]
      light box
    • 125. Implementation
      Components
      • 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]
    • Content-Adaptive Parallax Barriers
      L[i,k]
      `
      k
      g[k]
      i
      f[i]
      light box
    • 126. Content-Adaptive Parallax Barriers
      `
      =
    • 127. Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
      Rank-Constrained Displays and LF Adaptation
      `
      Content-Adaptive Parallax Barriers
      =
      All dual layer display = rank-1 constraint
      Light field display is a matrix approximation problem
      Exploit content-adaptive parallax barriers
    • 128. Optimization: Iteration 1
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 129. Optimization: Iteration 10
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 130. Optimization: Iteration 20
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 131. Optimization: Iteration 30
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 132. Optimization: Iteration 40
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 133. Optimization: Iteration 50
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 134. Optimization: Iteration 60
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 135. Optimization: Iteration 70
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 136. Optimization: Iteration 80
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 137. Optimization: Iteration 90
      rear mask: f1[i,j]
      front mask: g1[k,l]
      reconstruction (central view)
      Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999.
      Vincent Blondel et al. Weighted Non-negative Matrix Factorization. 2008.
    • 138. Content-Adaptive Front Mask (1 of 9)
    • 139. Content-Adaptive Rear Mask (1 of 9)
    • 140. Emitted 4D Light Field
    • 141. Conclusion
      `
      Content-Adaptive Parallax Barriers
      =
      • Described a rank constraint for all dual-layer displays
      • 142. With a fixed pair of masks, emitted light field is rank-1
      • 143. Achieved higher-rank approximation using temporal multiplexing
      • 144. With T time-multiplexed masks, emitted light field is rank-T
      • 145. Constructed a prototype using off-the-shelf panels
      • 146. Demonstrated light field display is a matrix approximation problem
      • 147. Introduced content-adaptive parallax barriers
      • 148. Applied weighted NMF to optimize weighted Euclidean distance to target
      Adaptation increases brightness and refresh rate of dual-stacked LCDs
    • 149. Parallax Barrier: Np=103 pix.
      Hologram: NH=105 pix.
      ϕP∝w/d
      ϕH∝λ/tH
      θp=10 pix
      θH =1000 pix
      Fourier Patch
      xH =100 patches
      w
      xp=100slits
      Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009
    • 150. Augmented Light Field
      118
      wave optics based
      rigorous but cumbersome
      Wigner Distribution Function
      WDF
      Augmented LF
      Traditional Light Field
      Traditional Light Field
      ray optics based
      simple and powerful
      Interference & Diffraction
      Interaction w/ optical elements
      Oh, Raskar, Barbastathis 2009: Augmented Light Field
    • 151. position
      light field transformer
      LF
      LF
      LF
      LF
      (diffractive)
      optical element
      Reference
      plane
      LF propagation
      LF propagation
      Light Fields
      Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters
      angle
    • 152. Augmented Lightfield for Wave Optics Effects
      WDF
      Wigner Distribution Function
      Augmented Light Field
      Light Field
      LF
      LF < WDF
      Lacks phase properties
      Ignores diffraction, interferrence
      Radiance = Positive
      ALF ~ WDF
      Supports coherent/incoherent
      Radiance = Positive/Negative
      Virtual light sources
    • 153. Free-space propagation
      Light field transformer
      Virtual light projector
      Possibly negative radiance
      121
    • 154. Lightfieldvs Hologram Displays
    • 155. Is hologram just another ray-based light field?
      Can a hologram create any intensity distribution in 3D?
      Why hologram creates a ‘wavefront’ but PB does not?
      Why hologram creates automatic accommodation cues?
      What is the effective resolution of HG vs PB?
    • 156. Zooming into the Light Field
      Rays: No Bending
      1 Fresnel HG Patch
      p Wm
      *
      *
      p d(θ)
      q d(θ)
      *
      q
      q
      p
      p
      *
      q Wm
      L(x,θ)
      W(x,u)
      Wm= sinc
      d = delta
      u
      θ
    • 157. Algebraic Rank Constraint
      Rank-3
      Rank-1
      Rank-1
      s1*
      s1
      m2
      s1*
      m2
      s1
      (a) Parallax Barrier
      (c) Hybrid
      (b) Hologram
      s1
      s1
    • 158. (a) Two Slits, Coherent
      Interference

      Rank-1
      -1
      Transform
      u
      -Transform
      R45, D
      x
      <t(x+xʹ/2)t*(x-xʹ/2)>
      t(x1)t*(x2)
      t(x+xʹ/2)t*(x-xʹ/2)
      W(x,u)
    • 159. L1(x,θ)
      (a)
      L1
      L2(x,θ)
      L2
      L3(x,θ)
      s1
      m2
      L3
      hH
      ϕ1
      ϕ1
      z2
      ϕ1
      ϕ1
      z1
      r
      d
      L3(x,θ)
      L1(x,θ)
      L2(x,θ)
    • 160. Is hologram just another ray-based light field?
      Can a hologram create any intensity distribution in 3D?
      Why hologram creates a ‘wavefront’ but PB does not?
      Why hologram creates automatic accommodation cues?
      What is the effective resolution of HG vs PB?
    • 161. Three Questions
      What are the benefits of higher dimensional imaging?
      Why is the algebraic rank of a Light Field not full?
      What makes looking around the corner possible?
    • 162. How to do Research in Imaging
      http://raskar.info
      How to come up w ideas: Idea Hexagon
      How to write a paper
      How to give a talk
      Open research problems
      How to decide merit of a project
      How to attend a conference, brainstorm
      Facebook.com/ rRaskar
      Tips
      Get on Seminar/Talks mailing lists worldwide
      http://www.cs.virginia.edu/~robins/YouAndYourResearch.html
      Why do so few scientists make significant contributions and so many are forgotten in the long run?
      Highly recommended Hamming talk at Bell Labs
    • 163. Take home points
      Co-design of hw/sw
      Avoid computational or optical chauvinism in imaging
      (Camera flash/Kinect)
      Hardware cost going to zero, Parallel technology trends
      Computer vision not just mimicking human vision/perception
      Borrow ideas from other fields: astronomy, scientific imaging, audio, communications
      Photons not just Pixels
      Change the rules of the game
      Optics, Sensors, Illum,
      Priors, Sparsity, Transforms
      Meta-data, Internet collection, Crowdsourcing
    • 164. MIT Media Lab raskar@mit.edu http://cameraculture.info fb.com/rraskar
      Inverse Problems
      How to do Research in Imaging
      • Inverse Problems, Reconstruction, Rank and Sparsity
      Co-design of Optics and Computation
      Photons not just pixels
      Mid-level cues
      Computational Photography
      Open research problems
      Compressive Sensing for High Speed Events
      Limits of CS for general imaging
      Computational Light Transport
      Looking Around Corners, trillion fps
      Lightfields: 3D Displays and Holograms
      Apply for internships/post-doc
      neXt

    ×