Your SlideShare is downloading. ×
MIT Camera Culture Group Update July 2009
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

MIT Camera Culture Group Update July 2009


Published on

Published in: Technology, Art & Photos, Business

1 Comment
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Since we are adapting LCD technology we can fit a BiDi screen into laptops and mobile devices.
  • 4 blocks : light, optics, sensors, processing, (display: light sensitive display)
  • Inference and perception are important. Intent and goal of the photo is important. The same way camera put photorealistic art out of business, maybe this new artform will put the traditional camera out of business. Because we wont really care about a photo, merely a recording of light but a form that captures meaningful subset of the visual experience. Multiperspective photos. Photosynth is an example.
  • Comparisons
  • Reversibly encode all the information in this otherwise blurred photo
  • The glint out of focus shows the unusual pattern.
  • put two projectors, one virtual projector at the middle, along this line, connecting the virtual light source, always destructive interference, does it make sense and right?
  • in wave optics, WDF exhibit similar property, compare the two,
  • the motivation, to augment lf, model diffraction in light field formulation
  • put two projectors, one virtual projector at the middle, along this line, connecting the virtual light source, always destructive interference, does it make sense and right?
  • Recall that one of our inspirations was this new class of optical multi-touch device. At the top you can see a prototype that Sharp Microelectronics has published. These devices are basically arrays of naked phototransistors. Like a document scanner, they are able to capture a sharp image of objects in contact with the surface of the screen. But as objects move away from the screen, without any focusing optics, the images captured this device are blurred.
  • This device would of course support multi-touch on-screen interaction, but because it can measure the distance to objects in the scene a user’s hands can be tracked in a volume in front of the screen, without gloves or other fiducials.
  • Since we are adapting LCD technology we can fit a BiDi screen into laptops and mobile devices.
  • Thus the ideal BiDi screen consists of a normal LCD panel separated by a small distance from a bare sensor array. This format creates a single device that spatially collocates a display and capture surface.
  • So here is a preview of our quantitative results. I’ll explain this in more detail later on, but you can see we’re able to accurately distinguish the depth of a set of resolution targets. We show above a portion of portion of views form our virtual cameras, a synthetically refocused image, and the depth map derived from it.
  • Our observation is that by moving the sensor plane a small distance from the LCD in an optical multitouch device, we enable mask-based light-field capture. We use the LCD screen to display the desired masks, multiplexing between images displayed for the user and masks displayed to create a virtual camera array. I’ll explain more about the virtual camera array in a moment, but suffice to say that once we have measurements from the array we can extract depth.
  • Transcript

    • 1. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab
    • 2.
      • How can we create an entirely new class of imaging platforms that have an understanding of the world that far exceeds human ability and produce meaningful abstractions that are well within human comprehensibility ?
      Ramesh Raskar
    • 3. Can you look around a corner ?
    • 4. Can you decode a 5 micron feature from 3 meters away with an ordinary camera ?
    • 5. Beyond Multi-touch: Mobile 3D Interfaces?
    • 6. 6D Display Light sensitive 4D display One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008
    • 7.  
    • 8.
      • A. Introduction‐‐5 minutes
      • B. Cameras of the future ( Raskar , 30 minutes) * Form factors, Modalities and Interaction * Enabling Visual Social Computing
      • C. Reconstruction the World ( Seitz , 30 minutes) * Photo tourism and beyond * Image‐based modeling and rendering on a massive scale * Scene summarization
      • D. Understanding a Billion Photos ( Efros , 30 minutes) * What will the photos depict? * Photos as visual content for computer graphics * Solving computer vision
      • E. Discussion‐‐10 minutes
      Course: Next Billion Cameras Wedn at 3:30pm
    • 9. Looking around a corner via Transient Imaging Coded Aperture Post-capture Focus Control Bokode: Long Distance Barcode Camera Culture Group, MIT Media Lab Ramesh Raskar BiDi Screen Theory of Light Propagation Light Field Augmented Light Field WDF
    • 10. Cameras and their Impact
      • Beyond Traditional Imaging Analysis and synthesis
        • Emerging optics, illumination, novel sensors
        • Exploit priors and online collections
      • Applications
        • Better scene understanding/analysis
        • Capture visual essence
        • Superior Metadata tagging for effective sharing
        • Fuse non-visual data
      • Impact on Society
        • Beyond entertainment and productivity
        • Sensors for disabled, new art forms, crowdsourcing, bridging cultures, Social stability
    • 11. New Topics in Camera Research
      • Imaging Devices, Modern Optics and Lenses
      • Emerging Sensor Technologies
      • Mobile Photography
      • Visual Social Computing and Citizen Journalism
      • Imaging Beyond Visible Spectrum
      • Computational Imaging in Sciences (Medical)
      • Trust in Visual Media
      • Solutions for Visually Challenged
      • Cameras in Developing Countries
      • Future Products and Business Models
    • 12. International Conference on Computational Photography Papers due November 2, 2009
    • 13.  
    • 14. Traditional Photography Lens Detector Pixels Image Mimics Human Eye for a Single Snapshot : Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy: Shree Nayar
    • 15. Computational Photography Computational Illumination Computational Camera Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing 4D Ray Bender Upto 4D Ray Sampler Ray Reconstruction Generalized Optics Recreate 4D Lightfield Light Sources Modulators 4D Incident Lighting 4D Light Field
    • 16. Computational Photography [Raskar and Tumblin]
      • Epsilon Photography
        • Low-level vision: Pixels
        • Multi-photos by perturbing camera parameters
        • HDR, panorama, …
        • ‘ Ultimate camera’
      • Coded Photography
        • Mid-Level Cues:
          • Regions, Edges, Motion, Direct/global
        • Single/few snapshot
          • Reversible encoding of data
        • Additional sensors/optics/illum
        • ‘ Scene analysis’
      • Essence Photography
        • High-level understanding
          • Not mimic human eye
          • Beyond single view/illum
        • ‘ New artform’
    • 17. Digital Epsilon Coded Essence Goal and Experience Low Level Mid Level High Level Hyper realism Raw Angle, spectrum aware Non-visual Data, GPS Metadata Priors Comprehensive Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision 8D reflectance field Dylan’s bicycle Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification
    • 18.
      • Ramesh Raskar and Jack Tumblin
      • Book Publishers: A K Peters
      • Siggraph 2009 booth: 20% off
      • Booth #2527
      • Meet the Authors
      • Thursday at 2pm-2:30pm
    • 19. Multi-flash Camera for Detecting Depth Edges
    • 20. Depth Edges Left Top Right Bottom Depth Edges Canny Edges
    • 21. Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] LCD opacity switched in coded sequence
    • 22. Traditional Coded Exposure Image of Static Object Deblurred Image Deblurred Image
    • 23. Can you look around a corner ?
    • 24. Can you look around a corner ? Kirmani, Hutchinson, Davis, Raskar 2009 Accepted for ICCV’2009, Oct 2009 in Kyoto Impulse Response of a Scene
    • 25. Femtosecond Laser as Light Source Pico-second detector array as Camera
    • 26. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern
    • 27. In Focus Photo LED
    • 28. Out of Focus Photo: Open Aperture
    • 29. Out of Focus Photo: Coded Aperture
    • 30. Captured Blurred Photo
    • 31. Refocused on Person
    • 32. Bokode: ankit mohan, grace woo, shinsaku hiura, quinn smithwick, ramesh raskar camera culture group, MIT media lab imperceptible visual tags for camera based interaction from a distance
    • 33. Barcodes markers that assist machines in understanding the real world
    • 34.
      • Smart Barcode size : 3mm x 3mm
      • Ordinary Camera: Distance 3 meter
      Computational Probes: Long Distance Bar-codes Mohan, Woo,Smithwick, Hiura, Raskar Accepted as Siggraph 2009 paper
    • 35. Bokode
    • 36. Defocus blur of Bokode
    • 37. Image greatly magnified. Simplified Ray Diagram
    • 38. Our Prototypes
    • 39. street-view tagging
    • 40. tabletop/surface interaction
    • 41. multi-user interaction
    • 42. Varying Exposure Video Amit Agrawal MERL , Yi Xu Purdue , Ramesh Raskar, MIT
    • 43. Deblurred Result Blurred Photo
    • 44. Varying Exposure Video DFT Exposure Time Exposure Time Exposure Time
    • 45. Can we deal with particle-wave duality of light with modern Lightfield theory ? Young’s Double Slit Expt first null (OPD = λ/2) Diffraction and Interferences modeled using Ray representation
    • 46. Light Fields
      • Radiance per ray
      • Ray parameterization:
        • Position : x
        • Direction : θ
      Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters Reference plane position direction
    • 47. Light Fields for Wave Optics Effects Wigner Distribution Function Light Field LF < WDF Lacks phase properties Ignores diffraction, phase masks Radiance = Positive ALF ~ WDF Supports coherent/incoherent Radiance = Positive/Negative Virtual light sources Light Field Augmented Light Field WDF
    • 48. Limitations of Traditional Lightfields Wigner Distribution Function Traditional Light Field ray optics based simple and powerful rigorous but cumbersome wave optics based limited in diffraction & interference holograms beam shaping rotational PSF
    • 49. Example: New Representations Augmented Lightfields Wigner Distribution Function Traditional Light Field WDF Traditional Light Field Augmented LF Interference & Diffraction Interaction w/ optical elements ray optics based simple and powerful limited in diffraction & interference rigorous but cumbersome wave optics based Non-paraxial propagation
    • 50. (ii) Augmented Light Field with LF Transformer WDF Light Field Augmented LF Interaction at the optical elements Augmenting Light Field to Model Wave Optics Effects , [Oh, Barbastathis, Raskar] LF propagation (diffractive) optical element LF LF LF LF LF propagation light field transformer negative radiance
    • 51. Virtual light projector with real valued (possibly negative radiance) along a ray real projector real projector first null (OPD = λ/2) virtual light projector
    • 52. (ii) ALF with LF Transformer
    • 53. Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar BiDi Screen *
    • 54. Light Sensing Pixels in LCD Display with embedded optical sensors Sharp Microelectronics Optical Multi-touch Prototype
    • 55. Beyond Multi-touch: Hover Interaction
      • Seamless transition of multitouch to gesture
      • Thin package, LCD
    • 56. Beyond Multi-touch: Mobile Laptops Mobile
    • 57. Design Vision Object Collocated Capture and Display Bare Sensor Spatial Light Modulator
    • 58. Touch + Hover using Depth Sensing LCD Sensor
    • 59. Overview: Sensing Depth from Array of Virtual Cameras in LCD
    • 60. Design Overview Display with embedded optical sensors LCD , displaying mask Optical sensor array ~2.5 cm ~50 cm
    • 61. International Conference on Computational Photography Papers due November 2, 2009
    • 62.  
    • 63. Looking around a corner via Transient Imaging Coded Aperture Post-capture Focus Control Bokode: Long Distance Barcode Camera Culture Group, MIT Media Lab Ramesh Raskar BiDi Screen Theory of Light Propagation Light Field Augmented Light Field WDF