Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Computational Photography http://raskar.info

Invertible Motion Blur in Video Photo 1 Photo 2 Photo 3 Deblurring Agrawal, Xu, Raskar, Siggraph 2009

Traditional Exposure Video DFT Motion PSF (Box Filter) Information is lost Exposure Time

Coded Exposure (Flutter Shutter) Raskar et al. 2006 Single Photo Deblurred Image

Varying Exposure Video DFT Exposure Time

Varying Exposure Video DFT Exposure Time Exposure Time

Varying Exposure Video DFT Exposure Time Exposure Time Exposure Time

Varying Exposure Video == PSF Null-Filling DFT Joint Frequency Spectrum Preserves High Frequencies

Varying Exposure Video: Exploit auto-exposure mode

Completely automatic: (i) Segmentation, (ii) PSF estimation, (iii) deblurring Blurred Photo

Deblurred Result Ground Truth Input Photos

Computational Photography [Raskar and Tumblin] Computational Imaging vs Computational Photography Synthesis Minimal change of hardware Goals are often opposite (human perception) Epsilon Photography Low-level vision: Pixels ‘ Ultimate camera’ Coded Photography Mid-Level Cues: Regions, Edges, Motion, Direct/global ‘ Scene analysis’ Essence Photography High-level understanding ‘ New artform’ captures a machine-readable representation of our world to hyper-realistically synthesize the essence of our visual experience.

Computational Imaging vs Computational Photography

Synthesis

Minimal change of hardware

Goals are often opposite (human perception)

Epsilon Photography

Low-level vision: Pixels

‘ Ultimate camera’

Coded Photography

Mid-Level Cues:

Regions, Edges, Motion, Direct/global

‘ Scene analysis’

Essence Photography

High-level understanding

‘ New artform’

Synthesis Low Level Mid Level High Level Hyper realism Raw Angle, spectrum aware Non-visual Data, GPS Metadata Priors Comprehensive 8D reflectance field Computational Photography (vs Imaging) Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism Resolution

Enhanced Defocus Blur Lots of glass; Heavy; Bulky; Expensive

Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh Raskar MIT Media Lab MIT Media Lab Camera Culture

Image Destabilization Lens Sensor Camera Static Scene

Image Destabilization Static Scene Lens Motion Sensor Motion Camera Mohan, Lanman,Hiura, Raskar ICCP 2009

Shifting Pinhole and Sensor A B A’ B’ Pinhole v p Sensor v s d a d b d s Focus Here

Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here

Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here

Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here

“ Time Lens” Ratio of speeds Lens Equation: Virtual Focal Length: Virtual F-Number:

Time Lens:

Adjusting the Focus Plane all-in-focus image

Adjusting the Focus Plane focused in the front using destabilization

Adjusting the Focus Plane focused in the middle using destabilization

Adjusting the Focus Plane focused in the back using destabilization

Bokode

Smart Barcode size : 3mm x 3mm Ordinary Camera: Distance 3 meter Long Distance Barcodes

Smart Barcode size : 3mm x 3mm

Ordinary Camera: Distance 3 meter

Defocus blur of Bokode

Coding in Angle Mohan, Woo, Smithwick, Hiura, Raskar [Siggraph 2009]

sensor Bokode (angle) Encoding in Angle , not space, time or wavelength camera

circle of confusion  circle of information camera Bokode (angle) Quote suggested by Kurt Akeley Encoding in Angle , not space, time or wavelength sensor

circle of confusion  circle of information

magnification = f c /f b (microscope) ; focus always at infinity Bokode camera f b f c ‘ long-distance microscope’

magnification = f c /f b (microscope) ;

focus always at infinity

Product labels Street-view Tagging

Product labels

capturing Bokodes cell-phone camera close to the Bokode (10,000+ bytes of data)

Augmenting Plenoptic Function Wigner Distribution Function Traditional Light Field WDF Traditional Light Field Augmented LF Interference & Diffraction Interaction w/ optical elements ray optics based simple and powerful wave optics based rigorous but cumbersome

Light Fields Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters Reference plane position angle LF propagation (diffractive) optical element LF LF LF LF LF propagation light field transformer

Free-space propagation Light field transformer Virtual light projector Possibly negative radiance

Free-space propagation

Light field transformer

Virtual light projector

Possibly negative radiance

LF Transformer: input to output LF Thin Elements: 6D General Case: 8D Angle Shift Invariant: 4D

Augmented LF framework 1. LF propagation (diffractive) optical element LF LF LF LF LF propagation 2. light field transformer 3. negative radiance Tech report, S. B. Oh et al. http://web.media.mit.edu/~raskar/RayWavefront/ 4. interference

Cubic phase plate Pure ray bending Positive only radiance Diffraction Interference of +/- rays Rotating PSF

Cubic phase plate

Interference received on Complex Geometry

Can you ‘see’ around a corner ?

Femto-Photography: Higher Dimensional LF FemtoFlash UltraFast Detector Computational Optics Serious Sync

Important Dates Submission: November 2, 2009 Notification : February 2, 2010 Topics Computational Cameras Multiple Images and Camera Arrays Computational Illumination Advanced Image and Video Processing Scientific Photography and Videography Organizing and Exploiting Photo & Video Collections Program Chairs Kyros Kutulakos, U. Toronto Rafael Piestun, U. Colorado Ramesh Raskar, MIT International Conference on Computational Photography (ICCP) March 29-30, 2010 MIT, Cambridge MA http://cameraculture.media.mit.edu/ iccp10 /

Beyond Multi-touch: Mobile Laptops Mobile

Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen

Overview: Sensing Depth from Array of Virtual Cameras in LCD

Beyond Traditional Imaging Invertible motion blur in video Looking around a corner LCDs as virtual cameras Computational probes (bokode) Image destabilization Augmented Light Field Rays for diffraction+interference Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info Computational Photography Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism WDF Light Field Augmented LF LF propagation (diffractive) optical element LF LF LF LF LF propagation light field transformer

Beyond Traditional Imaging

Invertible motion blur in video

Looking around a corner

LCDs as virtual cameras

Computational probes (bokode)

Image destabilization

Augmented Light Field

Rays for diffraction+interference