After X , what is ne X t Coming up with New Ideas in Imaging Ramesh Raskar, MIT Media Lab

X d X++ X X+Y X X ne X t Ramesh Raskar, MIT Media Lab

Camera Culture Ramesh Raskar Camera Culture MIT Media Lab http://raskar.info http://cameraculture.info Ramesh Raskar Associate Professor

Create tools to better capture and share visual information The goal is to create an entirely new class of imaging platforms that have an understanding of the world that far exceeds human ability and produce meaningful abstractions that are well within human comprehensibility

Create tools to better capture and share visual information

The goal is to create an entirely new class of imaging platforms that have an understanding of the world that far exceeds human ability and produce meaningful abstractions that are well within human comprehensibility

Camera Culture Course WebPage : http:// cameraculture.info /courses/

After X , what is ne X t Coming up with New Ideas in Imaging Ramesh Raskar, MIT Media Lab

X d X++ X X+Y X X ne X t Ramesh Raskar, MIT Media Lab

Simple Exercise .. What is ne X t

Strategy #1: X d Extend it to next dimension (or some other) dimension Context aware resizing Video Instead of square resizing-> CD cover (with a hole in center) resizing Text, Audio (Speech), Image, Video .. Whats next ? Video, 3D meshes, 4D lightfields Images to infrared, sound, ultrasound Macro scale to microscale (Levoy, Lightfield to Microscope) Time to space to angle to id (coded exposure <- coded aperture)

Extend it to next dimension (or some other) dimension

Context aware resizing

Video

Instead of square resizing-> CD cover (with a hole in center) resizing

Text, Audio (Speech), Image, Video .. Whats next ?

Video, 3D meshes, 4D lightfields

Images to infrared, sound, ultrasound

Macro scale to microscale (Levoy, Lightfield to Microscope)

Time to space to angle to id

(coded exposure <- coded aperture)

Coded-Aperture Imaging Lens-free imaging! Pinhole-camera sharpness, without massive light loss. No ray bending (OK for X-ray, gamma ray, etc.) Two elements Code Mask: binary (opaque/transparent) Sensor grid Mask autocorrelation is delta function (impulse) Similar to MotionSensor ?

Lens-free imaging!

Pinhole-camera sharpness, without massive light loss.

No ray bending (OK for X-ray, gamma ray, etc.)

Two elements

Code Mask: binary (opaque/transparent)

Sensor grid

Mask autocorrelation is delta function (impulse)

Similar to MotionSensor ?

Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] LCD opacity switched in coded sequence

Figure 2 results Input Image Problem: Motion Deblurring Ramesh Raskar, Camera Culture, MIT Media Lab

Image Deblurred by solving a linear system. No post-processing Blurred Taxi Ramesh Raskar, Camera Culture, MIT Media Lab

Flutter Shutter: Shutter is OPEN and CLOSED Sharp Photo Blurred Photo PSF == Broadband Function Fourier Transform Preserves High Spatial Frequencies

Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern

Out of Focus Photo: Coded Aperture

Captured Blurred Photo

Refocused on Person

Larval Trematode Worm

Strategy #2: X+Y Fusion of the dissimilar More dissimilar, more spectacular the output Example Scientific imaging + Photography Coded aperture Tomography Lightfields + User interfaces Projector = camera Spatial Augmented Reality

Fusion of the dissimilar

More dissimilar, more spectacular the output

Example

Scientific imaging + Photography

Coded aperture

Tomography

Lightfields + User interfaces

Projector = camera

Spatial Augmented Reality

Imaging in Sciences: Computer Tomography http://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_imaging/

http://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_imaging/

Borehole tomography receivers measure end-to-end travel time reconstruct to find velocities in intervening cells must use limited-angle reconstruction method (like ART) (from Reynolds)

receivers measure end-to-end travel time

reconstruct to find velocities in intervening cells

must use limited-angle reconstruction method (like ART)

Prototype camera 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125 μ square-sided microlenses

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Example of digital refocusing

Coded-Aperture Imaging Lens-free imaging! Pinhole-camera sharpness, without massive light loss. No ray bending (OK for X-ray, gamma ray, etc.) Two elements Code Mask: binary (opaque/transparent) Sensor grid Mask autocorrelation is delta function (impulse) Similar to MotionSensor ?

Lens-free imaging!

Pinhole-camera sharpness, without massive light loss.

No ray bending (OK for X-ray, gamma ray, etc.)

Two elements

Code Mask: binary (opaque/transparent)

Sensor grid

Mask autocorrelation is delta function (impulse)

Similar to MotionSensor ?

Mask in a Camera Mask Aperture Canon EF 100 mm 1:1.28 Lens, Canon SLR Rebel XT camera

Strategy #3: X Do exactly the opposite Processing, Memory, Bandwidth In Computing world, in any era, one of this is a bottleneck But overtime, they change. You can often take an older idea and do exactly the opposite. E.g. bandwidth is now considered virtually limitless In imaging: Larger sensors? Everyone is thinking about building cheaper, smaller pixel sensors and THEN improving SNR .. Maybe just build larger sensors? SLR: Faster mirror flip or no mirror flip Companies spent years improving mirror flip speed Why not just remove it? More computation Less light

Processing, Memory, Bandwidth

In Computing world, in any era, one of this is a bottleneck

But overtime, they change. You can often take an older idea and do exactly the opposite.

E.g. bandwidth is now considered virtually limitless

In imaging:

Larger sensors?

Everyone is thinking about building cheaper, smaller pixel sensors and THEN improving SNR .. Maybe just build larger sensors?

SLR: Faster mirror flip or no mirror flip

Companies spent years improving mirror flip speed

Why not just remove it?

More computation

Less light

e.g. Reverse Auction

e.g. Reverse Auction

Less is More Blocking Light == More Information Coding in Time Coding in Space

Larval Trematode Worm

Vicon Motion Capture High-speed IR Camera Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis

Towards ‘on-set’ motion capture 500 Hz with Id for each Marker Tag Visually imperceptible tags + Natural lighting Unlimited Number of Tags Base station and tags only a few 10’s $ Traditional: High-speed IR Camera + Body markers Second Skin : High-speed LED emitters+ Photosensing Body markers

500 Hz with Id for each Marker Tag

Visually imperceptible tags + Natural lighting

Unlimited Number of Tags

Base station and tags only a few 10’s $

R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns Siggraph 2007 Prakash: Lighting-Aware Motion Capture Using Photosensing Markers and Multiplexed Illuminators

Imperceptible Tags under clothing, tracked under ambient light Hidden Marker Tags Outdoors Unique Id

Labeling Space (Indoor GPS) Each location receives a unique temporal code But 60Hz video projector is too slow Projector Tags Pos=0 Pos=25 5 Time

Pattern MSB Pattern MSB-1 Pattern LSB For each tag From light sequence, decode x and y coordinate Transmit back to RF reader ( Id , x, y ) 0 1 1 0 0 X=12

For each tag

From light sequence, decode x and y coordinate

Transmit back to RF reader ( Id , x, y )

Inside of Multi-LED Emitter

Tag

When life gives you lemon, make lemonade

When life gives you lemon, make lemonade

Depth Edge Camera

Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk Mitsubishi Electric Research Labs (MERL), Cambridge, MA U of California at Santa Barbara U of North Carolina at Chapel Hill Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging

Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes

Depth Edges

Our Method Canny

Canny Intensity Edge Detection Our Method Photo Result

Car Manuals

What are the problems with ‘real’ photo in conveying information ? Why do we hire artists to draw what can be photographed ?

Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors

Shadows Clutter Many Colors Highlight Edges Mark moving parts Basic colors A New Problem

Canny Intensity Edge Detection Our Method

Strategy #4: X Given a Hammer .. Find all the nails Sometimes even screws and bolts Given a cool solution/technique, find other problems Good recent examples Gradient domain techniques Introduced in Graphics for High dynamic range tone mapping [Fattal Lischinski 2002] Now a major hammer Image editing, compositing, fusion, alpha matting, reflection layer recovery

Given a Hammer ..

Find all the nails

Sometimes even screws and bolts

Given a cool solution/technique,

find other problems

Good recent examples

Gradient domain techniques

Introduced in Graphics for High dynamic range tone mapping [Fattal Lischinski 2002]

Now a major hammer

Image editing, compositing, fusion, alpha matting, reflection layer recovery

A Night Time Scene: Objects are Difficult to Understand due to Lack of Context Dark Bldgs Reflections on bldgs Unknown shapes

Enhanced Context : All features from night scene are preserved, but background in clear ‘ Well-lit’ Bldgs Reflections in bldgs windows Tree, Street shapes

Background is captured from day-time scene using the same fixed camera Night Image Day Image Result: Enhanced Image

Flash Result Reflection Layer Ambient Flash and Ambient Images [ Agrawal, Raskar, Nayar, Li Siggraph05 ]

Agrawala et al, Digital Photomontage, Siggraph 2004

Agrawala et al, Digital Photomontage, Siggraph 2004

Agrawala et al, Digital Photomontage, Siggraph 2004

actual photomontage set of originals perceived

Source images Brush strokes Computed labeling Composite

Strategy #5: X Given a problem, find other solutions Given a nail, find all hammers Sometimes even screwdrivers and pliers may work High Dynamic Range Tone Mapping Started with Jack Tumblin’s LCIS Gradient domain Bilateral filter Filter banks etc .. About 6 years of heavy machinery Btw, the topic is done to death but continues to enthuse

Given a problem, find other solutions

Given a nail, find all hammers

Sometimes even screwdrivers and pliers may work

High Dynamic Range Tone Mapping

Started with Jack Tumblin’s LCIS

Gradient domain

Bilateral filter

Filter banks etc ..

About 6 years of heavy machinery

Btw, the topic is done to death but continues to enthuse

Strategy #6: X++ Pick your adjective .. Making it faster, better, cheaper neXt = adjective + X

Pick your adjective ..

Making it faster, better, cheaper

neXt = adjective + X

X++ : Add your favorite adjective Context aware, Adaptive (temporally) Coherent, Hierarchical, Progressive Efficient Parallelized Distributed Good example: Image or video compression schemes But X++ is a bad sign The field is dying in terms of research but booming in business impact

Context aware,

Adaptive

(temporally) Coherent,

Hierarchical,

Progressive

Efficient

Parallelized

Distributed

Good example: Image or video compression schemes

But X++ is a bad sign

The field is dying in terms of research but booming in business impact

Pitfalls These six ways are only a start They are a good mental exercise and will allow you to train as a researcher Great for class projects But Maynot produce radically new ideas Sometimes a danger of being labeled incremental Could be into ‘public domain ideas’

These six ways are only a start

They are a good mental exercise and will allow you to train as a researcher

Great for class projects

But

Maynot produce radically new ideas

Sometimes a danger of being labeled incremental

Could be into ‘public domain ideas’

What are Bad ideas to pursue X then Y (then Z) X+Y is great with true fusion, fusion of dissimilar is best But avoid a ‘pipeline’ systems paper, where the output of one is THEN channeled into the input of the next stage, and non of the components are novel E.g. I want to build a Follow the hype (too much competition) Do because it can be done (Why do we climb? because it is there! But only the first one gets a credit. May make you strong, and give you a sense of achievement but not a research project. )

X then Y (then Z)

X+Y is great with true fusion, fusion of dissimilar is best

But avoid a ‘pipeline’ systems paper, where the output of one is THEN channeled into the input of the next stage, and non of the components are novel

E.g. I want to build a

Follow the hype (too much competition)

Do because it can be done

(Why do we climb? because it is there!

But only the first one gets a credit.

May make you strong, and give you a sense of achievement but not a research project. )

X d X++ X X+Y X X ne X t Ramesh Raskar, MIT Media Lab

Camera Culture Ramesh Raskar Camera Culture MIT Media Lab http://raskar.info http://cameraculture.info