Successfully reported this slideshow.

Raskar 6Sight Keynote Talk Nov09

8

Share

Upcoming SlideShare
Raskar ideahexagonapr2010
Raskar ideahexagonapr2010
Loading in …3
×
1 of 93
1 of 93

Raskar 6Sight Keynote Talk Nov09

8

Share

Download to read offline

Though revolutionary in many ways, digital photography is essentially electronically implemented film photography. By contrast, computational photography exploits plentiful low-cost computing and memory, new kinds of digitally enabled sensors, optics, probes, smart lighting, and communication to capture information far beyond just a simple set of pixels. It promises a richer, even a multilayered, visual experience that may include depth, fused photo-video representations, or multispectral imagery. Professor Raskar will discuss and demonstrate advances he is working on in the areas of generalized optics, sensors, illumination methods, processing, and display, and describe how computational photography will enable us to create images that break from traditional constraints to retain more fully our fondest and most important memories, to keep personalized records of our lives, and to extend both the archival and the artistic possibilities of photography.

Though revolutionary in many ways, digital photography is essentially electronically implemented film photography. By contrast, computational photography exploits plentiful low-cost computing and memory, new kinds of digitally enabled sensors, optics, probes, smart lighting, and communication to capture information far beyond just a simple set of pixels. It promises a richer, even a multilayered, visual experience that may include depth, fused photo-video representations, or multispectral imagery. Professor Raskar will discuss and demonstrate advances he is working on in the areas of generalized optics, sensors, illumination methods, processing, and display, and describe how computational photography will enable us to create images that break from traditional constraints to retain more fully our fondest and most important memories, to keep personalized records of our lives, and to extend both the archival and the artistic possibilities of photography.

More Related Content

More from Camera Culture Group, MIT Media Lab

Raskar 6Sight Keynote Talk Nov09

  1. 1. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Photography Wishlist http://raskar.info
  2. 2. Film-like Digital Photography
  3. 3. Fernald, Science [Sept 2006] Shadow Refractive Reflective
  4. 4. ‘ film-like’ Photography Lens Detector Pixels Image Slide by Shree Nayar Reproduce for the eye
  5. 5. Wish List Today <ul><li>Consumers </li></ul><ul><ul><li>Super-human vision </li></ul></ul><ul><ul><li>Microscope like resolution </li></ul></ul><ul><ul><li>High speed </li></ul></ul><ul><ul><li>See inside the body (health) </li></ul></ul><ul><ul><li>Auto-trigger </li></ul></ul><ul><ul><li>Keep only ‘good’ pics </li></ul></ul><ul><ul><li>Find ‘relevant’ pics and better archiving/access </li></ul></ul><ul><ul><li>Put photographer back in photo! </li></ul></ul><ul><li>Companies </li></ul><ul><ul><li>Cost </li></ul></ul><ul><ul><li>Resolution </li></ul></ul><ul><ul><li>Low-light sensitivity, HDR </li></ul></ul><ul><ul><li>Stereo and 3D </li></ul></ul><ul><ul><li>Mecha-free zoom/focus </li></ul></ul><ul><ul><li>Auto-tagging for sharing </li></ul></ul><ul><ul><li>Recognition </li></ul></ul>
  6. 6. Computational Photography Computational Illumination Computational Camera Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing Ray Bender Ray Sampler Ray Reconstruction Generalized Optics Recreate 4D Lightfield Light Sources Modulators 4D Incident Lighting 4D Light Field
  7. 7. Can you look around a corner ?
  8. 9. Computational Photography [Raskar and Tumblin] <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level vision: Pixels </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>‘ Scene analysis’ </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>High-level understanding </li></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>captures a machine-readable representation of our world to hyper-realistically synthesize the essence of our visual experience.
  9. 10. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Computational Photography Wish List http://raskar.info
  10. 11. Synthesis Low Level Mid Level High Level Hyper realism Raw Angle, spectrum aware Non-visual Data, GPS Metadata Priors Comprehensive 8D reflectance field Computational Photography Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism Resolution
  11. 12. Camera Culture Ramesh Raskar Wish #1 Ultimate Post-capture Control
  12. 13. Digital Refocusing using Light Field Camera 125 μ square-sided microlenses [Ng et al 2005]
  13. 14. Zooming into the raw photo
  14. 15. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor
  15. 16. Captured 2D Photo
  16. 17. x y u v 121 Sub-aperture Views
  17. 18. <ul><li>Samples individual rays </li></ul><ul><li>Predefined spectrum for lenses </li></ul><ul><li>Chromatic abberration </li></ul><ul><li>High alignment precision </li></ul><ul><li>For each lenslet, peripheral pixels are wasted </li></ul><ul><li>Negligible Light Loss </li></ul><ul><li>Samples coded combination of rays </li></ul><ul><li>Supports any wavelength </li></ul><ul><li>Reconfigurable f/# , Easier alignment </li></ul><ul><li>No wastage </li></ul><ul><li>High resolution image for parts of scene in focus </li></ul><ul><li>50 % Light Loss due to mask </li></ul>Microlens array Sensor Plenoptic Camera Heterodyne Camera Mask Sensor
  18. 19. Figure 2 results Input Image Motion Blur in Low Light
  19. 20. Motion Blur in Low Light
  20. 21. Fluttered Shutter Camera Raskar, Agrawal, Tumblin Siggraph2006 Ferroelectric shutter in front of the lens is turned opaque or transparent in a rapid binary sequence
  21. 22. Image Deblurred by solving a linear system. No post-processing Blurred Taxi
  22. 23. Motion Blur in Low Light
  23. 24. Compact Programmable Lights ?
  24. 25. Image-Based Re-lighting Measure incoming light in Milan, Light the actress in LA Matte the background Matched LA and Milan lighting. Debevec et al., SIGG2001
  25. 26. Camera Culture Ramesh Raskar <ul><li>Wish #1 </li></ul><ul><li>Ultimate Post-capture Control </li></ul><ul><ul><li>Digital Refocus and Motion blur </li></ul></ul><ul><ul><li>Emulate studio light from compact flash </li></ul></ul>
  26. 27. Camera Culture Ramesh Raskar Wish #2 Freedom from Form
  27. 28. Convert LCD into a big flat camera ? Beyond Multi-touch: 3D Gestures
  28. 29. Large Virtual Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen
  29. 30. Touch + Hover using Thin, Depth Sensing LCD Sensor
  30. 31. Sensing Depth from Array of Virtual Cameras in LCD
  31. 32. Shallow DoF with Simple Lens Lots of glass; Heavy; Bulky; Expensive
  32. 33. Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh Raskar MIT Media Lab MIT Media Lab Camera Culture
  33. 34. Image De stabilization Lens Sensor Camera Static Scene
  34. 35. Image Destabilization Static Scene Lens Motion Sensor Motion Camera Mohan, Lanman,Hiura, Raskar ICCP 2009
  35. 36. Shifting Pinhole and Sensor A B A’ B’ Pinhole v p Sensor v s d a d b d s Focus Here
  36. 37. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here
  37. 38. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here
  38. 39. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here
  39. 40. “ Time Lens” Ratio of speeds Lens Equation: Virtual Focal Length: Virtual F-Number:
  40. 41. Time Lens:
  41. 42. Small Aperture Photo all-in-focus
  42. 43. Destabilized Small Aperture focused in the front using destabilization
  43. 44. Adjusting the Focus Plane focused in the middle using destabilization
  44. 45. Adjusting the Focus Plane focused in the back using destabilization
  45. 46. Camera Culture Ramesh Raskar Wish #3 Understand the World
  46. 47. Real or Fake ?
  47. 48. Direct Reflection
  48. 49. Fake Real Internal Reflection
  49. 50. High Frequency Pattern surface Camera Flash Nayar, Krishnan, Grossberg, Raskar [Siggraph 2006] i
  50. 51. Convert single 2D photo into 3D ? Snavely, Seitz, Szeliski U of Washington/Microsoft: Photosynth
  51. 52. Exploit Community Photo Collections U of Washington/Microsoft: Photosynth
  52. 53. Can you look around a corner ?
  53. 54. Can you ‘see’ around a corner ?
  54. 55. Femto-Photography: Higher Dimensional Capture FemtoFlash UltraFast Detector Computational Optics Serious Sync
  55. 57. Bokode
  56. 59. Defocus blur of Bokode
  57. 60. Coding in Angle Mohan, Woo, Smithwick, Hiura, Raskar [Siggraph 2009]
  58. 61. <ul><li>circle of confusion  circle of information </li></ul>camera Bokode (angle) Quote suggested by Kurt Akeley Encoding in Angle , not space, time or wavelength sensor
  59. 62. <ul><li>Product labels </li></ul>Street-view Tagging
  60. 63. capturing Bokodes cell-phone camera close to the Bokode (10,000+ bytes of data)
  61. 64. Camera Culture Ramesh Raskar <ul><li>Wish #3 </li></ul><ul><li>Understand the World </li></ul><ul><li>Identify/recognize Materials </li></ul><ul><li>3D Awareness </li></ul><ul><li>Interact with information </li></ul>
  62. 65. Camera Culture Ramesh Raskar Wish #4 Sharing Visual Experience
  63. 66. Camera Culture Ramesh Raskar <ul><li>Wish #4 </li></ul><ul><li>Sharing Visual Experience </li></ul><ul><li>LifeLog Auto-summary </li></ul><ul><li>Privacy in public and authentication </li></ul><ul><li>Great Photo-frames </li></ul>
  64. 67. 6D Photo Frames One Pixel of a 6D Display = 4D Display Single Pixel of 6D Frame 1 2 1 1 2D 2D 2D
  65. 68. 6D Photo Frames: Respond to Viewpoint + Ambient Light
  66. 69. Camera Culture Ramesh Raskar <ul><li>Wish #4 </li></ul><ul><li>Sharing Visual Experience </li></ul><ul><li>LifeLog Auto-summary </li></ul><ul><li>Privacy in public and authentication </li></ul><ul><li>Hyper-real Photo Frames </li></ul><ul><li>Print ‘material’ </li></ul>
  67. 70. Camera Culture Ramesh Raskar Wish #5 Capturing Essence
  68. 71. Essence Photography <ul><li>Beyond physical realism </li></ul><ul><li>New Visual Art Forms </li></ul>Yu, McMillan
  69. 72. What are the problems with ‘real’ photo in conveying information ? Why do we hire artists to draw what can be photographed ?
  70. 73. Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors
  71. 74. Depth Edges with MultiFlash Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004
  72. 79. Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes
  73. 80. Depth Edges
  74. 81. Our Method Canny
  75. 82. Canny Intensity Edge Detection Our Method Photo Result
  76. 86. Blind Camera Sascha Pohflepp, U of the Art, Berlin, 2006
  77. 87. Scene Completion Using Millions of Photographs Hays and Efros, Siggraph 2007
  78. 88. Camera Culture Ramesh Raskar Wish #5 …………………
  79. 89. Photos of tomorrow: computed not recorded http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/
  80. 90. Questions <ul><li>What will a camera look like in 10,20 years? </li></ul><ul><li>How will a billion networked and portable cameras change the social culture? </li></ul><ul><li>How will online photo collections transform visual social computing? </li></ul><ul><li>How will movie making/new reporting change? </li></ul><ul><li>computational-journalism.com </li></ul>
  81. 91. 2 nd International Conference on Computational Photography Papers due November 2, 2009 http://cameraculture.media.mit.edu/iccp10
  82. 92. <ul><li>Ramesh Raskar and Jack Tumblin </li></ul><ul><li>Book Publishers: A K Peters </li></ul><ul><li>ComputationalPhotography.org </li></ul>
  83. 93. <ul><li>Post-capture control </li></ul><ul><ul><li>Emulate studio lights with compact flash </li></ul></ul><ul><ul><li>Focus and motion blur </li></ul></ul><ul><li>New forms </li></ul><ul><ul><li>Flat camera, large LCDs as cameras </li></ul></ul><ul><ul><li>Image destabilization for larger aperture </li></ul></ul><ul><li>Understand the world </li></ul><ul><ul><li>Real or fake </li></ul></ul><ul><ul><li>Place 2D photo into 3D </li></ul></ul><ul><ul><li>Look around corner </li></ul></ul><ul><ul><li>Bokode: long distance barcode </li></ul></ul><ul><li>Sharing </li></ul><ul><ul><li>Lifelogs auto summary </li></ul></ul><ul><ul><li>Privacy/Verification </li></ul></ul><ul><ul><li>6D photoframes </li></ul></ul><ul><li>Essence </li></ul><ul><ul><li>New visual arts </li></ul></ul><ul><ul><li>Multi-flash camera </li></ul></ul><ul><ul><li>Delta-camera and Blind-camera </li></ul></ul>Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info Computational Photography Wish List Sensor

Editor's Notes

  • http://raskar.info http://cameraculture.info Beyond making it faster and cheaper, one can argue that digital photography has barely changed how we capture and share visual experiences.
  • Kodak DCS400 in Nikon F3 body in early 90’s Commendable first 1.3MP digital but film cartridge still there! (First one in 1991 but even in 1995 the space for cartridge) Quote from Jack Tumblin Digital photography is like a caged lion that is uncaged in a jungle after years .. The lion stays in place rather than rushing out to explore Billion cameras but they all look like human eye KODAK Professional Digital Camera DCS-100: a camera back and camera winder fitted to an unmodified Nikon F3 camera
  • CPUs and computers don’t mimic the human brain. And robots don’t mimic human activities. Should the hardware for visual computing which is cameras and capture devices, mimic the human eye? Even if we decide to use a successful biological vision system as basis, we have a range of choices. For single chambered to compounds eyes, shadow-based to refractive to reflective optics. So the goal of my group at Media Lab is to explore new designs and develop software algorithms that exploit these designs.
  • currently we solve the human visual perception problem by simply reproducing what the eye would see. (even for 3D, we show stereo pair) But this makes it difficult to understand or manipulate for computers. (machine readable rep)
  • Wishlist by consumers and companies today .. i.e. what is NOT available today but they wish it was So, I am not including Wifi, GPS, face detection etc in the list here. But let us dream beyond this list.
  • 4 blocks : light, optics, sensors, processing, (display: light sensitive display) + 5 th element: Network
  • Lets dream big .. Can we look around at something beyond the line of sight?
  • Can photos become emotive abstract renderings ?
  • Inference and perception are important. Intent and goal of the photo is important. Computational Photography pioneered by Nayar, Levoy, Debevec, Microsoft Research et al in late-90’s. The same way camera put photorealistic art out of business, maybe this new artform will put the traditional camera out of business. Because we wont really care about a photo, merely a recording of light but a form that captures meaningful subset of the visual experience. Multiperspective photos. Photosynth is an example.
  • My own wishlist .. By defn these problems are not solved .. But I will try to give you a flavor of things my group and others are doing in that direction
  • See computationalphotography.org
  • Simplify capture time decisions Fix everything in post if necessary
  • Stanford Plenoptic Camera
  • But you lose a lot of resolution, 16M reduced to 300x300 pixels
  • We can now create lightfield cameras by mechanism that don’t introduce new lenses. http://raskar.info/Mask/
  • Photo with Mask
  • Sub aperture views From these 121 views we can create stereo movies, compute depth, achieve digital refocusing etc.
  • The mask based lightfield camera is more suitable for post-capture control in many ways.
  • Wish: low light photography to deal with motion blur
  • License plate example: Blur = 60 pixels Can you guess what the car make is ?
  • Rudy Burger, ‘don’t use flash and destroy the image’ Can we use flash not just for improving scene brightness but for enhancing the mood? Like in studio lights? Main difference between professionals and consumers is lighting.
  • Debevec et al have shown for special effects terrific relighting strategies. Wish: do the same with a compact light source and exploit natural lighting.
  • Mechanical size and restrictions. Can we create a flat camera? But still capture enough light ? Current strategy is to shrink the camera in all dimensions: making it flat also means a tiny lens. Origami lens (Montage program, Joe Ford UCSD, ) Bidi Screen Image Destabilization New exciting projects at Stanford (Marc Levoy’s Open Source Camera) Shree Nayar’s BigShot ‘lego-camera’
  • How to exploit Sharp’s photosensing LCD originally designed for touch sensing and convert into a large area flat camera Since we are adapting LCD technology we can fit a BiDi screen into laptops and mobile devices.
  • So here is a preview of our quantitative results. I’ll explain this in more detail later on, but you can see we’re able to accurately distinguish the depth of a set of resolution targets. We show above a portion of portion of views form our virtual cameras, a synthetically refocused image, and the depth map derived from it.
  • Many are working on extending depth of field. But consumers pay quite a bit to actually reduce the depth of field. Can we support both options?
  • = Material index and compute bounces (real vs fake) = Automatic 3D, phototourism, and 3D awareness (look around a corner) = Find relationship (network) between all photos = Understand the world (recognize, categorize, make world smarter bokode)
  • Nayar, Krishnan, Grossberg, Raskar [Siggraph 2006]
  • Nayar, Krishnan, Grossberg, Raskar [Siggraph 2006]
  • Bokode.com
  • = Material index and compute bounces (real vs fake) = Automatic 3D, phototourism, and 3D awareness (look around a corner) = Find relationship (network) between all photos = Understand the world (recognize, categorize, make world smarter bokode)
  • Life Sharing = Automatic lifelogs and summaries (capture and render from other viewpoints, possibly retinal implants) = Privacy in public (smart probes, good recognition) and authentication = Ultimate photoframes, Print 3D and relightable photos (6D), print any material
  • Life Sharing = Automatic lifelogs and summaries (capture and render from other viewpoints, possibly retinal implants) = Privacy in public (smart probes, good recognition) and authentication = Ultimate photoframes, Print 3D and relightable photos (6D), print any material
  • Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008
  • This video is only for 4D display that responds to light Bonny’s lenticular prints outside
  • Life Sharing = Automatic lifelogs and summaries (capture and render from other viewpoints, possibly retinal implants) = Privacy in public (smart probes, good recognition) and authentication = Ultimate photoframes, Print 3D and relightable photos (6D), print any material
  • A great artifact in musuem which you can hold and observe Or that ride on a roller coaster Synthesize a new experience = Photoediting using learned models of earlier stroke activities, image filling etc = Artistic effects and NPR (like MS Word, but artistic ability to express) = Blind camera
  • Inference and perception are important. Intent and goal of the photo is important. The same way camera put photorealistic art out of business, maybe this new artform will put the traditional camera out of business. Because we wont really care about a photo, merely a recording of light but a form that captures meaningful subset of the visual experience. Multiperspective photos. Photosynth is an example.
  • Alexis Gerard threatens us that he has this camera which is so advanced, you don’t even need it .. Maybe all the consumer photographer wants is a black box with big red button. No optics, sensors or flash. If I am standing the middle of times square and I need to take a photo. Do I really need a fancy camera?
  • The camera can trawl on flickr and retrieve a photo that is roughly taken at the same position, at the same time of day. Maybe all the consumer wants is a blind camera.
  • Your wish here .. New Digital Imaging Workflow? Emerging opportunities with smarter cameras? Share over lunch, beer and cocktails
  • photos will be computed rather than recorded Comp photo will be there It will change the workflow, just with digital many pipeline have turned upside down and we will even more At the same time with cameras that understand our world better, there will be a lot of new opportunities http://scalarmotion.wordpress.com/2009/03/15/propeller-image-aliasing/
  • http:// ComputationalPhotography.org
  • http://cameraculture.info http://raskar.info
  • ×