Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Raskar 6Sight Keynote Talk Nov09


Published on

Though revolutionary in many ways, digital photography is essentially electronically implemented film photography. By contrast, computational photography exploits plentiful low-cost computing and memory, new kinds of digitally enabled sensors, optics, probes, smart lighting, and communication to capture information far beyond just a simple set of pixels. It promises a richer, even a multilayered, visual experience that may include depth, fused photo-video representations, or multispectral imagery. Professor Raskar will discuss and demonstrate advances he is working on in the areas of generalized optics, sensors, illumination methods, processing, and display, and describe how computational photography will enable us to create images that break from traditional constraints to retain more fully our fondest and most important memories, to keep personalized records of our lives, and to extend both the archival and the artistic possibilities of photography.

Published in: Technology, Art & Photos, Business

Raskar 6Sight Keynote Talk Nov09

  1. 1. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Photography Wish List
  2. 2. Film-like Digital Photography
  3. 3. Fernald, Science [Sept 2006] Shadow Refractive Reflective
  4. 4. ‘ film-like’ Photography Lens Detector Pixels Image Slide by Shree Nayar Reproduce for the eye
  5. 5. Wish List Today <ul><li>Consumers </li></ul><ul><ul><li>Super-human vision </li></ul></ul><ul><ul><li>Microscope like resolution </li></ul></ul><ul><ul><li>High speed (burst, video, no blur) </li></ul></ul><ul><ul><li>See inside the body (health) </li></ul></ul><ul><ul><li>Auto-trigger </li></ul></ul><ul><ul><li>Battery life </li></ul></ul><ul><ul><li>Zero start or shutter delay </li></ul></ul><ul><ul><li>Keep only ‘good’ pics </li></ul></ul><ul><ul><li>Find ‘relevant’ pics and better archiving/access </li></ul></ul><ul><ul><li>Put photographer back in photo! </li></ul></ul><ul><li>Companies </li></ul><ul><ul><li>Cost </li></ul></ul><ul><ul><li>Resolution </li></ul></ul><ul><ul><li>Low-light sensitivity, HDR </li></ul></ul><ul><ul><li>( For cameraphone ) Size and depth </li></ul></ul><ul><ul><li>Stereo and 3D </li></ul></ul><ul><ul><li>Mecha-free zoom/focus </li></ul></ul><ul><ul><li>Auto-tagging for sharing </li></ul></ul><ul><ul><li>Recognition </li></ul></ul>
  6. 6. Computational Photography Computational Illumination Computational Camera Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing Ray Bender Ray Sampler Ray Reconstruction Generalized Optics Recreate 4D Lightfield Light Sources Modulators 4D Incident Lighting 4D Light Field
  7. 7. Can you look around a corner ?
  8. 9. Computational Photography [Raskar and Tumblin] <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level vision: Pixels </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>‘ Scene analysis’ </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>High-level understanding </li></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>captures a machine-readable representation of our world to hyper-realistically synthesize the essence of our visual experience.
  9. 10. Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab Computational Photography Wish List
  10. 11. Synthesis Low Level Mid Level High Level Hyper realism Raw Angle, spectrum aware Non-visual Data, GPS Metadata Priors Comprehensive 8D reflectance field Computational Photography Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism Resolution
  11. 12. Camera Culture Ramesh Raskar Wish #1 Ultimate Post-capture Control
  12. 13. Digital Refocusing using Light Field Camera 125 μ square-sided microlenses [Ng et al 2005]
  13. 14. Zooming into the raw photo
  14. 15. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor
  15. 16. Captured 2D Photo
  16. 17. x y u v 121 Sub-aperture Views
  17. 18. <ul><li>Samples individual rays </li></ul><ul><li>Predefined spectrum for lenses </li></ul><ul><li>Chromatic abberration </li></ul><ul><li>High alignment precision </li></ul><ul><li>For each lenslet, peripheral pixels are wasted </li></ul><ul><li>Negligible Light Loss </li></ul><ul><li>Samples coded combination of rays </li></ul><ul><li>Supports any wavelength </li></ul><ul><li>Reconfigurable f/# , Easier alignment </li></ul><ul><li>No wastage </li></ul><ul><li>High resolution image for parts of scene in focus </li></ul><ul><li>50 % Light Loss due to mask </li></ul>Microlens array Sensor Plenoptic Camera Heterodyne Camera Mask Sensor
  18. 19. Figure 2 results Input Image Motion Blur in Low Light
  19. 20. Motion Blur in Low Light
  20. 21. Fluttered Shutter Camera Raskar, Agrawal, Tumblin Siggraph2006 Ferroelectric shutter in front of the lens is turned opaque or transparent in a rapid binary sequence
  21. 22. Image Deblurred by solving a linear system. No post-processing Blurred Taxi
  22. 23. Motion Blur in Low Light
  23. 24. Compact Programmable Lights ?
  24. 25. Image-Based Re-lighting Measure incoming light in Milan, Light the actress in LA Matte the background Matched LA and Milan lighting. Debevec et al., SIGG2001
  25. 26. Camera Culture Ramesh Raskar <ul><li>Wish #1 </li></ul><ul><li>Ultimate Post-capture Control </li></ul><ul><ul><li>Digital Refocus and Motion blur </li></ul></ul><ul><ul><li>Emulate studio light from compact flash </li></ul></ul>
  26. 27. Camera Culture Ramesh Raskar Wish #2 Freedom from Form
  27. 28. Convert LCD into a big flat camera ? Beyond Multi-touch: 3D Gestures
  28. 29. Large Virtual Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen
  29. 30. Touch + Hover using Thin, Depth Sensing LCD Sensor
  30. 31. Sensing Depth from Array of Virtual Cameras in LCD
  31. 32. Shallow DoF with Simple Lens Lots of glass; Heavy; Bulky; Expensive
  32. 33. Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh Raskar MIT Media Lab MIT Media Lab Camera Culture
  33. 34. Image De stabilization Lens Sensor Camera Static Scene
  34. 35. Image Destabilization Static Scene Lens Motion Sensor Motion Camera Mohan, Lanman,Hiura, Raskar ICCP 2009
  35. 36. Shifting Pinhole and Sensor A B A’ B’ Pinhole v p Sensor v s d a d b d s Focus Here
  36. 37. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here
  37. 38. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here
  38. 39. Shifting Pinhole and Sensor A B Pinhole A’ B’ v p Sensor v s d a d b d s Focus Here
  39. 40. “ Time Lens” Ratio of speeds Lens Equation: Virtual Focal Length: Virtual F-Number:
  40. 41. Time Lens:
  41. 42. Small Aperture Photo all-in-focus
  42. 43. Destabilized Small Aperture focused in the front using destabilization
  43. 44. Adjusting the Focus Plane focused in the middle using destabilization
  44. 45. Adjusting the Focus Plane focused in the back using destabilization
  45. 46. Camera Culture Ramesh Raskar Wish #3 Understand the World
  46. 47. Real or Fake ?
  47. 48. Direct Reflection
  48. 49. Fake Real Internal Reflection
  49. 50. High Frequency Pattern surface Camera Flash Nayar, Krishnan, Grossberg, Raskar [Siggraph 2006] i
  50. 51. Convert single 2D photo into 3D ? Snavely, Seitz, Szeliski U of Washington/Microsoft: Photosynth
  51. 52. Exploit Community Photo Collections U of Washington/Microsoft: Photosynth
  52. 53. Can you look around a corner ?
  53. 54. Can you ‘see’ around a corner ?
  54. 55. Femto-Photography: Higher Dimensional Capture FemtoFlash UltraFast Detector Computational Optics Serious Sync
  55. 57. Bokode
  56. 59. Defocus blur of Bokode
  57. 60. Coding in Angle Mohan, Woo, Smithwick, Hiura, Raskar [Siggraph 2009]
  58. 61. <ul><li>circle of confusion  circle of information </li></ul>camera Bokode (angle) Quote suggested by Kurt Akeley Encoding in Angle , not space, time or wavelength sensor
  59. 62. <ul><li>Product labels </li></ul>Street-view Tagging
  60. 63. capturing Bokodes cell-phone camera close to the Bokode (10,000+ bytes of data)
  61. 64. Camera Culture Ramesh Raskar <ul><li>Wish #3 </li></ul><ul><li>Understand the World </li></ul><ul><li>Identify/recognize Materials </li></ul><ul><li>3D Awareness </li></ul><ul><li>Interact with information </li></ul>
  62. 65. Camera Culture Ramesh Raskar Wish #4 Sharing Visual Experience
  63. 66. Camera Culture Ramesh Raskar <ul><li>Wish #4 </li></ul><ul><li>Sharing Visual Experience </li></ul><ul><li>LifeLog Auto-summary </li></ul><ul><li>Privacy in public and authentication </li></ul><ul><li>Great Photo-frames </li></ul>
  64. 67. 6D Photo Frames One Pixel of a 6D Display = 4D Display Single Pixel of 6D Frame 1 2 1 1 2D 2D 2D
  65. 68. 6D Photo Frames: Respond to Viewpoint + Ambient Light
  66. 69. Camera Culture Ramesh Raskar <ul><li>Wish #4 </li></ul><ul><li>Sharing Visual Experience </li></ul><ul><li>LifeLog Auto-summary </li></ul><ul><li>Privacy in public and authentication </li></ul><ul><li>Hyper-real Photo Frames </li></ul><ul><li>Print ‘material’ </li></ul>
  67. 70. Camera Culture Ramesh Raskar Wish #5 Capturing Essence
  68. 71. Essence Photography <ul><li>Beyond physical realism </li></ul><ul><li>New Visual Art Forms </li></ul>Yu, McMillan
  69. 72. What are the problems with ‘real’ photo in conveying information ? Why do we hire artists to draw what can be photographed ?
  70. 73. Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors
  71. 74. Depth Edges with MultiFlash Raskar, Tan, Feris, Jingyi Yu, Turk – ACM SIGGRAPH 2004
  72. 79. Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes
  73. 80. Depth Edges
  74. 81. Our Method Canny
  75. 82. Canny Intensity Edge Detection Our Method Photo Result
  76. 86. Blind Camera Sascha Pohflepp, U of the Art, Berlin, 2006
  77. 87. Scene Completion Using Millions of Photographs Hays and Efros, Siggraph 2007
  78. 88. Camera Culture Ramesh Raskar Wish #5 …………………
  79. 89. Photos of tomorrow: computed not recorded
  80. 90. Questions <ul><li>What will a camera look like in 10,20 years? </li></ul><ul><li>How will a billion networked and portable cameras change the social culture? </li></ul><ul><li>How will online photo collections transform visual social computing? </li></ul><ul><li>How will movie making/new reporting change? </li></ul><ul><li> </li></ul>
  81. 91. 2 nd International Conference on Computational Photography Papers due November 2, 2009
  82. 92. <ul><li>Ramesh Raskar and Jack Tumblin </li></ul><ul><li>Book Publishers: A K Peters </li></ul><ul><li> </li></ul>
  83. 93. <ul><li>Post-capture control </li></ul><ul><ul><li>Emulate studio lights with compact flash </li></ul></ul><ul><ul><li>Focus and motion blur </li></ul></ul><ul><li>New forms </li></ul><ul><ul><li>Flat camera, large LCDs as cameras </li></ul></ul><ul><ul><li>Image destabilization for larger aperture </li></ul></ul><ul><li>Understand the world </li></ul><ul><ul><li>Real or fake </li></ul></ul><ul><ul><li>Place 2D photo into 3D </li></ul></ul><ul><ul><li>Look around corner </li></ul></ul><ul><ul><li>Bokode: long distance barcode </li></ul></ul><ul><li>Sharing </li></ul><ul><ul><li>Lifelogs auto summary </li></ul></ul><ul><ul><li>Privacy/Verification </li></ul></ul><ul><ul><li>6D photoframes </li></ul></ul><ul><li>Essence </li></ul><ul><ul><li>New visual arts </li></ul></ul><ul><ul><li>Multi-flash camera </li></ul></ul><ul><ul><li>Delta-camera and Blind-camera </li></ul></ul>Camera Culture Group, MIT Media Lab Ramesh Raskar Computational Photography Wish List Sensor