Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Raskar Next Billion Cameras Siggraph 2009

3,491 views

Published on

http://raskar.scripts.mit.edu/~raskar/nextbillioncameras/

Siggraph 2009 Course with Alyosha Efros,
Ramesh Raskar,
Steve Seitz

  • Be the first to comment

Raskar Next Billion Cameras Siggraph 2009

  1. 1. Camera Culture Ramesh Raskar Alyosha Efros Ramesh Raskar Steve Seitz Siggraph 2009 Curated Course Next Billion Cameras http://raskar.scripts.mit.edu / nextbillioncameras /
  2. 2. <ul><li>A. Introduction‐‐5 minutes </li></ul><ul><li>B. Cameras of the future ( Raskar , 30 minutes) * Form factors, Modalities and Interaction * Enabling Visual Social Computing </li></ul><ul><li>C. Reconstruction the World ( Seitz , 30 minutes) * Photo tourism and beyond * Image‐based modeling and rendering on a massive scale * Scene summarization </li></ul><ul><li>D. Understanding a Billion Photos ( Efros , 30 minutes) * What will the photos depict? * Photos as visual content for computer graphics * Solving computer vision </li></ul><ul><li>E. Discussion‐‐10 minutes </li></ul>Next Billion Cameras
  3. 4. Alexei (Alyosha) Efros [CMU] <ul><li>Assistant professor at the Robotics Institute and the Computer Science Department at Carnegie Mellon University . </li></ul><ul><li>His research is in the area of computer vision and computer graphics, especially at the intersection of the two. He is particularly interested in using data-driven techniques to tackle problems which are very hard to model parametrically but where large quantities of data are readily available. Alyosha received his PhD in 2003 from UC Berkeley and spent the following year as a post-doctoral fellow in Oxford, England. Alyosha is a recipient of the NSF CAREER award (2006), the Sloan Fellowship (2008), the Guggenheim Fellowship (2008), and the Okawa Grant (2008). </li></ul><ul><li>http://www.cs.cmu.edu/~efros/ </li></ul>
  4. 5. Ramesh Raskar [MIT] <ul><li>Associate Professor at the MIT Media Lab and heads the Camera Culture research group. </li></ul><ul><li>The group focuses on creating a new class for imaging platforms to better capture and share the visual experience. This research involves developing novel cameras with unusual optical elements, programmable illumination, digital wavelength control, and femtosecond analysis of light transport, as well as tools to decompose pixels into perceptually meaningful components. </li></ul><ul><li>Raskar is a receipient of Alfred P Sloan research fellowship 2009, the TR100 Award 2004, Global Indus Technovator Award 2003. He holds 35 US patents and has received four Mitsubishi Electric Invention Awards. He is currently co-authoring, with Jack Tumblin, a book on computational photography. </li></ul><ul><li>http://www.media.mit.edu/~raskar </li></ul>
  5. 6. Steve Seitz [U-Washington] <ul><li>Professor in the Department of Computer Science and Engineering at the University of Washington. </li></ul><ul><li>He received Ph.D. in computer sciences at the University of Wisconsin, Madison in 1997. He was twice awarded the David Marr Prize for the best paper at the International Conference of Computer Vision, and has received an NSF Career Award, an ONR Young Investigator Award, and an Alfred P. Sloan Fellowship.  His work on Photo Tourism (joint with Noah Snavely and Rick Szeliski) formed the basis of Microsoft's Photosynth technology.  Professor Seitz is interested in problems in computer vision and computer graphics. His current research focuses on capturing the structure, appearance, and behavior of the real world from digital imagery. </li></ul><ul><li>http://www.cs.washington.edu/homes/seitz/ </li></ul>
  6. 7. Where are the ‘camera’s?
  7. 8. Where are the ‘camera’s?
  8. 10. Camera Culture Ramesh Raskar Alyosha Efros Ramesh Raskar Steve Seitz Siggraph 2009 Course Next Billion Cameras http://raskar.info/photo/
  9. 11. Camera Culture Ramesh Raskar Alyosha Efros Ramesh Raskar Steve Seitz Siggraph 2009 Course Next 100 Billion Cameras http://raskar.info/photo/
  10. 12. Key Message <ul><li>Cameras will not look like anything today </li></ul><ul><ul><li>Emerging optics, illumination, novel sensors </li></ul></ul><ul><li>Visual Experience will differ from viewfinder </li></ul><ul><ul><li>Photos will be ‘computed’ </li></ul></ul><ul><ul><li>Remarkable post-capture control </li></ul></ul><ul><ul><li>Crowdsource the photo collection </li></ul></ul><ul><ul><li>Exploit priors and online collections </li></ul></ul><ul><li>Visual Essence will dominate </li></ul><ul><ul><li>Superior Metadata tagging for effective sharing </li></ul></ul><ul><ul><li>Fusion with non-visual data </li></ul></ul>
  11. 13. Can you look around a corner ?
  12. 14. Can you decode a 5 micron feature from 3 meters away with an ordinary camera ?
  13. 15. Convert LCD into a big flat camera? Beyond Multi-touch
  14. 16. Pantheon
  15. 17. How do we move through a space?
  16. 18. What is ‘interesting’ here?
  17. 19. Record what you ‘feel’ not what you ‘see’
  18. 22. Camera Culture Ramesh Raskar Ramesh Raskar Camera Culture http://raskar.scripts.mit.edu / nextbillioncameras /
  19. 23. “ Visual Social Computing” <ul><li>Social Computing (SoCo) </li></ul><ul><ul><li>Computing </li></ul></ul><ul><ul><li>by the people, </li></ul></ul><ul><ul><li>for the people, </li></ul></ul><ul><ul><li>of the people </li></ul></ul><ul><li>Visual SoCo </li></ul><ul><ul><li>Participatory, Collaborative </li></ul></ul><ul><ul><li>Visual semantics </li></ul></ul><ul><ul><li>http://raskar.scripts.mit.edu / nextbillioncameras </li></ul></ul>?
  20. 24. Crowdsourcing http://www.wired.com/wired/archive/14.06/crowds.html Object Recognition Fakes Template matching Amazon Mechanical Turk: Steve Fossett search ReCAPTCHA=OCR
  21. 25. Participatory Urban Sensing <ul><li>Deborah Estrin et al </li></ul><ul><li>Static/semi-dynamic/dynamic data </li></ul><ul><li>A. City Maintenance </li></ul><ul><ul><li>Side Walks </li></ul></ul><ul><li>B. Pollution </li></ul><ul><li>-Sensor network </li></ul><ul><li>C. Diet, Offenders </li></ul><ul><ul><li>Graffiti </li></ul></ul><ul><ul><li>Bicycle on sidewalk </li></ul></ul><ul><li>Future .. </li></ul><ul><li>Citizen Surveillance Health Monitoring </li></ul>http://research.cens.ucla.edu/areas/2007/Urban_Sensing/ (Erin Brockovich) n
  22. 26. Community Photo Collections U of Washington/Microsoft: Photosynth
  23. 27. Beyond Visible Spectrum Cedip RedShift
  24. 28. Trust in Images From Hany Farid
  25. 29. Trust in Images From Hany Farid LA Times March’03
  26. 30. Cameras in Developing Countries http://news.bbc.co.uk/2/hi/south_asia/7147796.stm Community news program run by village women
  27. 31. Vision thru tongue http://www.pbs.org/kcet/wiredscience/story/97-mixed_feelings.html Solutions for the Visually Challenged http://www.seeingwithsound.com/
  28. 32. New Topics in Imaging Research <ul><li>Imaging Devices, Modern Optics and Lenses </li></ul><ul><li>Emerging Sensor Technologies </li></ul><ul><li>Mobile Photography </li></ul><ul><li>Visual Social Computing and Citizen Journalism </li></ul><ul><li>Imaging Beyond Visible Spectrum </li></ul><ul><li>Computational Imaging in Sciences (Medical) </li></ul><ul><li>Trust in Visual Media </li></ul><ul><li>Solutions for Visually Challenged </li></ul><ul><li>Cameras in Developing Countries </li></ul><ul><ul><li>Social Stability, Commerce and Governance </li></ul></ul><ul><li>Future Products and Business Models </li></ul>
  29. 33. Traditional Photography Lens Detector Pixels Image Mimics Human Eye for a Single Snapshot : Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy: Shree Nayar
  30. 34. Computational Photography Computational Illumination Computational Camera Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing 4D Ray Bender Upto 4D Ray Sampler Ray Reconstruction Generalized Optics Recreate 4D Lightfield Light Sources Modulators 4D Incident Lighting 4D Light Field
  31. 35. Computational Photography [Raskar and Tumblin] <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level vision: Pixels </li></ul></ul><ul><ul><li>Multi-photos by perturbing camera parameters </li></ul></ul><ul><ul><li>HDR, panorama, … </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><ul><li>Reversible encoding of data </li></ul></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><ul><li>‘ Scene analysis’ </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>High-level understanding </li></ul></ul><ul><ul><ul><li>Not mimic human eye </li></ul></ul></ul><ul><ul><ul><li>Beyond single view/illum </li></ul></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>captures a machine-readable representation of our world to hyper-realistically synthesize the essence of our visual experience.
  32. 36. Goal and Experience Low Level Mid Level High Level Hyper realism Raw Angle, spectrum aware Non-visual Data, GPS Metadata Priors Comprehensive 8D reflectance field Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism
  33. 37. 2 nd International Conference on Computational Photography Papers due November 2, 2009 http://cameraculture.media.mit.edu/iccp10
  34. 38. <ul><li>Ramesh Raskar and Jack Tumblin </li></ul><ul><li>Book Publishers: A K Peters </li></ul><ul><li>Siggraph 2009 booth: 20% off </li></ul><ul><li>Booth #2527 </li></ul><ul><li>ComputationalPhotography.org </li></ul><ul><li>Meet the Authors </li></ul><ul><li>Thursday at 2pm-2:30pm </li></ul>
  35. 39. Computational Photography [Raskar and Tumblin] <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level vision: Pixels </li></ul></ul><ul><ul><li>Multi-photos by perturbing camera parameters </li></ul></ul><ul><ul><li>HDR, panorama, … </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><li>Reversible encoding of data </li></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><ul><li>‘ Scene analysis’ : (Consumer software?) </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>Beyond single view/illum </li></ul></ul><ul><ul><li>Not mimic human eye </li></ul></ul><ul><ul><li>‘ New art form’ </li></ul></ul>
  36. 40. Epsilon Photography <ul><li>Dynamic range </li></ul><ul><ul><li>Exposure bracketing [Mann-Picard, Debevec] </li></ul></ul><ul><li>Wider FoV </li></ul><ul><ul><li>Stitching a panorama </li></ul></ul><ul><li>Depth of field </li></ul><ul><ul><li>Fusion of photos with limited DoF [Agrawala04] </li></ul></ul><ul><li>Noise </li></ul><ul><ul><li>Flash/no-flash image pairs </li></ul></ul><ul><li>Frame rate </li></ul><ul><ul><li>Triggering multiple cameras [Wilburn04] </li></ul></ul>
  37. 41. Dynamic Range Goal: High Dynamic Range Short Exposure Long Exposure
  38. 42. Epsilon Photography <ul><li>Dynamic range </li></ul><ul><ul><li>Exposure braketing [Mann-Picard, Debevec] </li></ul></ul><ul><li>Wider FoV </li></ul><ul><ul><li>Stitching a panorama </li></ul></ul><ul><li>Depth of field </li></ul><ul><ul><li>Fusion of photos with limited DoF [Agrawala04] </li></ul></ul><ul><li>Noise </li></ul><ul><ul><li>Flash/no-flash image pairs [ Petschnigg04, Eisemann04] </li></ul></ul><ul><li>Frame rate </li></ul><ul><ul><li>Triggering multiple cameras [Wilburn05, Shechtman02] </li></ul></ul>
  39. 43. Computational Photography <ul><li>Epsilon Photography </li></ul><ul><ul><li>Low-level Vision: Pixels </li></ul></ul><ul><ul><li>Multiphotos by perturbing camera parameters </li></ul></ul><ul><ul><li>HDR, panorama </li></ul></ul><ul><ul><li>‘ Ultimate camera’ </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Mid-Level Cues: </li></ul></ul><ul><ul><ul><li>Regions, Edges, Motion, Direct/global </li></ul></ul></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><ul><li>Reversible encoding of data </li></ul></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><ul><li>‘ Scene analysis’ </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>Not mimic human eye </li></ul></ul><ul><ul><li>Beyond single view/illum </li></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>
  40. 44. <ul><li>3D </li></ul><ul><ul><li>Stereo of multiple cameras </li></ul></ul><ul><li>Higher dimensional LF </li></ul><ul><ul><li>Light Field Capture </li></ul></ul><ul><ul><ul><li>lenslet array [Adelson92, Ng05] , ‘3D lens’ [Georgiev05] , heterodyne masks [Veeraraghavan07] </li></ul></ul></ul><ul><li>Boundaries and Regions </li></ul><ul><ul><li>Multi-flash camera with shadows [Raskar08] </li></ul></ul><ul><ul><li>Fg/bg matting [Chuang01,Sun06] </li></ul></ul><ul><li>Deblurring </li></ul><ul><ul><li>Engineered PSF </li></ul></ul><ul><ul><li>Motion: Flutter shutter [Raskar06] , Camera Motion [Levin08] </li></ul></ul><ul><ul><li>Defocus: Coded aperture [Veeraraghavan07,Levin07] , Wavefront coding [Cathey95] </li></ul></ul><ul><li>Global vs direct illumination </li></ul><ul><ul><li>High frequency illumination [Nayar06] </li></ul></ul><ul><ul><li>Glare decomposition [Talvala07, Raskar08] </li></ul></ul><ul><li>Coded Sensor </li></ul><ul><ul><li>Gradient camera [Tumblin05] </li></ul></ul>
  41. 45. Digital Refocusing using Light Field Camera 125 μ square-sided microlenses [Ng et al 2005]
  42. 46. <ul><li>3D </li></ul><ul><ul><li>Stereo of multiple cameras </li></ul></ul><ul><li>Higher dimensional LF </li></ul><ul><ul><li>Light Field Capture </li></ul></ul><ul><ul><ul><li>lenslet array [Adelson92, Ng05] , ‘3D lens’ [Georgiev05] , heterodyne masks [Veeraraghavan07] </li></ul></ul></ul><ul><li>Boundaries and Regions </li></ul><ul><ul><li>Multi-flash camera with shadows [Raskar08] </li></ul></ul><ul><ul><li>Fg/bg matting [Chuang01,Sun06] </li></ul></ul><ul><li>Deblurring </li></ul><ul><ul><li>Engineered PSF </li></ul></ul><ul><ul><li>Motion: Flutter shutter [Raskar06] , Camera Motion [Levin08] </li></ul></ul><ul><ul><li>Defocus: Coded aperture [Veeraraghavan07,Levin07] , Wavefront coding [Cathey95] </li></ul></ul><ul><li>Global vs direct illumination </li></ul><ul><ul><li>High frequency illumination [Nayar06] </li></ul></ul><ul><ul><li>Glare decomposition [Talvala07, Raskar08] </li></ul></ul><ul><li>Coded Sensor </li></ul><ul><ul><li>Gradient camera [Tumblin05] </li></ul></ul>
  43. 47. Multi-flash Camera for Detecting Depth Edges
  44. 48. Depth Edges Left Top Right Bottom Depth Edges Canny Edges
  45. 49. <ul><li>3D </li></ul><ul><ul><li>Stereo of multiple cameras </li></ul></ul><ul><li>Higher dimensional LF </li></ul><ul><ul><li>Light Field Capture </li></ul></ul><ul><ul><ul><li>lenslet array [Adelson92, Ng05] , ‘3D lens’ [Georgiev05] , heterodyne masks [Veeraraghavan07] </li></ul></ul></ul><ul><li>Boundaries and Regions </li></ul><ul><ul><li>Multi-flash camera with shadows [Raskar08] </li></ul></ul><ul><ul><li>Fg/bg matting [Chuang01,Sun06] </li></ul></ul><ul><li>Deblurring </li></ul><ul><ul><li>Engineered PSF </li></ul></ul><ul><ul><li>Motion: Flutter shutter [Raskar06] , Camera Motion [Levin08] </li></ul></ul><ul><ul><li>Defocus: Coded aperture [Veeraraghavan07,Levin07] , Wavefront coding [Cathey95] </li></ul></ul><ul><li>Global vs direct illumination </li></ul><ul><ul><li>High frequency illumination [Nayar06] </li></ul></ul><ul><ul><li>Glare decomposition [Talvala07, Raskar08] </li></ul></ul><ul><li>Coded Sensor </li></ul><ul><ul><li>Gradient camera [Tumblin05] </li></ul></ul>
  46. 50. Flutter Shutter Camera Raskar, Agrawal, Tumblin [Siggraph2006] LCD opacity switched in coded sequence
  47. 51. Traditional Coded Exposure Image of Static Object Deblurred Image Deblurred Image
  48. 52. <ul><li>3D </li></ul><ul><ul><li>Stereo of multiple cameras </li></ul></ul><ul><li>Higher dimensional LF </li></ul><ul><ul><li>Light Field Capture </li></ul></ul><ul><ul><ul><li>lenslet array [Adelson92, Ng05] , ‘3D lens’ [Georgiev05] , heterodyne masks [Veeraraghavan07] </li></ul></ul></ul><ul><li>Boundaries and Regions </li></ul><ul><ul><li>Multi-flash camera with shadows [Raskar08] </li></ul></ul><ul><ul><li>Fg/bg matting [Chuang01,Sun06] </li></ul></ul><ul><li>Deblurring </li></ul><ul><ul><li>Engineered PSF </li></ul></ul><ul><ul><li>Motion: Flutter shutter [Raskar06] , Camera Motion [Levin08] </li></ul></ul><ul><ul><li>Defocus: Coded aperture [Veeraraghavan07,Levin07] , Wavefront coding [Cathey95] </li></ul></ul><ul><li>Decomposition Problems </li></ul><ul><ul><li>High frequency illumination, Global/direct illumination [Nayar06] </li></ul></ul><ul><ul><li>Glare decomposition [Talvala07, Raskar08] </li></ul></ul><ul><li>Coded Sensor </li></ul><ul><ul><li>Gradient camera [Tumblin05] </li></ul></ul>
  49. 53. &quot;Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination,&quot; S.K. Nayar, G. Krishnan, M. D. Grossberg, R. Raskar, ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH), Jul, 2006.
  50. 54. Computational Photography [Raskar and Tumblin] <ul><li>Epsilon Photography </li></ul><ul><ul><li>Multiphotos by varying camera parameters </li></ul></ul><ul><ul><li>HDR, panorama </li></ul></ul><ul><ul><li>‘ Ultimate camera’ : (Photo-editor) </li></ul></ul><ul><li>Coded Photography </li></ul><ul><ul><li>Single/few snapshot </li></ul></ul><ul><ul><li>Reversible encoding of data </li></ul></ul><ul><ul><li>Additional sensors/optics/illum </li></ul></ul><ul><ul><li>‘ Scene analysis’ : (Next software?) </li></ul></ul><ul><li>Essence Photography </li></ul><ul><ul><li>High-level understanding </li></ul></ul><ul><ul><ul><li>Not mimic human eye </li></ul></ul></ul><ul><ul><ul><li>Beyond single view/illum </li></ul></ul></ul><ul><ul><li>‘ New artform’ </li></ul></ul>
  51. 56. Blind Camera Sascha Pohflepp, U of the Art, Berlin, 2006
  52. 57. Capturing the Essence of Visual Experience <ul><ul><li>Exploiting online collections </li></ul></ul><ul><ul><ul><li>Photo-tourism [Snavely2006] </li></ul></ul></ul><ul><ul><ul><li>Scene Completion [Hays2007] </li></ul></ul></ul><ul><ul><li>Multi-perspective Images </li></ul></ul><ul><ul><ul><li>Multi-linear Perspective [Jingyi Yu, McMillan 2004] </li></ul></ul></ul><ul><ul><ul><li>Unwrap Mosaics [Rav-Acha et al 2008] </li></ul></ul></ul><ul><ul><ul><li>Video texture panoramas [Agrawal et al 2005] </li></ul></ul></ul><ul><ul><li>Non-photorealistic synthesis </li></ul></ul><ul><ul><ul><li>Motion magnification [Liu05] </li></ul></ul></ul><ul><ul><li>Image Priors </li></ul></ul><ul><ul><ul><li>Learned features and natural statistics </li></ul></ul></ul><ul><ul><ul><li>Face Swapping: [Bitouk et al 2008] </li></ul></ul></ul><ul><ul><ul><li>Data-driven enhancement of facial attractiveness [Leyvand et al 2008] </li></ul></ul></ul><ul><ul><ul><li>Deblurring [Fergus et al 2006, Several 2008 and 2009 papers] </li></ul></ul></ul>
  53. 58. Scene Completion Using Millions of Photographs Hays and Efros, Siggraph 2007
  54. 59. Community Photo Collections U of Washington/Microsoft: Photosynth
  55. 60. Can you look around a corner ?
  56. 61. Can you look around a corner ? Kirmani, Hutchinson, Davis, Raskar 2009 Accepted for ICCV’2009, Oct 2009 in Kyoto Impulse Response of a Scene
  57. 62. Femtosecond Laser as Light Source Pico-second detector array as Camera
  58. 63. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern
  59. 64. In Focus Photo LED
  60. 65. Out of Focus Photo: Open Aperture
  61. 66. Out of Focus Photo: Coded Aperture
  62. 67. Captured Blurred Photo
  63. 68. Refocused on Person
  64. 69. <ul><li>Smart Barcode size : 3mm x 3mm </li></ul><ul><li>Ordinary Camera: Distance 3 meter </li></ul>Computational Probes: Long Distance Bar-codes Mohan, Woo,Smithwick, Hiura, Raskar Accepted as Siggraph 2009 paper
  65. 70. Bokode
  66. 71. Barcodes markers that assist machines in understanding the real world
  67. 72. Bokode: ankit mohan, grace woo, shinsaku hiura, quinn smithwick, ramesh raskar camera culture group, MIT media lab imperceptible visual tags for camera based interaction from a distance
  68. 73. Defocus blur of Bokode
  69. 74. Image greatly magnified. Simplified Ray Diagram
  70. 75. Our Prototypes
  71. 76. street-view tagging
  72. 77. Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar BiDi Screen *
  73. 78. Beyond Multi-touch: Mobile Laptops Mobile
  74. 79. Light Sensing Pixels in LCD Display with embedded optical sensors Sharp Microelectronics Optical Multi-touch Prototype
  75. 80. Design Overview Display with embedded optical sensors LCD , displaying mask Optical sensor array ~2.5 cm ~50 cm
  76. 81. Beyond Multi-touch: Hover Interaction <ul><li>Seamless transition of multitouch to gesture </li></ul><ul><li>Thin package, LCD </li></ul>
  77. 82. Design Vision Object Collocated Capture and Display Bare Sensor Spatial Light Modulator
  78. 83. Touch + Hover using Depth Sensing LCD Sensor
  79. 84. Overview: Sensing Depth from Array of Virtual Cameras in LCD
  80. 85. <ul><li>A. Introduction‐‐5 minutes </li></ul><ul><li>B. Cameras of the future ( Raskar , 30 minutes) * Form factors, Modalities and Interaction * Enabling Visual Social Computing </li></ul><ul><li>C. Reconstruction the World ( Seitz , 30 minutes) * Photo tourism and beyond * Image‐based modeling and rendering on a massive scale * Scene summarization </li></ul><ul><li>D. Understanding a Billion Photos ( Efros , 30 minutes) * What will the photos depict? * Photos as visual content for computer graphics * Solving computer vision </li></ul><ul><li>E. Discussion‐‐10 minutes </li></ul>Next Billion Cameras
  81. 86. <ul><li>Visual Social Computing </li></ul><ul><li>Computational Photography </li></ul><ul><ul><li>Digital </li></ul></ul><ul><ul><li>Epsilon </li></ul></ul><ul><ul><li>Coded </li></ul></ul><ul><ul><li>Essence </li></ul></ul><ul><li>Beyond Traditional Imaging </li></ul><ul><ul><li>Looking around a corner </li></ul></ul><ul><ul><li>LCDs as virtual cameras </li></ul></ul><ul><ul><li>Computational probes (bokode) </li></ul></ul>Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info Cameras of the Future Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism
  82. 87. Camera Culture Ramesh Raskar Alyosha Efros Ramesh Raskar Steve Seitz Siggraph 2009 Course Next Billion Cameras http://raskar.info/photo/
  83. 88. <ul><li>A. Introduction‐‐5 minutes </li></ul><ul><li>B. Cameras of the future ( Raskar , 30 minutes) * Form factors, Modalities and Interaction * Enabling Visual Social Computing </li></ul><ul><li>C. Reconstruction the World ( Seitz , 30 minutes) * Photo tourism and beyond * Image‐based modeling and rendering on a massive scale * Scene summarization </li></ul><ul><li>D. Understanding a Billion Photos ( Efros , 30 minutes) * What will the photos depict? * Photos as visual content for computer graphics * Solving computer vision </li></ul><ul><li>E. Discussion‐‐10 minutes </li></ul>Next Billion Cameras
  84. 89. <ul><li>Capture </li></ul><ul><li>Overcome Limitations of Cameras </li></ul><ul><li>Capture Richer Data </li></ul><ul><li>Multispectral </li></ul><ul><li>New Classes of Visual Signals </li></ul><ul><li>Lightfields, Depth, Direct/Global, Fg/Bg separation </li></ul><ul><li>Hyperrealistic Synthesis </li></ul><ul><li>Post-capture Control </li></ul><ul><li>Impossible Photos </li></ul><ul><li>Close to Scientific Imaging </li></ul>Computational Photography http://raskar.info/photo/
  85. 90. <ul><ul><li>http://raskar.scripts.mit.edu / nextbillioncameras </li></ul></ul>
  86. 92. Questions <ul><li>What will a camera look like in 10,20 years? </li></ul><ul><li>How will a billion networked and portable cameras change the social culture? </li></ul><ul><li>How will online photo collections transform visual social computing? </li></ul><ul><li>How will movie making/new reporting change? </li></ul><ul><li>computational-journalism.com </li></ul>
  87. 93. Fernald, Science [Sept 2006] Shadow Refractive Reflective Tools for Visual Computing
  88. 94. Cameras and their Impact <ul><li>Beyond Traditional Imaging Analysis and synthesis </li></ul><ul><ul><li>Emerging optics, illumination, novel sensors </li></ul></ul><ul><ul><li>Exploit priors and online collections </li></ul></ul><ul><li>Applications </li></ul><ul><ul><li>Better scene understanding/analysis </li></ul></ul><ul><ul><li>Capture visual essence </li></ul></ul><ul><ul><li>Superior Metadata tagging for effective sharing </li></ul></ul><ul><ul><li>Fuse non-visual data </li></ul></ul><ul><li>Impact on Society </li></ul><ul><ul><li>Beyond entertainment and productivity </li></ul></ul><ul><ul><li>Sensors for disabled, new art forms, crowdsourcing, bridging cultures, social stability </li></ul></ul>
  89. 95. 2 nd International Conference on Computational Photography Papers due November 2, 2009 http://cameraculture.media.mit.edu/iccp10
  90. 96. <ul><li>Ramesh Raskar and Jack Tumblin </li></ul><ul><li>Book Publishers: A K Peters </li></ul><ul><li>Siggraph 2009 booth: 20% off </li></ul><ul><li>Booth #2527 </li></ul><ul><li>ComputationalPhotography.org </li></ul><ul><li>Meet the Authors </li></ul><ul><li>Thursday at 2pm-2:30pm </li></ul>
  91. 97. <ul><li>Visual Social Computing </li></ul><ul><li>Computational Photography </li></ul><ul><ul><li>Digital </li></ul></ul><ul><ul><li>Epsilon </li></ul></ul><ul><ul><li>Coded </li></ul></ul><ul><ul><li>Essence </li></ul></ul><ul><li>Beyond Traditional Imaging </li></ul><ul><ul><li>Looking around a corner </li></ul></ul><ul><ul><li>LCDs as virtual cameras </li></ul></ul><ul><ul><li>Computational probes (bokode) </li></ul></ul><ul><ul><li>http://raskar.scripts.mit.edu / nextbillioncameras </li></ul></ul>Next Billion Cameras Digital Epsilon Coded Essence Computational Photography aims to make progress on both axis Camera Array HDR, FoV Focal stack Decomposition problems Depth Spectrum LightFields Human Stereo Vision Transient Imaging Virtual Object Insertion Relighting Augmented Human Experience Material editing from single photo Scene completion from photos Motion Magnification Phototourism
  92. 98. <ul><li>A. Cameras of the future ( Raskar , 30 minutes) * Enabling Visual Social Computing * Computational Photography * Beyond Traditional Imaging </li></ul><ul><li>B. Reconstruction the World ( Seitz , 30 minutes) * Photo tourism and beyond * Image‐based modeling and rendering on a massive scale * Scene summarization </li></ul><ul><li>C. Understanding a Billion Photos ( Efros , 30 minutes) * What will the photos depict? * Photos as visual content for computer graphics * Solving computer vision </li></ul>Next Billion Cameras <ul><ul><li>http://raskar.scripts.mit.edu / nextbillioncameras </li></ul></ul>Course Evaluation (prize: free mug for each course!) http://www.siggraph.org/courses_evaluation IntConf on Computational Photography, Mar’2010 Papers due Nov 2, 2009 http://cameraculture.info/iccp10 Book: Computational Photography [Raskar and Tumblin] AkPeters Booth #2527, 20% coupons here, Meet Authors Thu 2pm

×