Compressive DIsplays: SID Keynote by Ramesh Raskar


Published on

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Compressive DIsplays: SID Keynote by Ramesh Raskar

  1. 1. Compressive Displays:Managing Lightfield Bandwidth with Math + Layered LCDs Ramesh Raskar Asso. Prof. MIT Media Lab
  2. 2. Slow Glass: Time Shifted DisplayLight of Other Days by Bob Shaw
  3. 3. Shift Glass Space Shifting Angle Shifting Time ShiftingIllumination Shifting 4D 4D t 4D 4D
  4. 4. Motivation: Glasses Free 3D Displays Eliminating Eyewear Expanding FoV and DoF for Glasses-free DisplaysEliminating Moving Parts Thin Holographic Displays(Preserving Depth Cues) (Developing or Avoiding High-Res SLM) Favarola et al Michael Bove et al
  5. 5. Glasses-free 3D DisplaysBandwidth: 600 Gbytes/sec 1M * 60 Hz * 100x100 Views
  6. 6. Compressive Displays: Lossy Compression in OpticsBandwidth: 600 Gbytes/sec 2 Gbytes/sec
  7. 7. Opportunities: New Hardware + New Math Emerging Displays Multilayer High Frame Rate Directional Backlighting Compression and Embedded Processing Non-negative Matrix Factorization (NMF) Non-negative Tensor Factorization (NTF)
  8. 8. Camera Culture Creating new ways to capture and share visual information MIT Media Lab Ramesh Raskar Computational Photography Femtosecond Imaging 3D Displays 1. Looking around corners 1. Tensor Display 1.Light-Field Camera A family of compressive light field A new camera design exploiting the Using short laser pulses and fast detector, we aim to build a device that can look around corners with no imaging displays comprising all architectures fundamental dictionary of light-fields device in the line of sight using time resolved transient employing a stack of time-multiplexed, for a single-capture capture of light- imaging. light-attenuating layers illuminated by fields with full-resolution refocusing uniform or directional backlighting effects. 2. Layered 3D 2. Color Primaries Tomographic techniques for image A new camera design with synthesis on displays composed of switchable color filter arrays for compact volumes of light-attenuating optimal color fidelity and picture material. Such volumetric attenuators quality on scene geometry, color recreate a 4D light field or high-contrast and illumination. 2D image when illuminated by a uniform 2. Reflectance Recovery backlight. We demonstrate a new technique that allows a camera to 3. Flutter-Shutter rapidly acquire reflectance properties of objects in the wild 3. Glasses-free 3D HDTV A camera that codes the exposure from a single viewpoint, over relatively long distances and Light field displays with increased time with a binary pseudo-sequence without encircling equipment. brightness and refresh rate by stacking a to de-convolve and remove motion pair of modified LCD panels, exploiting blur in textured backgrounds and rank and constraint of 3D displays partial occluders. 3. Trillion Frames per Second Imaging A camera fast enough to capture light pulses moving through objects. We can use such a camera to understand 4. BIDI Screen reflectance, absorption and scattering properties of A thin, depth-sensing LCD for 3D 4. Compressive Capture materials. interaction using light fields which We analyze the gamut of visual supports both 2D multi-touch and signals from low-dimensional unencumbered 3D gestures. images to light-fields and propose non-adaptive projections for efficient sparsity exploiting 5. Living Windows 6D Display reconstruction. A completely passive display that responds to changes in viewpoint and changes in incident light conditions.May 2012
  9. 9. Health & Wellness Human Computer Interaction Visual Social Computing1. Retinal Imaging 1. Bokode 1. Photocloud With simplified optics and cleaver Low-cost, passive optical design so A near real-time system for illumination we visualize images of the that bar codes can be shrunk to fewer interactively exploring a collectively retina in a standalone device easily than 3mm and read by ordinary captured moment without explicit 3D operated by the end user. cameras several meters away. reconstruction.2. NETRA/CATRA Low-cost cell-phone attachments that 2. Specklesense 2. Vision Blocks measures eye-glass prescription and Set of motion-sensing configurations On-demand, in-browser, cataract information from the eye. based on laser speckle sensing . The customizable, computer-vision3. Cellphone Microscopy underlying principles allow application-building platform for the A platform for computational microscopy interactions to be fast, precise, masses. Without any prior and remote healthcare extremely compact, and low cost. programming experience, users can create and share computer vision4. High-speed Tomography 3. Sound Around applications. A compact, fast CAT scan machine using Soundaround is a multi-viewer no mechanical moving parts or interactive audio system, designed to synchronization. be integrated into multi-view displays 3. Lenschat presenting localized audio/video LensChat allows users to share5. Shield Fields channels with no need for glasses or mutual photos with friends or borrow 3D reconstruction of objects from a single headphones. the perspective and abilities of many shot photo using spatial heterodyning. cameras.6. Second Skin Using 3D motion tracking with real-time Visit us online at vibrotactile feedback aids the correct of movement and position errors to improve motor learning. Light Propagation Theory and Fourier Optics1. Augmented Light Fields 2. Hologram v Parallax Barrier Expands light field representations to Defines connections between parallax 3. Ray–Based Diffraction Model describe phase and diffraction effects by barrier displays and holographic displays by Simplified capture of diffraction model for using the Wigner Distribution Function analyzing their operations and limitations in computer graphics applications. phase space Post-Doctorial Researchers: Doug Lanman, Gordon Wetzstein, Alex Olwal, Christopher Barsi Research Assistants: Matthew Hirsch, Otkrist Gupta, Nikhil Naik, Jason Boggess, Everett Lawson, Aydın Arpa, Kshitij Marwah Visiting Researchers & Students: Di Wu, Daryl Lim
  10. 10. Office of the Future, UNC, 1997
  11. 11. Pocket Projectors 1999-2004 UNC Chapel Hill and Mitsubishi Electric Research Labs
  12. 12. Software CompressionScalable Display Inc.
  13. 13. Time Illumination Space/Angle Slow Display 6D Display Layered www.layered3d.infoHigh-Rank 3D (HR3D) BiDi Screen Polarization Fields
  14. 14. Optical Sensing: Touch + 3D Gestures
  15. 15. BiDi Screen
  16. 16. Light Field Transfer Application
  17. 17. MIT media lab camera culture NETRA: Refractive Error on Mobile Phone Siggraph 2010 Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar 18
  18. 18. 6D Display: Respond to View + Ambient Illumination
  19. 19. Camera Culture: Compressive Displays Team Gordon Wetzstein Matthew Hirsch Douglas Lanman Postdoctoral Associate Graduate Student Postdoctoral Associate Wolfgang Heidrich, Professor, University of British Columbia Yunhee Kim, Postdoctoral Fellow, MIT Media Lab
  20. 20. 3D Display: Light and Rank Deficient Parallax barrier Front Back LCD display
  21. 21. Input 4D Light Field: Horizontal and Vertical Parallax
  22. 22. Parallax Barrier: Front Layer
  23. 23. Parallax Barrier: Rear Layer
  24. 24. Light Field of Parallax Barriers: Rank 1 k L[i,k] i kg[k] i `f[i] light box L[i, k ]  f [i]  g[k ] L f g
  25. 25. Dual LCD: Content-Adaptive Parallax Barriers G L[i,k] kg[k] ~ i F L`f[i] light box ~ L  FG
  26. 26. Dual LCD: Content-Adaptive Parallax Barriers G L[i,k] kg[k] i F ~f[i] L` light box ~ L  FG Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010
  27. 27. High-Rank 3D (HR3D): Front Layer
  28. 28. High-Rank 3D (HR3D): Rear Layer
  29. 29. Algebraic Rank Constraint Rank-1 Rank-1 s1* m2 s1 s1
  30. 30. Light Field vs. Holographic Displays
  31. 31. Is a hologram just another ray-based light field?Can a hologram create any intensity distribution in 3D?Why does a hologram create a “wavefront”, but parallax barrier does not?Why does a hologram create accommodation cues?What are the effective resolution and depth of field for holograms vs. barriers?
  32. 32. Parallax Barrier: Np=103 pix. Hologram: NH=105 pix. ϕP∝w/d ϕH∝λ/tHθp=10 pix θH =1000 pix Fourier Patch w Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009
  33. 33. Augmented Light Field wave optics based rigorous but cumbersome Wigner WDF Distribution Function Augmented LF Traditional Traditional Light Field Light Fieldray optics based Interference & Diffractionsimple and powerful Interaction w/ optical elements Oh, Raskar, Barbastathis 2009: Augmented Light Field 34
  34. 34. Glasses-free 3D Displays Raw: 600 Gbytes/secCompressive: 2 Gbyte/sec
  35. 35. View Dependent Appearance and Iridescent colorCross section through a single M. Rhetenor scale
  36. 36. Generalizing Parallax Barriers: Rank 1 mask K … mask 3 mask 2 mask 2 mask 2 mask 1 mask 1 mask 1 light box light box light box Conventional Parallax Barrier High-Rank 3D (HR3D) Layered 3D and Polarization Fields Parallax barriers use heuristic design: front mask with slits/pinholes, rear mask with interlaced views High-Rank 3D (HR3D) considers dual-layer design with arbitrary opacity and temporal multiplexing Layered 3D and Polarization Fields considers multi-layer design without temporal multiplexing
  37. 37. Layered 3D: Multi-Layer Automultiscopic Displays mask K … mask 3 mask 2 mask 1 light box Layered 3D
  38. 38. Tomographic Light Field Synthesis virtual plane Image formation model: attenuator ò - m (r )dr L(x, q ) = I 0 e C æ L(x, q ) ö L(x, q ) = ln ç ÷ = - ò m (r)dr è I0 ø C backlight l = -Pa Tomographic synthesis: 2 arg min l + Pa , for a ³ 0 a 2D Light Field
  39. 39. Tomographic Light Field Synthesis virtual plane Image formation model: attenuator ò - m (r )dr L(x, q ) = I 0 e C æ L(x, q ) ö L(x, q ) = ln ç ÷ = - ò m (r)dr è I0 ø C backlight l = -Pa Tomographic synthesis: 2 arg min l + Pa , for a ³ 0 a 2D Light Field
  40. 40. Multi-Layer Light Field Decomposition Reconstructed Views Target 4D Light Field Multi-Layer Decomposition
  41. 41. Prototype Layered 3D Display Transparency stack with acrylic spacers Prototype in front of LCD (backlight source)
  42. 42. Polarization Fields Four Stacked Liquid Crystal Panels Two Crossed Polarizers
  43. 43. Simulation Results
  44. 44. Codesign of Optics + Computation Materials Shift GlassPhotons Optics Displays Lighting Capture Sensors HCI Signal Processing CV / Machine Learning Bits
  45. 45. Speed beats Resolution
  46. 46. Tensor Displays Siggraph 2012 Highlight Paper
  47. 47. Tensor Displays
  48. 48. Tensor Displays
  49. 49. Tensor Displays
  50. 50. Next Generation of Stacked LCDsAsksHigh Frame Rate (x 10)Moiré: Non-uniform pixel gridTransmission LimitationsOLED and emerging tech Needs ‘Compressible’ views Slightly lower brightnessCollaboration Within intrinsic display propertiesOpen ArchitectureCompDisp ConsortiumVisit ML,Join at
  51. 51. • Higher Dimensional Displays Compressive Display – 3D, video, wavelength ..• 600 GB  2GB/sec – Lossy compression in OPTICS• Emerging Hardware – Multi-layer, high frame rate, directional backlt – Frame rate beats resolution G• Mathematical Optimization ~ ` L = F – Tomography, sparsity, tensor factorization
  52. 52. Camera Culture: Compressive Displays Team Gordon Wetzstein Matthew Hirsch Douglas Lanman Postdoctoral Associate Graduate Student Postdoctoral Associate Wolfgang Heidrich, Professor, University of British Columbia Yunhee Kim, Postdoctoral Fellow, MIT Media Lab
  53. 53. Raskar, Lanman, Wetzstein, Hirsch MIT Media Lab Shift Glass Capture Display 5D: Looking Compressive Displays around corners 6D: View and Lighting Aware 4D: Rank Deficient, multilayer 4D: Netra for Optometry WDF Analyze G Augmented Light LF ~ ` L = F Field 4D, 6D, 8D: Augmented Light Field
  54. 54. Raskar, Lanman, Wetzstein, Hirsch MIT Media Lab Layered 3D Polarization Fields High-Rank 3D (HR3D) Slow Display 6D Display BiDi Screen
  55. 55. Compressive Display Research in Camera CultureRamesh Raskar, Douglas Lanman, Gordon Wetzstein, Matthew Hirsch
  56. 56. Faster Horse  Car