Successfully reported this slideshow.
BiDi Screen Depth and Lighting Aware Interactive Display Matthew Hirsch MIT Media Lab Douglas Lanman Brown University Rame...
BiDi Screen
Inspiration Light Sensitive Displays Depth Cameras Multitouch
Beyond Multi-touch: Hover Interaction <ul><li>Seamless transition of multitouch to gesture </li></ul><ul><li>Thin package,...
Beyond Multi-touch: Mobile Laptops Mobile
Results: Analysis
BiDi Screen <ul><li>Capture 3-D to enable depth and lighting aware interaction </li></ul><ul><li>Collocated image capture ...
Design Overview Display with embedded optical sensors Sharp Microelectronics Optical Multi-touch Prototype
Design Overview Display with embedded optical sensors LCD   ,  displaying   mask Optical sensor array ~2.5 cm ~50 cm
Design Vision Object Collocated Capture and Display Bare Sensor Spatial Light Modulator
Design Overview Mask Array of Virtual Cameras Sensor
Manipulating an object with 3D gesture
Alternatives to capture depth from an LCD <ul><li>Adapted Touch </li></ul><ul><ul><li>Capacitive </li></ul></ul><ul><ul><l...
Adapting Traditional Touch? <ul><li>Resistive touch screen </li></ul><ul><ul><li>Confined to screen </li></ul></ul><ul><li...
Camera Arrays <ul><li>Cameras behind screen </li></ul><ul><ul><li>Interfere with backlight </li></ul></ul><ul><ul><li>Expe...
Depth Cameras and Sensors <ul><li>Depth Cameras </li></ul><ul><ul><li>Behind screen w/ projector (MSR TouchlLight) </li></...
Theory: Depth from light-field capture <ul><li>LCD used for Spatial Heterodyne Light Field Capture and Display </li></ul><...
Theory: Spatial Heterodyning with MURA u d ref d m u  s u =0 s =0 = u +( d m / d ref ) s  L received ( f u , f s ) f u f...
Theory: Lightfield θ x y θ x . (Sensor) Sensor integrates these rays
Theory: Lightfield Frequency Domain f θ f x θ x Fourier Slice Theorm FT Lightfield FT
Theory: LF Skew in Free-Space Propogation θ x x y mask θ mask
Theory: Convolution with Delta Functions
Theory: Transforms f θ f x θ x f mask mask Tiled Broadband Mask Lightfield FT FT
Theory: Spatial Heterodyning x y mask Multiplication in prime domain is convolution in frequency domain
Theory: Spatial Heterodyning f θ f x f mask f θ f x
Theory: Spatial Heterodyning f θ f x Reconstructed Lightfield Spectrum Band Limited
Desired Prototype Actual Prototype
Prototype <ul><li>Camera / diffuser pair as sensor substitute </li></ul>
Prototype <ul><li>Camera / diffuser pair angle limiting property </li></ul>
Prototype Backlights Cameras
Pipeline Max Contrast Operator
Pipeline
Theory: Depth and Spatial Resolution
Theory: Depth and Spatial Resolution
Results: Analysis
Manipulating an object with 3D gesture
Navigating a 3D world with 3D gesture
Lighting a vritual scene with a real light
Limitations <ul><li>Requires separation between display and sensor (adds thickness) </li></ul><ul><ul><li>2.5 cm for 50 cm...
Conclusions <ul><li>Future Work </li></ul><ul><ul><li>Dynamic Masks </li></ul></ul><ul><ul><ul><li>Change frequency charac...
Conclusions <ul><li>Enable multitouch  and  gesture interaction on a thin display screen </li></ul><ul><ul><li>Walk-up int...
Upcoming SlideShare
Loading in …5
×

BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar

22,343 views

Published on

http://web.media.mit.edu/~mhirsch/bidi/

Watch video: http://www.youtube.com/watch?v=kXuxK6IeQfo

BiDi Screen
A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields

Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar

1MIT Media Lab

Published in: Technology, Business
  • Be the first to comment

BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar

  1. 1. BiDi Screen Depth and Lighting Aware Interactive Display Matthew Hirsch MIT Media Lab Douglas Lanman Brown University Ramesh Raskar MIT Media Lab Henry Holtzman MIT Media Lab
  2. 2. BiDi Screen
  3. 3. Inspiration Light Sensitive Displays Depth Cameras Multitouch
  4. 4. Beyond Multi-touch: Hover Interaction <ul><li>Seamless transition of multitouch to gesture </li></ul><ul><li>Thin package, LCD </li></ul>
  5. 5. Beyond Multi-touch: Mobile Laptops Mobile
  6. 6. Results: Analysis
  7. 7. BiDi Screen <ul><li>Capture 3-D to enable depth and lighting aware interaction </li></ul><ul><li>Collocated image capture and display </li></ul><ul><li>Walk-up interaction </li></ul><ul><li>Mobility and portability – thin device </li></ul>
  8. 8. Design Overview Display with embedded optical sensors Sharp Microelectronics Optical Multi-touch Prototype
  9. 9. Design Overview Display with embedded optical sensors LCD , displaying mask Optical sensor array ~2.5 cm ~50 cm
  10. 10. Design Vision Object Collocated Capture and Display Bare Sensor Spatial Light Modulator
  11. 11. Design Overview Mask Array of Virtual Cameras Sensor
  12. 12. Manipulating an object with 3D gesture
  13. 13. Alternatives to capture depth from an LCD <ul><li>Adapted Touch </li></ul><ul><ul><li>Capacitive </li></ul></ul><ul><ul><li>Resistive </li></ul></ul><ul><ul><li>Optical </li></ul></ul><ul><li>Camera arrays </li></ul><ul><ul><li>Behind screen </li></ul></ul><ul><ul><li>To sides </li></ul></ul><ul><li>Depth Sensors/Cameras </li></ul>
  14. 14. Adapting Traditional Touch? <ul><li>Resistive touch screen </li></ul><ul><ul><li>Confined to screen </li></ul></ul><ul><li>Capacitive </li></ul><ul><ul><li>Not lighting dependant </li></ul></ul><ul><ul><li>Calibration problems for gesture </li></ul></ul><ul><li>No existing technology trend for gesture </li></ul>?
  15. 15. Camera Arrays <ul><li>Cameras behind screen </li></ul><ul><ul><li>Interfere with backlight </li></ul></ul><ul><ul><li>Expensive, Large </li></ul></ul><ul><ul><li>Han , SecondLight, ThinSight </li></ul></ul><ul><ul><li>No tech curve </li></ul></ul><ul><li>Cameras to side </li></ul><ul><ul><li>Transition region </li></ul></ul><ul><ul><li>Second modality </li></ul></ul>?
  16. 16. Depth Cameras and Sensors <ul><li>Depth Cameras </li></ul><ul><ul><li>Behind screen w/ projector (MSR TouchlLight) </li></ul></ul><ul><ul><li>No longer a thin device </li></ul></ul><ul><ul><li>No light sensitivity </li></ul></ul><ul><ul><li>Touch difficult </li></ul></ul>
  17. 17. Theory: Depth from light-field capture <ul><li>LCD used for Spatial Heterodyne Light Field Capture and Display </li></ul><ul><ul><li>Pinholes or tiled broadband masks </li></ul></ul><ul><ul><li>Separate Sensor and Mask </li></ul></ul><ul><li>Fourier Refocusing </li></ul><ul><li>Depth from focus </li></ul><ul><li>Blob tracking (gesture) </li></ul>
  18. 18. Theory: Spatial Heterodyning with MURA u d ref d m u  s u =0 s =0 = u +( d m / d ref ) s  L received ( f u , f s ) f u f s f u 0 f s 0 f s R s   
  19. 19. Theory: Lightfield θ x y θ x . (Sensor) Sensor integrates these rays
  20. 20. Theory: Lightfield Frequency Domain f θ f x θ x Fourier Slice Theorm FT Lightfield FT
  21. 21. Theory: LF Skew in Free-Space Propogation θ x x y mask θ mask
  22. 22. Theory: Convolution with Delta Functions
  23. 23. Theory: Transforms f θ f x θ x f mask mask Tiled Broadband Mask Lightfield FT FT
  24. 24. Theory: Spatial Heterodyning x y mask Multiplication in prime domain is convolution in frequency domain
  25. 25. Theory: Spatial Heterodyning f θ f x f mask f θ f x
  26. 26. Theory: Spatial Heterodyning f θ f x Reconstructed Lightfield Spectrum Band Limited
  27. 27. Desired Prototype Actual Prototype
  28. 28. Prototype <ul><li>Camera / diffuser pair as sensor substitute </li></ul>
  29. 29. Prototype <ul><li>Camera / diffuser pair angle limiting property </li></ul>
  30. 30. Prototype Backlights Cameras
  31. 31. Pipeline Max Contrast Operator
  32. 32. Pipeline
  33. 33. Theory: Depth and Spatial Resolution
  34. 34. Theory: Depth and Spatial Resolution
  35. 35. Results: Analysis
  36. 36. Manipulating an object with 3D gesture
  37. 37. Navigating a 3D world with 3D gesture
  38. 38. Lighting a vritual scene with a real light
  39. 39. Limitations <ul><li>Requires separation between display and sensor (adds thickness) </li></ul><ul><ul><li>2.5 cm for 50 cm range in our prototype </li></ul></ul><ul><ul><li>750mm for 8 cm range in iPhone like device </li></ul></ul><ul><li>Sensitive to room lighting or requires its own light source </li></ul><ul><ul><li>This is true of many optical systems </li></ul></ul><ul><li>Time multiplexing of display/capture </li></ul><ul><ul><li>Requires fast capture and screen refresh to stay below flicker fusion rate </li></ul></ul><ul><ul><li>240 Hz LCDs coming to market </li></ul></ul>
  40. 40. Conclusions <ul><li>Future Work </li></ul><ul><ul><li>Dynamic Masks </li></ul></ul><ul><ul><ul><li>Change frequency characteristics to match scene </li></ul></ul></ul><ul><ul><li>Video capture / Video chat (higher resolution) </li></ul></ul><ul><li>SIGGRAPH Asia 2009 Paper </li></ul><ul><ul><li>BiDi Screen: Depth and Lighting Aware Interaction and Display </li></ul></ul>
  41. 41. Conclusions <ul><li>Enable multitouch and gesture interaction on a thin display screen </li></ul><ul><ul><li>Walk-up interaction does not require gloves / fiducials </li></ul></ul><ul><li>Capture depth using array of vitual cameras </li></ul><ul><li>Thin portable devices possible with area sensor </li></ul>

×