BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar

by MIT Media Lab, Camera Culture group on Dec 11, 2009

13,009 views

http://web.media.mit.edu/~mhirsch/bidi/ ...

http://web.media.mit.edu/~mhirsch/bidi/

Watch video: http://www.youtube.com/watch?v=kXuxK6IeQfo

BiDi Screen
A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields

Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar

1MIT Media Lab

Less

13 Embeds 9,879

http://web.media.mit.edu	9176
http://www.touchuserinterface.com	536
http://translate.googleusercontent.com	49
http://www.azoft.com	45
http://dsinsights.blogspot.com	40
http://www.slideshare.net	17
http://webcache.googleusercontent.com	6
http://www.linkedin.com	3
http://207.46.192.232	3
https://pww.intelligence.research.philips.com	1
http://mail.wamediagroup.com	1
http://rdbcci	1
http://translate.yandex.net	1

More...

Statistics

Favorites: 1
Downloads: 135
Comments: 0
Embed Views: 9,879
Views on SlideShare: 3,130
Total Views: 13,009

BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar — Presentation Transcript

BiDi Screen Depth and Lighting Aware Interactive Display Matthew Hirsch MIT Media Lab Douglas Lanman Brown University Ramesh Raskar MIT Media Lab Henry Holtzman MIT Media Lab

BiDi Screen

Inspiration Light Sensitive Displays Depth Cameras Multitouch

Beyond Multi-touch: Hover Interaction

Seamless transition of multitouch to gesture

Thin package, LCD

Beyond Multi-touch: Mobile Laptops Mobile

Results: Analysis

BiDi Screen

Capture 3-D to enable depth and lighting aware interaction

Collocated image capture and display

Walk-up interaction

Mobility and portability – thin device

Design Overview Display with embedded optical sensors Sharp Microelectronics Optical Multi-touch Prototype

Design Overview Display with embedded optical sensors LCD , displaying mask Optical sensor array ~2.5 cm ~50 cm

Design Vision Object Collocated Capture and Display Bare Sensor Spatial Light Modulator

Design Overview Mask Array of Virtual Cameras Sensor

Manipulating an object with 3D gesture

Alternatives to capture depth from an LCD

Adapted Touch

Capacitive

Resistive

Optical

Camera arrays

Behind screen

To sides

Depth Sensors/Cameras

Adapting Traditional Touch?

Resistive touch screen

Confined to screen

Capacitive

Not lighting dependant

Calibration problems for gesture

No existing technology trend for gesture

Camera Arrays

Cameras behind screen

Interfere with backlight

Expensive, Large

Han , SecondLight, ThinSight

No tech curve

Cameras to side

Transition region

Second modality

Depth Cameras and Sensors

Depth Cameras

Behind screen w/ projector (MSR TouchlLight)

No longer a thin device

No light sensitivity

Touch difficult

Theory: Depth from light-field capture

LCD used for Spatial Heterodyne Light Field Capture and Display

Pinholes or tiled broadband masks

Separate Sensor and Mask

Fourier Refocusing

Depth from focus

Blob tracking (gesture)

Theory: Spatial Heterodyning with MURA u d ref d m u  s u =0 s =0 = u +( d m / d ref ) s  L received ( f u , f s ) f u f s f u 0 f s 0 f s R s   

Theory: Lightfield θ x y θ x . (Sensor) Sensor integrates these rays

Theory: Lightfield Frequency Domain f θ f x θ x Fourier Slice Theorm FT Lightfield FT

Theory: LF Skew in Free-Space Propogation θ x x y mask θ mask

Theory: Convolution with Delta Functions

Theory: Transforms f θ f x θ x f mask mask Tiled Broadband Mask Lightfield FT FT

Theory: Spatial Heterodyning x y mask Multiplication in prime domain is convolution in frequency domain

Theory: Spatial Heterodyning f θ f x f mask f θ f x

Theory: Spatial Heterodyning f θ f x Reconstructed Lightfield Spectrum Band Limited

Desired Prototype Actual Prototype

Prototype

Camera / diffuser pair as sensor substitute

Prototype

Camera / diffuser pair angle limiting property

Prototype Backlights Cameras

Pipeline Max Contrast Operator

Pipeline

Theory: Depth and Spatial Resolution

Results: Analysis

Manipulating an object with 3D gesture

Navigating a 3D world with 3D gesture

Lighting a vritual scene with a real light

Limitations

Requires separation between display and sensor (adds thickness)

2.5 cm for 50 cm range in our prototype

750mm for 8 cm range in iPhone like device

Sensitive to room lighting or requires its own light source

This is true of many optical systems

Time multiplexing of display/capture

Requires fast capture and screen refresh to stay below flicker fusion rate

240 Hz LCDs coming to market

Conclusions

Future Work

Dynamic Masks

Change frequency characteristics to match scene

Video capture / Video chat (higher resolution)

SIGGRAPH Asia 2009 Paper

BiDi Screen: Depth and Lighting Aware Interaction and Display

Conclusions

Enable multitouch and gesture interaction on a thin display screen

Walk-up interaction does not require gloves / fiducials

Capture depth using array of vitual cameras

Thin portable devices possible with area sensor

BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar

by MIT Media Lab, Camera Culture group on Dec 11, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

13 Embeds 9,879

Statistics

BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar — Presentation Transcript