• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Raskar Keynote at Stereoscopic Display Jan 2011
 

Raskar Keynote at Stereoscopic Display Jan 2011

on

  • 1,770 views

Computational Displays in 4D, 6D, 8D ...

Computational Displays in 4D, 6D, 8D

We have explored how light propagates from thin elements into a volume for viewing for both automultiscopic displays and holograms. In particular, devices that are typically connected with geometric optics, like parallax barriers, differ in treatment from those that obey physical optics, like holograms. However, the two concepts are often used to achieve the same effect of capturing or displaying a combination of spatial and angular information. Our work connects the two approaches under a general framework based in ray space, from which insights into applications and limitations of both parallax-based and holography-based systems are observed.

Both parallax barrier systems and the practical holographic displays are limited in that they only provide horizontal parallax. Mathematically, this is equivalent to saying that they can always be expressed as a rank-1 matrix (i.e, a matrix in which all the columns are linearly related). Knowledge of this mathematical limitation has helped us to explore the space of possibilities and extend the capabilities of current display types. In particular, we have designed a display that uses two LCD panels, and an optimisation algorithm, to produce a content-adaptive automultiscopic display (SIGGRAPH Asia 2010).

(Joint work with R Horstmeyer, Se Baek Oh, George Barbastathis, Doug Lanman, Matt Hirsch and Yunhee Kim) http://cameraculture.media.mit.edu

In other work we have developed a 6D optical system that responds to changes in viewpoint as well as changes in surrounding light. Our lenticular array alignment allows us to achieve such a system as a passive setup, omitting the need for electrical components. Unlike traditional 2D flat displays, our 6D displays discretize the incident light field and modulate 2D patterns in order to produce super-realistic (2D) images. By casting light at variable intensities and angles onto our 6D displays, we can produce multiple images as well as store greater information capacity on a single 2D film (SIGGRAPH 2008).




Ramesh Raskar joined the Media Lab from Mitsubishi Electric Research Laboratories in 2008 as head of the Lab’s Camera Culture research group. His research interests span the fields of computational photography, inverse problems in imaging and human-computer interaction. Recent inventions include transient imaging to look around a corner, next generation CAT-Scan machine, imperceptible markers for motion capture (Prakash), long distance barcodes (Bokode), touch+hover 3D interaction displays (BiDi screen), low-cost eye care devices (Netra) and new theoretical models to augment light fields (ALF) to represent wave phenomena.

In 2004, Raskar received the TR100 Award from Technology Review, which recognizes top young innovators under the age of 35, and in 2003, the Global Indus Technovator Award, instituted at MIT to recognize the top 20 Indian technology innovators worldwide. In 2009, he was awarded a Sloan Research Fellowship. In 2010, he received the Darpa Young Faculty award. He holds over 40 US patents and has received four Mitsubishi Electric Invention Awards. He is currently co-authoring a book on Computational Photography. http://raskar.info

Statistics

Views

Total Views
1,770
Views on SlideShare
1,770
Embed Views
0

Actions

Likes
0
Downloads
71
Comments
1

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • This kind of a technique can be used in other scenarios as well such as Rescue and Planning
  • Robot and car navigation to avoid collisions by estimating position of objects around the bend
  • Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008
  • This video is only for 4D display that responds to light Bonny’s lenticular prints outside
  • Since we are adapting LCD technology we can fit a BiDi screen into laptops and mobile devices.
  • Here’s a quick teaser to illustrate the capabilities I’m describing. Here you see a user pulling her hands away to rotate and zoom a 3-D model. We also show a use of 3D gesture to navigate a 3D world. We support these modes by creating an array of virtual cameras on an LCD using a technique known as Spatial Heterodyning. Because we’re using an optical technique, we also enable dynamic relighting applications, where real-world lighting is transfered to a rendered scene.
  • So here is a preview of our quantitative results. I’ll explain this in more detail later on, but you can see we’re able to accurately distinguish the depth of a set of resolution targets. We show above a portion of portion of views form our virtual cameras, a synthetically refocused image, and the depth map derived from it.
  • A cross section through a single M. rhetenor scale. Light reflected off each level of the “Christmas tree structure” gives the butterfly its iridescent color. Credit: Pete Vukusic, University of Exeter
  • Augmented plenoptic function the motivation, to augment lf, model diffraction in light field formulation
  • In this paper, we show a self-optometry solution. You look at a cell phone display thru a clip-on eye piece, interactively align a few patterns, hit calculate and get data for your eye prescription.
  • We call our tool NETRA: near eye tool for refractive assessment such as nearsightedness/far/astigmatism Basic idea is to create a unique interactive lightfield display near the eye and is possible due to the highresolution of modern LCDs.
  • 2 billion people have refractive errors And half a billion in developing countries worldwide have uncorrected vision that affects their daily livelyhood. They don’t have access to an optometrist or it simply too expensive. While making and distributing of lenses has become quite easy now, surprisingly there is still no easy solution for measuring eyesight. Can we use a fraction of the 4.5B cellphone displays to address this problem?
  • For better precision, there are many kinds of solutions, some really clever. The beauty of netra is that it avoids moving parts or shining lasers, and all intelligence is in the software.
  • The most accurate method is based on a so called SH WS. It involves shining a laser at the back of the retina and observing the wavefront using a sophisticated sensor. We ask user to generate a spot diagram. But navigating in a high dimensional space is challenging so we come up with a strikingly simple approach to let the user interactively create the spot diagram. We are first to make connection between Shack Hartmann and Lightfields (and it goes well with recent work in computational photography about ALF and Zhang/Levoy). Connection to Adaptive optics/ Astronomy. The way that this device works is that, it shines a lasers in the eye, the laser is reflected in the retina and comes out of the eye being distorted by the cornea. These light rays reaches an array of lenses that focus them to dots in a sensor. The device measures how much this dots deviate from the ideal case. Since it uses lasers, the device is expensive and requires trained professionals
  • For a normal eye, the light coming out of the eye forms a parallel wavefront. The sensor has a lenslet array and we get a spot diagram of uniform dots. This lenslet should remind you of a lightfield camera, and in fact Levoy and others showed last year that there is a close relationship between the two. In addition, Zhang and Levoy, plus our grp has shown the relationship between wavefront sensing and lightfield sensing.
  • When the eye has a distortion, the spot diagram is not uniform. And the displacement of the spots from the center indicates the local slope of the wavefront. From the slope one can integrate and recover the wave shape.
  • NETRA uses an exact inverse of this sensor. We get rid of the laser and we instead show the same spot diagram in a cellphone display. For normal eye, it will appear as a dot to the user. And then we replace the sensor for a light field display. If the user sees a single red dot, he does not need glasses, but if he sees more than one, he interacts with this display.
  • NETRA uses an exact inverse of this sensor. We get rid of the laser and we instead show the same spot diagram in a cellphone display. For normal eye, it will appear as a dot to the user. And then we replace the sensor for a light field display. If the user sees a single red dot, he does not need glasses, but if he sees more than one, he interacts with this display.
  • For eye with distortion, the user will interactively displace the 25 points so that he will see a single spot. Of course changing 25 spot locations is cumbersome, but we realize that there are only 3 parameters for eye-prescription and we help the user navigate thru this space efficiently. But if you think about these theory, you will realize that we have the dual of the shack-hartmann. First we though out the laser.
  • For eye with distortion, the user will interactively displace the 25 points so that he will see a single spot. Of course changing 25 spot locations is cumbersome, but we realize that there are only 3 parameters for eye-prescription and we help the user navigate thru this space efficiently. But if you think about these theory, you will realize that we have the dual of the shack-hartmann. First we though out the laser.
  • Since we are relying on the user interaction, the subject has to be aware of the alignment tasks. So, very young Children may not be able to run the test. Instead of just one eye, one may use both eyes to exploit convergence. And of course, the resolution of NETRA itself is a function of the resolution of the display. With a 326 dpi display, resolution is 0.14 diopters and presciption glasses come in increments of 0.25 diopters. So our system is already sufficiently accurate.

Raskar Keynote at Stereoscopic Display Jan 2011 Raskar Keynote at Stereoscopic Display Jan 2011 Presentation Transcript

  • Camera Culture Ramesh Raskar Camera Culture MIT Media Lab Computational Displays in 4D, 6D and 8D
  • Slow Glass: Time Shift http:// baens-universe.com/articles/otherdays Light of Other Days by Bob Shaw http://www.fantasticfiction.co.uk/s/bob-shaw/other-days-other-eyes.htm
  • Shift Glass
  • Shift Glass Space Shifting Angle Shifting Time Shifting Illumination Shifting 4D 4D t 4D 4D
  • Capture Analyze Display Shift Glass
  • Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field Shift Glass
  • Can you look around a corner ? Without any device in the line of sight
  • Femto-Photography: Higher Dimensional LF FemtoFlash UltraFast Detector Computational Optics Serious Sync
  • Kirmani, Hutchinson, Davis, Raskar ICCV’2009, Marr Prize Honorable Mention
  • Streak Camera = Inverse of CRT/CRO Femto-laser
  • Multi-Dimensional Light Transport 5-D Transport
  •  
  • Rescue and Planning
  • Robot, Car Path Planning
  • Endoscopy
  • Camera Culture Ramesh Raskar Team Moungi G. Bawendi, Professor, Dept of Chemistry, MIT James Davis, UC Santa Cruz Andreas Velten, Postdoctoral Associate, MIT Media Lab Ahmed Kirmani, RA, MIT Media Lab Tyler Hutchison, RA, MIT Media Lab Rohit Pandharkar, RA, MIT Media Lab Andrew Matthew Bardagjy, RA, MIT Media Lab Everett Lawson, MIT Media Lab Ramesh Raskar, MIT Media Lab
  • Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field Shift Glass
  • Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field
  • Slow Display
  • Light Reactive Monostable Materials 16 Megapixel, 2 Watt
  • Day/Night visible
  • g SlowDisplay.org Saakes, Chiu, Hutchison, .., Inami, Raskar, Siggraph 2010 Etech Demo
  • 6D Photo Frames One Pixel of a 6D Display = 4D Display Single Pixel of 6D Frame Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch 1 2 1 1 2D 2D 2D
  • Respond to Viewpoint + Ambient Light
  • 6D Display Light sensitive 4D display One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008
  • Beyond Multi-touch: Thin LCD for touch+hover Laptops Mobile
  • BiDi Screen: Multi-touch + Hover 3D interface
  • Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen
  • Theory: Benefits of Tiled-broadband Masks 0 10 20 30 40 50 60 0 0.1 0.2 0.3 0.4 0.5 0.6 Angular Resolution Average Transmission (%) Pinholes Sum-of-Sinusoids MURA 11x11 23x23 43x43 Angular Resolution Tiled-Broadband Code MURA Sum-of-Sinusoids Pinholes
  • Overview: Sensing Depth from Array of Virtual Cameras in LCD
  • Bits Photons CV / Machine Learning Optics Sensors Computational Displays Signal Processing Light Transport Displays HCI
  •  
  • View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale
  • Two Layer Displays PB = dim displays Lenslets = fixed spatial and angular resolution Dynamic Masks = Brighter, High spatial resolution barrier sensor/display lenslet sensor/display
  • Limitations of 3D Display Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010 Front Back Parallax barrier LCD display
  • ` Light Field Analysis of Barriers L[i,k] L[i,k] i k light box g[k] k f[i] i
  • f[i] g[k] L[i,k] light box ` Content-Adaptive Parallax Barriers k i
  • Implementation
    • Components
    • 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]
  • f[i] g[k] L[i,k] light box ` Content-Adaptive Parallax Barriers k i
  • = Content-Adaptive Parallax Barriers `
  • Rank-Constrained Displays and LF Adaptation
    • All dual layer display = rank-1 constraint
    • Light field display is a matrix approximation problem
    • Exploit content-adaptive parallax barriers
    Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010 ` = Content-Adaptive Parallax Barriers
  • rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Optimization: Iteration 1 Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 10 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 20 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 30 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 40 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 50 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 60 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 70 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 80 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Optimization: Iteration 90 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.
  • Content-Adaptive Front Mask (1 of 9)
  • Content-Adaptive Rear Mask (1 of 9)
  • Emitted 4D Light Field
  • Conclusion
    • Described a rank constraint for all dual-layer displays
      • With a fixed pair of masks, emitted light field is rank- 1
    • Achieved higher-rank approximation using temporal multiplexing
      • With T time-multiplexed masks, emitted light field is rank- T
      • Constructed a prototype using off-the-shelf panels
    • Demonstrated light field display is a matrix approximation problem
    • Introduced content-adaptive parallax barriers
      • Applied weighted NMF to optimize weighted Euclidean distance to target
    • Adaptation increases brightness and refresh rate of dual-stacked LCDs
    ` = Content-Adaptive Parallax Barriers
  • Lightfield vs Hologram Displays
  • Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?
  • Parallax Barrier: N p =10 3 pix. Hologram: N H =10 5 pix. θ p =10 pix w θ H =1000 pix ϕ P ∝w/d ϕ H ∝λ/t H Fourier Patch Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009
  • Augmenting Plenoptic Function Wigner Distribution Function Traditional Light Field WDF Traditional Light Field Augmented LF Interference & Diffraction Interaction w/ optical elements ray optics based simple and powerful wave optics based rigorous but cumbersome Oh, Raskar, Barbastathis 2009: Augmented Light Field
  • Light Fields Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters Reference plane position angle LF propagation (diffractive) optical element LF LF LF LF LF propagation light field transformer
  • Augmented Lightfield for Wave Optics Effects Wigner Distribution Function Light Field LF < WDF Lacks phase properties Ignores diffraction, interferrence Radiance = Positive ALF ~ WDF Supports coherent/incoherent Radiance = Positive/Negative Virtual light sources LF Augmented Light Field WDF
      • Free-space propagation
      • Light field transformer
      • Virtual light projector
      • Possibly negative radiance
  • L(x, θ ) W(x,u) W m = sinc d = delta q W m p q p d(θ) p q q d(θ) * p W m * * Rays: No Bending 1 Fresnel HG Patch θ u * Zooming into the Light Field
  • Algebraic Rank Constraint s 1 m 2 s 1 m 2 s 1 * s 1 s 1 s 1 * Rank-1 Rank-1 Rank-3
  • - Transform <t(x+x ʹ /2)t*(x-x ʹ /2)> Interference xʹ x (a) Two Slits, Coherent t(x+x ʹ /2)t*(x-x ʹ /2) W(x,u) Rank-1 t(x 1 )t*(x 2 ) R 45 , D Transform -1 u
  • L2 L1 L3 ϕ 1 ϕ 1 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 h H r z 2 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) s 1 m 2 (a)
  • A B C Vary Illumination Direction: -5 ̊ , 0 ̊, 5 ̊ A B C A … -5 ̊ 5 ̊ 0 ̊ No Slits 24mm 36mm t H =25μm w=125μm z H =10cm (c)
  • M2 M1 M3 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 r z 2 s 1 m 2 s 1 m 2 s 1 m 2 s 1 * s 1 s 1 s 1 * Rank-1 Rank-1 Rank-3
  • Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?
  • NETRA: Interactive Display for Estimating Refractive Errors and Focal Range Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
  • Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar NETRA: N ear E ye T ool for R efractive A ssessment
  • 0.6B uncorrected refractive errors NETRA at LVP Eye Institute 6.5 Billion people 4.5B with Mobile phone 2B refractive errors
  • * Phoropter-based: $5,000.00 Needs expert, Moving parts, Shining lasers Retino scope w/ Lenses Auto-refracto-meter Chart with Lenses In-Focus: Focometer Optiopia Solo-health: EyeSite NETRA Technology Shining Light plus lenses Fundus Camera Moving lenses + target Moving lenses + target Reading chart on monitor Cellphone + eyepiece Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 -- $30 Cost per test ~$36 ~$36 ~$5 -- -- -- ~$1 Data capture No Comp. No No No Comp. Phone Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g Speed Fast Fast Medium Medium -- Fast Fast Scalability No No No Yes Probably No Yes Accuracy 0.15 0.15 0.5 0.75 -- -- <0.5 Self evaluation No No Yes Yes Yes Yes Yes Electricity Req No Yes No No -- Yes No Astigmatism Yes Yes Yes/No No -- Yes Yes Network No Yes No No No Yes Yes Training High High High Medium Medium Low Low
  • Shack-Hartmann Wavefront Sensor Expensive; Bulky, Requires trained professionals Wavefront aberrometer
  • Shack-Hartmann Wavefront Sensor Laser Sensor Microlens Array Planar Wavefront Shack & Platt 1971 Liang et al 1994 David Williams et al, Rochester Spot Diagram
  • Laser Sensor Displacement = Local Slope of the Wavefront Spot Diagram Shack-Hartmann Wavefront Sensor Shack-Hartmann ~ Lightfields Levoy et al 2009 Zhang and Levoy 2009: Observable Light Field Oh, Raskar, Barbastathis 2009: Augmented Light Field
  • NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece
  • NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece
  • Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points
  • Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points but 3 parameters
  • Limitations
    • Children
    • Ability to align lines
      • Retina, Animals
    • Single Eye test
      • Other eye for convergence-forced accommodation
    • Resolution is a function of the display DPI
      • Samsung Behold II – 160 DPI – 0.35D
      • Google Nexus One – 250 DPI – 0.2D
      • Apple iPhone 4G – 326 DPI – 0.14D
  • Capture Analyze Display 5D: Looking around corners 6D: View and Lighting Aware 4D: Rank Deficient, multilayer 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field MIT Media Lab Ramesh Raskar http://raskar.info Shift Glass ` = WDF Light Field Augmented LF