The document discusses advancements in computational displays at the MIT Media Lab, focusing on multi-dimensional light transport and new optical technologies like shift glass, which allows viewing around corners. It introduces concepts such as femto-photography, augmented light fields, and interactive displays for refractive error assessment through NETRA technology. Various experimental techniques and applications for enhancing optical performance are highlighted, along with insights into the optimization of display technologies.

1. Camera Culture Ramesh Raskar Camera Culture MIT Media Lab Computational Displays in 4D, 6D and 8D

2. Slow Glass: Time Shift http:// baens-universe.com/articles/otherdays Light of Other Days by Bob Shaw http://www.fantasticfiction.co.uk/s/bob-shaw/other-days-other-eyes.htm

3. Shift Glass

4. Shift Glass Space Shifting Angle Shifting Time Shifting Illumination Shifting 4D 4D t 4D 4D

5. Capture Analyze Display Shift Glass

6. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field Shift Glass

7. Can you look around a corner ? Without any device in the line of sight

8. Femto-Photography: Higher Dimensional LF FemtoFlash UltraFast Detector Computational Optics Serious Sync

9. Kirmani, Hutchinson, Davis, Raskar ICCV’2009, Marr Prize Honorable Mention

10. Streak Camera = Inverse of CRT/CRO Femto-laser

11. Multi-Dimensional Light Transport 5-D Transport

13. Rescue and Planning

14. Robot, Car Path Planning

15. Endoscopy

16. Camera Culture Ramesh Raskar Team Moungi G. Bawendi, Professor, Dept of Chemistry, MIT James Davis, UC Santa Cruz Andreas Velten, Postdoctoral Associate, MIT Media Lab Ahmed Kirmani, RA, MIT Media Lab Tyler Hutchison, RA, MIT Media Lab Rohit Pandharkar, RA, MIT Media Lab Andrew Matthew Bardagjy, RA, MIT Media Lab Everett Lawson, MIT Media Lab Ramesh Raskar, MIT Media Lab

17. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field Shift Glass

18. Capture Analyze Display 5D: Looking around corners 4D: Plenoptic Camera 3D: Flutter Shutter Camera 6D: View and Lighting Aware 4D: Rank Deficient 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field

19. Slow Display

20. Light Reactive Monostable Materials 16 Megapixel, 2 Watt

21. Day/Night visible

22. g SlowDisplay.org Saakes, Chiu, Hutchison, .., Inami, Raskar, Siggraph 2010 Etech Demo

23. 6D Photo Frames One Pixel of a 6D Display = 4D Display Single Pixel of 6D Frame Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch 1 2 1 1 2D 2D 2D

24. Respond to Viewpoint + Ambient Light

25. 6D Display Light sensitive 4D display One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008

26. Beyond Multi-touch: Thin LCD for touch+hover Laptops Mobile

27. BiDi Screen: Multi-touch + Hover 3D interface

28. Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing Matthew Hirsch, Henry Holtzman Doug Lanman, Ramesh Raskar Siggraph Asia 2009 BiDi Screen

29. Theory: Benefits of Tiled-broadband Masks 0 10 20 30 40 50 60 0 0.1 0.2 0.3 0.4 0.5 0.6 Angular Resolution Average Transmission (%) Pinholes Sum-of-Sinusoids MURA 11x11 23x23 43x43 Angular Resolution Tiled-Broadband Code MURA Sum-of-Sinusoids Pinholes

30. Overview: Sensing Depth from Array of Virtual Cameras in LCD

31. Bits Photons CV / Machine Learning Optics Sensors Computational Displays Signal Processing Light Transport Displays HCI

33. View Dependent Appearance and Iridescent color Cross section through a single M. rhetenor scale

34. Two Layer Displays PB = dim displays Lenslets = fixed spatial and angular resolution Dynamic Masks = Brighter, High spatial resolution barrier sensor/display lenslet sensor/display

35. Limitations of 3D Display Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010 Front Back Parallax barrier LCD display

36. ` Light Field Analysis of Barriers L[i,k] L[i,k] i k light box g[k] k f[i] i

37. f[i] g[k] L[i,k] light box ` Content-Adaptive Parallax Barriers k i

38. Implementation Components 22 inch ViewSonic FuHzion VX2265wm LCD [1680×1050 @ 120 fps]

39. f[i] g[k] L[i,k] light box ` Content-Adaptive Parallax Barriers k i

40. = Content-Adaptive Parallax Barriers `

41. Rank-Constrained Displays and LF Adaptation All dual layer display = rank-1 constraint Light field display is a matrix approximation problem Exploit content-adaptive parallax barriers Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010 ` = Content-Adaptive Parallax Barriers

42. rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Optimization: Iteration 1 Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

43. Optimization: Iteration 10 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

44. Optimization: Iteration 20 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

45. Optimization: Iteration 30 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

46. Optimization: Iteration 40 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

47. Optimization: Iteration 50 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

48. Optimization: Iteration 60 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

49. Optimization: Iteration 70 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

50. Optimization: Iteration 80 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

51. Optimization: Iteration 90 rear mask: f 1 [i,j] front mask: g 1 [k,l] reconstruction (central view) Daniel Lee and Sebastian Seung. Non-negative Matrix Factorization. 1999. Vincent Blondel et al . Weighted Non-negative Matrix Factorization. 2008.

52. Content-Adaptive Front Mask (1 of 9)

53. Content-Adaptive Rear Mask (1 of 9)

54. Emitted 4D Light Field

55. Conclusion Described a rank constraint for all dual-layer displays With a fixed pair of masks, emitted light field is rank- 1 Achieved higher-rank approximation using temporal multiplexing With T time-multiplexed masks, emitted light field is rank- T Constructed a prototype using off-the-shelf panels Demonstrated light field display is a matrix approximation problem Introduced content-adaptive parallax barriers Applied weighted NMF to optimize weighted Euclidean distance to target Adaptation increases brightness and refresh rate of dual-stacked LCDs ` = Content-Adaptive Parallax Barriers

56. Lightfield vs Hologram Displays

57. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?

58. Parallax Barrier: N p =10 3 pix. Hologram: N H =10 5 pix. θ p =10 pix w θ H =1000 pix ϕ P ∝w/d ϕ H ∝λ/t H Fourier Patch Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009

59. Augmenting Plenoptic Function Wigner Distribution Function Traditional Light Field WDF Traditional Light Field Augmented LF Interference & Diffraction Interaction w/ optical elements ray optics based simple and powerful wave optics based rigorous but cumbersome Oh, Raskar, Barbastathis 2009: Augmented Light Field

60. Light Fields Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters Reference plane position angle LF propagation (diffractive) optical element LF LF LF LF LF propagation light field transformer

61. Augmented Lightfield for Wave Optics Effects Wigner Distribution Function Light Field LF < WDF Lacks phase properties Ignores diffraction, interferrence Radiance = Positive ALF ~ WDF Supports coherent/incoherent Radiance = Positive/Negative Virtual light sources LF Augmented Light Field WDF

62. Free-space propagation Light field transformer Virtual light projector Possibly negative radiance

63. L(x, θ ) W(x,u) W m = sinc d = delta q W m p q p d(θ) p q q d(θ) * p W m * * Rays: No Bending 1 Fresnel HG Patch θ u * Zooming into the Light Field

64. Algebraic Rank Constraint s 1 m 2 s 1 m 2 s 1 * s 1 s 1 s 1 * Rank-1 Rank-1 Rank-3

65. - Transform <t(x+x ʹ /2)t*(x-x ʹ /2)> Interference xʹ x (a) Two Slits, Coherent t(x+x ʹ /2)t*(x-x ʹ /2) W(x,u) Rank-1 t(x 1 )t*(x 2 ) R 45 , D Transform -1 u

66. L2 L1 L3 ϕ 1 ϕ 1 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 h H r z 2 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) s 1 m 2 (a)

67. A B C Vary Illumination Direction: -5 ̊ , 0 ̊, 5 ̊ A B C A … -5 ̊ 5 ̊ 0 ̊ No Slits 24mm 36mm t H =25μm w=125μm z H =10cm (c)

68. M2 M1 M3 ϕ 1 ϕ 1 L 1 (x,θ) L 2 (x,θ) L 3 (x,θ) d z 1 r z 2 s 1 m 2 s 1 m 2 s 1 m 2 s 1 * s 1 s 1 s 1 * Rank-1 Rank-1 Rank-3

69. Is hologram just another ray-based light field? Can a hologram create any intensity distribution in 3D? Why hologram creates a ‘wavefront’ but PB does not? Why hologram creates automatic accommodation cues? What is the effective resolution of HG vs PB?

70. NETRA: Interactive Display for Estimating Refractive Errors and Focal Range Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar

71. Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar NETRA: N ear E ye T ool for R efractive A ssessment

72. 0.6B uncorrected refractive errors NETRA at LVP Eye Institute 6.5 Billion people 4.5B with Mobile phone 2B refractive errors

73. * Phoropter-based: $5,000.00 Needs expert, Moving parts, Shining lasers Retino scope w/ Lenses Auto-refracto-meter Chart with Lenses In-Focus: Focometer Optiopia Solo-health: EyeSite NETRA Technology Shining Light plus lenses Fundus Camera Moving lenses + target Moving lenses + target Reading chart on monitor Cellphone + eyepiece Cost to buy $2,000* ~$10,000 ~$100 ~$495 ~$200 -- $30 Cost per test ~$36 ~$36 ~$5 -- -- -- ~$1 Data capture No Comp. No No No Comp. Phone Mobility <500g >10Kg 2kg 1kg <5kg >10Kg <100g Speed Fast Fast Medium Medium -- Fast Fast Scalability No No No Yes Probably No Yes Accuracy 0.15 0.15 0.5 0.75 -- -- <0.5 Self evaluation No No Yes Yes Yes Yes Yes Electricity Req No Yes No No -- Yes No Astigmatism Yes Yes Yes/No No -- Yes Yes Network No Yes No No No Yes Yes Training High High High Medium Medium Low Low

74. Shack-Hartmann Wavefront Sensor Expensive; Bulky, Requires trained professionals Wavefront aberrometer

75. Shack-Hartmann Wavefront Sensor Laser Sensor Microlens Array Planar Wavefront Shack & Platt 1971 Liang et al 1994 David Williams et al, Rochester Spot Diagram

76. Laser Sensor Displacement = Local Slope of the Wavefront Spot Diagram Shack-Hartmann Wavefront Sensor Shack-Hartmann ~ Lightfields Levoy et al 2009 Zhang and Levoy 2009: Observable Light Field Oh, Raskar, Barbastathis 2009: Augmented Light Field

77. NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece

78. NETRA = Inverse of Shack-Hartmann Spot Diagram on LCD Cell Phone Display Eye Piece

79. Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points

80. Spot Diagram on LCD Inverse of Shack-Hartmann User interactively creates the Spot Diagram Displace 25 points but 3 parameters

81. Limitations Children Ability to align lines Retina, Animals Single Eye test Other eye for convergence-forced accommodation Resolution is a function of the display DPI Samsung Behold II – 160 DPI – 0.35D Google Nexus One – 250 DPI – 0.2D Apple iPhone 4G – 326 DPI – 0.14D

82. Capture Analyze Display 5D: Looking around corners 6D: View and Lighting Aware 4D: Rank Deficient, multilayer 4D: Netra for Optometry 4D, 6D, 8D: Augmented Light Field MIT Media Lab Ramesh Raskar http://raskar.info Shift Glass ` = WDF Light Field Augmented LF