Recommended
More Related Content
Similar to Computational Near-eye Displays with Focus Cues - SID 2017 Seminar
Similar to Computational Near-eye Displays with Focus Cues - SID 2017 Seminar (20)
More from StanfordComputationalImaging
More from StanfordComputationalImaging (17)
Recently uploaded
Recently uploaded (20)
Computational Near-eye Displays with Focus Cues - SID 2017 Seminar
- 1. Computational Near-eye Displays with Focus Cues Gordon Wetzstein Stanford University SID Display Week 2017 www.computationalimaging.org
- 3. Personal Computer e.g. Commodore PET 1983 Laptop e.g. Apple MacBook Smartphone e.g. Google Pixel AR/VR e.g. Microsoft Hololens ???
- 4. A Brief History of Virtual Reality 1838 1968 2012-2017 Stereoscopes Wheatstone, Brewster, … VR & AR Ivan Sutherland VR explosion Oculus, Sony, HTC, MS, … Nintendo Virtual Boy 1995 VR 2.0
- 5. Ivan Sutherland’s HMD • optical see-through AR, including: • displays (2x 1” CRTs) • rendering • head tracking • interaction • model generation • computer graphics • human-computer interaction I. Sutherland “A head-mounted three-dimensional display”, Fall Joint Computer Conference 1968
- 6. Where we are now IFIXIT teardown
- 8. Tutorial Overview • conventional, fixed-focus near-eye displays • focus cues & the vergence-accommodation conflict • advanced optics for VR with focus cues: • adaptive and gaze-contingent focus displays • monovision • volumetric and multi-plane displays • near-eye light field displays • Maxwellian-type displays • AR displays
- 10. Magnified Display 1 d + 1 d' = 1 f d d’ f • big challenge: virtual image appears at fixed focal plane! • no focus cues
- 12. Duane, 1912 Nearestfocusdistance Age (years) 8 16 24 32 40 48 56 64 72 4D (25cm) 8D (12.5cm) 12D (8cm) Importance of Focus Cues Decreases with Age - Presbyopia 0D (∞cm) 16D (6cm)
- 13. 4D / 25cm Optical Infinity Normal vision Nearsighted/myopic Farsighted/Hyperopic Presbyopic Focal range (range of clear vision) Modified from Pamplona et al, Proc. of SIGGRAPH 2010 Nearsightedness & Farsightedness
- 14. Relative Importance of Depth Cues Cutting&Vishton,1995
- 15. The Vergence-Accommodation Conflict (VAC)
- 23. Current VR Displays: Vergence & Accommodation Mismatch
- 24. Accommodation and Retinal Blur
- 25. Blur Gradient Driven Accommodation
- 26. Blur Gradient Driven Accommodation
- 27. Blur Gradient Driven Accommodation
- 28. Blur Gradient Driven Accommodation
- 29. Blur Gradient Driven Accommodation
- 30. Blur Gradient Driven Accommodation
- 31. Top View Real World: Vergence & Accommodation Match!
- 32. Top View Stereo Displays Today (including HMDs): Vergence-Accommodation Mismatch! Screen
- 33. VR Displays with Focus Cues 1. Adaptive and Gaze-contingent Focus
- 36. Adaptive Focus Magnified Display Display Lens focus-tunable lens vary f 1 d + 1 d' = 1 f
- 37. Adaptive Focus - History • M. Heilig “Sensorama”, 1962 (US Patent #3,050,870) • P. Mills, H. Fuchs, S. Pizer “High-Speed Interaction On A Vibrating-Mirror 3D Display”, SPIE 0507 1984 • S. Shiwa, K. Omura, F. Kishino “Proposal for a 3-D display with accommodative compensation: 3DDAC”, JSID 1996 • S. McQuaide, E. Seibel, J. Kelly, B. Schowengerdt, T. Furness “A retinal scanning display system that produces multiple focal planes with a deformable membrane mirror”, Displays 2003 • S. Liu, D. Cheng, H. Hua “An optical see-through head mounted display with addressable focal planes”, Proc. ISMAR 2008 manual focus adjustment Heilig 1962 automatic focus adjustment Mills 1984 deformabe mirrors & lenses McQuaide 2003, Liu 2008
- 38. Padmanaban et al., PNAS 2017
- 39. Padmanaban et al., PNAS 2017
- 40. Padmanaban et al., PNAS 2017
- 41. Padmanaban et al., PNAS 2017
- 42. Padmanaban et al., PNAS 2017
- 43. Padmanaban et al., PNAS 2017
- 44. Padmanaban et al., PNAS 2017
- 48. at ACM SIGGRAPH 2016 EyeNetra.com
- 49. at ACM SIGGRAPH 2016 participants of the study, 152 total EyeNetra.com
- 50. Participants - Prescription Padmanaban et al., PNAS 2017 n = 70, ages 21-64
- 51. vergence accommodation Conventional Stereo / VR Display
- 52. vergence accommodation Removing VAC with Adaptive Focus
- 53. Follow the target with your eyes 4D (0.25m) 0.5D (2m) Task
- 54. Stimulus Padmanaban et al., PNAS 2017 Accommodative Response RelativeDistance[D] Time [s]
- 55. Stimulus Accommodation n = 59, mean gain = 0.29 Padmanaban et al., PNAS 2017 Accommodative Response RelativeDistance[D] Time [s]
- 56. Stimulus Padmanaban et al., PNAS 2017 Accommodative Response RelativeDistance[D] Time [s]
- 57. Stimulus Accommodation Padmanaban et al., PNAS 2017 Accommodative Response RelativeDistance[D] Time [s] n = 24, mean gain = 0.77
- 58. Padmanaban et al., PNAS 2017 Do Presbyopes Benefit from Dynamic Focus? Gain Age
- 59. Padmanaban et al., PNAS 2017 Do Presbyopes Benefit from Dynamic Focus? Gain Age conventional
- 60. Padmanaban et al., PNAS 2017 Do Presbyopes Benefit from Dynamic Focus? Gain Age conventional dynamic
- 61. Gaze-contingent Focus • non-presbyopes: adaptive focus is like real world, but needs eye tracking! HMD lens micro display virtual image eye tracking Padmanaban et al., PNAS 2017
- 62. Gaze-contingent Focus Padmanaban et al., PNAS 2017
- 63. Gaze-contingent Focus Padmanaban et al., PNAS 2017
- 64. Gaze-contingent Focus Padmanaban et al., PNAS 2017
- 65. at ACM SIGGRAPH 2016
- 66. Gaze-contingent Focus – User Preference Padmanaban et al., PNAS 2017
- 67. VR Displays with Focus Cues 2. Monovision
- 68. Monovision VR Konrad et al., SIGCHI 2016; Johnson et al., Optics Express 2016; Padmanaban et al., PNAS 2017
- 69. Monovision VR Konrad et al., SIGCHI 2016; Johnson et al., Optics Express 2016; Padmanaban et al., PNAS 2017 • monovision did not drive accommodation more than conventional • visually comfortable for most; particularly uncomfortable for some users
- 70. VR Displays with Focus Cues 3. Multiplane Displays
- 71. Multiplane VR Displays • Rolland J, Krueger M, Goon A (2000) Multifocal planes head-mounted displays. Applied Optics 39 • Akeley K, Watt S, Girshick A, Banks M (2004) A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH) • Waldkirch M, Lukowicz P, Tröster G (2004) Multiple imaging technique for extending depth of focus in retinal displays. Optics Express • Schowengerdt B, Seibel E (2006) True 3-d scanned voxel displays using single or multiple light sources. JSID • Liu S, Cheng D, Hua H (2008) An optical see-through head mounted display with addressable focal planes in Proc. ISMAR • Love GD et al. (2009) High-speed switchable lens enables the development of a volumetric stereoscopic display. Optics Express • … many more ... idea introduced Rolland et al. 2000 benchtop prototype Akeley 2004 near-eye display prototype Liu 2008, Love 2009
- 72. Multiplane VR Displays • Rolland J, Krueger M, Goon A (2000) Multifocal planes head-mounted displays. Applied Optics 39 • Akeley K, Watt S, Girshick A, Banks M (2004) A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH) • Waldkirch M, Lukowicz P, Tröster G (2004) Multiple imaging technique for extending depth of focus in retinal displays. Optics Express • Schowengerdt B, Seibel E (2006) True 3-d scanned voxel displays using single or multiple light sources. JSID • Liu S, Cheng D, Hua H (2008) An optical see-through head mounted display with addressable focal planes in Proc. ISMAR • Love GD et al. (2009) High-speed switchable lens enables the development of a volumetric stereoscopic display. Optics Express • … many more ... idea introduced Rolland et al. 2000 benchtop prototype Akeley 2004 near-eye display prototype Liu 2008, Love 2009
- 73. VR Displays with Focus Cues 4. Light Field Displays
- 74. Light Field CamerasLight Field Stereoscope Huang et al., SIGGRAPH 2015
- 75. Backlight Thin Spacer & 2nd panel (6mm) Magnifying Lenses LCD Panel Light Field Stereoscope Huang et al., SIGGRAPH 2015
- 76. Near-eye Light Field Displays Idea: project multiple different perspectives into different parts of the pupil!
- 77. Target Light Field Input: 4D light field for each eye
- 78. Multiplicative Two-layer Modulation Input: 4D light field for each eye
- 79. Multiplicative Two-layer Modulation Input: 4D light field for each eye
- 80. Multiplicative Two-layer Modulation Input: 4D light field for each eye
- 81. Multiplicative Two-layer Modulation Reconstruction: for layer t1 Tensor Displays, Wetzstein et al. 2012 Input: 4D light field for each eye
- 82. Traditional HMDs - No Focus Cues The Light Field HMD Stereoscope Light Field Stereoscope Huang et al., SIGGRAPH 2015
- 83. Traditional HMDs - No Focus Cues The Light Field HMD Stereoscope Light Field Stereoscope Huang et al., SIGGRAPH 2015
- 84. Traditional HMDs - No Focus Cues The Light Field HMD Stereoscope Light Field Stereoscope Huang et al., SIGGRAPH 2015
- 85. Traditional HMDs - No Focus Cues The Light Field HMD Stereoscope Light Field Stereoscope Huang et al., SIGGRAPH 2015
- 86. Tensor Displays Wetzstein et al., SIGGRAPH 2012
- 87. Vision-correcting Display iPod Touch prototypeprinted transparency Huang et al., SIGGRAPH 2014
- 88. prototype 300 dpi or higher Huang et al., SIGGRAPH 2014
- 89. VR Displays with Focus Cues 5. Maxwellian-type Displays
- 90. Maxwellian-type Near-eye Displays • eyebox of display is a pinhole very large depth of field (no retinal blur cue) • exit pupil size of ≤0.5 mm accommodation in open loop • pinholes are dim and reduce eyebox severely! (not practical)
- 93. Accommodation-invariant Near-eye Displays Konrad et al., SIGGRAPH 2017
- 97. Accommodation-invariant Near-eye Displays Konrad et al., SIGGRAPH 2017
- 98. Other Upcoming Work at SIGGRAPH 2017 “Focal Surface Displays”, Matsuda et al. “Accommodation and Comfort in Head Mounted Displays”, Koulieris et al. “Holographic Near-eye Displays for Virtual and Augmented Reality”, Maimone et al.
- 99. Summary • focus cues in VR/AR are challenging • adaptive focus can correct for refractive errors (myopia, hyperopia) • gaze-contingent focus gives natural focus cues for non-presbyopes, but require eyes tracking • presbyopes require fixed focal plane with correction • multiplane displays require very high speed microdisplays • monovision has not demonstrated significant improvements • Maxwellian-type displays can be interesting, but provide small eyebox • light field displays may be the “ultimate” display need to solve “diffraction problem”
- 100. Overview of Optical See-through AR Displays
- 101. slidebyBernardKress Consumer Applications – Prescription Glasses
- 102. Google Glass small!
- 103. Google Glass
- 104. Pepper’s Ghost 1862
- 105. Microsoft Hololens – Waveguide with Holographic Optical Elements
- 106. Oculus DK2 (115 deg) FOV (diag) and FOV location in some of the current products Sony Morpheus (90 deg) Sony HMZ-T3 (51deg) Lumus DK-32 (40 deg) Zeiss cinemizer (35 deg) Optinvent ORA (24 deg) Epson Moverio (23 deg) Occlusion display See through display Google Glass (15deg) Vuzix M-100 (16deg) Recon Jet (16deg) slidebyBernardKress
- 107. FOV and angular resolution for current productsAngularresolution(pixels/deg) FOV (diag, deg) Eye resolution limit (1.2 arc min or around 50 pixels/deg) 50 40 30 20 10 10 20 30 40 50 60 70 80 90 100 110 Glass Epson Moverio Vuzix m100 Oculus DK1 Sony HMZ T3 Optinvent Lumus Sony Morpheus Resolution measured in dpd rather than dpi Oculus DK2 slidebyBernardKress
- 108. There are two main optical HMD architectures slidebyBernardKress
- 109. Opaque Bird bath Free space Waveguide EPE Lightguide There are 5 main head-worn optical platform architectures developed in industry today See through architectures slidebyBernardKress
- 110. Oculus DK2 Sony HMZ Vuzix Occlusion Huge FOV Large eyebox Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 111. slidebyBernardKress
- 112. Oculus DK2 Sony HMZ Vuzix Vuzix M100 MyVu … Occlusion Huge FOV Large eyebox Partially occluded Small FOV Medium eye box Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 113. Most of the early occlusion HMDs are based on the combination of a transmission lens and a 45 degrees mirror (either TIR or coated) , in either monocular or binocular version. MicroOptical Corp. – MyVu Corp Recon Instruments Corp. slidebyBernardKress
- 114. “Bird bath” optical architectures Google Glass RockChip Ltd ITRI Good see through Small FOV Medium eye box Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 116. Laster Sarl ODA Labs … Bug eye optics Large FOV Medium eyebox Temple projector Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 117. The notion of « Bug Eye » in HMD combiner opticsslidebyBernardKress
- 118. This is what we mean with the « bug-eye » concept slidebyBernardKress
- 119. ORA Associates (US) Reflective “bug eye” combiner Binocular version Laster Sarl (Fr) Reflective combiner (pupil forming architecture) slidebyBernardKress
- 120. Canon Ltd Motorola HC1 Kopin Golden eye Distorted see-through (or opaque) Medium FOV Medium eyebox Without complement piece Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 121. Canon Ltd Motorola HC1 Kopin Golden eye E-Magin Optics Fraunhofer Distorted see-through (or opaque) Medium FOV Medium eyebox Good see through Medium FOV Medium eye box Without complement piece With complement piece Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 122. TIR / free-form surfaces combiner (including eye tracking) Example with complement piece for see through slidebyBernardKress [Hua et al. 2013]
- 123. Epson Ltd Moverio 1 & 2 … OK see through Medium FOV Medium eyebox Curve coated reflector combiner Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 124. Epson Corp, Moverio (first and second generations) 3 collimation lenses, prism injector, 6 TIR bounces and curved 50/50 mirror Conventional optics (linear mirror in first gen and curved mirror in second gen) slidebyBernardKress
- 125. Optinvent Sarl See through OK Medium FOV Large eyebox Injection molded Lumus Ltd See through OK Medium FOV Large eyebox All glass various coatings Micro-prism combiner Cascaded coated mirrors combiner Opaque Bird bath Free space slidebyBernardKress Waveguide EPE Lightguide
- 126. Word side 6 mirrorsWorld side Lumus Ltd, DK32 (40deg FOV, 1.7mm) and DK40 (25 deg FOV. 2.0mm) (cascaded mirrors extractors) Uses side collimator and prism coupler slidebyBernardKress
- 127. Optinvent Sarl See through OK Medium FOV Large eyebox Injection molded Lumus Ltd See through OK Medium FOV Large eyebox All glass various coatings Sony Ltd Vuzix /Nokia (M2000AR) BAE Q-sight Good see through Medium FOV Large eyebox Photopolymer Micro-prism combiner Cascaded coated mirrors combiner Volume holographic combiner Diffractive combiner Opaque Bird bath Free space Waveguide EPE Lightguide slidebyBernardKress
- 128. H. Mukawa et al. A full color eyewear display using holographic planar waveguides. Proc. SID 2008.
- 129. (Some) Technology Challenges • Vergence-accommodation conflict (VAC) • Vestibular-visual conflict (motion sickness) • AR • occlusions • aesthetics / form factor • battery life • heat • wireless operation • low-power computer vision • registration of physical / virtual world and eyes • consistent lighting • scanning real world • VAC more important • display contrast & brightness • fast, embedded GPUs • …
- 130. Exciting Engineering Aspects of VR/AR • CPU, GPU • IPU, DPU? • sensors & imaging • computer vision • scene understanding • human perception • displays: visual, auditory, vestibular, haptic, …• VR cameras • cloud computing • shared experiences • HCI • compression, streaming
- 131. Stanford EE 267
- 132. Stanford Computational Imaging Group
- 133. Stanford Computational Imaging Lab Light Field Displays Time-of-Flight & NLOS Imaging Computational Microscopy Image Optimization Light Field Cameras
- 134. O’Toole et al., CVPR 2017
- 135. ProcessedMeasured
- 136. ProcessedMeasured
- 137. Gordon Wetzstein Computational Imaging Group Stanford University stanford.edu/~gordonwz www.computationalimaging.org