FocusAR: Auto-focus AR eyeglasses for real world and virtual imagery
•Download as PPTX, PDF•
1 like•168 views
Slides of Best Paper Award winning "FocusAR: Auto-focus AR eyeglasses for real world and virtual imagery" presented at ISMAR 2018 in Munich.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Recently uploaded
Recently uploaded (20)
FocusAR: Auto-focus AR eyeglasses for real world and virtual imagery
- 1. 1
- 2. Future of personal computing 2Image credits: framesdirect.com FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 3. 3 Wide Field of View High Resolution High Dynamic Range Large Eyebox Low Power consumption Compact Eyeglasses formfactor Occlusion support Low Latency Rendering and tracking Compelling Application development Mobile Computing LocalizationNetworking Ambient scene reconstruction Focal accommodation for Real and Augmented Imagery FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 4. Naive AR Display Cannot focus on closer distances 4 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 5. Basic AR Display ● Fixed focus depth ● Limited field of view ● Simple Optics ● Computationally inexpensive 5 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 6. Basic AR Display ● Fixed focus depth ● Wide field of view ● Simple Optics ● Computationally inexpensive 6 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 7. Focus Supporting AR Display ● Dynamic focus depth ● Wide field of view ● Simple Optics ● Computationally (slightly) expensive 7 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 8. Dunn et al. 2017 Aksit et al. 2017 Varifocal AR Displays Matsuda et al. 2017 Focal Surface DisplayLight field Display Maimone et al. 2014 Maimone et al. 2017 Shi et al. 2017 Holographic AR Displays
- 9. Viewing through AR eyeglasses Assume internal display renders realistic virtual content ● People with normal vision: ○ Well-focused real world ● People with near or far sightedness: ○ Well-focused view with prescription lens ● People over 40 years of age (presbyopia): ○ Multi-focal prescription lenses ○ limited FoV per depth range UNCOMFORTABLE AR EXPERIENCE! 9 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 10. Presbyopes (over 40 years old) ● Lose lens plasticity and so accommodation ● Bifocals invented 200 years ago ● Credited to Benjamin Franklin 10Image credits: David Fleishman’s collection from Benjamin Franklin Museum Ben Franklin’s Bifocals (from 1790s) FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 11. Some numbers… ● Less than 50% of the US population has 20/20 vision ● About 40% of the US population are presbyopic ● At least 40% of the US adults are near or far sighted or astigmatic 11 https://www.aoa.org/documents/optometrists/CPG-17.pdf FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 12. 12 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 13. Automatic focus adjusting AR glasses 13 AR with Trifocals Intended AR glasses FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 14. 14 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs Alvarez eyeglasses Joshua Silver’s fluid lens Image credits: Amazon, uk.blouinartinfo.com
- 15. 20/20 Vision for all with AR display glasses 15 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 16. Vergence and Accommodation 16Image credits: http://www.forbestvision.com/accommodation-and-convergence/ Vergence distance: - Distance to which eyes foveate - Drives retinal disparity Accommodation or focus distance: - Distance to which the eyes focus - Drives retinal blur Vergence and Accommodation are neurally coupled. FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 17. 17 What people need? FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs Young Marie Curie with Normal vision Old Ben Franklin with Presbyopia Myopic ( Nearsighted ) Hyperopic ( Farsighted )
- 18. Vision in VR 18 Real world: Vergence distance = focal distance Clear vision She is happy :) Virtual Reality Display: Vergence distance ≠ focal distance Changes focus Gets headaches :( Real world: Vergence distance ≠ focal distance Blurry vision He is unhappy :( Virtual Reality Display: Vergence distance ≠ focal distance Set focus of HMD to prescription He is very happy :) Normal vision (Plenty accommodation power) Presbyope (No accommodation power) FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 19. Vision in AR 19 Internal AR display: Vergence distance ≠ focal distance Changes focus Can see AR content clearly Real world appears blurry Change focus to real world object AR content usually not in focus Internal AR display: Vergence distance ≠ focal distance Set focus of HMD to prescription Can see AR content clearly Can’t see real world clearly, anyway Can’t change focus to real world object AR content can be in focus Normal vision (Plenty accommodation power) Presbyope (No accommodation power) FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 20. AR is more problematic Virtual and real can’t be in focus together 20Image credits: Dunn et al 2017 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 21. Vision in AR 21 Can change focus to real world objects Needs adjustable-focus internal display for virtual content Needs focus adjustment for viewing real world Fixed focus internal display for virtual content is okay Normal vision (Plenty accommodation power) Presbyope (No accommodation power) FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 22. 22 What people need? FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs Normal vision Presbyopic User vision type External Real world correction Internal AR display correction
- 23. 23 What people need? FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs Normal vision Presbyopic User vision type External Real world correction Internal AR display correction NO correction Dynamic correction Dynamic correction Static correction
- 24. 24 What people need? FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs User vision type External Real world correction Internal AR display correction Normal vision NO correction Dynamic correction Presbyopic Dynamic correction Static correction Myopic ( Nearsighted ) Static correction Dynamic correction Hyperopic ( Farsighted ) Static correction Dynamic correction
- 25. Everybody needs some form of correction 25 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 26. Vision in AR 26 Needs adjustable-focus internal display for virtual content Know where the user is looking Adjust the focus of internal display Render images with computational blur Needs focus adjustment for viewing real world Know where the user wants to fixate Estimate the eye lens power Automatically adjust focus for real and virtual world Render images with computational blur Normal vision (Plenty accommodation power) Presbyope (No accommodation power) FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 27. Focus Supporting AR Display ● Dynamic focus depth ● Dynamic real world correction ● Limited field of view ● Simple Optics ● Computationally (slightly) expensive 27 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 28. Focus Supporting AR Display ● Dynamic focus depth ● Dynamic real world correction ● Wide field of view ● Simple Optics ● Computationally (slightly) expensive 28 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 29. Adjustable-focus internal display 29 Wide Field of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors (Dunn et al., 2017) FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs Membrane Dynamic Pressure System Membrane Tracking System Display Dynamic Pressure Module Rigid Transparent Surface Deformable Membrane Pressure Chamber Observer
- 30. Adjustable-focus internal display 30 Dunn et al 2017 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs Display (rear) Membranes Vacuum input Ours
- 31. Pressure (and Focus) Control 31 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 32. Focus correction for external world 32 Optotune focus-tunable liquid lens (Optotune EL-10-30-TC-VIS-12D) FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 33. Display Prototype 33 Dynamic focus display prototype FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 34. Results 34 ConventionalARHMDVarifocalARHMD Near focus Mid focus Far focus Users with normal vision FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 35. FarfocusNearfocus Results 35 ConventionalARHMDOursolution Near focus Mid focus Far focus Real Virtual NearfocusFarfocus FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 36. FarfocusNearfocus Results 36 VarifocalARHMDOursolution Near focus Mid focus Far focus Real Virtual NearfocusFarfocus FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs
- 37. ● Increase the Field of View ● Integrate eye trackers and outward looking cameras ● Better methods for quantifying the results ● Complex perceptual experiments to validate the properties ● Minimize the form factor to that of regular eyeglasses 37 Future Work
- 38. Thank You 38 FocusAR: Auto-focus AR glasses P. Chakravarthula, D. Dunn, K. Aksit, H. Fuchs Acknowledgements: Co-authors and collaborators, Madhumita Mahadevan and Jim Mahaney This research is supported in part by the Being Together Center, a collaboration between Nanyang Technological University (NTU) Singapore and University of North Carolina (UNC) at Chapel Hill and a grant from NVIDIA Research. The Being Together Center is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its International Research Centers in Singapore Funding Initiative.
- 39. 39 AR with Trifocals Our AR glasses Praneeth Chakravarthula cpk@cs.unc.edu www.cs.unc.edu/~cpk FocusAR: Auto-focus AR glasses For questions and collaborations…
Editor's Notes
- Hi everyone! I am Praneeth, graduate student from UNC Chapel Hill. My talk today is going to be on focus AR, auto-focus AR glasses for both real world and virtual imagery. Arrow from blurry to sharp. Without and with focus adjustment. Can spend 2-3 min explaining the slide. Move leonard to left end
- Today, many experts predict that VR, especially AR, will be the next mass market platform, in the subsequent steps after PCs, laptops and mobile phones. If AR indeed is the next platform, then these AR systems will have to be comfortable enough to be worn for long periods, perhaps all day like an [press enter] ordinary pair of eyeglasses. Otherwise people will just continue to carry their smartphones in their pockets. But, if AR does become like one’s everyday prescription glasses, then it will be used for applications consisting of viewing both 2D and 3D content, and this demands that the user has a comfortable view of both real and virtual worlds simultaneously. Although we are slowly getting there, we are still away from realizing such a future. But to give a context to the importance of VR/AR systems in future.. [next slide]
- An eyeglasses style AR display would demand many things to be done right. Few of such important things would be wide fov, high res, eyebox,… Of course all of which needs to be in an eyeglasses formfactor, which in itself is a major challenge demanding unconventional optical designs. Once we get these things right, we would have to work hard for few more features and for a good application platform, and so on and so forth. But one of the very crucial aspect of AR displays is the focal accommodation for both real and virtual imagery. Over the last decade, there has been significant research into providing focus cues for virtual imagery. Let us spare a moment and quickly look at AR display designs
- Many different solutions have been proposed to address this problem, which can be broadly classified into light field and holographic displays, and varifocal displays. Presenting accurate depth cues requires the full light field to be rendered statically. Unfortunately, today’s light field displays offer poor resolution and demand heavy computation. While holographic displays can provide accurate depth cues and also superior image quality, the heavy computation and small eyebox for a wide field of view make the state-of-the-art displays currently impractical for real use. On the other hand varifocal displays provide profound depth cues, wide eyebox and a large field of view, all at lower computation, but requires objects out of focus to be blurred artificially leading to unnatural images. Recent work on focal surface displays provide near-correct focus cues in VR by warping the target image surface into content adaptive arbitrary shapes using a phase SLM. Unfortunately, the compromised image quality and low frame rates make the technology far from practical for commercial use.
- Let us suppose we have cracked a solution for the virtual display, which produces content indistinguishable from the real world. Will it cater to the uses of all users? Let’s see.. People with normal vision will have a well-focused view of both real and virtual imagery - that is good People with near or far sightedness will have some difficulty in viewing at the imagery, but will have a well-focused view with prescription lens But people over 40 years of age, who would have lost significant amount of their accommodation due to thickening of their eye lens, will have to use multi-focal prescription lenses. And these, as you all know, will limit the usable FoV per depth range, which will cause an uncomfortable AR experience.
- These days we do not use a mobile phone for more than 2 years. But we have been using the same bifocal design for over 200 years.
- Now the current problem is not only that we do not have an ultimate virtual display, but even if we do have one, that might not be all sufficient. If we just look at the US population.. Speaking from a decade old available statistics, only less than 50% of US population has 20/20 vision About 40% are presbyopic. And al least 40% of the adult population, which we expect to be a significant chunk of VR and AR consumers, are either near sighted or far sighted, If neither they are suffering from astigmatism. And the sad fact is that everyone, every AR user, would require some kind of correction as summarized in this table. This is probably the most important table that you’d want to remember from this talk..
- One needs to wear prescription glasses above AR display for better vision. You could argue that there could be slots over AR display to keep prescription lenses. Well, that’s alright. But what about old people who needs multiple lenses.. [next]
- Trifocal lenses brings into focus various depths only in limited field of view. And the user has to tilt his head to view objects at various distances through specific parts of the lens. And we just agreed that we do not want manual adjusting focus tunable eyeglasses. And what we need is a pair of AR glasses in which both real and virtual appear well-focused over the entire field of view of the user, “automatically” This is a huge opportunity and is a capability that is never before achieved. In other words .. [click] #### Make clear that you want ability to focus at various depths
- One needs to wear prescription glasses above AR display for better vision. You could argue that there could be slots over AR display to keep prescription lenses. Well, that’s alright. But what about old people who needs multiple lenses.. [next]
- While we have a major advantage in future AR glasses with the onboard computing, multiple of both outward looking and inward looking sensors and trackers. Leveraging such things like outward tracking, depth sensing, etc to determine the user’s scene of interest, we can actively correct the vision for a well-focused view of real world as well as the virtual. ** notion of : if we are going to have multiple trackers and sensors and on board compute, we might as well correct for real world along with virtual imagery ** Also get across the notion that people since at least Ben Franklin, for last 2 centuries, have been trying for auto-focals, but future AR displays can make this possible We need 20/20 vision for all with AR display glasses.
- When you try to fixate on a real world object, your eyes roll to foveate the object - and this is called vergence The crystalline lens of the eye now tries to adjust its focal power in order to focus on the scene of interest - and this adjustment of your eye lens power is called accommodation. Vergence causes retinal disparity accommodation causes retinal blur and vergence and accommodation are neurally coupled - meaning one drives the other.
- Since t
- There is a vergence-accommodation conflict in VR displays
- Define Internal and External display here..
- Note how the virtual image tags on the book appears blurred but are in focus, when the user focuses at far depth.
- For example, recently, from our lab, David Dunn has prototyped a varifocal display which uses deformable beamsplitter membrane to place virtual objects at various depths in space, and won best paper and best demo awards at IEEE VR and Siggraph E-Tech. What about adjustable focus prescription lenses for viewing real-world? #### Try to get the video
- For example, recently, from our lab, David Dunn has prototyped a varifocal display which uses deformable beamsplitter membrane to place virtual objects at various depths in space, and won best paper and best demo awards at IEEE VR and Siggraph E-Tech. What about adjustable focus prescription lenses for viewing real-world? #### Try to get the video
- We use a subwoofer to modulate the airpressure in the membrane housing for each eye. A pressure sensor provides feedback on the current pressure differential between ambient atmosphere and inside the membrane housing, thereby getting rid of power-draining cameras for pressure control. A solenoid valve is used for re-pressurizing the system as needed for leak correction, which in observation typically occurs during continuous operation, about thrice and hour.
- Now let us look at the dynamic focus for both real and virtual. Today there are commercially available tunable focus liquid lenses, like the ones from optotune. These lenses are great for prototyping and can offer a range of focal lengths. But as you can see these have a small aperture size. However, the good thing is that recently there have been multiple designs of large aperture tunable lenses, and is an active research field. We hope advances here. For the internal display, there are already many accommodation-supporting designs as we have seen. Even this is an active research area and we hope advances for a much better focus supporting internal display. An integration of robust eye tracking and such dynamic focus lenses will enable active vision correction for both real and virtual in AR displays. ### Should be much earlier