Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Tim Leland (Qualcomm): The Mobile Future of Extended Reality (XR)
Similar to Tim Leland (Qualcomm): The Mobile Future of Extended Reality (XR) (20)
More from AugmentedWorldExpo
More from AugmentedWorldExpo (20)
Recently uploaded
Recently uploaded (20)
Tim Leland (Qualcomm): The Mobile Future of Extended Reality (XR)
- 1. The Mobile Future of eXtended Reality (XR) Tim Leland Vice President, Product Management Qualcomm Technologies, Inc. May 31, 2017
- 2. 2 XR is the future Healthcare Industrial & manufacturing Education Military Retail Engineering Marketing & advertising Emergency response XR AR VR MR Entertainment
- 3. 3
- 4. 4
- 5. 5 Will the smartphone become an XR wearable? Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.
- 6. 6 XR is here today, but it is still in its infancy Analogy to smartphones: XR evolution will take years…opportunity will be immense XR will follow a similar ~30 year cycle of sleeker designs, with tremendously increasing functionality Technology Phase: Infancy Market: Mostly early adopter “Prosumers” Technology Phase: Rapid evolution Market: Surging consumer adoption Technology Phase: Maturity Market: Worldwide, ubiquitous use XR is here today XR by ~2020
- 7. 7 Solving the key XR technology challenges ahead Common illumination Making virtual objects in augmented worlds indistinguishable from real objects within the same view Connectivity The next level of ubiquitous, wireless connectivity for anywhere usage at fiber-optic speeds Motion tracking Intelligent, completely on-device tracking for intuitive head, hands, and eye interactions Display Displaying richer visual content, and switching seamlessly between fully and partially virtual worlds Power and thermal All day battery life, years of recharging, and compatible with sleek, thin, light passively cooled devices with no fans
- 8. 8 XR human factors challenges for displays Vergence and accommodation conflict and human field of view (FoV) Field of view (binocular) in XR glasses 190° 130° Vergence & accommodation
- 9. 9 Future XR needs a disruption in display technology First step towards high volume converged XR form factor are new displays • Solve the vergence accommodation conflict • Deliver necessary FoV both for immersive VR and useful AR • Be completely opaque for VR, yet at least ~85% transparent for AR • Support an angular resolution of at least 0.5 – 1.0 arc minutes per pixel • Drive HDR, at least Rec. 2020 gamut, with ~5X improvement in nits • Be capable of refreshing at a minimum of ~120Hz (per eye) • Be light, mechanically flexible, very durable, and eventually cost under ~$100 at very high volumes
- 10. 10 XR display questions that need answers • Can LCOS or DLP with a mechanical shutter, better optics, and improved wave guides get us there? • Can plastic AMOLED get us there? Can smaller transistor sizes improve transparency for AR? • Or should the industry be working together on something more exotic to meet these needs? • What’s the best technology to switch between opaque and nearly transparent display modes?
- 11. 11 Common illumination makes virtual objects look real
- 12. 1212 Where we are today Virtual objects look fake • In part due to mostly static lighting that’s often incorrect for the environment • Even when dynamic, the graphics shader’s lights don’t consistently match real world light sources or intensity • Consequently objects and materials look physically incorrect for the scene • It is always immediately obvious which objects are real and which are virtual
- 13. 1313 Where we must eventually be Virtual objects must look real • With sampled light from cameras or ALS used to determine final color of every pixel in the virtual object • Virtual lights should be very frequently updated with real world lights to be perceptually correct for real environment Making it possible • New, more intelligent, faster interaction between many different sensors and rendering systems • New computer vision and global illumination algorithms that use real world lights to dynamically render and overlay more realistic virtual objects
- 14. 1414 Taking immersive mobile XR experiences to the next level will require: • Improved head/body tracking o User friendly, inside-out 6 DoF head tracking o Power efficient, sub-10ms motion to photon latency with sub-millimeter drift o Functional at world scale with capability to appropriately alert for collision avoidance • Improved eye tracking o Automatic IPD calibration o Tracking accuracy for foveation/depth of field rendering and viewport aware video o Also for more natural intent-based interaction and interfaces Improvements needed in motion tracking
- 15. 1515 Intuitively interacting in virtual worlds • Controllers, when required, must be 6 DoF, responsive and low cost • However, for most use cases, the best controller is no controller Other improvements needed in motion tracking Making it possible • New, better 3DR so that virtual hands in VR mode look just like your own • CV, machine learning, and graphics convergence so they work just like them too
- 16. 16 Power and thermal efficiency is essential for XR The XR headset needs to be appropriate to wear and use all day Constrained mobile, wearable environment Must be thermally efficient for sleek, ultra-light designs Requires long battery life for all-day use Able to be quickly recharged at least 1,000 times The challenge of XR workloads Very compute intensive Complex concurrencies Always-on Real-time
- 17. 17 So how do we get there? Power & thermal efficiency is essential for XR • IHVs: Double perf/watt every ~3 years • Battery companies: Improve battery capacity per gram by at least ~5X over the next 10 years • Researchers: Innovate in mobile HMD materials science and passive cooling • App developers: Tune your code to be more power efficient • Consortiums: Standardize XR multimedia compression, foveation, and other areas that save power and boost performance • Network operators: Provide much more efficient wireless connectivity access to internet and cloud services
- 18. 18 5G enhanced mobile broadband is required for XR mass adoption Extreme throughput— multi Gbps Ultra-low latency — down to 1 ms Uniform experience — even at cell edge XR video will be the killer use case for 5G ~50 to 200 Mbps, lower latency Next-generation (2019) 3D 360° 8K/30fps viewport-aware HDR10 video ~10 to 50 Mbps Current-generation 360° 4K/30fps video ~200 to 5000 Mbps, very low latency Next-decade Interactive, real-time 3D “Free-Viewpoint” 6-DoF 8K/90-120fps HDR-next video Sustained network performance
- 19. 1919 XR is the next mobile computing platform Many technology breakthroughs are required Call to arms for XR market acceleration: cooperation is key
- 20. 2020 “You can’t depend on your eyes when your imagination is out of focus” - Mark Twain
- 21. Follow us on: For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog Thank you Nothing in these materials is an offer to sell any of the components or devices referenced herein. ©2017 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm and Snapdragon are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.
Editor's Notes
- Introduce self and role at Qualcomm - I’d like to start out by thanking the AWE planning committee for opportunity to present Qualcomm's vision for XR.
- - There’s lot of terminology already in this space, and we'd like to simplify it. - Extended Reality, or "XR" is not a new term - it’s an emerging umbrella term that is already being used to encapsulate AR, VR and everything in between. - XR isn't only a set of technologies but it's also used to describe new market verticals within industrial manufacturing, healthcare education, military, engineering, retail, marketing & advertising, and emergency services - There may be other talks at AWE that dive into each of these markets – but they won’t be our focus here. - Over the next 20 minutes we’ll focus on a few of the multidisciplinary technical challenges that we need to solve together for XR reach its full potential
- - We think a new wave of XR glasses is coming within the next few years that will be much sleeker and lighter. - Perhaps some of the models could be customized for specific jobs or functions. - Let‘s see an example of the sort of functionality one might expect from future XR glasses designed for "first responders" <<start animation>> - We think a “first responder” head mounted display will contain a tremendous amount of processing, connectivity, and sensors. - They’ll likely be built with new optics and projection technologies within a durable, display that can switch from being opaque to transparent. - Such a headset may have bright LED flashlights on the front, and hardware to help first responders see in the dark, detect areas of heat, or detect other hazards. - They’ll likely have many cameras, for eye, head and hand-motion tracking as well as recording and sensing objects outside of the users view. - Connectivity will likely be multimode with 5G support, for efficient interaction with mission critical cloud services.
- <<HMD animation from previous slide ends with this still image. Click to transition to first responder use case animation>> Imagine being a first responder at a hotel fire. There are many uncertainties when you show up at the scene – poor lighting, unknown fire conditions, unknown terrain, and unknown casualties and potential survivors. Technology could aid in helping with a lot of these challenges, from knowing how hot surface are, to seeing in the dark and through smoke, to being able to find survivors, and knowing both the terrain and where everyone on your rescue team is located. XR will help with all of this.
- - Most of us aren’t firemen though, so let’s ask a multi-billion dollar question: will the everyday, mass-market, smartphone for the common consumer eventually become an XR wearable? - At some point in the next decade, we expect the complete convergence of mobile VR and AR into a single HMD capable of both. - In this scenario, there is just a single headset that is primarily used for AR but can also do VR as an occasionally used mode. - We at Qualcomm think that mobile XR has the potential to become one of the world’s most ubiquitous and disruptive computing platforms, as the smartphone itself evolves in both function and form factor to become a wearable XR device that ships in the hundreds of millions per year. - The future of XR is one that's always connected to the cloud yet never constrained by clumsy, restrictive, and occasionally dangerous cable connections. - For mass market appeal, it's our belief that anything that requires a cable or a tethering of any kind to a personal computer, is an evolutionary dead end. - The solution isn't trying to cut the cord via wireless connectivity that still constrains use to the proximity of a PC - Instead the solution is to never have a cord to begin with; doing as much latency critical processing as possible on the HMD itself. - Achieving these goals will take a lot of hard work and cooperation by dozens of top companies and technology leaders (many of whom are here today).
- - So that was a brief glimpse into the future, but back here in 2017 we still have a very long ways to go. - By analogy with smartphones, XR will likely follow a similar ~30 year cycle of progressively sleeker designs, with tremendously increasing functionality - We're currently only the first few years into that multi-decade cycle. - It's only going to be possible for us turn XR into a multi-billion dollar market if we all work together to solve some of the difficult technical challenges that I'll discuss next.
- - Ultimately, there are at least a dozen different categories of technical problems that need to be solved before XR will reach its potential. - But for the scope of this short presentation, we'll just focus on five of them: 1) Displays: We need displays capable of delivering richer visual content seamlessly between fully and partially virtual worlds 2) Common Illumination: We want virtual objects in augmented worlds to be indistinguishable from real objects within the same view 3) Motion Tracking: Fast, intuitive, and accurate tracking of your head, hands, and eyes is very important 4) Power & Thermal: Users want all-day battery life, that’s thermally compatible with sleek & light XR glasses 5) Connectivity: XR will require seamless connectivity at fiber optic speeds and low latency of interaction with connected services
- There are a couple important human-factors challenges related to displays that need to be solved: the vergence accommodation conflict and field-of-view. - Accommodation is the process by which the lenses in our eyes stretch or retract to focus on an object as its distance varies. - Vergence describes how our eyes move in opposite directions, converging as objects get closer. - In the real world, our eye/brain physiology has evolved to match accommodation and vergence. - However in today's displays, there’s a disparity between the physical surface of the screen, and the apparent focal point of objects in the simulated world. - This conflict eventually causes discomfort & fatigue, and it gets even more uncomfortable if the focal depth of virtual objects changes rapidly. - One way of potentially solving this is to project virtual objects at multiple focal planes simultaneously, in conjunction with eye tracking technologies. At the same time, XR HMDs will need to deliver a wider FoV: - There are many opinions on this but a consensus seems to be that the binocular FoV should be roughly ~190° horizontal, ~130° vertical - It might be helpful at some point for the industry to somehow agree on a roadmap of FoV improvement that we could march towards it together.
- One thing for sure though: we need significant improvements in display technologies to solve the vergence accommodation conflict and boost FOV as we previously discussed. In addition to that: - Converged XR will require displays to be completely opaque for VR mode, yet at least ~85% transparent for the more commonly used AR mode - It’s generally accepted that immersive XR will eventually require an angular resolution of roughly 1.0 arc minute per pixel - We’ll want displays to drive HDR with support for at least the Rec. 2020 color gamut, and a ~5X improvement in luminance in AR mode. - They should be capable of refreshing at a minimum of ~120Hz (per eye), as well as being light, mechanically flexible, durable, and eventually cost under ~$100 in high volumes
- So there are several XR related display questions that still need answers: What’s the best technology to switch between opaque and nearly transparent display modes? Can LCOS or DLP with a mechanical shutter, better optics, and improved wave guides get us there? Can plastic AMOLED get us there? Can smaller transistor sizes improve transparency for AR? Or should the industry be working together on something more exotic to meet these needs?
- - Here is another big problem to solve: virtual objects placed in real worlds still look fake. - Here on the left is a real paper cup, and on the right side is a virtual, aluminum cola can. - You can see from the scene in the top, that the can looks obviously fake if it’s improperly lit, whereas the same virtual can below it looks more realistic. - For virtual objects to look as natural as the real objects that appear in the same XR scene, they must have “common lighting.”
- So where are we today? - Virtual objects look fake - That’ll be the case whenever the lighting of virtual objects is static. - It’s also true for dynamic lighting when the graphics shader’s lights don’t consistently match real world light sources or intensity. - Even if the objects are beautifully drawn, incorrect lighting will cause it to look fake.
- So where must we eventually be? - Virtual objects must look real, with frequently sampled light from cameras or ALS used to determine final color of every pixel in the virtual object. - To get there, we’ll need fast interaction between many different sensors & rendering systems that use real world lights to dynamically render & overlay more realistic virtual objects.
- Computer vision will help us take XR experiences to the next level. - The industry will collectively need to improve the accuracy of user-friendly, inside-out 6 DoF head motion tracking for outdoor XR use. - Eye tracking technology will be employed to a variety of tasks from automating IPD calibration to reducing the vergence-accomodation conflict we discussed earlier. - We think that eye tracking will become a foundational technology for improving visual quality within mobile constraints, through graphics & video foveation, and enabling more intuitive interactions
- - In the near term, some XR apps will need very responsive and accurate handheld 6 DOF controllers. - Hardware challenges aside, there is a need for a plug-n-play model with standardized APIs for developers. - In the long-term though, nobody wants to carry controllers with them. - CV and machine learning technologies will likely evolve to a degree that hand gestures could replace controllers entirely in AR mode, and utilize 3D reconstruction to display your own hands in VR mode.
- - We as an industry must come together to deliver a MASSIVE improvement in power & thermal efficiency - The XR headset must be appropriate to wear & use all day; that’s very difficult due to the constraints shown here, but it’s not impossible to achieve in the long term. - Firstly, XR workloads are “always on”, compute intensive, and relatively latency intolerant with complex concurrencies - Simultaneously, the constraints of a lightweight, sleek form factors also create extreme challenges related to thermal limits and battery life.
- So how do we get there? We need to attack this problem TOGETHER from all angles. - First, every XR IHV should have a common goal of doubling perf/watt every ~3 years; that’s a difficult goal, but it is possible. - However, improving power efficiency alone won’t get us there - simply put, we need more battery life and less battery weight. - Attention battery companies who may be listening to this: please find a way to improve battery life per gram by at least ~5X over the next 10 years. - That means getting to something like ~20 grams for a battery that can deliver roughly ~7 Amp-hours of sustained use. - We also need more innovation in mobile HMD materials science & passive cooling so that glasses can be lighter and stay cool - help us find ways to keep them under 30oC without fans. - There is a software angle to this too: XR App developers and middleware companies must work together with the hardware companies to tune their code to be more power efficient. - Consortiums must continue with further standardization of XR multimedia compression, foveation, and other means of improving power efficiency. - Finally, as XR advances, so too will access to internet cloud services, and for that we’ll need more efficient wireless connectivity as well.
- So that brings us to our final pillar: connectivity. - XR will only reach its potential if there can be seamless connectivity to the internet and useful cloud services - 5G will provide multi-gigabit throughputs, over the air latency down to 1ms, and a uniform experience irrespective of your distance from cell tower - We’ll need all this, for the killer use case for 5G: interactive next-generation, streaming video - Video content already dominates internet traffic – soon we expect to see an explosion of streaming, interactive 360o HDR video content from video broadcasters - There will be even more user generated 4K and then 8K video content from drones other XR headsets.
- In conclusion, it is our belief that: - XR is the next mobile computing platform – it’s already here in its infancy today, and will rapidly grow over the next several years. - We’ve discussed several areas where technology breakthroughs are required for XR convergence. - These technical challenges are too vast for anyone to solve all by themselves, so cooperation is key to achieving the goal of mass market XR adoption.
- Lastly, I’ll leave you with one parting thought: - Yes, it won’t be easy, but to anybody who thinks these technical barriers for XR are insurmountable, I’ll just mention with these words of wisdom from Mark Twain. - “You can’t depend on your eyes when your imagination is out of focus.” <<Thank you>>