Designing XR Experiences with Speech & Natural Language Understanding in Unity
Designing complex interactions for experiences that target XR headsets can be challenging due to the limited input schemes. Advances in Machine Learning (ML) now allow developers to easily leverage Natural Language Understanding through reusable techniques. The combination of XR+AI is a powerful integration that opens new possibilities for both gaming, entertainment and enterprise scenarios. This intermediate session is an exploration of how speech and language understanding can be used to augment Mixed Reality & VR experiences. We'll explore the use of speech recognition & Natural Language Understanding to build advanced voice commands, translate languages from within XR environments, and also look at the creation of intelligent conversation assistants to be used as interactive entities in Mixed Reality and VR apps & games. In a world where speech is the primary form of input, using Machine Learning to process language input and understand the user's intent is of paramount importance. Nick Landry - Microsoft
Recommended
Recommended
More Related Content
What's hot
What's hot (14)
Similar to Designing XR Experiences with Speech & Natural Language Understanding in Unity
Similar to Designing XR Experiences with Speech & Natural Language Understanding in Unity (20)
More from Unity Technologies
More from Unity Technologies (20)
Recently uploaded
Recently uploaded (20)
Designing XR Experiences with Speech & Natural Language Understanding in Unity
- 1. Nick Landry Designing XR Experiences with Speech & Natural Language Understanding in Unity
- 3. Computer Speech is the Dream
- 4. Azure AI Services Azure Infrastructure Tools
- 5. • Computer Vision + Holographic/AR • Language Services for MR Cognitive Services • Access to cloud data (SQL, Cosmos, etc.) from MR • Calling Azure ML APIs from MR Custom AI & Data Services • Smart assistants powered by Bots • Learning agents powered by ML Immersive Agents • Offline AI access via Windows ML • Access to Deep Learning frameworks (CNTK, TF, etc.) Local AI Services
- 6. Emotion Speaker Recognition Custom Speech Recognition (CRIS) Speech Synthesis & Recognition Computer Vision Face Video microsoft.com/cognitive Linguistic Analysis Language Understanding Bing Spell Check Entity Linking Knowledge Exploration Academic Knowledge Bing Image Search Bing Video Search Bing Web Search WebLM Text Analytics Recommendations Bing Autosuggest Bing News Search Translator Speech https://www.microsoft.com/cognitive-services/ Custom Vision Custom Voice
- 8. Why Speech Matters in XR Experiences • Speech is the most convenient input method • No keyboard, mouse, or touch screen (MR, VR) • Limited input options with gestures and/or motion controllers (MR, VR) • Limited ability to interact with screen (phone- based AR) • Voice recognition vs Speech recognition • Speech recognition vs Intent recognition
- 9. Not just for XR Many games can benefit from voice input, speech recognition & synthesis: FPS squad orders RPG dialogue RTS commands Space sim controls Interactive Fiction & trivia choices Hands-free gaming Voiced computers, robots, etc.
- 17. Thank You Nick Landry Senior Software Engineer, Microsoft activenick@microsoft.com slideshare.net/ActiveNick github.com/ActiveNick @ActiveNick
- 18. © 2018 Microsoft Corporation. All rights reserved. Microsoft, Windows, Microsoft Surface and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.