This presentation introduces HoloToolkit for Unity, which is an open-source framework started by the Microsoft HoloLens team, which encapsulates and simplifies key concepts of working with HoloLens: Gaze, Gesture, Voice, Spatial Audio, Spatial Mapping. The slides walk the audience through a concrete example illustrating Gaze and Voice Recognition, and show a few specific contributions to HoloToolkit that help developers avoid boilerplate code and focus on their applications.

1. HoloToolkit for Unity
Gaze & Voice
Dmitri Bouianov
HoloLens Developer

2. Background
• Waterloo CS, 16 years software engineering
• Unity + AR/VR/MR + full-stack development
• Multiple startups, technical & product roles
• Looking for experts: Unity, graphics, shaders

3. Overview
• HoloToolkit is an open-source framework
brought to you by the Microsoft HoloLens
team
• Encapsulates and simplifies key concepts
of working with HoloLens: Gaze, Gesture,
Voice, Spatial Audio, Spatial Perception
• Today: Gaze & Voice Recognition

4. Gaze & Keyword Managers
• HoloToolkit provides GazeManager prefab for getting
information about objects hit with user’s gaze
• HoloLens Unity SDK provides KeywordRecognizer API
to register voice commands and trigger event
callbacks
• Example: “Bigger”, “Smaller” to adjust object size
• HoloToolkit adds KeywordManager prefab for
registering keywords and handlers in Unity Editor

5. Problem
• Scenario: Looking at objects instantiated at runtime and
sending voice commands to those objects
• Problem: Objects need to exist in scene hierarchy to register
keywords and event handlers in KeywordManager
• Workaround: OnSelect sent by GestureManager,
OnGazeEnter & OnGazeLeave sent by GazeManager +
multiple dynamically created KeywordRecognizers
• Ideal: No boilerplate code when using voice commands +
solution that scales with large number of objects

6. Problem

7. Solution
• FocusedObjectMessageSender component is a proxy
that sends keyword events to any currently focused
object

8. FocusedObjectMessageSender

9. Solution
• SelectedObjectMessageSender component adds
persistent selection to explicitly select objects for voice
interaction

10. SelectedObjectMessageSender

11. Outcome
• Now: Sending voice commands to focused and
selected objects is easy, no code required
• Simple: Modular architecture decouples message
senders from message receivers
• Next: Automatic registration of keywords from
holographic UI elements and other objects
• Challenges: String message performance + type
safety

12. Join the Community!
• Discuss issues and challenges on HoloLens
Developer Forums & at HoloLens meetups
• Recognize common problems that can be solved
for everyone with some minimal code
• Fork HoloTookit-Unity on Github, open issues,
submit pull requests, address code reviews
• Spread the word and help grow the community

13. Thank You!
• @rocketalien
• dmitri@rocketalien.com
• http://rocketalien.com

14. Resources
• HoloToolkit for Unity
https://github.com/Microsoft/HoloToolkit-Unity
https://github.com/Microsoft/HoloToolkit-Unity/issues/71
https://github.com/Microsoft/HoloToolkit-Unity/pull/72
https://github.com/Microsoft/HoloToolkit-Unity/pull/69
• HoloLens Developer Forums
http://forums.hololens.com
• WinHUGR
http://www.meetup.com/Windows-Holographic-User-
Group-Redmond