Covers up concepts about XR, Potential projects & Hands-on development with Mixed Reality Toolkit (MRTK)

1. Introduction to Mixed Reality
(XR)
By Sanjit Singh

2. A B O U T M E
Sanjit Singh
Student, Hacker, Sub - Team lead @ NASA SUITS, Space Coach @ AFWERX,
Research Assistant @ Rutgers University & Mixed Reality Developer @ MPLEX
Co-Founder & Former VP, XBoost
Building the future – VR/AR, Robotics, Brain computer Interfaces,
Virtual Worlds, AI, Humanoids & more cutting-edge technologies.
Connect @ https://www.linkedin.com/in/sanjit-singh-66374b6a/
Follow @ https://twitter.com/sanjitpsingh

3. What is Virtual &
Augmented Reality
Virtual Reality: 3D generated virtual
environment in which users can interact with
as if they were in that simulation to gain
immersive experiences of there
surroundings.
Augmented Reality: 3D Computer generated
environment projected from the virtual
world onto the real world as holograms.

4. Early days Mixed Reality
Dr. Ivan Sutherland Morton Heilig
Dr. Tom Furness Dr. Grigore (Greg) Burdea Dr. Mark Billinghurst

5. Types of VR/AR Headsets
HoloLens
Magic Leap
Nreal
Project North Star
Oculus Rift &
Oculus Quest
Google Cardboard VR
HTC VIVE & Windows Mixed Reality Headset

6. Applications
Games Education
Aerospace &
Defense
Robotics Medicine
Brain
Computer
Interfaces

7. Demo of
Games

8. Educational Game

9. Medical Applications

10. Space
Space

11. Neurotechnology

12. Where The Future is Headed
• Metaverse – virtual world with artificial life (cyberspace) with users can interact with virtual agents
in a simulation
• Sci-fi fantasy into reality

13. Mixed Reality Toolkit (MRTK)
• MRTK – A library designed for XR developers use into there projects and provides set of
components for interaction, UI, rapid prototyping and more.
• Very cross platform
• Supports: Android, IOS, HoloLens, Oculus, Magic Leap, Nreal & more
• Game engines that use MRTK: Unity3d & Unreal
• Our focus will be Unity3D - No Unity or C# experiences needed

14. Unity
Installation
guide
• Unity Hub: https://unity3d.com/get-unity/download
• Create Unity Account
• Unity Version: 2021.1.24f1
• Note: iPhone users need to have MacBook to build on
IOS platforms along with XCode installed
• Android Users also have android studio installed with
API Level minimum set to 24 and API level highest to
latest version

15. Create New

16. Unity Scene & Layout

17. Package Manager
• Windows – Package Manager
• Install ARFoundation, ARcore XR Plugin & ARKit XR Plugin

18. Under Packages Folder

19. MRTK Packages
• https://github.com/microsoft/MixedRealityToolkit-Unity
- Under releases
• MRTK Version: 2.7.2
• Installation packages -
Microsoft.MixedReality.Toolkit.Unity.Examples.2.7.2.unitypackage
Microsoft.MixedReality.Toolkit.Unity.Extensions.2.7.2.unitypackage
Microsoft.MixedReality.Toolkit.Unity.Foundation.2.7.2.unitypackage
Microsoft.MixedReality.Toolkit.Unity.Tools.2.7.2.unitypackage

20. Integrating MRTK into Unity
• Click Import
• Do the same for rest of
packages!!

21. MRTK Popup
• If all goes well with
importing, then MRTK popup
from image to left is what
should be see
• Click Unity OpenXR plugin

23. Unity is Restarting
- Once restart completes, you will get another MRTK popup
- Click Show XR Plugin Management Settings

24. XR Plugin Management Settings
• Click on Android logo to the right (for android users)
• Check ARCore

25. Now you can click Skip this step
On the right side of image, there is apply default settings, Click Apply
Unity will restart again

26. Popup configuration shows up again. Click next
From Right side you will get importing TMP Essentials options
Click Import TMP Essentials

27. MRTK Import All Done
- After importing TMP Essentials,
you will get MRTK Setup Completed
message shown on the left
- Click Done

28. Building on Platform
• As mentioned, we will be building on Mobile Platform (Android)
• In Unity go to File-Build Settings
• In order to build apps, unity needs to know which scenes to build
• Click Add Open Scenes
• Then Click on Android under Platforms
• Click Switch Platform
• Note after platform switches to android incase you get Popups again from MRTK
configuration, just repeat it

29. Player Settings
• Player Settings-Android icon-Other Settings
• Right Side of player settings we see many options
• Make sure minimum API level is set to 24
• Under Graphic APIs click on Vulkan and underneath press
the minus button (that will delete Vulkan)

30. Setting up MRTK Camera
- On the left you have the original Unity Scene with the built-in main camera
- Right click on Main Camera under Sample Scene and click on Delete
- Go the very top left and look for Mixed Reality
- Click on that and then go to toolkit-Add to Scene and Configure
- You will get a new camera that comes with MRTK along with other gameobjects we will use to configure

31. MRTK Build
Configuration
• Click on MixedReality Toolkit
Gameobject under Hierarchy
• Go to the right and you will see
Experimental Settings
• On the top click on Clone
• You will have clone profile popup
• Under that go to profile name
• We will rename it
MobileMixedRealityToolkitConfiguratio
nProfile
• After doing that, click clone

32. • Click on Camera
• Do the same thing again
• Clone and then rename profile to
MobileMixedRealityCameraProfile
• Click Clone

33. MRTK Camera Settings
- Look for Camera Setting Providers to the
right
- You will the current camera providers set by
default
- To the right click the minus button for all 3
to delete them

34. - We will create new camera setting for mobile AR for MRTK
- Click Add Camera Settings Provider
- You will have a new data provider
- Click Type-Microsoft.MixedReality.Toolkit.Experimental.UnityAR-UnityARCameraSettings

35. All Good to Go

36. Simple Build
• Go to Mixed Reality-Toolkit-Utilities-UnityAR-
Update Scripting Defines
• Connect your Phone to Computer
• Go to File-Build Settings-Build And Run
• If all goes well and it builds on phone it should
look like the picture from the right side
• Now we can start adding some interactable
features

37. Interactable
Objects
• On project search bar type up
HandInteractionExample
• Drag that scene under Hierarchy

38. • Under Sample Scene Delete MixedRealitySceneContent
• Left Click MixedRealitySceneContent Under HandInteractionExample
• Drag it on top of HandInteractionExample
• Right Click on HandInteractionExample under Hierarchy and click Remove Scene
• You will get prompted an option to save. Click Don’t Save as we do not want to override the scene
Adding Interactable
Objects

39. Scene Completed
• Rebuild Scene
• File-Build Settings-Build and
Run

40. Final Build Video

41. Other Unity Resources
• https://learn.unity.com/tutorials
• https://learn.unity.com/search?k=%5B%22ind%3A5816ce9a32b3060
0171bef5a%22%2C%22sl%3Abeginner%22%2C%22sl%3Aintermediat
e%22%5D&ob=starts
• https://learn.unity.com/search/?k=%5B%22tag%3A5900b95a090915
001e654b47%22%2C%22sl%3Abeginner%22%2C%22lang%3Aen%22
%5D&ob=starts