Do you want to explore Mars? Model a building on your conference room table? Play Minecraft in your living room? All of these things are being done today using Augmented Reality. Augmented Reality has shifted from mobile app gimmick to an immersive UXD experience that is unrivaled.
This talk will use the HoloLens and Unity 3D to give a crash course in AR concepts and the code that makes them a reality. This includes:
Real World Understanding (Spatial Mapping, Plane Finding, Spatial Understanding)
Holograms (Billboarding, Placement, Physics)
Control (Gaze, Cursor, Gestures, Voice)
Persistence (World Anchors)
You'll leave with an understanding of HoloLens terminology and techniques, by watching a real HoloLens app get built step-by-step using each of these AR concepts.
5. Holographic Lens
1268 x 720 Per Eye
3 Waveguide Lenses Per Eye
Intel Atom CPU
X5-Z8100 1.4 GHZ
2GB Ram
Holographic Processing Unit
Custom build Microsoft GPU
Sensors
4 Environment Cameras
1 Depth Camera
1 Video Camera
4 Microphones
1 Light Sensor
What is the HoloLens?
www.cameronvetter.com 05
8. World Coordinate System
1 Unit = 1 Meter
User Start is Origin
X=0, Y=0, Z=0
Hidden Triangles
Called the Spatial Mesh
Spatial Anchors
Attach Hologram to Reference Points
in the Spatial Map
Spatial Mapping
www.cameronvetter.com 08
9. Room Mesh
Visualized with Solids in the
Illustration
Physics
Holograms and Real World interact
with each other through Game
Engine Physics
Occlusion
Works Both Ways
Plane Finding
Ability to Find Planes in the Real
World
What do we get?
www.cameronvetter.com 09
10. Uses Spatial Map
Spatial Map Data used to find Planes
Type of Planes
Walls
Floor
Ceiling
Horizontal Surfaces
Other
Skipping this Topic
We now have Spatial Understanding
Plane Finding
www.cameronvetter.com 010
11. World Understanding
More Detail than Spatial Map
Topology Queries
Simple Surface and Spatial Queries
Object Placement Solver
Constraint Based Placement of
Object Sets
Shape Queries
Look for Real World Shape Matching
Custom Shape by the User
Spatial Understanding
www.cameronvetter.com 011
14. Body Locked
Tags Along with the Display
Display Locked
Positionally Locked to the Display
Use Sparingly
Clinginess is Frustrating for Users
Breaks Illusion
Holograms are Intended to Blend,
Not Overlay
Billboarding
www.cameronvetter.com 014
15. Relative to Origin
Origin is Starting Position, Not
Current Position
Relative to Real World
Must take into Account, Real World
Objects
Spatial Understanding
Used to Accurately Place Objects
with Object Placement Solver
Plane Finding
Used to Quickly Find Ceiling, Tables,
Walls, and Other Surfaces
Placement
www.cameronvetter.com 015
16. Tracks World State
Real World Objects and Holograms
Placement Types
OnFloor, OnWall, OnCeiling,
OnShape, OnEdge,
OnFloorAndCeiling, RandomInAir,
InMidAir, UnderFurnitureEdge
Placement Constraint
NearPoint, NearWall,
AwayFromOtherObjects,
AwayFromPoint
Placement Rules
AwayFromPosition, AwayFromWalls,
AwayFromOtherObjects
Object Placement Solver
www.cameronvetter.com 016
17. Unity Engine Provided
Add a RigidBody to Any Hologram to
Make it Observe Game Engine
Physics
Spatial Map Collider
Each Plane in the Spatial Mesh
Receives a Unity Collider
Not the Other Way
Real World Objects are not Impacted
by Holograms
Real World Impact
Holograms are Impacted by Real
World
Physics
www.cameronvetter.com 017
20. Detects Using Raycast
Raycast from User’s Eyes to Unity
Collider
Primary Interaction
Looking At Holograms is the Primary
Way to Interact with Holograms
Real World has Colliders
Remember During Spatial Mapping
the Real World is given Colliders
Invisible Objects
Any Holograms missing a Collider
are not Selectable with Gaze
Gaze
www.cameronvetter.com 020
21. Center of View
Display Locked to Center of View
Requires Gaze
Cursor Depends on Gaze, Expect
Both to be in Every Project
Stick with Standard Cursor
Use Microsoft Provided Cursors
Unless you Have a Good Reason to
Customize, TIP: You Don’t
Selection
Primarily used for Selection
Confirmation in Conjunction with
Gaze and Gestures
Cursor
www.cameronvetter.com 021
The Cursor
22. Bloom
Task Switch / Start Menu
Air Tap
Similar to Primary Mouse Button
Manipulation
Typically Used to Move or Drag
Items
Hold
Similar to Right Clicking
Gestures
www.cameronvetter.com 022
23. Auxiliary Controls
The Real Right Click
Use Windows 10 Speech
Standard Engine Built into Windows
System Level Commands
Select == Air Tap
Face Me
Bigger
Smaller
Concise Commands
Too Long is Hard to Remember
Too Short is Hard to Differentiate
Voice Input
www.cameronvetter.com 023
26. Frame of Reference
Requires Stationary Frame of
Reference all Anchors are Tracked
Relative to
Attaches Hologram
Hologram is Attached to Frame of
Reference, Remains Stationary as
Coordinate System Changes
Persistable
HoloToolkit has built in Serializer
Separate Coordinate System
Anchor has own Coordinate System,
adjusts as World Coordinate System
changes to Keep Hologram in Place
Spatial Anchors
www.cameronvetter.com 026
27. Unity Specific for AR
Not Hololens Specific, Unity Specific
Uses Spatial Anchor
World Anchor is a Wrapped Spatial
Anchor
Persist Between Sessions
Survives App Restarts
Unity Anchor Store
World Anchors are Managed by
Unity Engine
World Anchors
www.cameronvetter.com 027
