Have you ever wondered why your virtual and real-world objects look disconnected from each other? Are you curious about how to blend virtual and physical spaces to create a seamless experience? This talk will cover advanced techniques using lighting, depth masks, occlusion, shadows and more to create a sense of immersion. You'll also get an introduction to a set of tools that will improve your development process when using Unity and AR Foundation. Jimmy Alamparambil - Unity Technologies Lukasz Pasek - Unity Technologies

1. GenerativeArt–MadewithUnity
AR FoundationA Look Ahead
Lukasz Pasek
Software Engineer, Unity
lukaszp@unity3d
1
Jimmy Alamparambil
Software Engineer, Unity
jimmy@unity3d

2. - Unity @ GDC - Architecture talk
- Part 1 @ Unite Berlin 2018 -
Rendering/Immersion talk
2
Creating immersive multi-
platform AR experiences
with Unity.

3. Agenda
●AR Foundation Updates
○ 2D Image Recognition
○ 3D Object Recognition
○ Face Tracking
○ Camera Image API
○ Environment Probes
○ Workflow improvements
■ Remoting
■ Simulation
○ LWRP + Shadergraph support
3

4. 2D Image Tracking API and Support Features
2D Image Tracking
4

5. 2D Image Tracking
5
Asset Creation
● Add image asset to Editor
● Add set of image assets
into a database
Session detects images
in image database
Events for 2D image
● Added
● Updated
● Removed
2D Image Pose

6. Image Tracking Editor Setup
6

7. Image Tracking Demo
7

8. 3D Object Tracker API and Features
3D Object Tracking
8

9. 3D Object Tracking
9
Asset Creation
● Scan Real Object
● Add Object asset to Editor
● Add set of object assets
into a database
Session detects objects
in object database
Events for 3D object
● Added
● Updated
● Removed
3D Object Pose

10. Face Tracking API and Features
Face Tracking
10

11. Face Tracking
11
Events for face
● Added
● Updated
● Removed
Face Pose
Face Mesh
● GetVertices
● GetUVs
● GetIndices
Face BlendShapes
● Coefficients

12. 12
FFAFace Pose Editor Setup

13. 13
FFAFace Pose Demo

14. 14
FFAFace Mesh Editor Setup

15. 15
FFAFace Mesh Demo

16. 16
FFAFace BlendShapes Editor Setup

17. 17
FFAFace BlendShapes Demo

18. ● Quick access to Camera Textureadb p
● Customized Request :
○ Partial Area
○ Different resolution
Allows to process camera feed for gameplay
orr machine learning purposes.d/screen.mp4.
Camera API
18

19. var conversionParams = new CameraImageConversionParams(image, format,
CameraImageTransformation.MirrorY);
19
var rawTextureData = m_Texture.GetRawTextureData<byte>();
image.Convert(conversionParams, new IntPtr(rawTextureData.GetUnsafePtr()), rawTextureData.Length););
CameraImage image;
if (!ARSubsystemManager.cameraSubsystem.TryGetLatestImage(out image))
return;
image.Dispose();
m_Texture.Apply();

20. Environment Probes API and Features
Environment Probes
20

21. Environment Probes
21
Auto or manual
Events for probes
● Added
● Updated
● Removed
Probe
● Pose
● BoundsExtent
● GetEnvironmentTexture

22. 22
FEnvironment Probe Editor Setup

23. 23
FEnvironment Probe Demo

24. Tools
24
- Mock Layer
- Remoting
- Simulation

25. Mock backend subsystem :
* Injecting data to the stack
* modifying current settings
* data is being propagated across all tools build on the top
of the mobile multi platform AR code.
Tools - Mock Backend

26. Tools - Remoting
26

27. Tools - Remoting
27
Quickly Visualize AR Experience

28. Tools - Remoting
28
- Sending over all AR data mimicking the state of the date within the editor
- Support for touch input
- Recording session allowing to replay them within the editor
- Remote App code available to modify

29. Tools - Remoting
29
Extending Remoting

30. Set of APIs allowing for easy access to Mock layer :
- Create, Modify planes
- Camera position
- Cloud points
- Reference points
- And more
Tools - Simulation
30

31. LWRP and AR Foundation
31

32. 32
FShadergraph with LWRP

33. Example shadergraph that works with
ARFoundation
33

34. GenerativeArt–MadewithUnity
Questions ?
Lukasz Pasek
Software Engineer, Unity
lukaszp@unity3d
34
Jimmy Alamparambil
Software Engineer, Unity
jimmy@unity3d
Twitter: @jimmy_jam_jam