https://youtu.be/YQlcwrTvDOU Learn how any creator with an iPhone X will soon be able to leverage the built-in camera to animate their characters' faces for use in any project on any platform. Also, get insight into authoring philosophy and the fun, nitty gritty details of developing digital experiences using real-world data from mobile devices, APIs, and even new wearable AR devices. Speakers: Stella Cannefax (Unity Technologies) Amy DiGiovanni (Unity Technologies) Jono Forbes (Unity Technologies) Matthew Schoen (Unity Technologies) Timoni West (Unity Technologies) https://unite.unity.com/2018/berlin/

1. Reality As Your Next Build Target
Mobile AR and the Future of Authoring
Authoring Tools Group, Unity Labs
Timoni West
Matt Schoen
Amy DiGiovanni
Stella Cannefax
Jono Forbes

2. Unity Labs
Animation
Augmented and Virtual Reality
Graphics Research
Future of Game Creation
Machine Learning
Today we are focusing on the
Authoring Tools Group, which has
been investigating how Unity will
both be used to make the future of
spatial computing, and what Unity
will look like in that future.

3. Authoring Tools Group
AnimateVR
ARView
Carte Blanche
EditorXR
MARS
XR Foundational Toolkit

5. Project MARS

6. All the world’s a stage, but you don’t know
what’s on it
• The way you author for augmented reality is uncharted territory
• In completely digitals worlds, you know everything that will
happen
• In partially digital worlds, you can only control the digital
• Your experience must be as robust, flexible, and responsive as
possible

7. SUPER robust, flexible, responsive
• You need to be able to test, test, test
• Test against unusual, inaccessible, or varied
environments
• Must have all the information about the world to edit
directly
• Machine learning can help, but is not widely available in a
way to do what we need—yet

8. The challenge with world data
• Usually only available in apps on the device after shipping
• We need to flip this
• Computer vision providers need to have their tech work
on many kinds of devices
• ML is often tied to specific hardware now—needs to
become more ubiquitous and consistent

9. Jono Forbes
UX Lead - MARS
Unity Labs

10. Designing for an
unpredictable real world

11. Use cases
• Sick table jump
• Essentially the demo you saw
• Zeldify the world
• Floor -> Water // Tables -> Grass // Walls -> Cliffs
• Character enters a room
• Semantically understanding a door and a chair
• How do I food?
• Great semantics & object tracking

12. Resonai

13. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)

14. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
• Analytics will be a big deal for AR devs

15. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based

16. Building up from nothing
• Start with the base layers (floor -> water..)
• Design simple queries (big surfaces, high surfaces..)
• Then more complex / rare (relationships to define a couch..)
• Finally, very context specific / trait-based
• Analytics will be a big deal for AR devs

17. Markers

18. 3D Markers / Geofences

19. Fallbacks

20. Procedural content

21. Procedural content

22. Amy DiGiovanni
Software Engineer
Unity Labs

23. Conditions
• Check against real world data
• Flexible
• Adaptable
• Author around the real world
Conditions specify what a MARSEntity
requires to perform some kind of function

24. Spatial conditions in the scene view

25. Miniature worlds

26. Scale the camera parent
“XR Cameras” GDC talk by Matt Schoen

27. World scale in AR

28. World scale in the scene view

29. It’s all relative
• MR authoring =/= traditional 3D authoring
• The scene is an abstract setup of conditions about the real world
• World scaling is necessary to support certain use cases - it must be
clear what scale your content is relative to real objects
• Positioning of entities is not relevant at runtime, but in editor is
meant to convey how the content is spatially related

30. Stella Cannefax
Software Engineer
Unity Labs

31. Contextual Authoring
Instead of explicitly designing scenes, think about
what context you need in the real world.

32. Data-Driven Authoring

33. Data-Driven Authoring
Each entity defines only what it depends on.

34. Data-Driven Authoring
Two more big advantages:
1.Easier multi-platform support
1.Simulation & testing

35. Data-Driven Authoring

36. Queries (editor)

37. Performance
Modern mobile devices experience performance drops due to heat
and processor throttling.
Graph is from our Mobile Performance Handbook:
http://on.unity.com/2Di8Hl7

38. Performance
MARS strives to be efficient in several ways:
• The behind-the-scenes work is distributed across time
• Built-in module to run processing tasks on an interval
• Managed memory is allocated only when absolutely necessary

39. Matt Schoen
Integrations Lead - MARS
Unity Labs

40. Hardware
Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Object recognition Light Estimation
ARKit devices X X X (X) X X X X
ARCore devices X X X (X) X (X)
Tango (defunct) X X X X
Hololens X X X
Magic Leap X X X X ? X X ? ? X
Vive Pro X X X X X
Windows MR X
Mirage Solo X
Santa Cruz X ? ? ? ? ? ? ? ? ?
Vive X
Rift X
Oculus Go (X)
GearVR (X)

41. Software
PC Mobile Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Obj rec Light Est Body Tracking Hand Tracking
Vuforia X X X X X X X X X
6d.ai X X
Placenote X X
Selerio X X X
ULsee X X X
Visage X X
Google Mobile
Vision
X (X) X
Apple Vision X X (X) X
Wrnch.ai X X X
Leap Motion* X X X
OpenCV X X (X) (X) X X X X
dlib X X
somewhere? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

42. • Faces
• Landmarks from (ARKit) face mesh
• Expressions from blendshapes
• 2D -> 3D landmark poses
• Expressions from landmark positions
• Surfaces
• “Meta-surface”
• Elevation / floor
• Rotation / alignment
• Overlap test
• Pause button
Room for improvement

43. Functionality Injection

44. Functionality Injection

45. Functionality Injection

46. MARS Provider Types (so far)
• CameraImage
• CameraIntrinsics
• CameraOffset
• CameraPose
• CameraPreview
• CompassHeading
• FaceTracking
• FacialExpressions
• FunctionalityInjection*
• LightEstimation
• MarkerTracking
• PlaneFinding
• PointCloud
• ReferencePoints
• WorldLocation

47. Editor Providers
• Must run in edit mode
• PC / mobile parity
• 3D face pose
• Markerless tracking
• Surface detection
• Remoting and Recording
• Local testing / debug
• Field recording
• Multi-user recording
• Generated Rooms
• ISimulatable and runInEditMode

48. Reasoning APIs
Fill in the missing pieces
• Which surface is the floor?
• Markers for relocalization
• Data correlation
• Which face is which?
• Which object is which?
• More to come

49. Thank you!

50. Hacked