A talk from the Developer Track at AWE Europe 2017 - the largest conference for AR+VR in Munich, Germany October 19-20, 2017 Jürgen Sturm (Google): The Computer Vision Technology Underlying ARCore What if your smartphone could perceive and understand the world in 3D as we do? Imagine what applications this would enable, such as augmented reality games that play directly in the space around you, professional applications such as building and object scanning, indoor navigation, and exciting robotics projects. ARCore provides visual-inertial 6-DOF motion estimation and 3D reconstruction in real-time and on-board the device, and can thereby kickstart your AR and VR project significantly. In my talk, I will give a technical introduction to the hardware and computer vision algorithms underlying the Tango platform, including visual-inertial odometry, SLAM, loop closure detection, global localization. The main focus of this talk will be on 3D reconstruction and semantic scene understanding. During my talk, I will show several live demos.

1. Augmented Reality at Google
Jürgen Sturm

2. Vision

3. Challenges
- Algorithms
- Computer vision
- Machine learning
- Applications & use-cases
- Hardware (form factor, battery life, displays, …)
- Price point
- Social acceptance
- ...
This talk

4. Understanding 3D Space
- Where am I?
- Relative to some (arbitrary) starting point
- In absolute coordinates on earth, cm accurate
- How does the world around me look like?
- Metrically accurate 3D scan
- Objects, scenes, semantics (persons, actions, …)

5. Applications
- Navigation (find a product in a shop, meeting room, ...)
- AR games (embedded in the real world)
- 3D scanning and measuring (3D selfies, buildings, objects)
- Robotics (drones, toys, cars,…)
Navigation Explore 3D Scanning Robots/Toys
Source: Matterport
Shopping Games

8. Scene understanding
3D Reconstruction
Mapping and Localization
Processing Pipeline
Visual-inertial odometry
Raw sensor data
(images, IMU)
Real-time 3D pose
Optimized 3D poses
Textured mesh
Floorplan
Meaningful semantic
labels, 3D objects
Visual feature tracking
MUC

9. Gyroscope
rotational
velocity
rotation angle
Accelerometer
acceleration velocity position
x
y
z

10. Visual Features
- Detect points (x, y) which are unique in a local region
e.g. corners, blobs

11. Feature Tracking

13. Cloud Mapping

14. Cloud Localization

15. Environmental Understanding
● ARCore computes
(sparse) point clouds of
the environment
● Extract surfaces
(currently planes)

16. 3D Reconstruction
- For devices that have depth sensors, we can additionally
generate 3D reconstructions
- Raw point clouds are noisy and hard to interpret
- How can we generate a 3D mesh of the environment?
- How can we generate a stylized building model?

17. Signed Distance Function (SDF)
[Curless and Levoy; SIGGRAPH 1996] [Newcombe et al; ISMAR 2011]

18. Signed Distance Function (SDF)
[Curless and Levoy; SIGGRAPH 1996] [Newcombe et al; ISMAR 2011]

19. Signed Distance Function (SDF)
[Curless and Levoy; SIGGRAPH 1996] [Newcombe et al; ISMAR 2011]

20. Signed Distance Function (SDF)
[Curless and Levoy; SIGGRAPH 1996] [Newcombe et al; ISMAR 2011]
- Compute SDF from a depth image
- Measure distance of each voxel to the observed surface

21. Signed Distance Function (SDF)
[Curless and Levoy; SIGGRAPH 1996] [Newcombe et al; ISMAR 2011]
- Assume known camera poses (for now)
- Calculate weighted average over all measurements

22. Mesh Extraction with Marching Cubes
Find zero-crossings in the signed distance function by
interpolation

23. Real-Time 3D Reconstruction

24. Semantic Scene Understanding
(* Walls omitted)
Semantic annotations3D Scan

25. Research Partners

26. Conclusion
- Augmented Reality increasingly becomes a commodity
- ARCore provides motion tracking and environmental
understanding out-of-the-box
- Pixel devices and Samsung S8 are already ARCore
compatible, more devices soon
- APIs for C++/Java/Unity
- Scene understanding is still an area of active research
- Exciting times ahead!

27. Live Demos!