Daimler Protics implemented mixed and augmented reality on mobile devices and used the Microsoft HoloLens for automotive production, training, and marketing. Discover the challenges Daimler Protics faced and the Unity solutions that eased the mixed reality implementation. Speakers: Daniel Keßelheim - Daimler Protics Sebastian Rigling - Daimler Protics Session available here: https://youtu.be/fTc1c8iTGqU

1. How Daimler used mobile mixed realities for
production, training and sales.

2. Daimler Protics GmbH
How Daimler used mobile mixed realities for
production, training and sales.
1. About us
2. Application of Mixed Reality in Automotive
3. Overcoming Challenges of Mobile Mixed Reality

3. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
About us
3

4. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
About us
4
Daniel Keßelheim Sebastian Rigling

5. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
About us
5

6. Daimler Protics GmbH
Daimler
How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 6

7. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 7
Daimler Protics
We design digital reality with
product and mobility data in
the Daimler Group.

8. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 8
Research
Development
Consulting
Prototypes
Demos
Products
Sales /
Aftersales
Poduction /
Engineering
Auto
Shows
Computer
Vision
Sensor
Fusion
AI
Data Prep
UI/UX
3D
Modelling
&
Animation
Development of Mobile Mixed Realities

9. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 9
Tablet Smartphone Smartglasses Projection
Development of Mobile Mixed Realities

10. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Applications of Mixed Reality in Automotive
10

11. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 11
Production

12. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 12
Production
• Safety Inspection
• Quality Control
• Planning
• Assembly Training

13. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 13

14. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 14

15. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 15
Sales

16. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 16
Sales
• Exciting Customer Experiences
• Explanation of Highlights and Features

17. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 17

18. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Powertrain Truck
18

19. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 19
After-Sales

20. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 20
After-Sales
• Repair Training
• Guided Repair
• Remote Assistance

21. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Training
21

22. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 22
In-Car

23. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 23
In-Car
• Navigation, Assistance Systems
• Information, Manuals
• Entertainment

24. Daimler Protics GmbH Titel der Präsentation | Abteilung | Datum |
Rear-Seat Entertainment
24

25. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Overcoming Challenges of Mobile Mixed Reality
25

26. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
High performance: Access CAD immediately. Excellent precision.
26
Customer Expectations

27. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Low performance: Data prep required. Camera-based tracking: Approximate precision.
27
Challenges for Mobile MR

28. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Challenges Overview
28
• Complex systems
• Complex data
• Precision
• Visual Quality
• Ease of Use
• Complex data
Production Sales After-Sales

29. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Challenges Overview
29
• Complex systems
• Complex data
• Precision
• Visual Quality
• Ease of Use
• Complex data
Production Sales After-Sales

30. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Challenges Overview
30
• Complex systems
• Complex data
• Precision
• Visual Quality
• Ease of Use
• Complex data
Production Sales After-Sales

31. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
31
Precision Performance Systems

32. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
32
Precision Performance Systems

33. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Feature-based Tracking
33
Marker Tracking
ARUCO QR Image

34. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Feature-based Tracking
34
Markerless Tracking
SLAM Model Tracking

35. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Feature-based Tracking
35
Markerless Tracking
SLAM Model Tracking

36. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Model Tracking
Continuous tracking: Better fit.
36
A
B
Model tracking from perspective A Model tracking from perspective B

37. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Model Tracking on HoloLens
Only initial position: Better Performance.
37
A
Initial pose from perspective A Visible offset from perspective B
B

38. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
38
WA

39. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
39
WA

40. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
40
WA

41. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
41
WA
WA

42. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
42
WA
WA

43. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Increasing Precision on HoloLens
• Initial Pose: Model Tracking
• Adjust for front and rear World Anchor
• Interpolate Content Root Position between World Anchors according to user‘s
position
43
Draufsicht Demonstration World
Anchors

44. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
44
Precision Performance Systems

45. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Performance
• Examine each usecase: Keep data at a minimum.
• Set up LOD Pipeline to be able to meet different demands over different
platforms or within one app.
• Budget onscreen polycount
45

46. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Polycount
46
Total: 100k Tris

47. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Polycount
47
Total: 70k Tris

48. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Polycount
48
Total: 80k Tris

49. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Physical Simulations
49
Active oil management
Target: Tablets and Smartphones
Approach : Fake using particles
Better approach: Render offline, visualize realtime
! yields physically correct fluid animations

50. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Physical Simulations
50
Simulat
e
Render Export Unity

51. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Physical Simulations
• Exporting the particle/fluid simulations to Unity was tricky
• Mobile devices are weak
• Solution: We used the Asset Store ! Export the simulation as obj-sequence
51

52. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
52
Precision Performance Systems

53. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
DA
TA
Fast In-Car Prototyping
53
Communicatio
n Interface
External
Devices
Video, Pose,
CAN, FlexRay,
etc.
visualize with
deployto
TCP/UDP, ROS,
etc.
Different
Platforms
AR/VR
Devices,
Smartphones,
Tablets, PCs,
etc.
Tracking
Cameras,
IMUs, RTK,
etc.
Easy co
like trac
Easy vis
data (lik
Easy de
Ra
Du
Un
ca
va

54. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Fast In-Car Prototyping
54
Screenshots of some prototyping tools
Easy co
like trac
Easy vi
data (lik
Easy de
Ra
Du
Un
ca
va

55. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Quick Deployment to Many Platforms
• HTC Vive
• Oculus Rift
• Oculus Go
• Microsoft HoloLens
• Google Daydream
• Lenovo Mirage
• Meta 2
• Smartphones
55

56. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Everything Comes Together
56
Design
RD
Services

57. Thank you!
Daimler Protics GmbH
628-G440 • 70546 Stuttgart • Telefon : +49 711 17-5 96 60 • Fax: +49 711 17-5 95 69
info-daimler-protics@daimler.com • www.daimler-protics.com • Intranet-Portal-Code:
@protics
Sitz in: Leinfelden-Echterdingen, Amtsgericht in: Stuttgart, HRB-Nr.: 23635, Geschäftsführung: Stefan Rosenwald (CEO), Antonio Del Mondo
(CFO)
ISO 9001:2015 Zertifizierung