Daimler Protics implemented mixed and augmented reality on mobile devices and used the Microsoft HoloLens for automotive production, training, and marketing. Discover the challenges Daimler Protics faced and the Unity solutions that eased the mixed reality implementation.
Speakers:
Daniel Keßelheim - Daimler Protics
Sebastian Rigling - Daimler Protics
Session available here: https://youtu.be/fTc1c8iTGqU
How Daimler uses mobile mixed realities for training and sales - Unite Copenhagen 2019
1. How Daimler used mobile mixed realities for
production, training and sales.
2. Daimler Protics GmbH
How Daimler used mobile mixed realities for
production, training and sales.
1. About us
2. Application of Mixed Reality in Automotive
3. Overcoming Challenges of Mobile Mixed Reality
3. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
About us
3
4. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
About us
4
Daniel Keßelheim Sebastian Rigling
5. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
About us
5
7. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 7
Daimler Protics
We design digital reality with
product and mobility data in
the Daimler Group.
8. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 8
Research
Development
Consulting
Prototypes
Demos
Products
Sales /
Aftersales
Poduction /
Engineering
Auto
Shows
Computer
Vision
Sensor
Fusion
AI
Data Prep
UI/UX
3D
Modelling
&
Animation
Development of Mobile Mixed Realities
9. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 9
Tablet Smartphone Smartglasses Projection
Development of Mobile Mixed Realities
10. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Applications of Mixed Reality in Automotive
10
11. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 11
Production
12. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 12
Production
• Safety Inspection
• Quality Control
• Planning
• Assembly Training
13. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 13
14. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 14
15. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 15
Sales
16. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 16
Sales
• Exciting Customer Experiences
• Explanation of Highlights and Features
17. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 17
18. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Powertrain Truck
18
19. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 19
After-Sales
20. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 20
After-Sales
• Repair Training
• Guided Repair
• Remote Assistance
21. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Training
21
22. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 22
In-Car
23. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 | 23
In-Car
• Navigation, Assistance Systems
• Information, Manuals
• Entertainment
24. Daimler Protics GmbH Titel der Präsentation | Abteilung | Datum |
Rear-Seat Entertainment
24
25. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Overcoming Challenges of Mobile Mixed Reality
25
26. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
High performance: Access CAD immediately. Excellent precision.
26
Customer Expectations
27. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Low performance: Data prep required. Camera-based tracking: Approximate precision.
27
Challenges for Mobile MR
28. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Challenges Overview
28
• Complex systems
• Complex data
• Precision
• Visual Quality
• Ease of Use
• Complex data
Production Sales After-Sales
29. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Challenges Overview
29
• Complex systems
• Complex data
• Precision
• Visual Quality
• Ease of Use
• Complex data
Production Sales After-Sales
30. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Challenges Overview
30
• Complex systems
• Complex data
• Precision
• Visual Quality
• Ease of Use
• Complex data
Production Sales After-Sales
31. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
31
Precision Performance Systems
32. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
32
Precision Performance Systems
33. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Feature-based Tracking
33
Marker Tracking
ARUCO QR Image
34. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Feature-based Tracking
34
Markerless Tracking
SLAM Model Tracking
35. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Feature-based Tracking
35
Markerless Tracking
SLAM Model Tracking
36. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Model Tracking
Continuous tracking: Better fit.
36
A
B
Model tracking from perspective A Model tracking from perspective B
37. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Model Tracking on HoloLens
Only initial position: Better Performance.
37
A
Initial pose from perspective A Visible offset from perspective B
B
38. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
38
WA
39. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
39
WA
40. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
40
WA
41. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
41
WA
WA
42. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
World Anchors on HoloLens
42
WA
WA
43. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Increasing Precision on HoloLens
• Initial Pose: Model Tracking
• Adjust for front and rear World Anchor
• Interpolate Content Root Position between World Anchors according to user‘s
position
43
Draufsicht Demonstration World
Anchors
44. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
44
Precision Performance Systems
45. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Performance
• Examine each usecase: Keep data at a minimum.
• Set up LOD Pipeline to be able to meet different demands over different
platforms or within one app.
• Budget onscreen polycount
45
46. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Polycount
46
Total: 100k Tris
47. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Polycount
47
Total: 70k Tris
48. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Polycount
48
Total: 80k Tris
49. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Physical Simulations
49
Active oil management
Target: Tablets and Smartphones
Approach : Fake using particles
Better approach: Render offline, visualize realtime
! yields physically correct fluid animations
50. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Physical Simulations
50
Simulat
e
Render Export Unity
51. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Physical Simulations
• Exporting the particle/fluid simulations to Unity was tricky
• Mobile devices are weak
• Solution: We used the Asset Store ! Export the simulation as obj-sequence
51
52. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Approaches to Overcoming Challenges
52
Precision Performance Systems
53. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
DA
TA
Fast In-Car Prototyping
53
Communicatio
n Interface
External
Devices
Video, Pose,
CAN, FlexRay,
etc.
visualize with
deployto
TCP/UDP, ROS,
etc.
Different
Platforms
AR/VR
Devices,
Smartphones,
Tablets, PCs,
etc.
Tracking
Cameras,
IMUs, RTK,
etc.
Easy co
like trac
Easy vis
data (lik
Easy de
Ra
Du
Un
ca
va
54. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Fast In-Car Prototyping
54
Screenshots of some prototyping tools
Easy co
like trac
Easy vi
data (lik
Easy de
Ra
Du
Un
ca
va
55. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Quick Deployment to Many Platforms
• HTC Vive
• Oculus Rift
• Oculus Go
• Microsoft HoloLens
• Google Daydream
• Lenovo Mirage
• Meta 2
• Smartphones
55
56. Daimler Protics GmbH How Daimler used mobile mixed realities for production, training and sales| IT/PRO | 09/25/2019 |
Everything Comes Together
56
Design
RD
Services
57. Thank you!
Daimler Protics GmbH
628-G440 • 70546 Stuttgart • Telefon : +49 711 17-5 96 60 • Fax: +49 711 17-5 95 69
info-daimler-protics@daimler.com • www.daimler-protics.com • Intranet-Portal-Code:
@protics
Sitz in: Leinfelden-Echterdingen, Amtsgericht in: Stuttgart, HRB-Nr.: 23635, Geschäftsführung: Stefan Rosenwald (CEO), Antonio Del Mondo
(CFO)
ISO 9001:2015 Zertifizierung