June 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Kubernetes based
connected vehicle platform
DENSO CORPORATION Digital Innovation Div
Kenta Suzuki / Aman Gupta

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Agenda
2
Introduction about DENSO and our team (Japanese)
What we’re building and why we’re using k8s (Japanese)
Technical details and demo of our project (English)

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Introduction
Kenta Suzuki / @suusan2go
3
Senior Software Engineer@DENSO CORPORATION
Digital Innovation Div.
Previously worked for several internet companies
and worked as freelance, joined DENSO in 2020.
Certified ScrumMaster /
Certified Kubernetes Application Developer

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Introduction
Aman Gupta / theguptaji
4
Software Engineer @DENSO CORPORATION
Joined DENSO in October 2019.
AWS Solution Architect Associate /
Certified Kubernetes Application Developer/
❤ Golang

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
About DENSO
5
https://www.denso.com/global/en/

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
6
As a global manufacturer of automotive parts, DENSO is expanding its
business operations around the world.
DENSO supplies a wide range of products and systems
as a Tier 1* supplier trusted by the world’s automobile manufacturers.
Consolidated net sales of 5.2 trillion yen. World’s No.2
About DENSO

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
7
https://www.slideshare.net/devsumi/16e410
About DENSO

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
8
https://www.slideshare.net/devsumi/16e410
About DENSO

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Our team
DENSO + Creationline = Our team!
9

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
10
Why it is difficult to develop vehicle applications.
Applications on vehicle is becoming more and more important!
Develop application on
vehicles

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Develop application on
vehicles
Difficult to Develop - Need to learn embedded things.
11
There are many ECU(Electric control unit)s in a car.
rader
accessmeter
rain sensor
camera
battery sensor
ECU

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Difficult to Develop - Need to learn embedded things.
12
There are many ECU(Electric control unit)s in a car.
https://www.denso-ten.com/jp/story/ecu/
Not linux in most cases.
CPU/Memory resources are limited and distributed.
Develop application on
vehicles

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
No easy way to update applications on many cars at once.
13
It's not always possible to run the application on your PC as it is.
Develop application on
vehicles
Difficult to Develop - Need to learn embedded things.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
There are several network disconnection.
Because cars are moving around!
14
If
connection
breaks..
Cars are like highly distributed system.
Developers have to deal with network
disconnection for each apps manually.
(retry / store data locally…)
Develop application on
vehicles
Difficult to Develop - Unstable Network.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
But application on vehicle is becoming more and more important!
15
It’s important to handle data on edge and and keep updating the application
to minimize communications expenses.
e.g: Picture / Video / Traffic Data / Metadata
The Global Connected Car Market size is projected to reach USD 212.7 billion by
2027, from an estimated value of USD 42.6 billion in 2019, at a CAGR of 22.3%.
Current vehicles are like feature phone at 2000s. Smart "vehicle" era is coming
soon. "Applications" will be a selling point of smart vehicles same as smart phone.
Develop application on
vehicles
https://www.marketsandmarkets.com/Market-Reports/connected-car-market-102580117.html

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
What we are building?
Kubernetes based connected vehicle platform!
16
Minimize need to learn embedded things.
Easy to deploy/update application on vehicles.
No need for application developers to think about
the network issues.
It's still a prototype.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Architecture Overview
17
Ideal situation for
connected cars
Kubernetes cluster
Extending cloud to vehicle

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Architecture Overview
18
App
Edge Cloud
App
K8s
Agent
VPN
VPN
Applying
Manifest
Choose
HELM
Chart
Generate
K8s
Manifest
List
Share
Ideal
Manifest
b/w Edge and Cloud
DigitalTwin API UI
K8s
Agent

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Why Kubernetes?
Flexibility of resources
19
rader
accessmeter
rain sensor
battery sensor
ECU
ECUs will unified into several High Performance ECU in the future. We can use it
as k8s worker node.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Why Kubernetes?
Flexibility of resources
20
rader
accessmeter
rain sensor
battery sensor
High Performance ECU
Easy to orchestrate and manage ECU resources.
These ECUs will be separated from the basic operation of the car (running,
stopping, turning)

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Why Kubernetes?
Flexibility of resources
21
Easy to orchestrate and manage ECU resources.
These ECUs will be separated from the basic operation of the car (running,
stopping, turning)
https://www.nvidia.com/ja-jp/autonomous-machines/embedded-systems/jetson-nano/

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
22
Edge
A
Cloud
High Spec Car
We can use resources on cloud.
We can run same applications
even if edge resources are not enough
Of course we can run container image
in different vehicles and cloud.
Why Kubernetes?
Flexibility of resources
B C
VPN
A
Low Spec Car
B C
VPN

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Why Kubernetes?
ServiceMesh
23
Vehicle application it self will not need to care
about network connection because of our
service mesh.
Our custom control plane will handle network
connection issue.
App
Edge Cloud
Retry / store results manually
when network connection is
down
Without
ServiceMesh
App Queue
Store requests in queue when network is
down, and send it
when network connection is back.
With
ServiceMesh

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Why Kubernetes?
Now Kubernetes is already running on U.S. Air Force too!
24
https://www.youtube.com/watch?time_continue=89&v=YjZ4AZ7hRM0&feature=emb_title

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Why Kubernetes?
25
https://www.theregister.com/2020/06/03/kubernetes_b_21_bomber/

June 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Technical details

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Technical details about
Misaki
27
App
Edge Cloud
App
VPN
Vehicle cluster
B C

June 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Misaki
Misaki Orchestrator
Misaki Service mesh

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Technical details about
Misaki orchestrator
29
App
Edge Cloud
App
K8s
Agent
VPN
Apply
Manifest
To
Cluster
Choose
HELM
Chart
Generate
K8s
Manifest
List
Share
Ideal
Manifest
b/w Edge and Cloud
DigitalTwin API UI
4 3 2 1
4 components

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Misaki-UI
30
Technical details about
Misaki orchestrator
Administrator
Chart repository
Misaki-API
Can be any Helm chart
repository i.e. gitlab,
GitHub
Returns the Vehicles
That belongs to the
User. i.e which are
present in the DITTO
Can choose any vehicle,
and then the helm chart
they want to deploy
Misaki-UI
Implemented in NuxtJs,
A Vue framework
1

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Misaki-api
31
Technical details about
Misaki orchestrator
Misaki-api
2
Rest API made with
Golang
HELM chart

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Digital-twin
32
Technical details about
Misaki orchestrator
3
Digital-twin
An open source database
solution for IOT devices
Misaki-k8s-agent Misaki-APIConnected via
WebSocket
Connected via
WebSocket
REPORTEDDESIRED
DESIRED
REPORTED

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Misaki-k8s-agent
33
Technical details about
Misaki orchestrator
4
Misaki-k8s-agent
A Web-socket client to ditto,
which lives in vehicle’s cloud
Kubectl apply
Fetch manifest
From ditto
Via WS

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
34
Demo Misaki orchestrator
Applications
Repository
Data
Processcor
Data
Uploader
Sensory data
APP1
APP2
APP3
APP4
APP1
APP2
APP3
APP4
https://www.flaticon.com/authors/vitaly-gorbachev
https://www.flaticon.com/authors/xnimrodx
https://www.flaticon.com/authors/good-ware
Video scenario overview

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
35
Demo Misaki orchestrator
Data
Receiver
Data
Uploader
Sensory data
Collector
App:
Engine-checker
Video scenario setup
Pod3Pod2Pod1
Data received
{x…, y….}
Data sent!
{x…, y….}
logslogs

June 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Technical Detail about
Misaki Service Mesh
Overview
37
Service-mesh decouples network concerns from applications.
Applications no longer needs to implement distributed system practices (i.e timeouts, service
discoveries etc.)
Application developers can focus on their business values/logics
Why
Envoy as a sidecar proxy
How
Misaki control plane

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Technical Detail about
Misaki Service Mesh
Architecture
38
• Provides policies
and configurations
to each Envoy
• Connects individual
envoys into a
distributed system.
Routing the application
to appropriate service(s)
Control plane: Envoy:
App A
Queue
App B
App C
Cloud node
Edge node

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Technical Detail about
Misaki Service Mesh
Queue
39
Acts as a proxy server.
Store HTTP Request/streaming-data from application when network disconnect.
Resends the HTTP Request/streaming-data to server when network available again.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
40
Demo Misaki Service Mesh
With a good WIFI/LTE connection,
the mp4 will be uploaded directly from the vehicle to the cloud
Kubernetes cluster
Connected Scenario
App queue

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
41
Demo Misaki Service Mesh
Disconnected Scenario
Kubernetes cluster
App
HTTP Request will be saved by the Queue.
Queue acts as proxy server and will resend the Request when reconnected.
queue

June 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Summary
43
We developed a prototype of k8s based connected vehicle platform.
K8s helps us to develop / deploy applications on vehicle.
But there's still a lot of challenges to be done.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Challenges
44
How to update k8s worker nodes in edge vehicles?
How to manage many k8s master ?
Or we should think about one k8s master and
many edge vehicles architecture ?
Our journey is just beginning!
We have to try light weight k8s like k3s and kube edge to minimize
required cpu/memory resources on edges.

May 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Visit us
45
misaki-io
currently empty, but
we have a plan to make it open
source this year
Thanks