Integrating Service Mesh with Kubernetes-based connected vehicle platform

June 2020 / Digital Innovation, Engineering Research & Development
ｩ DENSO CORPORATION All Rights Reserved.
Integrating Service Mesh
with Kubernetes-based
connected vehicle platform
DENSO CORPORATION Digital Innovation Div
JunKai Yong / Tomoya Amachi

May 2020 / Digital Innovation, Engineering Research & Development
Agenda
2
Introduction of DENSO and our team
What we are building
Technical details and demonstration

Introduction
JunKai Yong/@yjk_miloaddict
3
Software Engineer@DENSO CORPORATION
Digital Innovation Div.
Graduated from Nanyang Technological University
(NTU) in May 2018
Joined DENSO in Oct 2018.
Working on IoT and Service-Mesh related
topics

Introduction
Tomoya Amachi/@tomoyamachi
4
Software Engineer
Joined misaki project as part of CREATIONLINE team

About DENSO
5
https://www.denso.com/global/en/

6
As a global manufacturer of automotive parts, DENSO is expanding its
business operations around the world.
DENSO supplies a wide range of products and systems
as a Tier 1* supplier trusted by the world’s automobile manufacturers.
World’s No.2
Consolidated net sales of 5.2 trillion yen.
About DENSO

Our team
DENSO + Creationline = Our team!
7

Abstracts of the Misaki

9
Why it is difficult to develop vehicle applications?
Applications on vehicle is becoming more and more important!
Develop application on
vehicles

vehicles
10
There are many ECU(Electric control unit)s in a
car.
rader
accessmete
r
rain
sensor
camer
a
battery sensor
EC
U
Difficult to Develop - Need to learn about embedded systems.

11
There are many ECU(Electric control unit)s in a
car.
Not using linux in most
cases.
CPU/Memory resources are limited and distributed.
vehicles

No easy way to update applications on many cars
simultaneously.
12
It's not always possible to run the application on your
PC as it is.
vehicles

Network connection can be unstable
Because cars are moving around!
13
If connection
breaks..
Cars are like highly distributed system.
Developers have to deal with network
disconnection for their apps.
(retries/store data locally…)
vehicles
Difficult to Develop - Unstable Network.

But application on vehicle is becoming more and more important!
14
It’s important to handle data on edge and and keep updating the application
to minimize communications expenses.
e.g: Picture / Video / Traffic Data / Metadata
The Global Connected Car Market size is projected to reach USD 212.7 billion by
2027, from an estimated value of USD 42.6 billion in 2019, at a CAGR of 22.3%.
Current vehicles are like feature phone at 2000s. Smart "vehicle" era is coming soon.
"Applications" will be a selling point of smart vehicles same as smart phone.
vehicles
https://www.marketsandmarkets.com/Market-Reports/connected-car-market-102580117.html

What we are building?
Kubernetes based connected vehicle platform!
15
Minimize the need to learn about embedded
systems.
Easy to deploy/update application to vehicles.
Remove the need for application developers to
handle network issues.
It's still a prototype.

Architecture Overview

Architecture Overview
17
App
Edge Cloud
App
K8s
Agent
VP
N
VP
N
Applying
Manifest
Choos
e
HELM
Chart
Generat
e
K8s
Manifest
List
Share
Ideal
Manifest
b/w Edge and
Cloud
DigitalTwin API UI
K8s
Agent

Why Kubernetes?
Flexibility of resources
18
radar
accessmete
r
rain
sensor
battery sensor
EC
U
ECUs will unified into several High Performance ECU in the future. We can use it
as k8s worker node.

19
radar
accessmete
r
rain
sensor
battery sensor
High Performance
ECU
Easy to orchestrate and manage ECU resources.
These ECUs will be separated from the basic operation of the car (running, stopping,
turning)
Why Kubernetes?

High performance ECU:
NVIDIA Jetson AGX Xavier Developer Kit
20
https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/
Why Kubernetes?

21
Edge
A
Cloud
High Spec Car
We can use resources on
cloud.
We can run same applications
even if edge resources are not enough
We can run container image
in different vehicles and cloud.
B C
VP
N
A
Low Spec Car
B C
VP
N
Why Kubernetes?

Technical Details

Components of Misaki
Misaki
23
Misaki Orchestrator
Misaki Service mesh

Misaki Orchestrator
For more information:
https://youtu.be/2x7jQTBUT5w

Technical details about
Misaki orchestrator
25
App
Edge Cloud
App
K8s
Agent
VP
N
Apply
Manifest
To
Cluster
Choos
e
HELM
Chart
Generat
e
K8s
Manifest
List
Share
Ideal
Manifest
b/w Edge and
Cloud
DigitalTwin API UI
4 3 2 1
4 components

26
Misaki orchestrator
Applications
Repository
Data
Processcor
Data
Uploader
Sensory data
APP1
APP2
APP3
APP4
APP1
APP2
APP3
APP4
https://www.flaticon.com/authors/vitaly-gorbachev
https://www.flaticon.com/authors/xnimrodx
https://www.flaticon.com/authors/good-ware
How it works:

27
Demo: Misaki orchestrator Video scenario
GPS managerCan manager
Deleted Service:
CAN-uploader-for-prius-phv
Pod 3 Pod 4
Pod 1 Pod 2
Local Broker Redis to Ambass
GPS managerCan manager
New service deployed:
via-containers-for-prius
Pod 3
Pod 4
Pod 1 Pod 2
Local Broker
Redis to Ambass
Pod 5
File uploader
Pod 6
Ring Buffer
Pod 7
Steady video
Pod 8
Video
segment

DEMO VIDEO (orchestrator)
https://www.youtube.com/watch?v=2x7jQTBUT5w&t=1145s

Misaki Service-Mesh

What is Service
Mesh?
30
Traffic control (load balancing, routing rules, failovers etc...)
Policy control
Secure communications
Service monitoring (metrics, logs, traces)
https://www.abhishek-tiwari.com/a-sidecar-for-your-service-mesh/
A dedicated infrastructure layer mainly for microservices

Service Mesh
Architecture
31
https://servicemesh.es/

Why Service Mesh?
32
App Proxy
Service-mesh decouples network concerns
from applications.
https://philcalcado.com/img/service-mesh/mesh1.png
https://qiita.com/Ladicle/items/4ba57078128d6affadd5
Application developers can focus on their
business values/logics
Applications no longer needs to implement
distributed system practices (i.e timeouts, service
discoveries etc.)

Technical Detail about
Misaki Service Mesh
Overview
33
Misaki Control Plane
Misaki Data Plane

Misaki Service Mesh
Architecture
34
• Provides policies
and configurations
to each Envoy
• Connects individual
envoys into a
distributed system.
Routing the application
to appropriate service(s)
Envoy:
Cloud node
Edge node
Queue
Filter
Queue
Filter
Misaki Control
Plane
App A App B
App C
Proxy

What is needed to deploy an app?
36
K8s deployment manifest
Envoy Custom Resource Definitions (CRD)

Deployment Manifest
37
Create, modify or delete K8s resources
(i.e pods, deployment)
Commonly written in .yaml
- App
- Envoy side-car
- Control plane configmap
- Initialization settings

Custom Resource Definition
(CRD)
38
Custom k8s API endpoint that stores a collection of API objects
I.E route CRD contains route CRD contains all the route information values

39
Custom k8s controller
Assign appropriate settings to envoy side-car of newly deployed application, or
when changes are made to an existing application

40
App C
Prox
y
New k8s
deployment
Misaki Control
Plane
App C
CRD
Obtain Pod.Spec
information1
Retrieve App C’s CRD
Generates envoy setting
snapshot and sets it to the
envoy side-car of the new app
3
2
App B
Prox
y
App A
Prox
y

Misaki Data Plane

What’s going on
normal service-mesh Data Plane?
42
https://www.slideshare.net/ThomasGraf5/accelerating-envoy-and-istio-with-cilium-and-the-linux-kernel

What’s going on
Misaki Data Plane?
43
Buffering HTTP Request from application when network disconnect or not
applicable network interface(3g, 4g, wifi etc).
Re-sends the HTTP Request/streaming-data to server when network available
again.
Store HTTP Request from application if network did not recover during buffering
duration.

Prototype Outbounds
on Misaki Data Plane
44
volume
App
Prox
y
Prox
y
Server
Online1 Offline2
volume
App
Prox
y
Prox
y
Server
Name resolve
in /etc/hosts
Recover3
volume
App
Prox
y
Prox
y
Server

Problems
45
Server footprint is not small
Not supported Dynamic DNS resolve
Network security problems

Current Outbounds
46
volume
App Proxy
volume
App Proxy
Local
DNS
Local
DNS
Online1 Offline2 Recover during buffering time3
App Proxy
Local
DNS

Current Outbounds
47
Recover after buffering time4
volume
App Proxy
Local
DNS
Replay
Async
response
volume
App Proxy
Local
DNS
Replay
Next request5

DEMO VIDEO (service mesh)
https://www.youtube.com/watch?v=2x7jQTBUT5w&t=1630s

Troubles
49
Envoy does not suppose offline communications.
We need to customize Envoy code (written by C++11).
Network security problems
Create an original DNS Resolver (based on tomoyamachi/go-dnsmasq)
All component images need to build for cross platforms(ARM, AMD etc).

Summary

Summary
51
We developed a k8s-based connected vehicle platform prototype
Misaki service-mesh removes the need for developers to
handle network problems
But there is still many challenges to be solved.

Challenges
52
Misaki service-mesh currently only support simple HTTP/1.1 and HTTP/2.0.
We want it to support more protocols (WebSocket, gRPC etc)
Misaki control plane needs to be more flexible
Our journey is just beginning!
Minimizing cpu/memory usage on edge devices.

Visit us
53
misaki-io
currently empty, but
we have a plan to
make it open source
this year
Thank You

Integrating Service Mesh with Kubernetes-based connected vehicle platform

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Integrating Service Mesh with Kubernetes-based connected vehicle platform

Similar to Integrating Service Mesh with Kubernetes-based connected vehicle platform (20)

Recently uploaded

Recently uploaded (20)

Integrating Service Mesh with Kubernetes-based connected vehicle platform

Editor's Notes