1. ONAP was evolving from a "SOA-like" architecture to a "pure microservices" architecture. MSB provided transparent service registration and communication for ONAP microservices. 2. Istio is an open source service mesh that provides reliability, security, observability and manageability for microservices. It introduces a centralized control plane to manage distributed sidecars. 3. Integrating Istio with ONAP would provide benefits like distributed tracing, metrics visibility and service graph for ONAP microservices. It is important to address challenges in supporting multiple network interfaces and coarse-grained services.

1. Service Mesh in Action with ONAP
Huabing Zhao, ONAP MSB Project PTL, Istio Contributor

2. 2
ONAP Amsterdam Architecture Retrospect
A&AI/ESR
DCAE
Correlation Engine
Policy Framework
Service Orchestration
SDNC APPC
“SOA like” Components
VFC
VFC -MS1
VFC -MS2
VFC -MS3
VFC -MSN
Multi-Cloud
MC-MS1
MC-MS2
MC-MS3
MC-MSN
Holmes
HM-MS1
HM-MS2
HM-MS3
HM-MSN
“Microservice” Components
ONAP was a “SOA like” and “Microservice” combined architecture, and it's
evolving towards a “pure Microservice” architecture

3. 3
“SOA like” Components Communication
Approach:
 Services find each other via DNS lookup
 Services IPs are put into hosts file by scripts or manually
Challenge:
 Hard to maintain the Hosts files across multiple hosts
 No consistent way across system to handle
communication policies and failures
(Rate limiting, LB, Timeout, Circuit Breaker, etc.)
 Difficult to scale out
Client
DNS Lookup
Service 1
Service 2
Service 3
Update
Hosts
Hosts
Service Request

4. 4
MSB Solution
Microservices Bus(MSB) is an ONAP common service component providing a reliable, resilient and scalable
communication and governance infrastructure to support ONAP Microservice Architecture.
MSB Component:
Service Registry
 External API gateway
 Internal API gateway
 JAVA Client SDK
 Pluggable architecture
 Service Portal
MSB Features:
 Pluggable architecture
 Transparent service registration
 Transparent service access proxy
 Reliable service communication(LB, Retry, Failover, Health check)
MSB
MSB
Discovery
External service Internal service
External Gateway Internal Gateway
Registry
ONAP Microservices
Microservice A
Microservice B
Microservice C
MSB SDK
1 Service discovery
2 Service request
1 Service request
2 Route request to service provider
Desktop External SystemsMobile
External service request
MSB
Discovery
External Gateway Internal Gateway

5. 5
Transparent Service Registration
 Kubernetes deploy/start/stop ONAP components.
 Registrator watches the kubernetes pod event .
 Registrator registers service endpoint info to MSB. It also updates the service info to MSB when ONAP
components are stopped/restarted/scaled.
 Flexible: A signle component can register multiple REST APIs, which is a common situation for “SOA like”
components
MSB
MSB
Discovery
External service Internal service
External Gateway Internal Gateway
Registry
ONAP Operations Manager
ONAP
A&AI
Kube2msb registrator
Pod Even (Start, Stop etc.)
Policy
SDC
SO
VF-CVF-C
Register Service
Update Service
k8s
deployment
specs
(annotation)
Kube2msb registrator can register service endpoints for the microservices deployed by K8s

6. 6
Trasparent Service Communication
MSB
Discovery
External
services
Internal services
External Gateway Internal Gateway
Registry
ONAP System
Microservice A
Microservice C
Microservice B
MSB
Discovery
External Gateway Internal Gateway
internal request
External request
Service Information
Desktop
OtherSystems
Mobile
Service Request
Route Request to C
 API Gatway can be deployed as Internal or External API Gateway.
 Internal services are pushed to InternalAPI Gateway and external services are pushed
to External API Gateway
 Client just sends a request to API Gateway, API Gateway handles the details such as
service discovery/LB/rate limiting/retry and route the request to the destination.
MSB API Gatway provide transparent, reliable communication for ONAP microservices

7. 7
What's Next -> Service Mesh
A service mesh is a dedicated infrastructure layer for handling service-to-service communication.
It's responsible for the reliable delivery of requests through the complex topology of services that
comprise a modern, cloud native application. In practice, the service mesh is typically implemented
as an array of lightweight network proxies that are deployed alongside application code, without
the application needing to be aware. -- Willian Morgan
client server
registry
1. register2. lookup
3. call
client server
registry 1. register
3. lookup
2. call
Proxy
4. forward
Separation of Concers：
• The remote communication
logic is moved into the proxy
and deployed as “sidecar”
• Service developers only need
to take care of the business
logic
• With sidecar approach,
there're much more
possibilities to seperate the
Microservice Infrastructure
from business logic
Service Mesh sidecar is an evolved version of MSB API Gateway approach

8. 8
Istio service mesh
 Stability and Reliability: Reliable communication with
retries and circuit breaker
 Security : Secured communication with TLS
 Observability: Metrics measurement and distributed
tracing without instrumenting application
 Manageability: Routing rule , rate limiting
enforcement, canary deployment
 Testability: Fault injection to test resilience of the
services
Istio is an open source service mesh project.
Istio features:
Istio Architecture

9. 9
Why Istio?
The main advantage of Istio is introducing a
centralized Control Plane to manage the distributed
sidecars across the mesh, Control plane is a set of
centralized management utilities including:
• Pilot: routing tables, service discovery, and load
balancing pools
• Mixer: Policy enforcement and telemetry
collection
• Citadel: TLS mutual service authentication and
fine-grained RBAC
The Istio control plane is highly extendable by design.
 Multiple adapters can be plugged into Pilot to populate the services: Kubernetes, Consul, Mesos...
 Different backends can be connected to Mixer without modification at the application side: Prometheus, Heapster,AWS
CloudWatch...
 Standard API between Pilot and data plane for service discovery, LB pool and routing tables which decouples the sidecar
implementation and Pilot: Envoy, Linkerd, Nginmesh are all support Istio now and can work along with Istio as sidecar
Standard API
Kubernetes
Consul
Mesos
Logging
Metrics
Quota
Data Plane
Control Plane

10. Casablanca Istio Tryout
MSB Service Portal
Distributed Tracing
Service Graph
Metrics Visibility
 Integrate Istio with Multi-Cloud and VFC
microservices
 Integrate with CNCF projects
 jaeger to provide distributed tracing
 prometheus and grafana for metrics
collection and display
 Add MSB Portal to control plane to provide
service Catalog ,swagger UI of Restful API
How to setup Istio with ONAP:
 Without Mutual TLS:
https://wiki.onap.org/display/DW/Manage+ONAP+Microservices+with+I
stio+Service+Mesh
 With Mutal TLS:
https://wiki.onap.org/display/DW/Manage+ONAP+Microservices+with+I
stio+Service+Mesh-Mutual+TLS+Authentication+Enabled

11. How to Support Multiple Network Interfaces?
Istio only support service deployments with a single network interface. But in production, deployment
with multiple network interfaces is a very common senario.
Pilot Service Registry
Register service
(IP :10.75.8.101)
10.75.8.101
10.75.8.0/24
192.168.10.0/24
192.168.10.63
Envoy Service A
Host/Pod
xDS Request
(Node: 192.168.10.63)
Sevice A as
Outboud Listener
service A (IP :10.75.8.101)
Infinite
Loop
Request to Service A
10.75.8.101
Crash!
No Service A instance
whose ip is 192.168.63 -->
Service A is not located on
this Node
This is an outbound
request, it should be
send out to 10.75.8.101

12. How to Support Multiple Network Interfaces?
Enhance Istio to support multiple network interfaces.
Pilot Service Registry
Envoy Service A
Register service
(IP :10.75.8.101)
10.75.8.101
10.75.8.0/24
192.168.10.0/24
192.168.10.63
Host/Pod
xDS Request
Node:
192.168.10.63
10.7.8.101
Sevice A as
InBound Listener
service A (IP :10.75.8.101)
Request to Service A 127.0.0.1
There is a Service A
instance whose ip is
10.7.8.101 --> Service A is
colocated on this Node
This is an intbound
request, it should be
send to 127.0.0.1

13. How to Support “Coarse Grained” Service?
Kubernetes and Istio assumes that one progress/container is a service.
While this assumption is fine with “pure” Microservices architecture, it does have
problems supporting “Coarse Grained” Service.
Process
Service-A
Service-B
Service-C
Coarse grained Service
Application
Process
Process
Service B
Process
(Service-E)
Process
(Service-F)
Fine grained Service
(Service-D)
An example application: Combination of “fine Grained” and “Coarse Grained“ Services
How to map multiple
logic services inside
one process to Istio
service?

14. How to Support “Coarse Grained” Service?
Process
Service-A
Service-B
Service-C
Sidecar
Process
Process
Service B
Process
Service-E
Fine grained Service
(Service-D)
Coarse grained Service
Sidecar
Sidecar
MSB
Service Registry
Registration
Registration
Pilot
Pull Services
xDS
Transparent Service
Registration via
K8S Annotation
Client Request
Client Request
Client Request

15. K8S Ingress, Istio Gateway and API Gateway
K8S Ingress
Load balancing
SSL termination
Virtual hosting
Istio Gateway
Load balancing
SSL termination
Virtual hosting
Advanced traffic routing
Fault injection
Other benifits brought
by Istio
API Gateway
Load balancing
SSL termination
Virtual hosting
Traffic routing
API lifecycle management
API Operation Monitoring
API access authorization
API key management
API transformation
API Billing and Rate limiting

16. Istio Ingress and API Gateway should work together
MSB API Gateway
Sidecar
ServerSidecarSidecarClient
External request
Service Mesh
API Management
Service Discovery &
Traffic Management
Service Mesh
Control Plane

17. Service Mesh
The Missing Part: Async Messaging
Server
(V2)
Sidecar
SidecarClient
Server
(V1)
Sidecar
Partition 0
Partition 1
90%
10%
90%
10%
Microservice has two communication approaches : Sync RPC and Async Messaging
Kafaka Topic
A possible use of service mesh with Kafka in canary deployment
Control PlaneOperation
Rules

18. Network Latency Brought by Sidecar
It doesn't make sense for
local traffics to go through
TCP/IP and Ethernet

19. Cilium: Transparent acceleration on datapath
Direct communication between sockets, bypass the TCP/IP Stack
Works like unix domain socket, but behaviour stays the same with network socket
Based on BPF sockmap

20. 20
Service Mesh as Part of Infrastructure
PaaS
Servers
Virtualization
O/S
Container
Orchestration
Application
Fuction
Network
Storage
PaaS
Servers
Virtualization
O/S
Container
Orchestration
Service Mesh
Application
Fuction
Network
Storage
FaaS
Servers
Virtualization
O/S
Container
Orchestration
Service Mesh
Serverless
Fuction
Network
Storage
Leave all the others to the platform except business logic

21. 21
Thank You
Blog https://zhaohuabing.com
Github https://github.com/zhaohuabing