Presented at: All Things Open 2019 Presented by: John Joyce, Cisco

1. Istio with Network Service Mesh
Speaker: John Joyce
email: joycej@cisco.com

2. What you will see
• Two open source projects - Istio and Network Service Mesh working
together
• Network Service Mesh will be used to connect two clusters
• Network Service Mesh enables sophisticated layer 2 and 3 network
configuration and options
• Istio will be used to layer an Application Service Mesh on top of the
Network Service Mesh orchestrated connectivity
• Istio enables sophisticated layer 7 functionality
• Result is application request routing across the extended Service Mesh

3. Why should you care
• NSM provides a framework to offload complex hybrid cloud
requirements
• In hybrid cloud environments each cloud or domain has very different
requirements on layer 2 & 3 setup and protocols supported.
• SecOPs and NetOPs teams enforce complex requirements to allow applications
to communicate in hybrid environments
• NSM allows these requirements to be fulfilled in a cloud native way
• Istio provides a framework to off-load many application level tasks
• Security
• Retries
• Logging & tracing
• Application developer is unencumbered from both layer 2 & 3
requirements and layer 7 requirements and can focus on business logic

4. Network Service Mesh
(NSM) Intro

5. What is Network Service Mesh
• A CNCF Sandbox project - https://networkservicemesh.io/
• Network Service Mesh (NSM) is a novel approach to solving
complicated L2/L3 use cases in Kubernetes
• Provides the following properties to networking in Kubernetes
• Heterogeneous network configurations
• Exotic protocols
• Tunneling as a first-class citizen
• Networking context as a first-class citizen
• Policy-driven service function chaining (SFC)
• On-demand, dynamic, negotiated connections
• Main goal is to provide ability to insert network functions between
workloads in a cloud native way.

6. Node(Network Service Manager Domain)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
App Pod
(w/ NSC client)
App Pod
(w/ NSC client)
...
Network Service
Endpoint (NSE)
(Pod)
Network Service
Endpoint (NSE)
(Pod)
...
...
Kubernetes API Server
(Network Service Registry via CRDs)
Network
Service
Manager
(NSM)
(Daemonset)
Node(Network Service Manager Domain)
Network
Service
Manager
(NSM)
(Daemonset)
Network Service Mesh Dataplane (NSMD)
(kernel/vswitch)
App Pod
(w/ NSC client)
...
App Pod
(w/ NSC client)
...
Network Service
Endpoint (NSE)
(Pod)
Network Service
Endpoint (NSE)
(Pod)
Network Service Mesh Architecture Components
Network Service Domain

7. NSM Value Proposition
vWireWorkload
NSE function and implementation
can be defined outside of NSM
project
NSM can stich arbitrary network functions between workloads
vWire WorkloadvWire
NSE
DPI
vWire
NSE
(router)
NSE
FW
Different Vwire choices
for both intra and inter
cluster connections
NSM supports both intra-cluster and inter-cluster connections

8. Service Mesh Intro
(or Application Service Mesh)

9. 10
What is a Service Mesh
Gateway
/Ingress
External
Requests
Business
Logic2
Web UI
Accounts
Database
Billing
Database
Business
Logic1
Proxy
More completely referred to as
Application Service Mesh
Wiki Definition - A mesh of proxies
• Infrastructure layer for secure
service-to-service
communication
• Supports numerous service to
service API formats (HTTP
1/2, gRPC, TCP, UDP)
• Can inspect API transactions
at Layer 7 or layer 3/4.
• Intelligent routing rules can
be applied between
endpoints
• Supports advance policy,
logging and telemetry
Proxy
Proxy
Proxy
Proxy

10. Service Mesh Offerings
• Numerous Service Mesh offerings available in the market
• Different proxies or dataplanes
• Envoy
• eBPF (via Cilium)
• FD.io
• Others and non-open source
• Different control planes for the proxies
• Istio – Apache License
• Linkerd – CNCF
• Cilium
• Others and non-open source

11. Istio Architecture
(https://istio.io/docs/concepts/what-is-istio/arch.svg)

12. Install Istio & NSM

13. Application Service Mesh
Layered on top of
Network Service Mesh

14. NSM + vL3 NSE creates a vL3 topology
vWires
Workload
Work load to NSE vWire
Ex. kernel intf - NSM dataplane –
kernel Intf
NSE – NSE vwire
Ex. vxlan
NSM stiches components together with vWires
Various options are supported
vWires Workload
vWires Workload
NSE
(router)
NSE
(router)
NSE
(router)

15. Istio on top of NSM
vWires
Workload
vWires
Istio
Control
Plane*
Envoy
* Different Istio deployment models can be supported. This Demo installs a control plane on each cluster
vWires WorkloadEnvoy
WorkloadEnvoy
NSE
(router)
NSE
(router)
NSE
(router)

16. NSM FD.io dataplane
vL3 NSE vL3 NSE
NSM fd.io dataplane
Istio layered on top of NSM vl3
App container
App Pod
App container
App Pod
K8s Cluster 2K8s Cluster 1
App container
App Pod
App container
App Pod
NS Dataplane
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
NSM init
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
NSM init
NS Dataplane
FD.io
FD.io FD.ioEnvoy Sidecar
Envoy Sidecar
FD.io
Istio Control
Plane
Istio Control
Plane
External
Service
Registry
Envoy Sidecar
Envoy Sidecar
Base Application
Data Plane
NSM Control Plane
Istio Control Plane

17. Istio layered on top of NSM vl3 – complete view
NSMgr
NS Registry
FD.io
NS Dataplane
nsmmgr
NSM FD.io dataplane
vL3 NSE
control
FD.io
vL3 NSE
NSM init
App container
App Pod
NSM init
App container
App Pod
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
FD.io
vL3 NSE
K8s Cluster 2K8s Cluster 1 NSM Control
App Data
Istio Control
Envoy Sidecar
NSM init
App container
App Pod
NSM init
App container
App Pod
Envoy Sidecar
Istio Control
Plane
Istio Control
Plane
External
Service
Registry
FD.io
NS Dataplane
NSM fd.io dataplane
SVC discover

18. NSM Extensibility
foo
NS Dataplane
NSM Foo dataplane
vL3 NSE
NSM init
App container
App Pod
NSM init
App container
App Pod
Bar NSE
Dataplane
vL3 NSE
K8s Cluster 2K8s Cluster 1
NSM init
App container
App Pod
NSM init
App container
App Pod
foo
NS Dataplane
NSM foo dataplane
NSM
Project allows NSE
details to be
opaque
NSM allows any dataplane
implementation
Bar NSE
Dataplane

19. Pod connection details
Application Pod
Application
Container
Envoy
Sidecar
Network
Service
Client
NSM Service
Traffic
K8s service &
endpoint
traffic
All app traffic
redirected to
Envoy
Envoy has SVC and EP
data for both NSM and
non-NSM services
CNI
Interface
NSM
Managed
Interface
IPtablesRoute

20. Service Registration and Discovery
K8s Cluster App
Container
Istio
Pilot
External Registry
Service
Discovery
Controller
Implemented as
a KIND cluster
Via Multicluster
Config
App Pod

21. TCP?
Redirect
to proxy
NAT
Prerouting
chain
Dest port
= App
Port?
TCP?
NAT
Output
chain
Owner
==
proxy?
Proxy
Route
App TCP?
NAT
Output
chain
Owner
!=
proxy?
dest !=
localhost
?
Included
IP CIDR
Redirect
to proxy
iptables / ip6tables context
DNS packets
TCP/HTTP packets
How Envoy directs packets
Pod Interface Layer
CNI
Managed IF
NSM
Managed IF
Routes to NSM IF injected by NSM
Routes to CNI IF by default
Route

22. Demo Details

23. Install App Pods and
Network Service
Endpoints

24. Demo sequence
1. Deploy the Istio control plane
1. Deploy all the essential Istio components on 2 clusters
2. Configure Istio to insert Envoy proxies in front of all application workloads and configure
Envoy
3. The Istio service mesh will be layered on top of the NSM vL3 Inter-domain connectivity
2. Deploy NSM Control components to create the necessary vWires and NSE
discovery
3. Deploy Network Service Endpoints (NSEs) to provide the L3 networking
foundation Istio requires
4. Populate a service registry
1. Based the application deployment graph mapping to the NSM assigned network addressing
2. Use Istio multicluster to allow Istio to learn the service graph
3. Envoy provided the service mappings (via the Istio control plane)
5. Show app level service mesh connectivity
1. Load-Balancing
2. Canary deployments etc.

25. Demo Dataplane Topology
helloworld
NSE
(router)
vWires
NSE
(router)
Application
Service Mesh
helloworld
helloworld
helloworld
Cluster 1
Cluster 2
All Helloworld Apps can reach each
other via NSM interface IP or cluster IP
Once Envoy has mesh configururation

26. Current state
• Four helloworld apps deployed. Two on each Cluster.
• The pods are inter-connected via NSM and pingable
• The service mesh for the helloworld service and endpoint has not been
populated.
• Can only access the local helloworld Instance

27. Install Controller to
Populate External
Service Registry

28. Final state
• A controller on each node has been started
• That controller watches for NSM related pod events
• The controller has populated the service and endpoints in an external
registry (KIND cluster)
• Istio Pilot is watching that registry via Multicluster configuration
• Istio Pilot has populated the Envoy sidecar with the service mesh
topology
• Now all the helloworld endpoints are available to each other
• Envoy will load balance across all the endpoints if cluster IP used

29. Questions & Answers

30. References
• NSM
• https://networkservicemesh.io/
• https://github.com/networkservicemesh/networkservicemesh
• https://github.com/networkservicemesh/examples
• Istio
• https://istio.io/
• https://github.com/istio/istio
• https://github.com/istio/cni
• Demo Material
• https://github.com/john-a-joyce/nsm-istio
• https://github.com/tiswanso/networkservicemesh
• https://github.com/john-a-joyce/examples
• https://github.com/tiswanso/examples
• Contact me:
• Slack – Active on both Istio and NSM communities
• Email: joycej@cisco.com

31. Backup Slides

32. Service Registration Details
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2019-09-24T14:01:27Z"
generateName: helloworld-v1-5bc5c999d6-
labels:
app: helloworld
nsm/servicename: helloworld
nsm/serviceport: "5000"
pod-template-hash: 5bc5c999d6
version: v1
Service
Discovery
Controller
(New)
Create service helloworld
RegistryCreate Endpoint IP = PodIP
Port = nsm/serviceport
Pod
Numerous
implementation
options

33. Istio - vL3 Control Plane Only
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
vL3 NSE
NSM init
App container
App Pod
NSM init
App container
App Pod
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
vL3 NSE
K8s Cluster 2K8s Cluster 1 NSM Control
App Data
Istio Control
NSM init
App container
App Pod
NSM init
App container
App Pod
Envoy Sidecar
Istio Control
Plane
Istio Control
Plane
External
Service
Registry
SVC discover

34. Istio - vL3 Control Plane Only
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
vL3 NSE
NSM init
App container
App Pod
App container
App Pod
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
vL3 NSE
K8s Cluster 2K8s Cluster 1 NSM Control
App Data
Istio Control
NSM init
App container
App Pod
NSM init
App container
App Pod
Istio Control
Plane
Istio Control
Plane
External
Service
Registry
SVC discover

35. NSM FD.io dataplane
vL3 NSE vL3 NSE
NSM fd.io dataplane
Istio layered on top of NSM vl3
NS Dataplane
App container
App Pod
App container
App Pod
K8s Cluster 2K8s Cluster 1 NSM Control
App container
App Pod
App container
App Pod
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
NSM init
NSMgr
NS Registry
nsmmgr
vL3 NSE
control
NSM init
NS Dataplane
FD.io
FD.io FD.ioEnvoy Sidecar
Envoy Sidecar
FD.io
Istio Control
Plane
Istio Control
Plane
External
Service
Registry
App Data
Istio Control
SVC discover

36. NSM vL3 Dataplane only view
FD.io
NS Dataplane
NSM FD.io dataplane
FD.io
vL3 NSE
NSM init
App container
App Pod
App container
App Pod
FD.io
vL3 NSE
K8s Cluster 2K8s Cluster 1
Envoy Sidecar
NSM init
App container
App Pod
App container
App Pod
Envoy Sidecar
FD.io
NS Dataplane
NSM fd.io dataplane

37. Service Registration and Discovery
K8s Cluster CoreDNS
Server
App
Container
Istio
Pilot
External Registry
Service
Discovery
Controller
Implemented as
a KIND cluster
Via Multicluster
Config
App Pod
Local server
Fwd. to Ext.
Registry

38. Pod connection details
Application Pod
Application
Container
K8s CNI managed
Interface
Envoy
Sidecar
Network
Service
Client
NSM managed
interfaces
NSM Service
Traffic
Non-NSM
Traffic
All app traffic
redirected to
Envoy
Pilot
K8s API
Server
External
Service
Registry
Envoy has SVC and EP
data for both NSM and
non-NSM services