PKS Networking with NSX-T
“you focus on your app, we'll take care of the rest !”
Gaetano Borgione, @TanoBorgione
Angela Chin, @AngelaSChin
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Agenda
● What is PKS
● Networking in Kubernetes
● What is NSX
● NSX + PKS
● Cluster Creation
● Cluster Upgrade
2
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Disclaimer
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal
offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to
change. This information is provided without warranty or any kind, express or implied, and is not a commitment to
deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding
Pivotal's offerings. These purchasing decisions should only be based on features currently available. The
development, release, and timing of any features or functionality described for Pivotal's offerings in this
presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking
information in this presentation.
4
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
What is PKS
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
6
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
7
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
8
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
9
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
● Aimed at Day 2 Operations
1
0
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Pivotal Container Service
● Managed Kubernetes for multiple clusters
● Aimed at Day 2 Operations
● Uses BOSH!
1
1
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
2
BOSH
Infrastructure
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
3
BOSH
PKS
Infrastructure
watches
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
4
BOSH
PKS K8s
Infrastructure
watches
deploys
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
5
BOSH
PKS K8s
Infrastructure
watches watches
deploys
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
6
BOSH
PKS K8s K8s
Infrastructure
watches watches
deploys
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
1
7
BOSH
PKS K8s K8s
Infrastructure
watches watches watches
deploys
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
1
8
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/ 19
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
2
0
● Deploys as a tile in Ops Manager
PKS
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
2
1
PKS
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
2
2
PKS
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
2
3
PKS k8s
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
pks get-credentials cluster-name
2
4
PKS k8s
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
pks get-credentials cluster-name
2
5
PKS k8s
kubeconfig
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
PKS
● Deploys as a tile in Ops Manager
● Interact via the pks cli
pks create-cluster cluster-name...
pks get-credentials cluster-name
kubectl get pods
2
6
PKS k8s
kubeconfig
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking in Kubernetes
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
2
8
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
2
9
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
● Ingress and LoadBalancers for ingress traffic
3
0
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
● Ingress and LoadBalancers for ingress traffic
● NetworkPolicy for enforcing network rules
3
1
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Networking
● CNI (Container Network Interface): choose your own adventure
● Baseline Flannel for pod communication
● Ingress and LoadBalancers for ingress traffic
● NetworkPolicy for enforcing network rules
○ ...but not provided with Flannel alone
3
2
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
What is NSX-T ?
The NSX-T Platform
Single platform for Networking, Security & Monitoring for Containers & VMs
PKS PAS/PCFOpenShift VM
Networking, Security, Monitoring
Kubernetes
34
Address all K8s Networking Functions
Load Balancing, IPAM, Routing, Firewalling
Complete automation and dynamic provisioning of
Network Objects required for K8s and Workloads
Support for different topology choices for
Pod and Node Networks (NAT/No-NAT)
Network Security Policies for Kubernetes Clusters,
Namespaces and Individual Services
Full Network traceability/visibility using NSX-T in-built
operational tools for Kubernetes
How does PKS with NSX-T add value over opensource solutions
35
NSX-T: operational tools and automation
36
Traceflow, Port-Connection,
Topology View
Back-Up & Restore
Selective Tech Support Logs
Monitoring & Stats
REST API support
Dynamic libraries available in
multiple languages (Java, Python,
Terraform)
Upstream OpenStack Support and
Partner Ecosystem
Getting started wizards
Dashboards
IPFIX, Port Mirroring, Port
Connection
Granular RBAC
Upgrade Coordinator
vRealize Log Insight / Splunk
plugins
Automation Operations Troubleshooting
36
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
NSX-T + PKS
38
Pivotal
Operations Manager
BOSH
Master Worker Worker Worker
Master Worker Worker Worker
Master Worker Worker Worker
PKS-API
UAA
ODB
kubo
service
adapter
MySQL
NSX-T
Proxy
Broker
Telemetry
Aggregator
PKS
Control Plane
adminserver
jobservice
ui registry
notary
clair
NSX-T
vSphere
vRLI
Wavefront
VAC
PKS: High Level Architecture
PKS CLI
38
38
PRINCIPAL
IDENTITY
39
K8s Cluster Management Nodes
T1
T1 for LB NSX-T LB
K8s cluster 1
K8s cluster 2
K8s cluster n
Ops-Manager BOSH PKS
Control Plane
Harbor
T1
PKS
mgmt
plane
NSX-T
Edge Cluster
NSX-T
Controllers
vCenter
NSX-T
Manager
vCenter
Physical
L2/L3
switches
Internet
K8s Cluster and NSX-T Provisioning
VIP
API Server https://api.cluster.pks.customer.com
Master VM#1, VM#2, VM#3
39
pks create-cluster my-cluster --plan small --num-nodes 3
Worker ‘VM’ Worker ‘VM’ Worker ‘VM’
Master ‘VM’
Master ‘VM’
Master ‘VM’
etcd
API
srv
sched
ctrlr
mgr
Worker ‘VM’ Worker ‘VM’ Worker ‘VM’
NSX
NCP
dash
board
NA NANA
Kube
DNS
Cluster Provisioning Workflow
41
PKS API
1
NSX-T
Proxy Broker
named_cloud_config
2
• pre-deployment provisioning
o allocate cluster subnet
o create named_cloud_config
o create Node Network
o create SNAT rule, if required, from k8s nodes to external
• post-destroy decommissioning
o delete SNAT rule, if required
o release Virtual IP for API Server
o delete Node Network
o delete named_cloud_config
o remove Principal Identity
o run clean-up script for NSX-T cluster resources
3
manifest
Service
Adapter
5
CFCR
release
pks-nsx-t
release
7
Ops Manager OD-Broker
cloud_config
4
vSphere CPI
6
BOSH
pks create-cluster my-cluster --plan small --num-nodes 3
41
41
Cluster Provisioning Workflow (cont.)
42
o NCP
o NSX-CNI
o OpenvSwitch
o NSX scripts
BOSH
CFCR
release
vSphere CPI
pks-nsx-t
release
7
Worker ‘VM’ Worker ‘VM’ Worker ‘VM’
Master ‘VM’
Master ‘VM’
Master ‘VM’
8
VMs tagged with BOSH ID
9
BOSH Job on Master Node to perform cluster-level provisioning
o create Principal Identity
o create Load Balancer for the new Cluster
o reserve Virtual IP for API Server
10
42
42
PKS and NSX-T integration: Key Components
NSX Container Plugin (NCP)
43
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
NS: foo NS: bar
NSX Example K8s topology
K8s master
etcd
API-Server
Scheduler
• NCP is a software component
provided by Vmware, running on a
K8s Node..
• NCP is built in a modular way, so
that individual adapters can be
added for different CaaS and
PaaS systems
• NCP ‘listens’ to Kubernetes/CF to
create the required NSX-T
constructs.
• CNI Integration is used.
PKS + NSX-T: built-in Load Balancing
Kubernetes Ingress and Svc Type LB support
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server
10.114.209.209HTTP and/or
HTTPS traffic
Server Pool 1
Server Pool 2Rule 2
/bar/
Rule 1
/foo/
LB Service
NCM
Infra
K8s / OS
Adapter
CloudFoundry
Adapter
Libnetwork
Adapter
NSX Container Plugin
More…
NSX
Manager
API Client
NSX
Manager
K8s master
etcd
API-Server
Scheduler
Virtual Server
10.114.209.212TCP and/or
UDP traffic
Server Pool
LB Service
Built-in support for Ingress (L7) and Svc Type LB (L4) w/t NSX-T integration. Most other K8s
networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX
from Ingress (L7).
44
admin@k8s-master:~$ kubectl create namespace foo
namespace ”foo" created
admin@k8s-master:~$ kubectl create namespace bar
namespace ”bar" created
admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo
deployment "nginx-foo" created
admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar
deployment "nginx-bar" created
Namespace: foo Namespace: bar
NSX / K8s topology
10.24.0.0/24 10.24.1.0/24 10.24.2.0/24
NAT
boundary
NAT
boundary
K8s nodesK8s Masters
45
PKS + NSX-T: Namespaces
45
45
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Cluster Upgrades
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
● Install the new tile
● Update any new fields
● Deploy!
4
7
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
4
8
BOSH
PKS K8s K8s
Infrastructure
watches watches watches
deploys
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
4
9
BOSH
PKS K8s K8s
Infrastructure
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
0
BOSH
PKS K8s K8s
Infrastructure
upgrade
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
1
BOSH
PKS K8s K8s
Infrastructure
upgrade upgrade
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
52
BOSH
PKS K8s K8s
Infrastructure
upgrade upgrade upgrade
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
● NCP components upgrade within PKS
5
3
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
4
BOSH
PKS K8s K8s
Infrastructure
upgrade upgrade upgrade
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
5
Master
Worker
MasterMaster
WorkerWorker
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
6
Master
Worker
MasterMaster
WorkerWorker
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
7
Master
Worker
MasterMaster
WorkerWorker
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
8
Master
Worker
MasterMaster
WorkerWorker
NCP
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
5
9
Master
Worker
MasterMaster
WorkerWorker
NCP
Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial
license: http://creativecommons.org/licenses/by-nc/3.0/
Upgrades
● NCP components upgrade within PKS
● NSX-T can be upgraded independent of PKS
6
0

PKS Networking with NSX-T: You Focus on your App, We'll Take Care of the Rest!

  • 1.
    PKS Networking withNSX-T “you focus on your app, we'll take care of the rest !” Gaetano Borgione, @TanoBorgione Angela Chin, @AngelaSChin
  • 2.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Agenda ● What is PKS ● Networking in Kubernetes ● What is NSX ● NSX + PKS ● Cluster Creation ● Cluster Upgrade 2
  • 3.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Disclaimer
  • 4.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation. 4
  • 5.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is PKS
  • 6.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Pivotal Container Service 6
  • 7.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Pivotal Container Service ● Managed Kubernetes for multiple clusters 7
  • 8.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Pivotal Container Service ● Managed Kubernetes for multiple clusters 8
  • 9.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Pivotal Container Service ● Managed Kubernetes for multiple clusters 9
  • 10.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Pivotal Container Service ● Managed Kubernetes for multiple clusters ● Aimed at Day 2 Operations 1 0
  • 11.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Pivotal Container Service ● Managed Kubernetes for multiple clusters ● Aimed at Day 2 Operations ● Uses BOSH! 1 1
  • 12.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 2 BOSH Infrastructure
  • 13.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 3 BOSH PKS Infrastructure watches
  • 14.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 4 BOSH PKS K8s Infrastructure watches deploys
  • 15.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 5 BOSH PKS K8s Infrastructure watches watches deploys
  • 16.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 6 BOSH PKS K8s K8s Infrastructure watches watches deploys
  • 17.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 1 7 BOSH PKS K8s K8s Infrastructure watches watches watches deploys
  • 18.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Deploys as a tile in Ops Manager 1 8
  • 19.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 19
  • 20.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS 2 0 ● Deploys as a tile in Ops Manager PKS
  • 21.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Deploys as a tile in Ops Manager ● Interact via the pks cli 2 1 PKS
  • 22.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Deploys as a tile in Ops Manager ● Interact via the pks cli pks create-cluster cluster-name... 2 2 PKS
  • 23.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Deploys as a tile in Ops Manager ● Interact via the pks cli pks create-cluster cluster-name... 2 3 PKS k8s
  • 24.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Deploys as a tile in Ops Manager ● Interact via the pks cli pks create-cluster cluster-name... pks get-credentials cluster-name 2 4 PKS k8s
  • 25.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Deploys as a tile in Ops Manager ● Interact via the pks cli pks create-cluster cluster-name... pks get-credentials cluster-name 2 5 PKS k8s kubeconfig
  • 26.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PKS ● Deploys as a tile in Ops Manager ● Interact via the pks cli pks create-cluster cluster-name... pks get-credentials cluster-name kubectl get pods 2 6 PKS k8s kubeconfig
  • 27.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking in Kubernetes
  • 28.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking ● CNI (Container Network Interface): choose your own adventure 2 8
  • 29.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking ● CNI (Container Network Interface): choose your own adventure ● Baseline Flannel for pod communication 2 9
  • 30.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking ● CNI (Container Network Interface): choose your own adventure ● Baseline Flannel for pod communication ● Ingress and LoadBalancers for ingress traffic 3 0
  • 31.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking ● CNI (Container Network Interface): choose your own adventure ● Baseline Flannel for pod communication ● Ingress and LoadBalancers for ingress traffic ● NetworkPolicy for enforcing network rules 3 1
  • 32.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking ● CNI (Container Network Interface): choose your own adventure ● Baseline Flannel for pod communication ● Ingress and LoadBalancers for ingress traffic ● NetworkPolicy for enforcing network rules ○ ...but not provided with Flannel alone 3 2
  • 33.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is NSX-T ?
  • 34.
    The NSX-T Platform Singleplatform for Networking, Security & Monitoring for Containers & VMs PKS PAS/PCFOpenShift VM Networking, Security, Monitoring Kubernetes 34
  • 35.
    Address all K8sNetworking Functions Load Balancing, IPAM, Routing, Firewalling Complete automation and dynamic provisioning of Network Objects required for K8s and Workloads Support for different topology choices for Pod and Node Networks (NAT/No-NAT) Network Security Policies for Kubernetes Clusters, Namespaces and Individual Services Full Network traceability/visibility using NSX-T in-built operational tools for Kubernetes How does PKS with NSX-T add value over opensource solutions 35
  • 36.
    NSX-T: operational toolsand automation 36 Traceflow, Port-Connection, Topology View Back-Up & Restore Selective Tech Support Logs Monitoring & Stats REST API support Dynamic libraries available in multiple languages (Java, Python, Terraform) Upstream OpenStack Support and Partner Ecosystem Getting started wizards Dashboards IPFIX, Port Mirroring, Port Connection Granular RBAC Upgrade Coordinator vRealize Log Insight / Splunk plugins Automation Operations Troubleshooting 36
  • 37.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ NSX-T + PKS
  • 38.
    38 Pivotal Operations Manager BOSH Master WorkerWorker Worker Master Worker Worker Worker Master Worker Worker Worker PKS-API UAA ODB kubo service adapter MySQL NSX-T Proxy Broker Telemetry Aggregator PKS Control Plane adminserver jobservice ui registry notary clair NSX-T vSphere vRLI Wavefront VAC PKS: High Level Architecture PKS CLI 38 38
  • 39.
    PRINCIPAL IDENTITY 39 K8s Cluster ManagementNodes T1 T1 for LB NSX-T LB K8s cluster 1 K8s cluster 2 K8s cluster n Ops-Manager BOSH PKS Control Plane Harbor T1 PKS mgmt plane NSX-T Edge Cluster NSX-T Controllers vCenter NSX-T Manager vCenter Physical L2/L3 switches Internet K8s Cluster and NSX-T Provisioning VIP API Server https://api.cluster.pks.customer.com Master VM#1, VM#2, VM#3 39 pks create-cluster my-cluster --plan small --num-nodes 3 Worker ‘VM’ Worker ‘VM’ Worker ‘VM’ Master ‘VM’ Master ‘VM’ Master ‘VM’ etcd API srv sched ctrlr mgr Worker ‘VM’ Worker ‘VM’ Worker ‘VM’ NSX NCP dash board NA NANA Kube DNS
  • 40.
    Cluster Provisioning Workflow 41 PKSAPI 1 NSX-T Proxy Broker named_cloud_config 2 • pre-deployment provisioning o allocate cluster subnet o create named_cloud_config o create Node Network o create SNAT rule, if required, from k8s nodes to external • post-destroy decommissioning o delete SNAT rule, if required o release Virtual IP for API Server o delete Node Network o delete named_cloud_config o remove Principal Identity o run clean-up script for NSX-T cluster resources 3 manifest Service Adapter 5 CFCR release pks-nsx-t release 7 Ops Manager OD-Broker cloud_config 4 vSphere CPI 6 BOSH pks create-cluster my-cluster --plan small --num-nodes 3 41 41
  • 41.
    Cluster Provisioning Workflow(cont.) 42 o NCP o NSX-CNI o OpenvSwitch o NSX scripts BOSH CFCR release vSphere CPI pks-nsx-t release 7 Worker ‘VM’ Worker ‘VM’ Worker ‘VM’ Master ‘VM’ Master ‘VM’ Master ‘VM’ 8 VMs tagged with BOSH ID 9 BOSH Job on Master Node to perform cluster-level provisioning o create Principal Identity o create Load Balancer for the new Cluster o reserve Virtual IP for API Server 10 42 42
  • 42.
    PKS and NSX-Tintegration: Key Components NSX Container Plugin (NCP) 43 NCM Infra K8s / OS Adapter CloudFoundry Adapter NSX Container Plugin More… NSX Manager API Client NSX Manager NS: foo NS: bar NSX Example K8s topology K8s master etcd API-Server Scheduler • NCP is a software component provided by Vmware, running on a K8s Node.. • NCP is built in a modular way, so that individual adapters can be added for different CaaS and PaaS systems • NCP ‘listens’ to Kubernetes/CF to create the required NSX-T constructs. • CNI Integration is used.
  • 43.
    PKS + NSX-T:built-in Load Balancing Kubernetes Ingress and Svc Type LB support NCM Infra K8s / OS Adapter CloudFoundry Adapter Libnetwork Adapter NSX Container Plugin More… NSX Manager API Client NSX Manager K8s master etcd API-Server Scheduler Virtual Server 10.114.209.209HTTP and/or HTTPS traffic Server Pool 1 Server Pool 2Rule 2 /bar/ Rule 1 /foo/ LB Service NCM Infra K8s / OS Adapter CloudFoundry Adapter Libnetwork Adapter NSX Container Plugin More… NSX Manager API Client NSX Manager K8s master etcd API-Server Scheduler Virtual Server 10.114.209.212TCP and/or UDP traffic Server Pool LB Service Built-in support for Ingress (L7) and Svc Type LB (L4) w/t NSX-T integration. Most other K8s networking choice don't support Svc Type LB (L4), and you need an additional technology like NGINX from Ingress (L7). 44
  • 44.
    admin@k8s-master:~$ kubectl createnamespace foo namespace ”foo" created admin@k8s-master:~$ kubectl create namespace bar namespace ”bar" created admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foo deployment "nginx-foo" created admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bar deployment "nginx-bar" created Namespace: foo Namespace: bar NSX / K8s topology 10.24.0.0/24 10.24.1.0/24 10.24.2.0/24 NAT boundary NAT boundary K8s nodesK8s Masters 45 PKS + NSX-T: Namespaces 45 45
  • 45.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cluster Upgrades
  • 46.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades ● Install the new tile ● Update any new fields ● Deploy! 4 7
  • 47.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 4 8 BOSH PKS K8s K8s Infrastructure watches watches watches deploys
  • 48.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 4 9 BOSH PKS K8s K8s Infrastructure
  • 49.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 0 BOSH PKS K8s K8s Infrastructure upgrade
  • 50.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 1 BOSH PKS K8s K8s Infrastructure upgrade upgrade
  • 51.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 52 BOSH PKS K8s K8s Infrastructure upgrade upgrade upgrade
  • 52.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades ● NCP components upgrade within PKS 5 3
  • 53.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 4 BOSH PKS K8s K8s Infrastructure upgrade upgrade upgrade
  • 54.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 5 Master Worker MasterMaster WorkerWorker
  • 55.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 6 Master Worker MasterMaster WorkerWorker
  • 56.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 7 Master Worker MasterMaster WorkerWorker
  • 57.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 8 Master Worker MasterMaster WorkerWorker NCP
  • 58.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades 5 9 Master Worker MasterMaster WorkerWorker NCP
  • 59.
    Unless otherwise indicated,these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Upgrades ● NCP components upgrade within PKS ● NSX-T can be upgraded independent of PKS 6 0