Deploying WSO2 Middleware
on Kubernetes
Imesh Gunaratne
Senior Technical Lead, WSO2
Agenda
● PaaS Solutions for WSO2 Middleware
● Deploying WSO2 Middleware on Kubernetes
○ Kubernetes Architecture
○ Kubernetes Features Used
○ WSO2 Docker Images
○ WSO2 Carbon Reference Architectures for Kubernetes
○ Deployment Workflow
○ WSO2 Kubernetes Artifacts
○ Monitoring
PaaS Solutions for
Middleware
PaaS Solutions for WSO2 Middleware
Core PaaS Features Needed for a
Production Deployment
● Auto healing
● Autoscaling
● Automatic cluster discovery
● Dynamic load balancing
● VM/Container support
● Multi-tenancy
● Configuration orchestration
● Artifact distribution
● Multi-region/cloud deployments
● Centralized logging
● Monitoring
● Metering
Deploying WSO2 Middleware
on Kubernetes
An Introduction to Kubernetes
● An open source container cluster management system
● Started by Google with decade and half experience on
running containers at scale (Borg)
● Written in Go
● Can be deployed on any Infrastructure/IaaS
● Over 600 contributors
http://kubernetes.io/v1.0/docs/whatisk8s.html
Kubernetes High Level Architecture
Node1 Node2 Node n
Physical Network
Master
Overlay Network (Flannel/OpenVSwitch/Weave)APIServer
Scheduler
ControllerManager
etcd
Kubernetes Component Architecture
Kubernetes Features used by WSO2
Middleware
● Pods
● Replication Controllers (RC)
● Horizontal Pod Autoscalers (HPA)
● Services
● Ingress/Service Load Balancer
● Overlay Network
● Service Accounts & Secrets
● Namespaces
Kubernetes Pods
● A group of containers which can share
resources and context
● Shared namespaces:
○ PID namespace (processes visible globally)
○ network namespace (same IP and port
space)
○ IPC namespace (SystemV IPC or POSIX)
○ UTS namespace (share a hostname)
http://kubernetes.io/v1.0/docs/user-guide/pods.html
Pod
C1 C2 C3
Kubernetes Replication Controllers
● Provides features for replicating pods:
○ Auto-healing
○ Manual scaling
○ Rolling updates
○ Multiple release tracks
http://kubernetes.io/v1.0/docs/user-guide/replication-controller.html
Pod
Template
Replicas = n
Pod 1 Pod 2 Pod n
Replication
Controller
Kubernetes Horizontal Pod Autoscaler
● Provides features for horizontally
autoscaling pods
○ CPU consumption based
○ Currently in Alpha
http://kubernetes.io/v1.1/docs/user-guide/horizontal-pod-autoscaler.html
Pod
Template
Replicas = n
Pod 1 Pod 2 Pod n
Replication
Controller
Horizontal Pod
Autoscaler
● Service types:
○ ClusterIP (default)
○ NodePort
○ LoadBalancer
● Service discovery:
○ DNS
○ Environment variables
● Client IP based Session Affinity
http://kubernetes.io/v1.0/docs/user-guide/services.html
Kubernetes Services
Node IP: 172.17.8.102
Port: 9443
Domain Name: service1
IP: 10.2.10.20
Port: 9443
NodePort: 32001
Protocol: TCP
Pod 1 Pod 2 Pod n
Service
L1
L1 L1 L1
Node
● Unique IP addresses
● Provides pod to pod networking
● Options available:
○ Flannel (default)
○ OpenVSwitch
○ Weave
○ Calico
http://kubernetes.io/v1.1/docs/admin/networking.html
Kubernetes Overlay Network
WSO2 Docker Images
https://github.com/wso2/dockerfiles
WSO2 Carbon Reference Architecture 1
Pod 1 Pod 2
Manager Cluster
Services
Pod 3 Pod 4 Pod n
Worker Cluster
Services
Manager
Replication
Controller
Worker Replication
Controller
Client
Gov
Reg
Conf
Reg
User
Store
WSO2 Carbon Reference Architecture 2
Pod 1
Gateway Mgr
Services
Pod 2 Pod 3
Gateway Worker
Services
Client
Pod 4 Pod 5
Key Manager
Services
Pod 6 Pod 7
Store Services
Gateway
Mgr RC
Gateway
Worker RC
Key
Manager RC
Store RCGov
Reg
Conf
Reg
User
Store
API-M
DB
A volume mount for
synchronizing API
artifacts
Pod 8 Pod 9
Publisher Services
Publisher
RC
Carbon Cluster Discovery on Kubernetes
Pod 1 Pod 2 Pod n
Service
Server 1 Server 2 Server n
Pod 1
Service
Server 1
Pod 1 Pod 2
Service
Server 1 Server 2
Hazelcast member
initialization
Hazelcast member
initialization
Hazelcast member
initialization
1 2 n
Deployment Workflow
3
1 2
Build Docker
Image
Import Docker Image
to Docker Registry
Application Artifacts
Kubernetes
Membership
Scheme
Product Distribution
Docker Image (DI)
Puppet Module
+ Hiera Files
JDK Deploy Kubernetes
Artifacts
Kubernetes
Artifacts
WSO2 Kubernetes Artifacts
● WSO2 Kubernetes Artifacts Git Repository
○ Kubernetes Membership Scheme
○ Kubernetes Services
○ Kubernetes Replication Controllers
https://github.com/wso2/kubernetes-artifacts
Monitoring Kubernetes Cluster Health
Monitoring Kubernetes Cluster/Node/Pod
Health
cAdvisorcAdvisor
Node 1
cAdvisor
Heapster
Node 2 Node n
Grafana
InfluxDB
Thank you!

Deploying WSO2 Middleware on Kubernetes

  • 1.
    Deploying WSO2 Middleware onKubernetes Imesh Gunaratne Senior Technical Lead, WSO2
  • 2.
    Agenda ● PaaS Solutionsfor WSO2 Middleware ● Deploying WSO2 Middleware on Kubernetes ○ Kubernetes Architecture ○ Kubernetes Features Used ○ WSO2 Docker Images ○ WSO2 Carbon Reference Architectures for Kubernetes ○ Deployment Workflow ○ WSO2 Kubernetes Artifacts ○ Monitoring
  • 3.
  • 4.
    PaaS Solutions forWSO2 Middleware
  • 5.
    Core PaaS FeaturesNeeded for a Production Deployment ● Auto healing ● Autoscaling ● Automatic cluster discovery ● Dynamic load balancing ● VM/Container support ● Multi-tenancy ● Configuration orchestration ● Artifact distribution ● Multi-region/cloud deployments ● Centralized logging ● Monitoring ● Metering
  • 6.
  • 7.
    An Introduction toKubernetes ● An open source container cluster management system ● Started by Google with decade and half experience on running containers at scale (Borg) ● Written in Go ● Can be deployed on any Infrastructure/IaaS ● Over 600 contributors http://kubernetes.io/v1.0/docs/whatisk8s.html
  • 8.
    Kubernetes High LevelArchitecture Node1 Node2 Node n Physical Network Master Overlay Network (Flannel/OpenVSwitch/Weave)APIServer Scheduler ControllerManager etcd
  • 9.
  • 10.
    Kubernetes Features usedby WSO2 Middleware ● Pods ● Replication Controllers (RC) ● Horizontal Pod Autoscalers (HPA) ● Services ● Ingress/Service Load Balancer ● Overlay Network ● Service Accounts & Secrets ● Namespaces
  • 11.
    Kubernetes Pods ● Agroup of containers which can share resources and context ● Shared namespaces: ○ PID namespace (processes visible globally) ○ network namespace (same IP and port space) ○ IPC namespace (SystemV IPC or POSIX) ○ UTS namespace (share a hostname) http://kubernetes.io/v1.0/docs/user-guide/pods.html Pod C1 C2 C3
  • 12.
    Kubernetes Replication Controllers ●Provides features for replicating pods: ○ Auto-healing ○ Manual scaling ○ Rolling updates ○ Multiple release tracks http://kubernetes.io/v1.0/docs/user-guide/replication-controller.html Pod Template Replicas = n Pod 1 Pod 2 Pod n Replication Controller
  • 13.
    Kubernetes Horizontal PodAutoscaler ● Provides features for horizontally autoscaling pods ○ CPU consumption based ○ Currently in Alpha http://kubernetes.io/v1.1/docs/user-guide/horizontal-pod-autoscaler.html Pod Template Replicas = n Pod 1 Pod 2 Pod n Replication Controller Horizontal Pod Autoscaler
  • 14.
    ● Service types: ○ClusterIP (default) ○ NodePort ○ LoadBalancer ● Service discovery: ○ DNS ○ Environment variables ● Client IP based Session Affinity http://kubernetes.io/v1.0/docs/user-guide/services.html Kubernetes Services Node IP: 172.17.8.102 Port: 9443 Domain Name: service1 IP: 10.2.10.20 Port: 9443 NodePort: 32001 Protocol: TCP Pod 1 Pod 2 Pod n Service L1 L1 L1 L1 Node
  • 15.
    ● Unique IPaddresses ● Provides pod to pod networking ● Options available: ○ Flannel (default) ○ OpenVSwitch ○ Weave ○ Calico http://kubernetes.io/v1.1/docs/admin/networking.html Kubernetes Overlay Network
  • 16.
  • 17.
    WSO2 Carbon ReferenceArchitecture 1 Pod 1 Pod 2 Manager Cluster Services Pod 3 Pod 4 Pod n Worker Cluster Services Manager Replication Controller Worker Replication Controller Client Gov Reg Conf Reg User Store
  • 18.
    WSO2 Carbon ReferenceArchitecture 2 Pod 1 Gateway Mgr Services Pod 2 Pod 3 Gateway Worker Services Client Pod 4 Pod 5 Key Manager Services Pod 6 Pod 7 Store Services Gateway Mgr RC Gateway Worker RC Key Manager RC Store RCGov Reg Conf Reg User Store API-M DB A volume mount for synchronizing API artifacts Pod 8 Pod 9 Publisher Services Publisher RC
  • 19.
    Carbon Cluster Discoveryon Kubernetes Pod 1 Pod 2 Pod n Service Server 1 Server 2 Server n Pod 1 Service Server 1 Pod 1 Pod 2 Service Server 1 Server 2 Hazelcast member initialization Hazelcast member initialization Hazelcast member initialization 1 2 n
  • 20.
    Deployment Workflow 3 1 2 BuildDocker Image Import Docker Image to Docker Registry Application Artifacts Kubernetes Membership Scheme Product Distribution Docker Image (DI) Puppet Module + Hiera Files JDK Deploy Kubernetes Artifacts Kubernetes Artifacts
  • 21.
    WSO2 Kubernetes Artifacts ●WSO2 Kubernetes Artifacts Git Repository ○ Kubernetes Membership Scheme ○ Kubernetes Services ○ Kubernetes Replication Controllers https://github.com/wso2/kubernetes-artifacts
  • 22.
  • 23.
    Monitoring Kubernetes Cluster/Node/Pod Health cAdvisorcAdvisor Node1 cAdvisor Heapster Node 2 Node n Grafana InfluxDB
  • 24.