This document discusses implementing multi-cloud deployments using Kubernetes. It outlines two reference architectures for Kubernetes multi-cloud - a monolithic approach with a single load balancer and a decoupled hierarchical approach. The core challenges of location affinity, cross-cluster service discovery, scheduling and migration are also covered. WSO2 Private PaaS is presented as a solution that addresses these challenges through network partitioning, a composite application model and runtime topology definitions. A comparison of features between Kubernetes and WSO2 Private PaaS is provided. The document concludes with WSO2's vision to contribute to Kubernetes and potentially use it as the foundation for their PaaS and microservices platform going forward.

1. Scale into Multi Cloud
with Containers
Imesh Gunaratne
Senior Technical Lead, WSO2

2. Agenda
• WSO2 Carbon Reference Architectures for Kubernetes
• Implementing a Multi-Cloud with Kubernetes
• WSO2 Private PaaS Multi-Cloud Solution
• Kubernetes, WSO2 Private PaaS Feature Comparison
• WSO2 Vision on Kubernetes

3. WSO2 Carbon Reference Architectures
for Kubernetes

4. WSO2 Carbon Worker/Manager
Reference Architecture for Kubernetes
Pod 1 Pod 2
Manager Cluster
Services
Pod 3 Pod 4 Pod n
Worker Cluster
Services
Manager
Replication
Controller
Worker Replication
Controller
Client
Gov
Reg
Conf
Reg
User
Store

5. WSO2 API-M Reference Architecture
for Kubernetes
Pod 1
Gateway Mgr
Services
Pod 2 Pod 3
Gateway Worker
Services
Client
Pod 4 Pod 5
Key Manager
Services
Pod 6 Pod 7
Store Services
Gateway
Mgr RC
Gateway
Worker RC
Key
Manager RC
Store RCGov
Reg
Conf
Reg
User
Store
API-M
DB
A volume mount for
synchronizing API
artifacts
Pod 8 Pod 9
Publisher Services
Publisher
RC

6. Implementing a Multi-Cloud with
Kubernetes

7. Why do we need a Multi-Cloud?
● Capacity overflow
○ Overflow from on-premise clusters to public when run out of
on-premise capacity
● Sensitive workloads
○ Privacy-sensitive workloads to run on on-premise clusters
● Vendor lock-in avoidance
○ Workloads to run across multiple cloud providers
● High availability
○ Multiple availability zones within a single cloud provider
https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md

8. Kubernetes Multi-Cloud Reference
Architecture 1 (monolithic)
IaaS Provider 1
Kubernetes Cluster 1
Load
Balancer 1
Load
Balancer 2
Load
Balancer n
Central Load
Balancer
N1 N2 N3 NnM N1 N2 N3 NnM N1 N2 N3 NnM
IaaS Provider n
Client
IaaS Provider 2

9. Kubernetes Multi-Cloud Reference
Architecture 2 (decoupled & hierarchical)
IaaS Provider 1
Kubernetes Cluster 1
Load
Balancer 1
Load
Balancer 2
Load
Balancer n
Central Load
Balancer
N1 N2 N3 NnM
Kubernetes Cluster 2
N1 N2 N3 NnM
Kubernetes Cluster n
N1 N2 N3 NnM
IaaS Provider n
Client
Federation
Controller

10. Core Challenges of Implementing a Multi-
Cloud
● Location affinity
○ Distributed applications
■ Strictly coupled
■ Strictly decoupled
■ Preferentially coupled
○ Can pods of a single distributed application partitioned across
more than one cluster?
● Cross-cluster service discovery
○ Use DNS
https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md

11. Core Challenges of Implementing a Multi-
Cloud
● Cross-cluster scheduling
○ Closely related to location affinity
○ Have a layer on top of k8s
clusters to handle resource
creation
○ Make policy-based decisions
● Cross-cluster migration
○ Moving a distributed application
from one k8s cluster to another
https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md

12. Kubernetes Cluster Federation Proposed
Architecture (decoupled & hierarchical)
https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md
Ubernetes Control Plane
Kubernetes Cluster 1
GCE
Kubernetes Cluster 2
AWs
Kubernetes Cluster n
Private
Standard
Kubernetes API
Ubernetes API
Policy Engine
Auth
Desired
Federation
State
Migration
Controllers

13. WSO2 Private PaaS Multi-Cloud Solution
+

14. WSO2 Private PaaS Architecture

15. Network Partitioning
IaaS Provider 1 IaaS Provider n
Region 1 Region 2
Zone A Zone B
Network Partition 1
Partition 1 Partition 2
Network Partition 2 Network Partition 3
Region 1
https://docs.wso2.com/display/PP410/Network+Partitions

16. Composite Application Model
https://docs.wso2.com/display/PP410/WSO2+API+Manager+as+a+Service
Gateway Mgt
Cartridge
Gateway Worker
Cartridge
Key Manager
Cartridge
Publisher/Store
Group
Gateway/KM Group
API-M App
Network Partition 1
Network Partition 2
Publisher Cartridge Store Cartridge
Deployment
Policy
Autoscaling
Policy Deployment
Policy
Autoscaling
Policy

17. Runtime Model (Topology)
https://docs.wso2.com/display/PP410/WSO2+API+Manager+as+a+Service
API-M App
NP1 App Instance NP2 App Instance
Gateway Mgt
Cluster
Gateway Worker
Cluster
Key Manager Cluster
Pub/Store Cluster
Gateway/KM Group
Gateway Mgt
Cluster
Gateway Worker
Cluster
Key Manager Cluster
Pub/Store Cluster
Gateway/KM Group

18. WSO2 Private PaaS Multi-Cloud Solution
IaaS Provider 1
Kubernetes Cluster 1
Load
Balancer 1
Load
Balancer 2
Load
Balancer n
Central Load
Balancer
N1 N2 N3 NnM
Kubernetes Cluster 2
N1 N2 N3 NnM
Kubernetes Cluster n
N1 N2 N3 NnM
IaaS Provider n
Client

19. Kubernetes, WSO2 Private PaaS Feature
Comparison
≅

20. Feature Comparison
Kubernetes WSO2 Private PaaS
Carbon Cluster Discovery Available Available
Artifact Distribution Available (Rolling update) Available (Git based)
Centralized Logging Available (kubectl) Available (Thrift, DAS)
Monitoring & Metering Available (cAdvisor,
heapster, grafana)
Available (DAS)
Multi-Tenancy Available (Namespaces) Available (Carbon)
Load Balancing Available (nginx) Available (nginx, haproxy,
LVS, ec2)

21. Feature Comparison (cont.)
Kubernetes WSO2 Private PaaS
Horizontal Pod
Autoscaling
Available with v1.2.0-
alpha.2
Available
Multi-Cloud Deployment A proposal available Available
Horizontal VM
Autoscaling
Being discussed Available
Composite Application
Model
Being discussed Available

22. WSO2 Vision on Kubernetes

23. WSO2 Vision on Kubernetes
● Contribute to Kubernetes on implementing:
○ Autoscaling (Horizontal Pod/Node + Vertical Pod scaling)
○ Kubernetes Cluster Federation (Ubernetes)
○ Composite Application Model (may be using TOSCA, CAMP,
etc)
● Use Kubernetes in the future:
○ As the WSO2 PaaS
○ As the Microservices solution platform
○ Replace in JVM multi-tenancy using Kubernetes namespaces

24. Thank You