Kubernetes, Containers and Cloud - The MoD Story. by Mete Atamel, Google and Steve Latchem, UK Ministry of Defence

1. Confidential & ProprietaryGoogle Cloud Platform 1
Kubernetes, Containers and Cloud
The MoD Story
Mete Atamel
Developer Advocate
Google
Steve Latchem
Head of App Services & DevOps
UK Ministry of Defence

2. Confidential & ProprietaryGoogle Cloud Platform 2
Agenda
The Monolith and Microservices
What is the Monolith and how do Microservices help (or not)?
Containers and Kubernetes
What are they? What problems do they solve?
Kubernetes Building Blocks
Pods, services, replication controllers/set and more
Defence-as-a-Platform
The MoD Journey
@meteatamel

3. Confidential & ProprietaryGoogle Cloud Platform 3
The Monolith
@meteatamel

4. Confidential & ProprietaryGoogle Cloud Platform 4
What is the Monolith?
@meteatamel
APP SERVER
Module1 Module2 Module3
DB

5. Confidential & ProprietaryGoogle Cloud Platform 5
Problems with the Monolith
Unnecessary tight coupling among different modules
All at once update policy, ignores different development velocities
Hard to scale different parts independently
Hard to establish ownership of the whole system
Hard to debug and test in general, hard to run on a single development machine
@meteatamel

6. Confidential & ProprietaryGoogle Cloud Platform 6
Breaking the Monolith into Microservices
@meteatamel

7. Confidential & ProprietaryGoogle Cloud Platform 7
The Monolith to Microservices
@meteatamel
Microservice1
DB1
Microservice2
DB2
Microservice3
DB3

8. Confidential & ProprietaryGoogle Cloud Platform 8
Problems with Microservices
Need to worry about multiple independent systems instead of one
Debugging and testing across multiple services can be hard without proper
instrumentation
“But it works on my machine!” problem still applies
Common maintenance problems still apply: Redundancy, resilience,
upgrades/downgrades, scaling up/down
@meteatamel

9. Confidential & ProprietaryGoogle Cloud Platform 9
Containers and Kubernetes
@meteatamel

10. Confidential & ProprietaryGoogle Cloud Platform 10
Quick recap of Containers
@meteatamel
Lightweight
Hermetically sealed
Isolated
Easily deployable
Introspectable
Runnable
Linux processes
Docker
A lightweight way to virtualize applications

11. Confidential & ProprietaryGoogle Cloud Platform 11
Everything at Google runs on containers
Gmail, Web Search, Maps, ...
MapReduce, batch, ...
GFS, Colossus, ...
Google’s Cloud Platform: VMs run in containers!
We launch over 2 billion containers per week

12. Confidential & ProprietaryGoogle Cloud Platform 12
Containers are great but not enough
Containers help to create a lightweight and consistent environment for apps
But they do not solve common app management problems:
● Deploy your a new version of your app reliably
● Create resiliency
● Scale up and down
● Update to a new version
● Rollback to a previous version
● Health checks
● Graceful shutdown
@meteatamel

13. Confidential & ProprietaryGoogle Cloud Platform 13
Kubernetes comes to rescue
https://kubernetes.io
Open source container management platform. Based on years of experience
running Borg at Google
Runs everywhere: your laptop, on-prem, different cloud platforms
Provides a high level API to manage containers
Helps with reliable deployment of apps, scaling, roll out and roll back of versions,
autoscaling, health checks and much more!
@meteatamel

14. Confidential & ProprietaryGoogle Cloud Platform 14
Kubernetes Cluster
@meteatamel
K8s Master
API Server
Dash Board
scheduler
Kubelet Kubelet Kubelet Kubelet
Container
Registry
etcdControllers
web browsers
kubectl
web browsers
Config
file
Image

15. Confidential & ProprietaryGoogle Cloud Platform 15
Kubernetes Cluster on GKE
@meteatamel

16. Confidential & ProprietaryGoogle Cloud Platform 16
Kubernetes Building Blocks
@meteatamel

17. Confidential & ProprietaryGoogle Cloud Platform 17
Pods
@meteatamel
The atom of scheduling for containers
Represents an application specific logical
host
Hosts containers and volumes
Each has its own routable (no NAT) IP
address
Ephemeral
• Pods are functionally identical and therefore
ephemeral and replaceable
Pod
Web Server
Volume
Consumers

18. Confidential & ProprietaryGoogle Cloud Platform 18
Pods
@meteatamel
Pod
Git
Synchronizer
Node.js App
Container
Volume
Consumersgit Repo
Can be used to group multiple containers &
shared volumes
Containers within a pod are tightly coupled
Shared namespaces
• Containers in a pod share IP, port and IPC
namespaces
• Containers in a pod talk to each other through
localhost

19. Confidential & ProprietaryGoogle Cloud Platform 19
Labels
@meteatamel
Pod
Pod
frontend
Pod
frontend
Pod Pod
type = FE
version =
v2
type = FE version =
v2
● Metadata with semantic meaning
● Membership identifier
● The only Grouping Mechanism
Behavior Benefits
➔ Allow for intent of many users (e.g. dashboards)
➔ Build higher level systems …
➔ Queryable by Selectors
Dashboard
selector:
type = FE
Dashboard
selector:
version = v2

20. Confidential & ProprietaryGoogle Cloud Platform 20
Label Expressions
@meteatamel
Pod
Pod
frontend
Pod
frontend
Pod Pod
env = qa env = test
● env = prod
● tier != backend
● env = prod, tier !=backend
Expressions
● env in (test,qa)
● release notin (stable,beta)
● tier
● !tier
env = prod
Pod
env = prod
Dashboard
selector:
env = notin(prod)

21. Confidential & ProprietaryGoogle Cloud Platform 21
Services
@meteatamel
Client
Pod
Container
Pod
Container
Pod
Container
A logical grouping of pods that perform the
same function (the Service’s endpoints)
• grouped by label selector
Load balances incoming requests across
constituent pods
Choice of pod is random but supports
session affinity (ClientIP)
Gets a stable virtual IP and port
• also a DNS nametype =
Service
Label selector:
type = FE
VIP
type = FE type = FE type = FE

22. Confidential & ProprietaryGoogle Cloud Platform 22
Replica Sets
@meteatamel
Replication
Controller Pod
frontend
Pod
frontend
app = demo app = demo app = demo
ReplicaSet
#pods = 3
app = demo
color in (blue,grey)
show: version = v2
color = blue color = blue color = grey
Behavior Benefits
● Keeps Pods running
● Gives direct control of Pod #s
● Grouped by Label Selector
➔ Recreates Pods, maintains desired state
➔ Fine-grained control for scaling
➔ Standard grouping semantics
Pod Pod Pod

23. Confidential & ProprietaryGoogle Cloud Platform 23
Replica Sets
@meteatamel
ReplicaSet
- Name = “backend”
- Selector = {“name”: “backend”}
- Template = { ... }
- NumReplicas = 4
API Server
3
Start 1
more
OK 4
How
many?
How
many?
Canonical example of control loops
Have one job: ensure N copies of a pod
if too few, start new ones
if too many, kill some
group == selector
Replicated pods are fungible
No implied order or identity

24. Confidential & ProprietaryGoogle Cloud Platform 24
Scaling
@meteatamel
Service
Label selectors:
version = 1.0
type = Frontend
Service
name = frontend
Label selector:
type = BE
Replication
Controller Pod
frontend
Pod
version= v1
version =
v1
ReplicaSet
version = v1
#pods = 1
show: version = v2
type = FE type = FE
Pod
frontend
Pod
version =
v1
type = FE
ReplicaSet
version = v1
#pods = 2
show: version = v2
Pod Pod
ReplicaSet
version = v1
type = FE
#pods = 4
show: version = v2
version =
v1
type = FE

25. Confidential & ProprietaryGoogle Cloud Platform 25
Canary Deployments
@meteatamel
Service
Label selectors:
version = 1.0
type = Frontend
Service
name = backend
Label selector:
type = BE
Replication
Controller
Pod
Pod
frontend
Pod
version= v1
version =
v1
ReplicaSet
version = v1
type = BE
#pods = 2
show: version = v2
type = BE type = BE
Replication
Controller
ReplicaSet
version = v2
type = BE
#pods = 1
show: version = v2
Pod
frontend
Pod
version =
v2
type = BE

26. Confidential & ProprietaryGoogle Cloud Platform 26
Autoscaling
@meteatamel
Replication
Controller Pod
frontend
Pod
name=locust name=locust
ReplicaSet
name=locust
role=worker
#pods = 1
show: version = v2
Pod
frontend
Pod
name=locust
ReplicaSet
name=locust
role=worker
#pods = 2
show: version = v2
Pod Pod
name=locust
Scale
CPU Target% = 50
Heapster
role=worker role=worker role=worker role=worker
ReplicaSet
name=locust
role=worker
#pods = 4
70% CPU
40% CPU
> 50% CPU< 50% CPU

27. Confidential & ProprietaryGoogle Cloud Platform 27
Rollout
@meteatamel
API
DeploymentDeployment
Create frontend-1234567
Deployment
Create frontend-1234567
Scale frontend-1234567 up to 1
Deployment
Create frontend-1234567
Scale frontend-1234567 up to 1
Scale frontend-7654321 down to 0
Pod Pod
frontend
Pod
version = v1
ReplicaSet
frontend-1234567
version = v2
type = BE
#pods = 0
show: version = v2
ReplicaSet
frontend-7654321
version = v1
type = BE
#pods = 2
version: v2
ReplicaSet
frontend-7654321
version: v1
type: BE
#pods = 0
version: v1
ReplicaSet
frontend-1234567
version = v2
type = BE
#pods = 1
show: version = v2
ReplicaSet
frontend-1234567
version: v2
type: BE
#pods = 2
type = BE type = BE
Pod
version: v2
type = BE
Servic
e
be-svc
Deployment
Create frontend-1234567
Scale frontend-1234567 up to 1
Scale frontend-7654321 down to 0
Scale frontend-1234567 up to 2
kubectl edit deployment ...

28. Confidential & ProprietaryGoogle Cloud Platform 28
There is much more!
Namespace, Deployment, StatefulSet,
DaemonSet, Job, ConfigMap, Secret,
Federation
@meteatamel

29. 2929
kubernetes.io
cloud.google.com/container-engine
Mete Atamel
@meteatamel
atamel@google.com
meteatamel.wordpress.com
Thank You
@meteatamel

30. Defence-as-a-Platform
The MOD Journey
Steve Latchem
Head of App Services & DevOps

31. Hybrid Cloud
Hybrid Cloud
MOD Remote/
Home Worker
Organisations
with existing
RLI Connections
Trust Zone
2a
Industry
RLI gateway
Organisations
with existing
SLI connections
Deployed
rented
servers
Trust Zone
1
Trust Zone
4
Industry
SLI gateway
Including non-RLI/SLI
connected organisations,
through horizontal
collaboration
Internet
gateway
Internet
Public
MOD
Wide Area
Network
Shared
O-Cloud
Shared
S-Cloud

32. 32 DEFENCE-AS-A-PLATFORM
Ruggedised Containers
Containers Everywhere

33. Containers Everywhere
Containers Everywhere
Empty OS
Container
Catalogue of Pre-Accredited
Container Categories
Catalogue of
Server Types / Host OS / Security Levels
LINUX VM HOSTMS VM HOST RENTED SERVER
FROM:
Microsoft
Web over SQL
AS
DT
SUPPLI
ERS
ACCREDITATIONSCOPE
FROM: FROM:
Microsoft
Web over SQL
ActiveMQ,
JBOSS, SQL
Hypervisor Neutral VM Configuration (e.g. TOSCA)
Ruggedised Container Library, ready for Orchestration
Container Orchestration of Choice, e.g. Kubernetes (Linux & MSFT)

34. DevOps Containers Everywhere
Print/
File Service
Hypervisor plus
OS from the
Cloud
SDN
IdAM
Cyber
Monitor
Env.
Monitor
Gateway
DataSync /
SIEPatch
Service NATO C3
Taxonomy
Funct.
Test Suite
Non-Funct.
Test Suite
Accredited Container
Apps
DBs
Apps
DBs
Apps
DBs
Containers Everywhere

35. 35 DEFENCE-AS-A-PLATFORM
Apps and Container Lifecycle

36. Defence-as-a-Platform
steve.latchem100@mod.gov.uk
Thank you