Container Camp London (2016-09-09)

Kubernetes, beyond the core
Craig Box
@craigbox

Container Camp London // @craigboxGoogle Cloud Platform 2

● Lightweight
● Hermetically sealed
● Isolated
● Easily deployable
● Introspectable
● Runnable
Containers: a quick recap
i'm boring core
infrastructure!
Linux processes
● Improves overall developer experience
● Fosters code and component reuse
● Simplifies operations for cloud native applications

● Scheduling: Decide where my containers should run
● Lifecycle and health: Keep my containers running despite
failures
● Scaling: Make sets of containers bigger or smaller
● Naming and discovery: Find where my containers are now
● Load balancing: Distribute traffic across a set of containers
● Storage volumes: Provide data to containers
● Logging and monitoring: Track what’s happening with my
containers
● Debugging and introspection: Enter or attach to containers
● Identity and authorization: Control who can do things to my
containers
Kubernetes: a quick recap

● Kubernetes is one of the three legs of Cloud Native
○ Takes in container packaged apps
○ Emits microservices architectures
● Announced June 2014, in GA since June 2015
● 1.4 is due out in 1 week!
● Under half the code is now written by Google
● Stewarded by the Cloud Native Compute Foundation™
○ A Linux Foundation Collaborative Project™
Kubernetes: a quick recap

Kubernetes is stable
● Concrete ideas from 10 years of production experience
○ and mistakes!
● v1 API; breaking changes held until v2
● Alpha, Beta and GA tracks for new features
● Thorough end-to-end testing
● New work taking place outside of core
○ Volume & network plugins
○ Custom controllers
○ ThirdPartyResources

Kubernetes has a solid core
● Core primitives:
○ pods, services, volumes, labels, controllers, etc
● Continual improvement using these basic concepts
○ Ingress: connect a load balancer to a Service
○ ReplicaSet: fungible replicas
○ DaemonSet: put a pod on every node
○ Job: batch workloads
○ ScheduledJob: run a Job at a certain time (cron)

Kubernetes has
a healthy ecosystem
Some examples:
● Cloud providers: Azure, VMware, Openstack, Rackspace, CenturyLink
● Distros: CoreOS Tectonic, Mirantis Murano (OpenStack), RedHat
Atomic, Hyper.sh, VMTurbo
● PaaS: RedHat OpenShift, Deis, Rancher, WSO2, Gondor/Kel, Apcera
● CD: Fabric8, Shippable, CloudBees, Solano
● Deployment: Kumoru, Redspread, Spinnaker
● Package managers: Helm, KPM
● Monitoring: Prometheus, Sysdig, Datadog
● Networking: Weaveworks, Tigera, OpenContrail
● Storage: NetApp, ClusterHQ
● Appliances: Redapt, Diamante

Kubernetes has great momentum

To host a similar set of services on our older Openstack environment
would require at least 2-3x the number of servers. The cost savings
isn't even the best part. Kubernetes has allowed us to build a
completely self-service pipeline for our devs and has taken the ops
team out of day-to-day app management. The nodes update
themselves with the latest OS and Kube shifts the workload around
as they do. This infrastructure is faster, more nimble, more
cost-effective and so much easier to run.
This is the best infrastructure I've ever used in twenty years
of doing ops and leading ops teams.

Since we started using kubernetes, we reduced our bill to 30%
of its original price, and it made everything easier and scalable
just as if we were using the costly [alternative]

Your cluster turnup story is bad, and you should feel bad
awwwwwwkward

laptop$ kubeadm --help
kubeadm: bootstrap a secure Kubernetes cluster easily.
/==========================================================
| KUBEADM IS ALPHA, DO NOT USE IT FOR PRODUCTION CLUSTERS! |
| |
| But, please try it out! Give us feedback at: |
| https://github.com/kubernetes/kubernetes/issues |
| and at-mention @kubernetes/sig-cluster-lifecycle |
==========================================================/
Example usage:
Create a two-machine cluster with one master (which controls the cluster),
and one node (where workloads, like pods and containers run).
On the first machine
====================
master# kubeadm init master
Your token is: <token>
On the second machine
=====================
node# kubeadm join node --token=<token> <ip-of-master>

The problems to be solved
1. Preserve individual identity for fungible entities
2. Provide predictable ordering and control as those entities change
3. To enable the software entities to identify and recognize the other entities
by those identities
4. To get access to a consistent storage mechanism (because their identity
also corresponds to data)

I apologise in advance for this horrible metaphor

Pets vs Cattle
It's so horrible it can only be written in Comic Sans

Sometimes pets are
Indistinguishable
from cattle

Sometimes cattle are
just plain adorable

Sometimes cattle
come pre-numbered

It's not just the "pet" part

Naming things is hard

(In fairness, I can tell you the UDP joke, but I can't guarantee you will get it)

What is a pet?
A Pet Set ensures that a specified number of “pets” with unique identities are
running at any given time.
The identity of a Pet is comprised of:
● a stable hostname, available in DNS
● an ordinal index
● stable storage: linked to the ordinal & hostname

you're awesomethree please
ReplicaSets
Master

you're awesome
ReplicaSets
web-7ci7o
web-kzszj
web-qqcnn
Master

bleep, bloopsix please
ReplicaSets
web-7ci7o
web-kzszj
web-qqcnn
Master

bleep, bloop
ReplicaSets
web-7ci7o
web-kzszj
web-qqcnn
Master
web-khku8
web-nacti
web-z9gth

ReplicaSets
web-7ci7o
web-kzszj
web-qqcnn
Master
web-khku8
web-nacti
web-z9gth
at least i'm not
passing the butter
lol jk, 2 is plenty

ReplicaSets
web-7ci7o
Master
web-z9gth
at least i'm not
passing the butter

you're awesome
three please,
with storage
PetSets
Master

you're awesome
three please,
with storage
PetSets
db-0
Master
pvc-db-0pv-db-0
1:1 mapping

you're awesome
three please,
with storage
PetSets
db-0
db-1
Master
pvc-db-0
pvc-db-1
pv-db-0
pv-db-1
1:1 mapping

you're awesome
PetSets
db-0
db-1
db-2
Master
pvc-db-0
pvc-db-1
pvc-db-2
pv-db-0
pv-db-1
pv-db-2
1:1 mapping

no robots shall
touch my pets
PetSets
db-0
db-1
db-2
Master
pvc-db-0
pvc-db-1
pvc-db-2
pv-db-0
pv-db-1
pv-db-2
scale down to
2, please

no robots shall
touch my pets
PetSets
db-0
db-1
Master
pvc-db-0
pvc-db-1
pvc-db-2
pv-db-0
pv-db-1
I might come in
useful some day

What other problems do I have?
● Discovery of peers for quorum
○ Sidecars and peer finder scripts
● Startup/teardown ordering
○ Init containers
○ Implicit ordering

InitContainers
db-0
Mount some
things

InitContainers
db-0
Mount some
things
pv-db-0

InitContainers
db-0
Copy some
stuff
pv-db-0

InitContainers
db-0
Write some
configs
pv-db-0
DNS

InitContainers
db-0
Be a database
pv-db-0

● InitContainers and PetSet introduced in 1.3
● InitContainers are Beta in 1.4
● PetSet remain in Alpha
"The real P0 beta blocker is solid prototypes that
increase our confidence in the core feature set."
https://github.com/kubernetes/charts/tree/master/incubator
Status: Alpha
Thanks to Christian and Matt from

Some terminology
● What is a cluster?
○ A bunch of machines on a high-speed network
● What is high-speed?
○ Generally "in the same building"
○ Same latency and throughput between any two machines
● How much is a bunch?
○ Enough to get the benefits of packing
○ Not too many to Accidentally Kill Everything

shared cell
(original)
shared cell
(compacted)
non-prod load
(compacted)
prod-only load
(compacted)
# machines
25% overhead
The bigger
the bin,
the better
the packing

How to separate
● Within a cluster
○ Use namespaces
● Within a region
○ Use NodePools to create "regional" cluster
● With multiple regions
○ Use cluster federation

etcd
scheduler
controllers
apiserver
Users Master Nodes
kubelet
kubelet
kubelet
CLI
UI
API
Single Kubernetes cluster

Container
Cluster
All you care about
API

kubelet
Control Plane
Users Control Plane Clusters
Federation
APICLI
UI
API

Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API CLI
UI
Create the clusters
API API API API

Federation consists of
● Namespace
● API Server Service with public VIP
● API Server Deployment with 2 replicas
● Controller Manager Pod with 1 replica
● Database key/value store
Familiar? --context=federation-cluster
Deploy the Federated Control Plane
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API

Add clusters to federation
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
Federation Control Plane
kubectl --context=federation-cluster create -f clusters/gce-asia-east1.yaml
apiVersion: federation/v1beta1
kind: Cluster
metadata:
name: gce-asia-east1
spec:
serverAddressByClientCIDRs:
- clientCIDR: "0.0.0.0/0"
serverAddress: "https://257.100.194.68"
secretRef:
name: gce-asia-east1

Deploy a federated ReplicaSet
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
kubectl --context=federation-cluster create -f rs/nginx.yaml
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: nginx
spec:
replicas: 4
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.10
me again

Deploy a federated Service
Cluster 2
us-central1-b
Cluster 1
us-east1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
API API API API
kubectl --context=federation-cluster create -f service/nginx.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
name: http
selector:
app: nginx
type: LoadBalancer

Each service shard gets a load balancer
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b

...and each service creates a DNS entry

Cross-cluster service discovery
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
nslookup nginx.default.federation.svc.federation.com

Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
nslookup nginx.default.federation.svc.federation.com
DNS
Clusters 3 and 4

Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b

Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 4
asia-east1-b

Stop the presses: Federated Ingress
Cluster 1
us-east1-b
Cluster 2
us-central1-b
Cluster 3
europe-west1-b
Cluster 4
asia-east1-b
Google's Magic Global Load Balancer

Q2 2016 Q3 2016 Q4 2016 (**) 2017 and beyond (**)
Beta 1
● Public facing,
multi-region/Cloud,
cross-cluster
service discovery
(internal/external DNS)
● Service object API
support
Beta 2
● Replica Sets
● Multi-region Ingress (L7)
Load Balancing across
clusters for GCP only
Beta 3
● Cross-provider,
multi-region Ingress (L7)
Load Balancing
● GKE IAM Integration
GA!
● Non-public-facing
cross-cluster
service discovery
● Full support for
Kubernetes API objects
● UI support for
Federated Clusters
● Federated IAM
● GKE hosted control plane
(**) - this is a proposed roadmap. Items listed here are subject to change.
Status: Beta

● Kubernetes Cluster Federation Sneak Peak
● Kubernetes Cluster Federation using GKE
● Cluster Federation Admin Guide
● Cross Cluster Service Discovery Deployment Guide
● Cross Cluster Services - Achieving Higher Availability for your Kubernetes Applications
Also,
● Participate with us on the Kubernetes #sig-federation
● Post issues or feature requests on GitHub
● Join us in the #federation channel on Slack
Want to learn more?

1.4 is coming soon!
● Use Swagger 2.0, enabled non-go clients
● StorageClass
● AppArmor Support
● PodSecurityPolicy
● New Volume Plugins: Quobyte and Azure Data Disk
● Dashboard UI
● ScheduledJobs
● InitContainers
● Workloads installable with 1 command
● GCI as default Node Image
● GKE: Curated IAM Roles
● GKE: Alpha Clusters
● GKE: Available in Oregon (and soon in Japan)
● Federate all the things:
○ Ingress for GCP
○ Namespaces
○ Services
○ Secrets
○ ReplicaSets
● Federation Events
● PodDisruptionBudget
● Ingress for Multizone Clusters
● Prioritized Scheduling of Cluster Add-ons
● Container Image Policy
● Workload spreading across failure domains
● Kubelet TLS Bootstrap
● External Source IP Preservation
● Audit Logging

Container Camp London (2016-09-09)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Container Camp London (2016-09-09)

Similar to Container Camp London (2016-09-09) (20)

Recently uploaded

Recently uploaded (20)

Container Camp London (2016-09-09)