SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
1.
Kubernetes in Production,
From the Ground Up
Mike Splain | @mikesplain
2.
About Me
• Lead DevOps Engineer at Barkly
• Formerly PayPal / eBay & OurStage
• Running applications at scale with the least amount of human interaction
• Current Technologies: Kubernetes, Docker, Terraform, Ansible
• Previous Technologies: Chef, Test-Kitchen….
5.
Our requirements for running our apps
• Fast recovery without human intervention
• Nodes are ephemeral
• Autoscaling
• Testable on developer machines
• Infrastructure as an artifact
7.
Hasn’t someone else already built this?
… March 2015
8.
K8s official scripts
• Simple bash scripts
• “Just Works!”
• Sets up Autoscaling group for
minions
• Uses Salt
Problems? (at the time)
• No etcd HA
• No Master HA
• Salt Master is coupled with K8s
Master
• Ubuntu / Fedora
9.
CoreOS’s official scripts
• Cool go app to start it!
• Or Cloud formation?
• But what now?
Problems? (at the time)
• No etcd HA
• No Master HA
• Lots of magic
17.
Master Node
• Kubelet service
• API Server
• Controller Manager
• Scheduler
• Proxy
• Flannel
18.
Consider Master pods as
additional artifact
• Docker container that takes env
variables
• Outputs templated pods to disk for
Kubelet to load
• We use j2cli to template these files
with little overhead
apiVersion: v1
kind: Pod
metadata:
name: kube-podmaster
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: scheduler-elector
image: gcr.io/google_containers/podmaster:1.1
imagePullPolicy: Always
command:
- /podmaster
- --etcd-servers={{ ETCD_ENDPOINTS }}
- --key=scheduler
- --whoami={{ ADVERTISE_IP }}
- --source-file=/src/manifests/kube-scheduler.yaml
- --dest-file=/dst/manifests/kube-scheduler.yaml
volumeMounts:
- mountPath: /src/manifests
name: manifest-src
readOnly: true
- mountPath: /dst/manifests
name: manifest-dst
19.
Master Cloud Config
• Setup box for services needed for
real Docker to run
• Get etcd server IPs and write to file
• Start flannel with those IPs
#cloud-config
coreos:
units:
- name: etcd.service
command: stop
- name: etcd2.service
command: stop
- name: early-docker.service
command: start
- name: kub_get_etcd.service
command: start
content: |
[Unit]
Description= Write K8s etcd urls to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c "/usr/bin/docker pull
registry.barklyprotects.com/kubernetes/kub-get-etcd"
ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/
barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{
KUB_ETCD_ASG }} > /etc/etcd_servers.env"
- name: flanneld.service
command: start
drop-ins:
- name: 10-environment_vars.conf
content: |
[Unit]
After=kub_get_etcd.service
[Service]
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n
FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env
>> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE=
$private_ipv4 >> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/
flannel/options.env
Restart=always
RestartSec=10
20.
Other config
• Grab certs from S3
• Terraform only allows
permissions to specific files
• Format Master pod files to disk
- name: kub_certs.service
command: start
content: |
[Unit]
Description=Writes kubernetes cluster certs to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
Before=kubelet.service
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c /usr/bin/mkdir -p /etc/kubernetes/ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/ca.pem /ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/apiserver.pem /ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/apiserver-key.pem /ssl
- name: kub_pods.service
command: start
content: |
[Unit]
Description=Writes kubernetes pod files to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
Before=kubelet.service
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c "/usr/bin/mkdir -p /etc/kubernetes/ssl"
ExecStart=/usr/bin/docker run --net=host -v /etc/barkly/:/etc/
barkly/ -e K8S_VERSION='1.1.7' -e CLOUD_PROVIDER='--cloud-provider=aws' -e
SERVICE_IP_RANGE="10.3.0.0/16" -e ADVERTISE_IP="$private_ipv4" -e
ETCD_AUTOSCALE_GROUP_NAME="our_etcd_autoscaling_group_name" -v /srv/
kubernetes/manifests:/output_src -v /etc/kubernetes/manifests:/output_dst
registry.barklyprotects.com/kubernetes/kub-master-pods
21.
Start Docker & Kubelet
• Kubelet will wait for docker and
flannel to be ready
• Kubelet will load manifests for
Master services outputted from
previous container
- name: docker.service
command: start
drop-ins:
- name: 40-flannel.conf
content: |
[Unit]
Requires=flanneld.service
After=flanneld.service
- name: kubelet.service
command: start
content: |
[Unit]
Requires=docker.service
After=docker.service
After=fluentd-elasticsearch.service
[Service]
ExecStartPre=/usr/bin/mkdir -p /var/log/containers
ExecStart=/etc/bin/kubelet
--hostname-override="$private_ipv4"
--api_servers=http://127.0.0.1:8080
--register-node=false
--allow-privileged=true
--config=/etc/kubernetes/manifests
--cluster-dns=10.3.0.10
--cluster-domain=cluster.local
--cloud-provider=aws
--v=4
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
22.
Get Kubelet
• /usr/bin & /usr/local/bin are read
only in CoreOS
- name: kubelet.service
command: start
drop-ins:
- name: 10-download-binary.conf
content: |
[Service]
ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet"
write_files:
# Since systemd needs these files before it will start
- path: /etc/bin/download-k8s-binary
permissions: '0755'
content: |
#!/usr/bin/env bash
export K8S_VERSION=“v1.2.4"
mkdir -p /etc/bin
FILE=$1
if [ ! -f /usr/bin/$FILE ]; then
curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly-
kubernetes-builds/${K8S_VERSION}/bin/$FILE
chmod +x /etc/bin/$FILE
else
# we check the version of the binary
INSTALLED_VERSION=$(/etc/bin/$FILE --version)
MATCH=$(echo "${INSTALLED_VERSION}" | grep -c "${K8S_VERSION}")
if [ $MATCH -eq 0 ]; then
# the version is different
curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly-
kubernetes-builds/${K8S_VERSION}/bin/$FILE
chmod +x /etc/bin/$FILE
fi
fi
31.
K8s does not so well:
• Still relatively young,
• Not extremely google-able
• Leaves managing manifests to user with little guidance
• Docker only (rkt coming soon)
• No DNS support (coming soon)
• SaltStack by default
32.
K8s does really well:
• Keeping applications running
• Managing failover, liveness,readiness of applications
• Mounting volumes, load balancers (cloud resources)
• Internal DNS, Namespacing
• Basic secret / config management
• Easily Horizontally Scalable services
• Rolling updates / rollback via Deployments
• Support via Github / Slack chat
• Easy debugging of actual app problems via kubectl logs and kubectl exec