About Me
• Lead DevOps Engineer at Barkly
• Formerly PayPal / eBay & OurStage
• Running applications at scale with the least amount of human interaction
• Current Technologies: Kubernetes, Docker, Terraform, Ansible
• Previous Technologies: Chef, Test-Kitchen….
Our requirements for running our apps
• Fast recovery without human intervention
• Nodes are ephemeral
• Autoscaling
• Testable on developer machines
• Infrastructure as an artifact
K8s official scripts
• Simple bash scripts
• “Just Works!”
• Sets up Autoscaling group for
minions
• Uses Salt
Problems? (at the time)
• No etcd HA
• No Master HA
• Salt Master is coupled with K8s
Master
• Ubuntu / Fedora
CoreOS’s official scripts
• Cool go app to start it!
• Or Cloud formation?
• But what now?
Problems? (at the time)
• No etcd HA
• No Master HA
• Lots of magic
Master Node
• Kubelet service
• API Server
• Controller Manager
• Scheduler
• Proxy
• Flannel
Consider Master pods as
additional artifact
• Docker container that takes env
variables
• Outputs templated pods to disk for
Kubelet to load
• We use j2cli to template these files
with little overhead
apiVersion: v1
kind: Pod
metadata:
name: kube-podmaster
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: scheduler-elector
image: gcr.io/google_containers/podmaster:1.1
imagePullPolicy: Always
command:
- /podmaster
- --etcd-servers={{ ETCD_ENDPOINTS }}
- --key=scheduler
- --whoami={{ ADVERTISE_IP }}
- --source-file=/src/manifests/kube-scheduler.yaml
- --dest-file=/dst/manifests/kube-scheduler.yaml
volumeMounts:
- mountPath: /src/manifests
name: manifest-src
readOnly: true
- mountPath: /dst/manifests
name: manifest-dst
Master Cloud Config
• Setup box for services needed for
real Docker to run
• Get etcd server IPs and write to file
• Start flannel with those IPs
#cloud-config
coreos:
units:
- name: etcd.service
command: stop
- name: etcd2.service
command: stop
- name: early-docker.service
command: start
- name: kub_get_etcd.service
command: start
content: |
[Unit]
Description= Write K8s etcd urls to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c "/usr/bin/docker pull
registry.barklyprotects.com/kubernetes/kub-get-etcd"
ExecStart=/usr/bin/sh -c "/usr/bin/docker run --net=host -v /etc/
barkly/:/etc/barkly/ registry.barklyprotects.com/kubernetes/kub-get-etcd {{
KUB_ETCD_ASG }} > /etc/etcd_servers.env"
- name: flanneld.service
command: start
drop-ins:
- name: 10-environment_vars.conf
content: |
[Unit]
After=kub_get_etcd.service
[Service]
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo -n
FLANNELD_ETCD_ENDPOINTS= > /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/cat /etc/etcd_servers.env
>> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/sh -c "/usr/bin/echo FLANNELD_IFACE=
$private_ipv4 >> /etc/flannel_etcd_servers.env"
ExecStartPre=/usr/bin/ln -sf /etc/flannel_etcd_servers.env /run/
flannel/options.env
Restart=always
RestartSec=10
Other config
• Grab certs from S3
• Terraform only allows
permissions to specific files
• Format Master pod files to disk
- name: kub_certs.service
command: start
content: |
[Unit]
Description=Writes kubernetes cluster certs to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
Before=kubelet.service
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c /usr/bin/mkdir -p /etc/kubernetes/ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/ca.pem /ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/apiserver.pem /ssl
ExecStart=/usr/bin/docker run --net=host -v /etc/kubernetes/ssl:/
ssl registry.barklyprotects.com/ops/awscli s3 cp s3://
our_k8s_cluster_bucket/apiserver-key.pem /ssl
- name: kub_pods.service
command: start
content: |
[Unit]
Description=Writes kubernetes pod files to disk.
Requires=early-docker.service
After=early-docker.service
Before=early-docker.target
Before=kubelet.service
[Service]
Type=oneshot
Environment="DOCKER_HOST=unix:///var/run/early-docker.sock"
ExecStart=/usr/bin/sh -c "/usr/bin/mkdir -p /etc/kubernetes/ssl"
ExecStart=/usr/bin/docker run --net=host -v /etc/barkly/:/etc/
barkly/ -e K8S_VERSION='1.1.7' -e CLOUD_PROVIDER='--cloud-provider=aws' -e
SERVICE_IP_RANGE="10.3.0.0/16" -e ADVERTISE_IP="$private_ipv4" -e
ETCD_AUTOSCALE_GROUP_NAME="our_etcd_autoscaling_group_name" -v /srv/
kubernetes/manifests:/output_src -v /etc/kubernetes/manifests:/output_dst
registry.barklyprotects.com/kubernetes/kub-master-pods
Start Docker & Kubelet
• Kubelet will wait for docker and
flannel to be ready
• Kubelet will load manifests for
Master services outputted from
previous container
- name: docker.service
command: start
drop-ins:
- name: 40-flannel.conf
content: |
[Unit]
Requires=flanneld.service
After=flanneld.service
- name: kubelet.service
command: start
content: |
[Unit]
Requires=docker.service
After=docker.service
After=fluentd-elasticsearch.service
[Service]
ExecStartPre=/usr/bin/mkdir -p /var/log/containers
ExecStart=/etc/bin/kubelet
--hostname-override="$private_ipv4"
--api_servers=http://127.0.0.1:8080
--register-node=false
--allow-privileged=true
--config=/etc/kubernetes/manifests
--cluster-dns=10.3.0.10
--cluster-domain=cluster.local
--cloud-provider=aws
--v=4
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
Get Kubelet
• /usr/bin & /usr/local/bin are read
only in CoreOS
- name: kubelet.service
command: start
drop-ins:
- name: 10-download-binary.conf
content: |
[Service]
ExecStartPre=/bin/bash -c "/etc/bin/download-k8s-binary kubelet"
write_files:
# Since systemd needs these files before it will start
- path: /etc/bin/download-k8s-binary
permissions: '0755'
content: |
#!/usr/bin/env bash
export K8S_VERSION=“v1.2.4"
mkdir -p /etc/bin
FILE=$1
if [ ! -f /usr/bin/$FILE ]; then
curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly-
kubernetes-builds/${K8S_VERSION}/bin/$FILE
chmod +x /etc/bin/$FILE
else
# we check the version of the binary
INSTALLED_VERSION=$(/etc/bin/$FILE --version)
MATCH=$(echo "${INSTALLED_VERSION}" | grep -c "${K8S_VERSION}")
if [ $MATCH -eq 0 ]; then
# the version is different
curl -sSL -o /etc/bin/$FILE https://s3.amazonaws.com/barkly-
kubernetes-builds/${K8S_VERSION}/bin/$FILE
chmod +x /etc/bin/$FILE
fi
fi
K8s does not so well:
• Still relatively young,
• Not extremely google-able
• Leaves managing manifests to user with little guidance
• Docker only (rkt coming soon)
• No DNS support (coming soon)
• SaltStack by default
K8s does really well:
• Keeping applications running
• Managing failover, liveness,readiness of applications
• Mounting volumes, load balancers (cloud resources)
• Internal DNS, Namespacing
• Basic secret / config management
• Easily Horizontally Scalable services
• Rolling updates / rollback via Deployments
• Support via Github / Slack chat
• Easy debugging of actual app problems via kubectl logs and kubectl exec